MCMC的链式特性是否决定了其无法Parallel computing？

YSU

自己对MCMC数学计算算是知道点皮毛上的尘埃，对计算机怎么实现parallel computation一窍不通。想问一下明白人：MCMC的链式特性是否决定了其无法Parallel computing？

micro@

More difficult but still possible.

Usually not embarrassingly parallelizable across iterations, but still possible within iterations. The gain could potentially be huge if you have a large number of parameters.

There are actually some more general parallel algorithms for certain MCMC problems, but it seems to me that they rarely provide as much gain as usual parallelism in pure Monte Carlo procedures.

YSU

回复第2楼的 micro@：

Thanks for this information. Good to know that parallel computing can be done for each iteration, if the time for each iteration can be reduced to 1/10 of single node computing, then my 24 hours computation can be shorten to 2.4 hour, that's a huge gain. But in case I have 1000 data sets, I still have to wait for 240 hours (10 days) to get my computation done.[s:12]

I hope the math and computer science guys can come up with some better way, then we end users can benefit from it.

micro@

回复第3楼的 YSU：If you have independent data sets, parallelism would be trivial, because the term independence has told you that you can analyzie each of them without needing to worry about any other data sets.

YSU

回复第4楼的 micro@：

Not sure if my 1000 data sets are "independent" as the meaning of this word sometimes is vague to me.

Nevertheless, this example pertains to a typical model evaluation process: A model was built on one original data set, then 1000 data sets were created by bootstrapping from the original data set, and model parameters were re-estimated in these 1000 data sets using the model built originally.

Now without parallel computing, re-estimating these 1000 data sets seems to be mission impossible if each data set needs 24 hours to finish.

micro@

回复第5楼的 YSU：Normally, you can (and should) treat bootstrapped data sets as independent when fitting your time consuming model.

However, it's not that "typical" for me to combine MCMC with bootstrap this way. Not sure how many Bayesians will buy this... For a Bayesian, data are conditioned upon and fixed. Why should there be any bootstrapped data sets? Why not check the posterior / posterior predictive distribution?

YSU

回复第6楼的 micro@：

You raised good point. I particularly agree with your last point: posterior predictive check after the model building on the original data set is a good way for model check, and people normally do this.

Now the discussion is beyond my initial question. So I want to make my example more transparent, and want to explain to you why I have a bootstrapped data and then do MCMC.

1. A model was built on one data set using parametric modeling technique.

2. I assume my model in step 1 is TRUE. Then 1000 data sets were replicated by simulating this model under different set of covariates which refer to different population, different experiment conditions etc. (I call this "bootstrap" as I see people refer to both parametric and non parametric resampling as bootstrap). My goal is to make sure that my proposed experimental conditions can enable me to estimate the model parameters reasonably good if the experiment are carried out in future.

3. This set of 1000 data sets were then modeled again. Say, I use WinBugs and set prior to non-informative, so posterior distribution from MCMC iterations are mainly from the current data.

4. Posterior distributions of model parameters estimates from each replicate were summarized, say, geometric mean and its 95% CI assuming these parameters are log-normally distributed.

5. I then check the proportion of replicates with 95% CI falling in some range, say

60 to 150% of geometric mean of estimates.

6. If >80% is obtained in step 5. I am happy that the proposed experimental condition can enable me to estimate the model parameters reasonably good.

micro@

回复第7楼的 YSU：It reads more weired now... So far, I'm not quite convinced that this has anything to do with bootstrap, whether parametric or not. It troubles me to have completely different sets of covariates from different populations... If this is indeeded true, some mixture type models would make more sense. Otherwise I can hardly understand what the term "probability" means.

YSU

回复第8楼的 micro@：

I think what bothers you is the terminology of "bootstrap", and I guess I should NOT ambiguously use this word before statisticians who have a strict and dogmatic definition of "bootstrap".[s:12][s:12][s:12]

Let's forget about this terminology for now, and focus on the procedure. For instance, in clinical research, suppose you have a model establihsed in adult, and now you want to extend this model to research in the paeditric population. In other words, you want to re-establish a model in Children, but Children has lots of limitations on experimental conditions. and your goal is to make sure that your study design in children can ensure you to get a reasonably good model.

Assume that the model in the adult can be adjusted to paeditric population, and we can get a TRUE model in paeditic. <<<<<<<<<<<< This is my big assumption and it is often TRUE based on the historic data and our thorough understanding of biological system.

Then we simulate this TRUE model in Children for many replicates under the proposed study design (say, 500-1000. I probably should NOT call this bootstrap). The possible variability can be introduced over replicates, so these replicates can represent trials.

Finally, we fit these replicates again in using MCMC method instead of maximum likelihood. We could use Winbugs here but set the prior to non-informative and let the current data drive the posterior distribution (we regard the MCMC iteration in posterior distribution as the parameters uncertainty), and obtain the descriptive statistics of model parameters from replicates, compare them to the "TRUE" model, and conclude the probability of successful estimation of model parameters under the proposed study design (as I described in the previous reply).

My contact info is yaming dot su at gmail dot com. We can discuss over the phone or email if you want.