A counterexample to the use of diffuse priors?

rtist

<bblatex>Y|p \sim N(\mu,\exp(-p^2))</bblatex>,

<bblatex> p \sim N(0, \sigma^2)</bblatex>.

If the prior variance <bblatex>\sigma^2</bblatex> is infinite, Y is degenerate, i.e., a most informative model that discards any information in Y.

Counter-intuitive?

yihui

你是说Y的什么分布退化？prior predictive？

rtist

I mean the marginal of <bblatex>Y</bblatex> (do you call that prior predictive?):

<bblatex>V(Y)=V(E(Y|p))+E(V(Y|p))=V(\mu)+E(\exp(-p^2)))=V(\mu)+\frac{1}{\sqrt{2\sigma^2+1}}</bblatex>.

If <bblatex>\mu</bblatex> is a constant, then <bblatex>V(Y)\rightarrow 0</bblatex> as <bblatex>\sigma\rightarrow\infty</bblatex>.

Even if we give <bblatex>\mu</bblatex> another prior, <bblatex>V(Y)</bblatex> is still decreasing in <bblatex>\sigma^2</bblatex>. In other words, even if we mix <bblatex>Y</bblatex> over <bblatex>p</bblatex>, increasing the variance (vagueness) of <bblatex>p</bblatex> will limit the flexibility of the model on <bblatex>Y</bblatex>...

stella2011

It's dangrous to set prior variance to infinite.

If you want to express a vague prior ( no prior information), you may try a very small variance like "1.0E-6".

If you see the density plot,that will become a very flat curve ,quite similar as a flat line( no infor prior)

Hope I didn't make you more confusing[s:13]

stella2011

Variance in bayesian is actrually inverse to the variance in frequency analysis.

I should type "precision" in stead of variance just now[s:12]

rtist

回复第4,5楼的 stella2011：I understand what you mean. But this is not what I am discussing here...

My point here is the so-called 'vague' can be very informative in this example problem.

yihui

The problem is that the vagueness in p does not imply vagueness in Y; on the contrary, the more vague p is, the more likely that exp(-p^2) falls into a small neighborhood of 0. But I agree it looks very counter-intuitive at the first glance.

(there seems to be a typo in the last term of the variance (should be \sqrt{2/(1+2\sigma^2)}?))

rtist

回复第7楼的谢益辉：That's exactly how I constructed this example, by making it sufficiently concentrated around zero. I'm happy to see that you agree with this.

This came up to me during the analysis of a real data set (nothing to do with Bayesian). When I removed a random effect from a hierarchical model, the marginal variance got increased. That is, the contribution of this random factor to the variance in the data is negative. Maybe more counter-intuitive... But the problem is the same as the above artificial example. Thankfully, such ill cases won't happen for usual mixed linear models.

(why should there be an extra 2?)

rtist

Back the Bayesian world, it seems signifying the role of parametrization on the conceptual vagueness, and hence the development of Jeffrey's invariant prior.

yihui

回复第8楼的 rtist：Oh, forget it... you are right

little_stat

'noninformative' does not necessarily mean 'vague'. 'vague' priors are noninformative usually only for centroid parameters (for instance 'mean'), but not for variance components. the parameter p in your case is a reparameterized variance component. hence vague normal should never be used for that case.

the idea of noninformative priors (nowadays referred as objective priors) is to maximize information gained from data and alleviate affect from priors. it's not easy to handle objective bayes. this is still an active field.

rtist

回复第11楼的 little_stat：

Thanks. That makes sense to me.

My artificial example is not a very good one. Especially, p is not identifiable from the data model alone, as -p and p cannot be distinguished. Probably I should treat <bblatex>\theta=p^2</bblatex> as the parameter, then the equivalent prior is a scaled chi-square with 1 df. As <bblatex>\theta</bblatex> is simply log(precision), a large scale factor (<bblatex>\sigma^4</bblatex>) corresponds to large precision. So, it is highly informative, instead of non-informative.

Still, this seems to me primarily the problem of parameterization.

Do you mean that no such counter-examples can be constructed through reparameterization for location parameters?