Bdtyktl

If I have a physical model

begin{equation}
y = frac{1}{beta_0} (beta_1 x_1 + beta_2 x_2)
end{equation}

and want to estimate coefficients $beta_0$, $beta_1$, and $beta_2$ from given data set $(y, x_1, x_2)$, with regression I would typically have to combine them into two coefficients $beta_1/beta_0$ and $beta_2/beta_0$ since that would be what I would be able to solve (contributions from $1/beta_0$ and $beta_1$ cannot be distinguished from each other individually).

However, if I have separate prior distributions for $beta_0$, $beta_1$, and $beta_2$ - is it possible to obtain sensible posterior distributions for each of them using MCMC or Laplace's method without combining the variables?

Edits Corrected title and variable name error based on comments and answers.

This is an case where the parameters are non-identifiable in your model. As you point out, contributions from the individual non-identifiable parameters in these ratios cannot be distinguished using the data. When using Bayesian analysis with a non-identifiable model, specification of a prior for all the individual non-identifiable parameters will still lead you to a valid posterior, and this posterior will converge to a point mass with more and more data. This often gives the misleading impression that Bayesian analysis renders the identifiability problem benign. However, it is crucial to note that the posterior in these cases is strongly affected by the prior in ways that do not vanish as we get more and more data. The identifiability problem is not rendered benign merely by using Bayesian analysis with priors.

To see exactly what I mean, define the minimal sufficient parameters $phi_1 equiv beta_1 / beta_0$ and $phi_2 equiv beta_2 / beta_0$. These are the parameters that are identified in the present model. Using the rules for density transformation, the posterior distribution for the three non-identifiable parameters of interest can be written as:

$$begin{equation} begin{aligned}
pi(beta_0, beta_1, beta_2 | mathbf{x}, mathbf{y})
&= frac{1}{beta_0^2} cdot pi(beta_0, phi_1, phi_2 | mathbf{x}, mathbf{y}) \[6pt]
&= frac{1}{beta_0^2} cdot p(beta_0 | phi_1, phi_2) cdot pi(phi_1, phi_2 | mathbf{x}, mathbf{y}). \[6pt]
end{aligned} end{equation}$$

Now, the posterior $pi(phi_1, phi_2 | mathbf{x}, mathbf{y})$ for the minimal sufficient parameters (which are identifiable) is determined by the prior assumptions and the data as normal. However, the density $p(beta_0 | phi_1, phi_2)$ is determined purely by the prior (i.e., it does not change as you get more data). This latter density is just an aspect of the assumed prior on the three non-identifiable parameters. Hence, the posterior of the non-identifiable parameters will be determined in large measure by a part that is purely a function of the prior.

Bayesian asymptotic theory tells us that, under broad conditions, the posterior distribution of identifiable parameters converges towards a point-mass on the true values. (More specifically, there are a number of convergence results that show asymptotic convergence to a normal distribution with mean that approaches the true parameter values and variance that approaches zero.) In the context of regression there are some conditions on the explanatory variables, but again, that convergence result holds broadly. Hence, under appropriate conditions, as $n rightarrow infty$ the density $pi(phi_1, phi_2 | mathbf{x}, mathbf{y})$ will converge closer and closer to a point-mass distribution on the true values $(phi_1^*, phi_2^*)$, and so in the limit you have:

$$pi(beta_0, beta_1, beta_2 | mathbf{x}, mathbf{y}) rightarrow
pi_infty(beta_0, beta_1, beta_2 | mathbf{x}_infty, mathbf{y}_infty)
propto frac{1}{beta_0^2} cdot p(beta_0 | phi_1^*, phi_2^*) .$$

We can see that this asymptotic density is affected by the data only through the true values of the minimal sufficient parameters. It is still heavily affected by the form of the density $p(beta_0 | phi_1, phi_2)$, which is a function of the prior. In other words, despite the fact that we get a valid posterior, and asymptotic convergence of the posterior, don't let that fool you into thinking that you have "solved" the identifiabiity problem.

Yes, with a Bayesian analysis you can get sensible posteriors and concentrate around sensible values to the point that the combination of information in prior and likelihood allow it. In this sense the Bayesian analysis can deal with this kind of situation a lot better than a frequentist analysis. However, it cannot get around the fundamental non-identifiability of the parameters in a model and the posteriors will still reflect the lack of identifiability of the parameters.

To use an example, let's use a simpler model that is just $log Z_i sim N(theta_1/theta_2, sigma^2)$ and let's assume that we have a huge amount of data. We'll have hardly any uncertainty around the ratio $theta_1/theta_2$, but lots of values for each parameter are still getting support in the likelihood. Thus, we just get still a wide marginal posterior distribution and the joint distribution does have a very strong correlation between the two parameters.

This is illustrated with example code below (using the re-parameterization $Y_i := log Z_i$ and $beta_j := log theta_j$).

Obviously, the marginal posterior would be wider, if our prior for the parameters had been wider and the posterior correlation less strong, if we had less data (then the parameter identifiability problem would be less obvious).

I would expect something similar to happen in your example - whether that's a problem or not is a different matter.

library(rstan)

library(bayesplot)

y <- exp(rnorm(10000, 0,1))



stancode <- "

data {

  int n;

  real y[n];

}



parameters{

  real beta0;

  real beta1;

  real<lower=0> sigma;

}



model {

  beta0  ~ normal(0,1);

  beta1 ~ normal(0,1);

  sigma ~ normal(0,1);

  y ~ normal(exp(beta1-beta0), sigma);

}

"



stanfit <- stan(model_code=stancode, data=list(n=length(y), y=y))



posterior <- as.matrix(stanfit)



mcmc_pairs(posterior, pars = c("beta0", "beta1"))