Introduction
Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).
Self-consistency of any Bayesian analysis with a proper prior:
\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]
SBC procedure:
Repeat \(s=1, ..., S\) times:
Sample from the prior \[\tilde{\theta} \sim p(\theta)\]
Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]
Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]
Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]
The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]
Hierarchical intercept only (random-effects intercept) model for binomial, gaussian and Poisson likelihood
Likelihood:
- Binary \[y_i|\theta_{j} \sim \mbox{Bernoulli}(\theta_j), \] \[g(\theta) = \mbox{logit}(\theta)\]
- Normal \[y_i|\theta_{j} \sim \mbox{Normal}(\theta_j, \sigma^2), \] \[g(\theta) = \theta\]
- Poisson \[y_i|\theta_{j} \sim \mbox{Poisson}(\theta_j), \] \[g(\theta) = \log(\theta)\]
Hierarchical prior:
\[ g(\theta_j)|\mu,\tau \sim \mbox{Normal}(\mu, \tau^2)\]
\[\mu \sim \mbox{Normal}(m_\mu, s^2_\mu)\] \[\tau \sim \mbox{Normal}^+(0, s^2_\tau)\]
The fake data simulation function returns for binomial and Poisson data the sum of the responses while for normal the mean summary is used. Please refer to the sbc_tools.R and make_reference_rankhist.R R programs for the implementation details.
The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.
Dense Scenario, \(\mu\)
Dense Scenario, \(\tau\)
Sparse Scenario, \(\mu\)
Sparse Scenario, \(\tau\)
\(\chi^2\) Statistic, \(\mu\)
| problem | likelihood | sd_tau | statistic | df | p.value |
|---|---|---|---|---|---|
| dense | binomial | 0.5 | 46.886 | 63 | 0.936 |
| dense | binomial | 1 | 64.717 | 63 | 0.416 |
| dense | gaussian | 0.5 | 74.099 | 63 | 0.160 |
| dense | gaussian | 1 | 46.170 | 63 | 0.945 |
| dense | poisson | 0.5 | 44.467 | 63 | 0.963 |
| dense | poisson | 1 | 57.254 | 63 | 0.680 |
| sparse | binomial | 0.5 | 43.994 | 63 | 0.967 |
| sparse | binomial | 1 | 53.734 | 63 | 0.791 |
| sparse | gaussian | 0.5 | 50.266 | 63 | 0.877 |
| sparse | gaussian | 1 | 57.446 | 63 | 0.674 |
| sparse | poisson | 0.5 | 62.502 | 63 | 0.494 |
| sparse | poisson | 1 | 70.490 | 63 | 0.242 |
\(\chi^2\) Statistic, \(\tau\)
| problem | likelihood | sd_tau | statistic | df | p.value |
|---|---|---|---|---|---|
| dense | binomial | 0.5 | 56.141 | 63 | 0.717 |
| dense | binomial | 1 | 69.568 | 63 | 0.266 |
| dense | gaussian | 0.5 | 67.635 | 63 | 0.322 |
| dense | gaussian | 1 | 61.056 | 63 | 0.546 |
| dense | poisson | 0.5 | 73.741 | 63 | 0.167 |
| dense | poisson | 1 | 54.080 | 63 | 0.781 |
| sparse | binomial | 0.5 | 66.790 | 63 | 0.348 |
| sparse | binomial | 1 | 69.734 | 63 | 0.261 |
| sparse | gaussian | 0.5 | 53.363 | 63 | 0.801 |
| sparse | gaussian | 1 | 61.069 | 63 | 0.545 |
| sparse | poisson | 0.5 | 56.627 | 63 | 0.701 |
| sparse | poisson | 1 | 65.331 | 63 | 0.396 |
Session Info
## R version 3.5.3 (2019-03-11)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 16.04.6 LTS
##
## Matrix products: default
## BLAS: /usr/lib/libblas/libblas.so.3.6.0
## LAPACK: /usr/lib/lapack/liblapack.so.3.6.0
##
## locale:
## [1] C
##
## attached base packages:
## [1] tools stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] bindrcpp_0.2.2 rstan_2.18.2 StanHeaders_2.18.0-1
## [4] RBesT_1.4-0 testthat_2.0.1 Rcpp_1.0.1
## [7] usethis_1.4.0 devtools_2.0.1 ggplot2_3.1.0
## [10] broom_0.5.1 tidyr_0.8.2 dplyr_0.7.8
## [13] assertthat_0.2.1 knitr_1.22 rmarkdown_1.11
##
## loaded via a namespace (and not attached):
## [1] mvtnorm_1.0-8 lattice_0.20-38 prettyunits_1.0.2
## [4] ps_1.2.1 rprojroot_1.3-2 digest_0.6.18
## [7] R6_2.4.0 plyr_1.8.4 ggridges_0.5.1
## [10] backports_1.1.3 stats4_3.5.3 evaluate_0.13
## [13] highr_0.8 pillar_1.3.0 rlang_0.3.3
## [16] lazyeval_0.2.1 rstudioapi_0.10 callr_3.1.0
## [19] checkmate_1.8.5 labeling_0.3 desc_1.2.0
## [22] stringr_1.4.0 loo_2.0.0 munsell_0.5.0
## [25] compiler_3.5.3 xfun_0.6 pkgconfig_2.0.2
## [28] pkgbuild_1.0.2 htmltools_0.3.6 tidyselect_0.2.5
## [31] tibble_1.4.2 gridExtra_2.3 codetools_0.2-16
## [34] matrixStats_0.54.0 crayon_1.3.4 withr_2.1.2
## [37] grid_3.5.3 nlme_3.1-137 gtable_0.2.0
## [40] magrittr_1.5 scales_1.0.0 cli_1.0.1
## [43] stringi_1.4.3 reshape2_1.4.3 fs_1.2.6
## [46] remotes_2.0.2 generics_0.0.2 Formula_1.2-3
## [49] glue_1.3.1 purrr_0.2.5 processx_3.2.1
## [52] pkgload_1.0.2 parallel_3.5.3 yaml_2.2.0
## [55] inline_0.3.15 colorspace_1.3-2 sessioninfo_1.1.1
## [58] bayesplot_1.6.0 memoise_1.1.0 bindr_0.1.1