This report documents the results of a simulation based calibration (SBC) run for OncoBayes2
. TODO
The calibration data presented here has been generated at and with the OncoBayes
git version as:
## Created: 2019-11-12 13:52:03 UTC
## git hash: c0c7b22c8dcb7e8bddd937392dad247a164c0ac2
## MD5: 227529ff1e63b08a064d8b0a702d37a5
The MD5 hash of the calibration data file presented here must match the above listed MD5:
## calibration.rds
## "227529ff1e63b08a064d8b0a702d37a5"
Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).
Self-consistency of any Bayesian analysis with a proper prior:
\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]
SBC procedure:
Repeat \(s=1, ..., S\) times:
Sample from the prior \[\tilde{\theta} \sim p(\theta)\]
Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]
Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]
Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]
The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]
The fake data simulation function returns … TODO. Please refer to the sbc_tools.R
and make_reference_rankhist.R
R programs for the implementation details.
The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.
param | statistic | df | p.value |
---|---|---|---|
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] | 41.805 | 31 | 0.093 |
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] | 17.395 | 31 | 0.977 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] | 33.165 | 31 | 0.362 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] | 45.555 | 31 | 0.044 |
param | statistic | df | p.value |
---|---|---|---|
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] | 26.298 | 31 | 0.707 |
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] | 32.698 | 31 | 0.384 |
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] | 39.814 | 31 | 0.133 |
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] | 32.627 | 31 | 0.387 |
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] | 46.701 | 31 | 0.035 |
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] | 26.112 | 31 | 0.716 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] | 27.930 | 31 | 0.625 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] | 21.421 | 31 | 0.900 |
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] | 32.442 | 31 | 0.396 |
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] | 24.134 | 31 | 0.805 |
param | statistic | df | p.value |
---|---|---|---|
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] | 21.184 | 31 | 0.907 |
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] | 23.072 | 31 | 0.847 |
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] | 26.163 | 31 | 0.713 |
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] | 25.933 | 31 | 0.725 |
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] | 42.125 | 31 | 0.088 |
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] | 27.430 | 31 | 0.650 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] | 37.325 | 31 | 0.201 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] | 23.258 | 31 | 0.840 |
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] | 25.190 | 31 | 0.759 |
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] | 21.216 | 31 | 0.906 |
param | statistic | df | p.value |
---|---|---|---|
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2] * DosesAdm3/dref[3])] | 29.434 | 31 | 0.547 |
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] | 40.326 | 31 | 0.122 |
mu_eta[I(DosesAdm1/dref[1] * DosesAdm3/dref[3])] | 21.530 | 31 | 0.897 |
mu_eta[I(DosesAdm2/dref[2] * DosesAdm3/dref[3])] | 39.136 | 31 | 0.150 |
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] | 24.954 | 31 | 0.770 |
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] | 32.589 | 31 | 0.389 |
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] | 27.258 | 31 | 0.659 |
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] | 24.870 | 31 | 0.773 |
mu_log_beta[I(log(DosesAdm3/dref[3])),intercept] | 32.371 | 31 | 0.399 |
mu_log_beta[I(log(DosesAdm3/dref[3])),log_slope] | 27.245 | 31 | 0.660 |
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2] * DosesAdm3/dref[3])] | 19.840 | 31 | 0.939 |
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] | 26.285 | 31 | 0.708 |
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm3/dref[3])] | 42.432 | 31 | 0.083 |
tau_eta[1,I(DosesAdm2/dref[2] * DosesAdm3/dref[3])] | 26.355 | 31 | 0.704 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] | 15.872 | 31 | 0.989 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] | 23.418 | 31 | 0.834 |
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] | 25.766 | 31 | 0.732 |
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] | 15.008 | 31 | 0.993 |
tau_log_beta[1,I(log(DosesAdm3/dref[3])),intercept] | 16.896 | 31 | 0.981 |
tau_log_beta[1,I(log(DosesAdm3/dref[3])),log_slope] | 25.504 | 31 | 0.745 |
## R version 3.6.1 (2019-07-05)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 16.04.6 LTS
##
## Matrix products: default
## BLAS: /usr/lib/libblas/libblas.so.3.6.0
## LAPACK: /usr/lib/lapack/liblapack.so.3.6.0
##
## locale:
## [1] C
##
## attached base packages:
## [1] tools stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] abind_1.4-5 Formula_1.2-3 mvtnorm_1.0-11
## [4] RBesT_1.4-0 rstan_2.19.2 StanHeaders_2.19.0
## [7] checkmate_1.9.4 OncoBayes2_0.5-3 testthat_2.2.1
## [10] Rcpp_1.0.2 devtools_2.2.1 usethis_1.5.1
## [13] ggplot2_3.2.1 broom_0.5.2 tidyr_1.0.0
## [16] dplyr_0.8.3 assertthat_0.2.1 knitr_1.25
## [19] rmarkdown_1.16
##
## loaded via a namespace (and not attached):
## [1] lattice_0.20-38 prettyunits_1.0.2 ps_1.3.0
## [4] zeallot_0.1.0 rprojroot_1.3-2 digest_0.6.21
## [7] plyr_1.8.4 R6_2.4.0 ggridges_0.5.1
## [10] backports_1.1.5 stats4_3.6.1 evaluate_0.14
## [13] highr_0.8 pillar_1.4.2 rlang_0.4.0
## [16] lazyeval_0.2.2 rstudioapi_0.10 callr_3.3.2
## [19] labeling_0.3 desc_1.2.0 stringr_1.4.0
## [22] loo_2.1.0 munsell_0.5.0 compiler_3.6.1
## [25] xfun_0.10 pkgconfig_2.0.3 pkgbuild_1.0.6
## [28] rstantools_2.0.0 htmltools_0.4.0 tidyselect_0.2.5
## [31] tibble_2.1.3 gridExtra_2.3 codetools_0.2-16
## [34] matrixStats_0.55.0 crayon_1.3.4 withr_2.1.2
## [37] grid_3.6.1 nlme_3.1-141 gtable_0.3.0
## [40] lifecycle_0.1.0 magrittr_1.5 scales_1.0.0
## [43] cli_1.1.0 stringi_1.4.3 fs_1.3.1
## [46] remotes_2.1.0 ellipsis_0.3.0 generics_0.0.2
## [49] vctrs_0.2.0 glue_1.3.1 purrr_0.3.3
## [52] processx_3.4.1 pkgload_1.0.2 parallel_3.6.1
## [55] yaml_2.2.0 inline_0.3.15 colorspace_1.4-1
## [58] sessioninfo_1.1.1 bayesplot_1.7.0 memoise_1.1.0