Simulation based calibration for OncoBayes2

Fri Nov 15 20:49:46 2019

This report documents the results of a simulation based calibration (SBC) run for OncoBayes2. TODO

The calibration data presented here has been generated at and with the OncoBayes git version as:

## Created:  2019-11-12 13:52:03 UTC
## git hash: c0c7b22c8dcb7e8bddd937392dad247a164c0ac2
## MD5:      227529ff1e63b08a064d8b0a702d37a5

The MD5 hash of the calibration data file presented here must match the above listed MD5:

##                    calibration.rds 
## "227529ff1e63b08a064d8b0a702d37a5"

Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

  1. Sample from the prior \[\tilde{\theta} \sim p(\theta)\]

  2. Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]

  3. Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]

  4. Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

Model description TODO

The fake data simulation function returns … TODO. Please refer to the sbc_tools.R and make_reference_rankhist.R R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

SBC results

Model 1: Single-agent logistic regression

Component intercept/slopes

Means

Standard deviations (tau’s)

Model 2: Double combination, fully exchangeable

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (from exchangeable part)

Mean

Standard deviation

Model 3: Double combination, EXchangeable/NonEXchangeable model

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (from exchangeable part)

Mean

Standard deviation (tau)

Model 4: Triple combination, EX/NEX model

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (means from exchangeable part)

Mean

Standard deviation (tau)

\(\chi^2\) Statistic, Model 1: Single-agent logistic regression

param statistic df p.value
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] 41.805 31 0.093
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] 17.395 31 0.977
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] 33.165 31 0.362
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] 45.555 31 0.044

\(\chi^2\) Statistic, Model 2: Double combination, fully exchangeable

param statistic df p.value
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 26.298 31 0.707
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] 32.698 31 0.384
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] 39.814 31 0.133
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] 32.627 31 0.387
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] 46.701 31 0.035
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 26.112 31 0.716
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] 27.930 31 0.625
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] 21.421 31 0.900
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] 32.442 31 0.396
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] 24.134 31 0.805

\(\chi^2\) Statistic, Model 3: Double combination, EXchangeable/NonEXchangeable model

param statistic df p.value
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 21.184 31 0.907
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] 23.072 31 0.847
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] 26.163 31 0.713
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] 25.933 31 0.725
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] 42.125 31 0.088
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 27.430 31 0.650
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] 37.325 31 0.201
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] 23.258 31 0.840
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] 25.190 31 0.759
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] 21.216 31 0.906

\(\chi^2\) Statistic, Model 4: Triple combination, EX/NEX model

param statistic df p.value
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2] * DosesAdm3/dref[3])] 29.434 31 0.547
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 40.326 31 0.122
mu_eta[I(DosesAdm1/dref[1] * DosesAdm3/dref[3])] 21.530 31 0.897
mu_eta[I(DosesAdm2/dref[2] * DosesAdm3/dref[3])] 39.136 31 0.150
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] 24.954 31 0.770
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] 32.589 31 0.389
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] 27.258 31 0.659
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] 24.870 31 0.773
mu_log_beta[I(log(DosesAdm3/dref[3])),intercept] 32.371 31 0.399
mu_log_beta[I(log(DosesAdm3/dref[3])),log_slope] 27.245 31 0.660
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2] * DosesAdm3/dref[3])] 19.840 31 0.939
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 26.285 31 0.708
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm3/dref[3])] 42.432 31 0.083
tau_eta[1,I(DosesAdm2/dref[2] * DosesAdm3/dref[3])] 26.355 31 0.704
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] 15.872 31 0.989
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] 23.418 31 0.834
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] 25.766 31 0.732
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] 15.008 31 0.993
tau_log_beta[1,I(log(DosesAdm3/dref[3])),intercept] 16.896 31 0.981
tau_log_beta[1,I(log(DosesAdm3/dref[3])),log_slope] 25.504 31 0.745

Session Info

## R version 3.6.1 (2019-07-05)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 16.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/libblas/libblas.so.3.6.0
## LAPACK: /usr/lib/lapack/liblapack.so.3.6.0
## 
## locale:
## [1] C
## 
## attached base packages:
## [1] tools     stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] abind_1.4-5        Formula_1.2-3      mvtnorm_1.0-11    
##  [4] RBesT_1.4-0        rstan_2.19.2       StanHeaders_2.19.0
##  [7] checkmate_1.9.4    OncoBayes2_0.5-3   testthat_2.2.1    
## [10] Rcpp_1.0.2         devtools_2.2.1     usethis_1.5.1     
## [13] ggplot2_3.2.1      broom_0.5.2        tidyr_1.0.0       
## [16] dplyr_0.8.3        assertthat_0.2.1   knitr_1.25        
## [19] rmarkdown_1.16    
## 
## loaded via a namespace (and not attached):
##  [1] lattice_0.20-38    prettyunits_1.0.2  ps_1.3.0          
##  [4] zeallot_0.1.0      rprojroot_1.3-2    digest_0.6.21     
##  [7] plyr_1.8.4         R6_2.4.0           ggridges_0.5.1    
## [10] backports_1.1.5    stats4_3.6.1       evaluate_0.14     
## [13] highr_0.8          pillar_1.4.2       rlang_0.4.0       
## [16] lazyeval_0.2.2     rstudioapi_0.10    callr_3.3.2       
## [19] labeling_0.3       desc_1.2.0         stringr_1.4.0     
## [22] loo_2.1.0          munsell_0.5.0      compiler_3.6.1    
## [25] xfun_0.10          pkgconfig_2.0.3    pkgbuild_1.0.6    
## [28] rstantools_2.0.0   htmltools_0.4.0    tidyselect_0.2.5  
## [31] tibble_2.1.3       gridExtra_2.3      codetools_0.2-16  
## [34] matrixStats_0.55.0 crayon_1.3.4       withr_2.1.2       
## [37] grid_3.6.1         nlme_3.1-141       gtable_0.3.0      
## [40] lifecycle_0.1.0    magrittr_1.5       scales_1.0.0      
## [43] cli_1.1.0          stringi_1.4.3      fs_1.3.1          
## [46] remotes_2.1.0      ellipsis_0.3.0     generics_0.0.2    
## [49] vctrs_0.2.0        glue_1.3.1         purrr_0.3.3       
## [52] processx_3.4.1     pkgload_1.0.2      parallel_3.6.1    
## [55] yaml_2.2.0         inline_0.3.15      colorspace_1.4-1  
## [58] sessioninfo_1.1.1  bayesplot_1.7.0    memoise_1.1.0