Fixed models / data files for cmdstan

[PhilClemson]

This basically fixes the majority of models that were either broken or had missing / incorrect syntax for the R dump data files used by cmdstan. They can be checked using the script found in the performance tests repository (https://github.com/stan-dev/performance-tests-cmdstan), e.g.:

python ./performance-tests-cmdstan/runPerformanceTests.py directories "performance-tests-cmdstan/example-models" --overwrite-gold --method=sample --num-samples=1000

Here is a list of the updated models:

misc/ecology/occupancy/occupancy
misc/ecology/mark-recapture/mark-recapture
misc/ecology/mark-recapture/cjs-K
misc/ecology/mark-recapture/mark-recapture-2
misc/ecology/mark-recapture/mark-recapture-3
misc/ecology/mark-recapture/cjs
misc/linear-regression_regression
misc/linear-regression_regression_std
misc/garch_arch1
misc/garch_koyck
misc/garch_garch1_1
misc/hier_multivariate_hier_multivariate
misc/moving-avg/arma11
misc/moving-avg/arma11_alt
misc/moving-avg/maQ
misc/moving-avg/stochastic-volatility
misc/moving-avg/ma2
misc/moving-avg/stochastic-volatility-optimized
misc/gaussian-process/gp-predict-analytic
misc/gaussian-process/gp-sim-multi
misc/gaussian-process/gp-fit-ARD
misc/gaussian-process/gp-fit-latent
misc/dlm/fx/equicorr
misc/dlm/fx/factor
misc/sur/sur/imrpoper
misc/sur/sur
misc/hmm/hmm
misc/hmm/hmm-analytic
misc/hmm/hmm-semisup
misc/hmm/hmm-sufficient
misc/nnmf/nnmf
misc/nnmf/nnmf/vec
knitr/simplest-regression/fake-data
knitr/pool-binary-trials/pool
knitr/pool-binary-trials/no-pool
knitr/pool-binary-trials/hier
knitr/pool-binary-trials/hier-logit-centered
knitr/pool-binary-trials/hier-logit
knitr/chapter2/worldcup/fixed
knitr/chapter2/worldcup/first/try
knitr/chapter2/worldcup/first/try/noprior
knitr/chapter2/worldcup/with/replication
knitr/chapter2/golf1
knitr/chapter2/worldcup/no/sqrt
knitr/car-iar-poisson/pois/re
knitr/car-iar-poisson/pois
knitr/car-iar-poisson/bym2/islands
knitr/car-iar-poisson/bym2/offset/only
knitr/car-iar-poisson/pois/icar
knitr/car-iar-poisson/bym2
knitr/car-iar-poisson/simple/iar
knitr/chapter1/fake-data
knitr/golf/golf1
knitr/world-cup/worldcup/fixed
knitr/world-cup/worldcup/first/try
knitr/world-cup/worldcup/first/try/noprior
knitr/world-cup/worldcup/with/replication
knitr/world-cup/worldcup/no/sqrt
ARM/Ch.25/earnings2
ARM/Ch.25/earnings
ARM/Ch.25/earnings/pt2
ARM/Ch.25/earnings/pt1
education/tutorial/twopl/reg/dif
education/tutorial/twopl/reg/noncentered
education/tutorial/twopl/reg/centered
education/tutorial/twopl/twopl
education/hierarchical/2pl/hierarchical/2pl
education/dina/independent/dina/independent
education/dina/independent/dina/nostructure
BPA/Ch.04/GLMM1
BPA/Ch.04/GLMM2
BPA/Ch.13/bluebug
BPA/Ch.13/owls/ms1
BPA/Ch.13/owls/ms2
BPA/Ch.13/Dynocc2
BPA/Ch.10/js/ms
BPA/Ch.12/binmix/cov
BPA/Ch.12/binmix
BPA/Ch.12/Nmix0
BPA/Ch.12/Nmix2
BPA/Ch.12/Nmix1
BPA/Ch.07/cjs/trap
BPA/Ch.07/cjs/temp/corr

[mitzimorris]

where did you get the data from?
I realize that in the past there have been attempts to add data - cf #151 - these were done in a brute-force way. After the PR associated with that issue went in, I cleaned up part of it by removing generated data from case studies where there wasn't any data.

it's important that the data used to test the model should be generated by the model or the process which the model is trying to approximate. using just the specified constraints on data and parameters to generate test is not enough - there is often domain knowledge which is not specified, e.g., for the icar models, the input data describes an areal map, and there are constraints on the relations between the map edges which aren't directly expressed in the data declarations.

for testing purposes, I would imagine that we're interested in testing well-specified models which are fitting data generated from the model and/or which is somehow pathological. this requires that all test data be annotated explaining what kind of data it is and what kind of behavoir we expect when fitting the model to the data.

[mitzimorris]

hi @PhilClemson - please allow me to clarify:

I think it would be better to curate the models used for testing, not make sure that we can use every single one of them. so I don't think this PR is a good idea.

[bob-carpenter]

I also think a curated set of test models makes much more sense for testing than all the stuff that's accumulated in our example-models repo.

Having said that, I would like to see something done to clean up example-models, which means:

• fixing models that we want to keep
• removing junk

In the first category are models there for pedagogical purposes, not for testing purposes, like the ones in directory BPA (from the Bayesian Population Analysis book). It'd be nice to at least clean those up. Pretty much everything in misc is just junk I threw up over the years for the manual or just as an example for someone. That can all be eliminated.

How did you generate new data for things like my hierarchical binary pooling example in knitr/pool-binary-trials?

[mitzimorris]

@PhilClemson - w/r/t to the icar-car-poisson models, I don't think your correction of the stan code is correct. did you test it on an areal map data that contained islands/singletons?

[PhilClemson]

where did you get the data from?
I realize that in the past there have been attempts to add data - cf #151 - these were done in a brute-force way. After the PR associated with that issue went in, I cleaned up part of it by removing generated data from case studies where there wasn't any data.

it's important that the data used to test the model should be generated by the model or the process which the model is trying to approximate. using just the specified constraints on data and parameters to generate test is not enough - there is often domain knowledge which is not specified, e.g., for the icar models, the input data describes an areal map, and there are constraints on the relations between the map edges which aren't directly expressed in the data declarations.

for testing purposes, I would imagine that we're interested in testing well-specified models which are fitting data generated from the model and/or which is somehow pathological. this requires that all test data be annotated explaining what kind of data it is and what kind of behavoir we expect when fitting the model to the data.

Hi @mitzimorris, thanks for taking a look through this.

Most of the data came from the .R and documentation found in the same folder as the stan files. For the BPA models I downloaded the data from the links provided. My main task was to standardise the models so that they could all be run using cmdstan. Fortunately in most cases it was just a case of generating the data.R file from existing code / data.

There were a few cases where we had to generate "compatible" data but not "curated" data. The knitr models are a good example of this.

hi @PhilClemson - please allow me to clarify:

I think it would be better to curate the models used for testing, not make sure that we can use every single one of them. so I don't think this PR is a good idea.

I understand, we just thought we would share these changes in the hope that they could be applied in general. We've found that having a large selection of models / data is useful, even if just for identifying bugs. It would definitely not be appropriate to judge performance statistics based on individual models, although looking at performance over a large population of models might still be useful.

I also think a curated set of test models makes much more sense for testing than all the stuff that's accumulated in our example-models repo.

Having said that, I would like to see something done to clean up example-models, which means:

* fixing models that we want to keep

* removing junk

In the first category are models there for pedagogical purposes, not for testing purposes, like the ones in directory BPA (from the Bayesian Population Analysis book). It'd be nice to at least clean those up. Pretty much everything in misc is just junk I threw up over the years for the manual or just as an example for someone. That can all be eliminated.

How did you generate new data for things like my hierarchical binary pooling example in knitr/pool-binary-trials?

That sounds like a good idea, although in that case maybe the junk models like those in misc could become a separate repository? I'd have to check with @alecksphillips on those specific examples, but as I've said above I imagine it will be compatible rather than curated data.

@PhilClemson - w/r/t to the icar-car-poisson models, I don't think your correction of the stan code is correct. did you test it on an areal map data that contained islands/singletons?

For these I generated the data from the corresponding .R files. I remember for bym2_islands.stan I was getting errors which seemed to be an indexing issue (which I fixed) but I might be wrong. I believe I used the nycTracts10 data.

[alecksphillips]

That sounds like a good idea, although in that case maybe the junk models like those in misc could become a separate repository? I'd have to check with @alecksphillips on those specific examples, but as I've said above I imagine it will be compatible rather than curated data.

Hi all,
Just to add, the bits of data I contributed I generated using the R markdown workbooks that were in those directories.

[mitzimorris]

For these I generated the data from the corresponding .R files. I remember for bym2_islands.stan I was getting errors which seemed to be an indexing issue (which I fixed) but I might be wrong. I believe I used the nycTracts10 data.

please revert all changes to the car-icar-poisson models and delete the data files you added. I will try to add appropriate datasets myself.

the bym2_islands.stan is correctly coded - it requires input where the areal map contains at least 2 disconnected subcomponents, whereas the data you're using is an areal map which is fully connected. the bym2_islands.stan model is checked into the case study because it's the next level of in complexity for that model, but I haven't yet extended the case study.

I agree that we should expand the suite of test models, but in the icar model case study, the models that consist of a regression plus just an icar or just a random effects are degenerate versions of the bym2 model when there's no spatial correlation or when there's perfect spatial correlation, so adding these to the test suite isn't going to tell you anything new. those models were included mainly to generate visualizations and stan programming pedegogy. they're just not worth running in a test suite - it's a waste of compute cycles / carbon credits.

[PhilClemson]

please revert all changes to the car-icar-poisson models and delete the data files you added. I will try to add appropriate datasets myself.

Thanks for the explanation, that makes a lot of sense and I agree. I've reverted the relevant commits.

[jgabry]

Thanks for doing all this work!

@mitzimorris Now that the icar-poisson commits are reverted, is this ready to merge?

[mitzimorris]

yes!

[jgabry]

Ok, great, will merge now. Thanks again @PhilClemson for doing all this!

[PhilClemson]

Thanks both, although I can't take all of the credit as I had help!