Merge pull request #1021 from ICB-DCM/develop

* Warn only once if using ineffiecient objective settings (#996)

* warn only once about inefficient objective settings regarding hess

* sacess optimizer (#988)

Adds a shared-memory implementation of SaCeSS (https://doi.org/10.1186/s12859-016-1452-4).

Closes #284

* Fix misuse of start indices in waterfall plot (#1000)

* pypesto.Result: allow passing results to `__init__` (#998)

* Updated flake8 to ignore Error B028 from bugbear until support for python 3.8 runs out. (#1005)

closes #1003

* Fix large function values in clustering for visualizations (#999)

* Dynesty sampler: get MCMC-like samples; save/load internal sampler (#1002)

* Doc typo fixes (#995)

* fixing typos

Co-authored-by: PaulJonasJost <[email protected]>

* Hierarchical: separate standard and hierarchical PEtab validators (#1006)

* Add CODEOWNERS (#1001)


Co-authored-by: Yannik Schaelte <[email protected]>
Co-authored-by: Fabian Fröhlich <[email protected]>
Co-authored-by: Stephan Grein <[email protected]>

* Fix test/sample/test_sample.py::test_samples_cis failures (#1004)

Fixed recurrent issue with `test/sample/test_sample.py::test_samples_cis` by:
- Manually setting burn-in index
- Shorten the chain length down to 1000 as now the burn-in is manually set

fixes #897

* Fix cma documentation (#987)

Co-authored-by: Paul Jonas Jost <[email protected]>
Co-authored-by: Daniel Weindl <[email protected]>

* Add list of publications using pypesto (#1008)

Closes #820

* Sacess fixes (#997)

* Fix TypeError for size-1-arrays
* Optional arguments
* logging
* optional multiprocessing instead of multithreading
* cleanup

* black update (#1010)

* black update

* Ignoring D202 now as it clashes with black

* update to parameter correlation (#1009)

* added a save and show parameter. changed default colors to blue-white-red, added the option to return the list of parameters for other inspections.

* removing suggestions of autosave again.

* Doc: Install amici on RTD (#1016)

To run example notebooks (-> https://readthedocs.org/projects/pypesto/builds/19417900/)

* added option to profile the whole parameter bounds. (#1014)

* added option to profile the whole parameter bounds.

* combined stop profile check into one

* Optimization Parameter scatter plot (#1015)

* Small update to parameter historgram

* Added parameter scatter plot for optimization.

* Renamed to optimization_scatter, added two tests.

* Added to an example doc. Fixed test

* Added id to optimization.

* Added correct id to optimization. Increased possible runtime

* small change

* fixed typo

* More informative doc

---------

Co-authored-by: Polina Lakrisenko <[email protected]>

* remove alternative formats build bc nobody uses those (#1022)

* PetabImporter.create_startpoint_method - kwargs, not-None (#1018)

* Add kwarg forwarding to PetabImporter.create_startpoint_method (to use checked startpoints more conveniently)
* Always return a startpoint method

* Prepare release 0.2.16 (#1020)

* initial commit

* Updated changelog and version

* Updated changelog

---------

Co-authored-by: Simon Merkt <[email protected]>
Co-authored-by: Daniel Weindl <[email protected]>
Co-authored-by: Dilan Pathirana <[email protected]>
Co-authored-by: Lea Seep <[email protected]>
Co-authored-by: Yannik Schaelte <[email protected]>
Co-authored-by: Fabian Fröhlich <[email protected]>
Co-authored-by: Stephan Grein <[email protected]>
Co-authored-by: Elba Raimúndez <[email protected]>
Co-authored-by: Yannik Schälte <[email protected]>
Co-authored-by: Polina Lakrisenko <[email protected]>

.flake8

@@ -19,6 +19,10 @@ extend-ignore =
   D107
   # Empty method in abstract base class
   B027
+  # Using f"{foo!r}" instead of f"'{foo}'" (only until April 2023)
+  B028
+  # No blank lines allowed after function docstring. (Clashing with black)
+  D202
 
 per-file-ignores =
     # Imported but unused

.github/CODEOWNERS

@@ -0,0 +1,46 @@
+# see https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners
+
+# default owners
+* @ICB-DCM/pypesto-maintainers
+
+/doc/example/hdf5_storage.ipynb @PaulJonasJost
+/doc/example/hierarchical.ipynb @dilpath @dweindl @yannikschaelte
+/doc/example/julia.ipynb @yannikschaelte
+/doc/example/model_selection.ipynb @dilpath
+/doc/example/petab_import.ipynb @dweindl @FFroehlich
+/doc/example/sampler_study.ipynb @dilpath @yannikschaelte
+/doc/example/sampling_diagnostics.ipynb @dilpath @yannikschaelte
+/doc/example/store.ipynb @PaulJonasJost
+/doc/example/synthetic_data.ipynb @dilpath
+/docker/ @dweindl
+/pypesto/engine/ @yannikschaelte
+/pypesto/engine/mpi_pool.py @PaulJonasJost
+/pypesto/ensemble/ @dilpath
+/pypesto/hierarchical/ @dilpath @dweindl @yannikschaelte
+/pypesto/hierarchical/optimal_scaling_approach/ @doresic
+/pypesto/history/ @PaulJonasJost
+/pypesto/objective/ @yannikschaelte
+/pypesto/objective/amici/ @yannikschaelte @dweindl @FFroehlich
+/pypesto/objective/jax/ @FFroehlich
+/pypesto/objective/aesara/ @FFroehlich
+/pypesto/optimize/ @yannikschaelte
+/pypesto/petab/ @dweindl @FFroehlich
+/pypesto/predict/ @PaulJonasJost @dilpath
+/pypesto/problem/ @yannikschaelte
+/pypesto/profile/ @PaulJonasJost
+/pypesto/result/ @yannikschaelte
+/pypesto/sample/ @dilpath @yannikschaelte
+/pypesto/select/ @dilpath
+/pypesto/startpoint/ @yannikschaelte
+/pypesto/store/ @PaulJonasJost
+/pypesto/visualize/ @plakrisenko @stephanmg
+/test/base/ @yannikschaelte
+/test/doc/ @yannikschaelte
+/test/hierarchical/ @dilpath @dweindl @yannikschaelte
+/test/julia/ @yannikschaelte
+/test/optimize/ @yannikschaelte
+/test/petab/ @dweindl @FFroehlich
+/test/profile/ @PaulJonasJost
+/test/sample/ @dilpath @yannikschaelte
+/test/select/ @dilpath
+/test/visualize/ @plakrisenko @stephanmg

.github/workflows/ci.yml

@@ -38,7 +38,7 @@ jobs:
       run: .github/workflows/install_deps.sh amici
 
     - name: Run tests
-      timeout-minutes: 15
+      timeout-minutes: 25
       run: tox -e base
 
     - name: Coverage
@@ -286,7 +286,7 @@ jobs:
       run: pip install cffconvert && cffconvert --validate
 
     - name: Build doc
-      timeout-minutes: 5
+      timeout-minutes: 10
       run: tox -e doc
 
   notebooks1:

.pre-commit-config.yaml

@@ -11,12 +11,12 @@
 
 repos:
 - repo: https://github.com/psf/black
-  rev: 22.12.0
+  rev: 23.1.0
   hooks:
   - id: black
     description: The uncompromising code formatter
 - repo: https://github.com/pycqa/isort
-  rev: 5.11.1
+  rev: 5.12.0
   hooks:
   - id: isort
     name: isort (python)
@@ -27,7 +27,7 @@ repos:
     name: isort (pyi)
     types: [pyi]
 - repo: https://github.com/nbQA-dev/nbQA
-  rev: 1.5.3
+  rev: 1.6.3
   hooks:
   - id: nbqa-black
   - id: nbqa-pyupgrade

.readthedocs.yml

@@ -8,17 +8,21 @@ version: 2
 # Build documentation in the doc/ directory with Sphinx
 sphinx:
   configuration: doc/conf.py
+  fail_on_warning: True
 
 # python requirements
 python:
-  version: 3.8
   install:
     - method: pip
       path: .
       extra_requirements:
         - doc
+        - amici
 
-# what to build
-formats:
-  - htmlzip
-  - pdf
+build:
+  os: "ubuntu-20.04"
+  apt_packages:
+    - libatlas-base-dev
+    - swig
+  tools:
+    python: "3.10"

CHANGELOG.rst

@@ -6,6 +6,38 @@ Release notes
 ..........
 
 
+0.2.16 (2023-02-23)
+-------------------
+
+* Optimize:
+    * sacess optimizer (#988, #997)
+    * Warn only once if using ineffiecient objective settings (#996)
+    * Hierarchical Optimization (#1006)
+    * Fix cma documentation (#987)
+* Petab
+    * Improvement to create_startpoint_method() (#1018)
+* Sampling:
+    * Dynesty sampler (#1002)
+    * Fix test/sample/test_sample.py::test_samples_cis failures (#1004)
+* Visualization:
+    * Fix misuse of start indices in waterfall plot (#1000)
+    * Fix large function values in clustering for visualizations (#999)
+    * parameter correlation diverging color scheme (#1009)
+    * Optimization Parameter scatter plot (#1015)
+* Profiling:
+    * added option to profile the whole parameter bounds. (#1014)
+* General
+    * Add CODEOWNERS (#1001)
+    * Add list of publications using pypesto (#1008)
+    * allow passing results to __init__  of pypesto.Result (#998)
+    * Updated flake8 to ignore Error B028 from bugbear until support for python 3.8 runs out. (#1005)
+    * black update (#1010)
+    * Doc typo fixes (#995)
+    * Doc: Install amici on RTD (#1016)
+    * Add getting_started notebook (#1023)
+    * remove alernative formats build (#1022)
+
+
 0.2.15 (2022-12-21)
 -------------------
 

doc/conf.py

@@ -48,6 +48,8 @@
     'IPython.sphinxext.ipython_console_highlighting',
     # support markdown-based docs
     'myst_parser',
+    # bibtex references
+    'sphinxcontrib.bibtex',
 ]
 
 # default autodoc options
@@ -75,6 +77,8 @@
     'amici': ('https://amici.readthedocs.io/en/latest/', None),
 }
 
+bibtex_bibfiles = ["using_pypesto.bib"]
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 

doc/example.rst

@@ -23,6 +23,7 @@ Getting started
 .. toctree::
    :maxdepth: 2
 
+   example/getting_started.ipynb
    example/rosenbrock.ipynb
 
 PEtab and AMICI

doc/example/boehm_JProteomeRes2014/Boehm_JProteomeRes2014.yaml

@@ -0,0 +1,13 @@
+format_version: 1
+parameter_file: parameters_Boehm_JProteomeRes2014.tsv
+problems:
+- condition_files:
+  - experimentalCondition_Boehm_JProteomeRes2014.tsv
+  measurement_files:
+  - measurementData_Boehm_JProteomeRes2014.tsv
+  observable_files:
+  - observables_Boehm_JProteomeRes2014.tsv
+  sbml_files:
+  - boehm_JProteomeRes2014.xml
+  visualization_files:
+  - visualizationSpecification_Boehm_JProteomeRes2014.tsv

doc/example/boehm_JProteomeRes2014/experimentalCondition_Boehm_JProteomeRes2014.tsv

@@ -0,0 +1,2 @@
+conditionId	conditionName
+model1_data1	condition1

doc/example/boehm_JProteomeRes2014/measurementData_Boehm_JProteomeRes2014.tsv

@@ -0,0 +1,49 @@
+observableId	preequilibrationConditionId	simulationConditionId	measurement	time	observableParameters	noiseParameters	datasetId
+pSTAT5A_rel		model1_data1	7.901072999	0.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	66.36349397	2.5		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	81.17132392	5.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	94.73030806	10.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	95.11648305	15.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	91.44171655	20.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	91.25709923	30.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	93.67229784	40.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	88.75423282	50.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	85.26970322	60.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	81.13239534	80.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	76.13592848	100.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	65.24805913	120.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	42.59965871	160.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	25.15779754	200.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5A_rel		model1_data1	15.4301824	240.0		sd_pSTAT5A_rel	model1_data1_pSTAT5A_rel
+pSTAT5B_rel		model1_data1	4.596533343	0.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	29.63454599	2.5		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	46.04380647	5.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	81.97473362	10.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	80.5716093	15.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	79.03571964	20.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	75.67238037	30.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	71.62471986	40.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	69.06286328	50.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	67.14738432	60.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	60.89947629	80.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	54.80925777	100.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	43.98128998	120.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	29.77145816	160.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	20.08901656	200.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+pSTAT5B_rel		model1_data1	10.96184517	240.0		sd_pSTAT5B_rel	model1_data1_pSTAT5B_rel
+rSTAT5A_rel		model1_data1	14.72316822	0.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	33.76234229	2.5		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	36.79985129	5.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	49.71760229	10.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	46.9281201	15.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	47.83657456	20.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	46.92872725	30.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	40.59775294	40.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	43.78366389	50.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	44.45738765	60.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	41.32715926	80.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	41.06273321	100.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	39.23583003	120.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	36.61946054	160.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	34.8937144	200.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel
+rSTAT5A_rel		model1_data1	32.21107716	240.0		sd_rSTAT5A_rel	model1_data1_rSTAT5A_rel