Merge pull request #160 from PEtab-dev/release_0.1.27

Release 0.1.27

.github/workflows/ci_tests.yml

@@ -11,7 +11,7 @@ jobs:
     strategy:
       matrix:
         platform: [windows-latest, macos-latest, ubuntu-latest]
-        python-version: [3.7, 3.8, 3.9, "3.10"]
+        python-version: ["3.7", "3.10"]
     runs-on: ${{ matrix.platform }}
 
     steps:
@@ -38,9 +38,9 @@ jobs:
 
     - name: Install dependencies
       run: |
-        python -m pip install --upgrade pip
+        python -m pip install --upgrade pip wheel
         pip install -r .ci_pip_reqs.txt
-        pip install .[reports,combine]
+        pip install .[reports,combine,tests]
 
     - name: Run flake8
       run: |

CHANGELOG.md

@@ -2,6 +2,62 @@
 
 ## 0.1 series
 
+### 0.1.27
+
+Features:
+* Added method to check if measurement time is at steady-state by @dilpath in
+  https://github.com/PEtab-dev/libpetab-python/pull/124
+* Create dummy simulation conditions dataframe for empty measurements by
+  @dilpath in https://github.com/PEtab-dev/libpetab-python/pull/127
+* Validator: Report empty noiseFormula by @dweindl in
+  https://github.com/PEtab-dev/libpetab-python/pull/134
+* Speedup visspec assembly / fix deprecation warning by @dweindl in
+  https://github.com/PEtab-dev/libpetab-python/pull/135
+* Handle incomplete PEtab problems in `petab.Problem.from_yaml` by @dweindl in
+  https://github.com/PEtab-dev/libpetab-python/pull/138
+* Argument forwarding for
+  `Problem.get_optimization_to_simulation_parameter_mapping` by @dweindl in
+  https://github.com/PEtab-dev/libpetab-python/pull/159
+* Added candidate schema for version 2  by @dweindl in
+  https://github.com/PEtab-dev/libpetab-python/pull/142
+* `get_parameter_df`: Allow any collection of parameter tables by @dweindl in
+  https://github.com/PEtab-dev/libpetab-python/pull/153,
+  @m-philipps in https://github.com/PEtab-dev/libpetab-python/pull/156,
+  @dilpath in https://github.com/PEtab-dev/libpetab-python/pull/157
+* Updated visualization example notebooks
+  by @dweindl in https://github.com/PEtab-dev/libpetab-python/pull/137,
+  by @plakrisenko in https://github.com/PEtab-dev/libpetab-python/pull/146,
+  by @plakrisenko in https://github.com/PEtab-dev/libpetab-python/pull/147
+* Added support for PEtab problems with multiple condition files
+  by @dweindl in https://github.com/PEtab-dev/libpetab-python/pull/152
+* Added abstraction for (SBML) models by @dweindl in
+  https://github.com/PEtab-dev/libpetab-python/pull/133
+
+Fixes:
+* Apply get table method before write table method to ensure correct index
+  by @dilpath in https://github.com/PEtab-dev/libpetab-python/pull/126
+* petablint: Fix incorrect noise-parameter-mismatch error message
+  by @dweindl in https://github.com/PEtab-dev/libpetab-python/pull/129
+* Fixed handling of NaN values for parameters in condition table
+  by @dweindl in https://github.com/PEtab-dev/libpetab-python/pull/150
+* More informative `petab.calculate` errors
+  by @dweindl in https://github.com/PEtab-dev/libpetab-python/pull/151
+
+Removals:
+* Removed ancient/deprecated default file naming scheme
+  by @dweindl in https://github.com/PEtab-dev/libpetab-python/pull/132
+* Removed ancient deprecated functions related to specifying observables/noise
+  models inside SBML
+  by @dweindl in https://github.com/PEtab-dev/libpetab-python/pull/140
+  https://github.com/PEtab-dev/libpetab-python/pull/131
+* Removed deprecated visualization functions
+   by @dweindl in https://github.com/PEtab-dev/libpetab-python/pull/130
+
+**New Contributors**
+* @m-philipps made their first contribution in https://github.com/PEtab-dev/libpetab-python/pull/156
+
+**Full Changelog**: https://github.com/PEtab-dev/libpetab-python/compare/v0.1.26...v0.1.27
+
 ### 0.1.26
 
 * Fix SBML Rule handling logic by @dweindl in

MANIFEST.in

@@ -1,2 +1,2 @@
-include petab/petab_schema.yaml
-recursive-include petab/visualize/templates
+recursive-include petab/schemas *.yaml
+recursive-include petab/visualize/templates *

README.md

@@ -1,5 +1,5 @@
-[![CI](https://github.com/PEtab-dev/libpetab-python/actions/workflows/ci_tests.yml/badge.svg?branch=master)](https://github.com/PEtab-dev/libpetab-python/actions/workflows/ci_tests.yml)
-[![codecov](https://codecov.io/gh/PEtab-dev/libpetab-python/branch/master/graph/badge.svg)](https://codecov.io/gh/PEtab-dev/libpetab-python)
+[![CI](https://github.com/PEtab-dev/libpetab-python/actions/workflows/ci_tests.yml/badge.svg?branch=main)](https://github.com/PEtab-dev/libpetab-python/actions/workflows/ci_tests.yml)
+[![codecov](https://codecov.io/gh/PEtab-dev/libpetab-python/branch/main/graph/badge.svg)](https://codecov.io/gh/PEtab-dev/libpetab-python)
 [![PyPI version](https://badge.fury.io/py/petab.svg)](https://badge.fury.io/py/petab)
 
 # petab - a Python package for handling PEtab files
@@ -50,8 +50,8 @@ be:
 
 Examples for PEtab Python library usage:
 
-* [Validation](https://github.com/PEtab-dev/libpetab-python/blob/master/doc/example/example_petablint.ipynb)
-* [Visualization](https://github.com/PEtab-dev/libpetab-python/blob/master/doc/example/example_visualization.ipynb)
+* [Validation](https://github.com/PEtab-dev/libpetab-python/blob/main/doc/example/example_petablint.ipynb)
+* [Visualization](https://github.com/PEtab-dev/libpetab-python/blob/main/doc/example/example_visualization.ipynb)
 
 
 ## Getting help

doc/conf.py

@@ -9,14 +9,6 @@
 import sys
 import warnings
 
-# Sphinx autosummary does not handle well submodules and functions with
-#  identical names if the function is part of __all__ of the parent module.
-#  This is way, we get autosummary to create the submodule documentation:
-import petab
-from petab.visualize.plot_data_and_simulation import plot_data_and_simulation
-petab.visualize.plot_data_and_simulation = \
-    sys.modules['petab.visualize.plot_data_and_simulation']
-
 # -- Path setup --------------------------------------------------------------
 
 # If extensions (or modules to document with autodoc) are in another directory,
@@ -50,9 +42,8 @@
     'sphinx.ext.autosummary',
     'sphinx.ext.intersphinx',
     'sphinx.ext.viewcode',
-    'myst_parser',
     'sphinx_markdown_tables',
-    'nbsphinx',
+    'myst_nb',
 ]
 
 intersphinx_mapping = {
@@ -90,10 +81,14 @@
 # For some reason causes sphinx import errors otherwise
 autodoc_mock_imports = ['yaml']
 
+# myst_nb options
+#  https://myst-nb.readthedocs.io/en/latest/configuration.html
+nb_execution_mode = "force"
+
+
 source_suffix = {
     '.rst': 'restructuredtext',
     '.txt': 'restructuredtext',
-    '.md': 'markdown',
 }
 
 # ignore numpy warnings

doc/example/example_Isensee/Isensee_visualizationSpecification.tsv

@@ -30,6 +30,6 @@ plot4	JI09_151102_Drg421_418_Age	LinePlot	MeanAndSD	JI09_151102_Drg421_418_Age__
 plot5	JI09_160126_Drg449_444_CycNuc	BarPlot	MeanAndSD	JI09_160126_Drg449_444_CycNuc__ctrl	condition	0	Condition	pRII_Western	0	pRII_Western	Ctrl	lin	lin
 plot5	JI09_160126_Drg449_444_CycNuc	BarPlot	MeanAndSD	JI09_160126_Drg449_444_CycNuc__Fsk10_and_IBMX100	condition	0	Condition	pRII_Western	0	pRII_Western	IBMX(100)/Fsk(10)	lin	lin
 plot5	JI09_160126_Drg449_444_CycNuc	BarPlot	MeanAndSD	JI09_160126_Drg449_444_CycNuc__Sp8_Br_cAMPS_AM10	condition	0	Condition	pRII_Western	0	pRII_Western	Sp8-Br-cAMPS-AM(10)	lin	lin
-plot6	JI09_160201_Drg453-452_CycNuc	ScatterPlot	MeanAndSD	JI09_160201_Drg453-452_CycNuc__ctrl	time	0	Time [min]	Calpha_Microscopy	0	Calpha_Microscopy	Ctrl	lin	lin
-plot6	JI09_160201_Drg453-452_CycNuc	ScatterPlot	MeanAndSD	JI09_160201_Drg453-452_CycNuc__Fsk	time	0	Time [min]	Calpha_Microscopy	0	Calpha_Microscopy	Fsk(10)	lin	lin
-plot6	JI09_160201_Drg453-452_CycNuc	ScatterPlot	MeanAndSD	JI09_160201_Drg453-452_CycNuc__Sp8_Br_cAMPS_AM	time	0	Time [min]	Calpha_Microscopy	0	Calpha_Microscopy	Sp8-Br-cAMPS-AM(10)	lin	lin
+plot6	JI09_160201_Drg453-452_CycNuc	LinePlot	MeanAndSD	JI09_160201_Drg453-452_CycNuc__ctrl	time	0	Time [min]	Calpha_Microscopy	0	Calpha_Microscopy	Ctrl	lin	lin
+plot6	JI09_160201_Drg453-452_CycNuc	LinePlot	MeanAndSD	JI09_160201_Drg453-452_CycNuc__Fsk	time	0	Time [min]	Calpha_Microscopy	0	Calpha_Microscopy	Fsk(10)	lin	lin
+plot6	JI09_160201_Drg453-452_CycNuc	LinePlot	MeanAndSD	JI09_160201_Drg453-452_CycNuc__Sp8_Br_cAMPS_AM	time	0	Time [min]	Calpha_Microscopy	0	Calpha_Microscopy	Sp8-Br-cAMPS-AM(10)	lin	lin

doc/example/example_Isensee/Isensee_visualizationSpecification_scatterplot.tsv

@@ -0,0 +1,7 @@
+plotId	plotName	plotTypeSimulation	plotTypeData	datasetId	xValues	xOffset	xLabel	yValues	yOffset	yLabel	legendEntry	xScale	yScale
+plot5	JI09_160126_Drg449_444_CycNuc	ScatterPlot	MeanAndSD	JI09_160126_Drg449_444_CycNuc__ctrl	condition	0	Condition	pRII_Western	0	pRII_Western	Ctrl	lin	lin
+plot5	JI09_160126_Drg449_444_CycNuc	ScatterPlot	MeanAndSD	JI09_160126_Drg449_444_CycNuc__Fsk10_and_IBMX100	condition	0	Condition	pRII_Western	0	pRII_Western	IBMX(100)/Fsk(10)	lin	lin
+plot5	JI09_160126_Drg449_444_CycNuc	ScatterPlot	MeanAndSD	JI09_160126_Drg449_444_CycNuc__Sp8_Br_cAMPS_AM10	condition	0	Condition	pRII_Western	0	pRII_Western	Sp8-Br-cAMPS-AM(10)	lin	lin
+plot6	JI09_160201_Drg453-452_CycNuc	ScatterPlot	MeanAndSD	JI09_160201_Drg453-452_CycNuc__ctrl	time	0	Time [min]	Calpha_Microscopy	0	Calpha_Microscopy	Ctrl	lin	lin
+plot6	JI09_160201_Drg453-452_CycNuc	ScatterPlot	MeanAndSD	JI09_160201_Drg453-452_CycNuc__Fsk	time	0	Time [min]	Calpha_Microscopy	0	Calpha_Microscopy	Fsk(10)	lin	lin
+plot6	JI09_160201_Drg453-452_CycNuc	ScatterPlot	MeanAndSD	JI09_160201_Drg453-452_CycNuc__Sp8_Br_cAMPS_AM	time	0	Time [min]	Calpha_Microscopy	0	Calpha_Microscopy	Sp8-Br-cAMPS-AM(10)	lin	lin

doc/md2rst.py

@@ -1,4 +1,4 @@
-import m2r
+import m2r2
 import re
 
 
@@ -26,7 +26,7 @@ def absolute_links(txt):
 
 def md2rst(source: str, target: str):
     txt = absolute_links(read(source))
-    txt = m2r.convert(txt)
+    txt = m2r2.convert(txt)
     with open(target, 'w') as f:
         f.write(txt)
 

petab/C.py

@@ -3,7 +3,7 @@
 This file contains constant definitions.
 """
 
-import math
+import math as _math
 
 
 # MEASUREMENTS
@@ -24,7 +24,7 @@
 TIME = 'time'
 
 #: Time value that indicates steady-state measurements
-TIME_STEADY_STATE = math.inf
+TIME_STEADY_STATE = _math.inf
 
 #:
 OBSERVABLE_PARAMETERS = 'observableParameters'
@@ -264,6 +264,8 @@
 OBSERVABLE_FILES = 'observable_files'
 #:
 VISUALIZATION_FILES = 'visualization_files'
+#:
+EXTENSIONS = 'extensions'
 
 
 # MORE
@@ -274,3 +276,6 @@
 RESIDUAL = 'residual'
 #:
 NOISE_VALUE = 'noiseValue'
+
+# separator for multiple parameter values (bounds, observableParameters, ...)
+PARAMETER_SEPARATOR = ';'

petab/calculate.py

@@ -1,14 +1,15 @@
 """Functions performing various calculations."""
 
+import numbers
+from functools import reduce
+from typing import Dict, List, Union
+
 import numpy as np
 import pandas as pd
-from functools import reduce
-from typing import List, Union
+import petab
 import sympy
-import numbers
 
 from .C import *
-import petab
 
 __all__ = ['calculate_residuals', 'calculate_residuals_for_table',
            'get_symbolic_noise_formulas', 'evaluate_noise_formula',
@@ -121,7 +122,7 @@ def calculate_residuals_for_table(
     return residual_df
 
 
-def get_symbolic_noise_formulas(observable_df) -> dict:
+def get_symbolic_noise_formulas(observable_df) -> Dict[str, sympy.Expr]:
     """Sympify noise formulas.
 
     Arguments:
@@ -144,9 +145,10 @@ def get_symbolic_noise_formulas(observable_df) -> dict:
 
 def evaluate_noise_formula(
         measurement: pd.Series,
-        noise_formulas: dict,
+        noise_formulas: Dict[str, sympy.Expr],
         parameter_df: pd.DataFrame,
-        simulation: numbers.Number) -> float:
+        simulation: numbers.Number,
+) -> float:
     """Fill in parameters for `measurement` and evaluate noise_formula.
 
     Arguments:
@@ -165,10 +167,11 @@ def evaluate_noise_formula(
     # extract measurement specific overrides
     observable_parameter_overrides = petab.split_parameter_replacement_list(
         measurement.get(NOISE_PARAMETERS, None))
-    overrides = {}
     # fill in measurement specific parameters
-    for i_obs_par, obs_par in enumerate(observable_parameter_overrides):
-        overrides[f"noiseParameter{i_obs_par+1}_{observable_id}"] = obs_par
+    overrides = {
+        f"noiseParameter{i_obs_par + 1}_{observable_id}": obs_par
+        for i_obs_par, obs_par in enumerate(observable_parameter_overrides)
+    }
 
     # fill in observables
     overrides[observable_id] = simulation
@@ -190,10 +193,12 @@ def evaluate_noise_formula(
     # conversion is possible if all parameters are replaced
     try:
         noise_value = float(noise_value)
-    except TypeError:
-        raise TypeError(
+    except TypeError as e:
+        raise ValueError(
             f"Cannot replace all parameters in noise formula {noise_value} "
-            f"for observable {observable_id}.")
+            f"for observable {observable_id}. "
+            f"Missing {noise_formula.free_symbols}. Note that model states "
+            "are currently not supported.") from e
     return noise_value
 
 
@@ -230,12 +235,12 @@ def calculate_chi2(
         normalize, scale)
     chi2s = [calculate_chi2_for_table_from_residuals(df)
              for df in residual_dfs]
-    chi2 = sum(chi2s)
-    return chi2
+    return sum(chi2s)
 
 
 def calculate_chi2_for_table_from_residuals(
-        residual_df: pd.DataFrame) -> float:
+        residual_df: pd.DataFrame,
+) -> float:
     """Compute chi2 value for a single residual table."""
     return (np.array(residual_df[RESIDUAL])**2).sum()
 
@@ -278,8 +283,7 @@ def calculate_llh(
         _llh = calculate_llh_for_table(
             measurement_df, simulation_df, observable_df, parameter_df)
         llhs.append(_llh)
-    llh = sum(llhs)
-    return llh
+    return sum(llhs)
 
 
 def calculate_llh_for_table(
@@ -326,8 +330,7 @@ def calculate_llh_for_table(
         llh = calculate_single_llh(
             measurement, simulation, scale, noise_distribution, noise_value)
         llhs.append(llh)
-    llh = sum(llhs)
-    return llh
+    return sum(llhs)
 
 
 def calculate_single_llh(
@@ -371,5 +374,4 @@ def calculate_single_llh(
         raise NotImplementedError(
             "Unsupported combination of noise_distribution and scale "
             f"specified: {noise_distribution}, {scale}.")
-    llh = - nllh
-    return llh
+    return -nllh

petab/conditions.py

@@ -52,6 +52,7 @@ def write_condition_df(df: pd.DataFrame, filename: Union[str, Path]) -> None:
         df: PEtab condition table
         filename: Destination file name
     """
+    df = get_condition_df(df)
     df.to_csv(filename, sep='\t', index=True)
 
 

petab/core.py

@@ -15,7 +15,7 @@
 logger = logging.getLogger(__name__)
 __all__ = ['get_simulation_df', 'write_simulation_df', 'get_visualization_df',
            'write_visualization_df', 'get_notnull_columns',
-           'get_observable_id', 'flatten_timepoint_specific_output_overrides',
+           'flatten_timepoint_specific_output_overrides',
            'concat_tables', 'to_float_if_float', 'is_empty',
            'create_combine_archive', 'unique_preserve_order']
 
@@ -90,30 +90,6 @@ def get_notnull_columns(df: pd.DataFrame, candidates: Iterable):
             if col in df and not np.all(df[col].isnull())]
 
 
-def get_observable_id(parameter_id: str) -> str:
-    """Get PEtab observable ID from PEtab-style sigma or observable
-    `AssignmentRule`-target ``parameter_id``.
-
-    e.g. for 'observable_obs1' -> 'obs1', for 'sigma_obs1' -> 'obs1'
-
-    Arguments:
-        parameter_id: Some parameter ID
-
-    Returns:
-        Observable ID
-    """
-    warn("This function will be removed in future releases.",
-         DeprecationWarning)
-
-    if parameter_id.startswith(r'observable_'):
-        return parameter_id[len('observable_'):]
-
-    if parameter_id.startswith(r'sigma_'):
-        return parameter_id[len('sigma_'):]
-
-    raise ValueError('Cannot extract observable id from: ' + parameter_id)
-
-
 def flatten_timepoint_specific_output_overrides(
         petab_problem: 'petab.problem.Problem') -> None:
     """Flatten timepoint-specific output parameter overrides.
@@ -144,8 +120,8 @@ def flatten_timepoint_specific_output_overrides(
         replacement_id = ''
         for field in possible_groupvars:
             if field in groupvars:
-                val = str(groupvar[groupvars.index(field)
-                                   ]).replace(';', '_').replace('.', '_')
+                val = str(groupvar[groupvars.index(field)])\
+                    .replace(PARAMETER_SEPARATOR, '_').replace('.', '_')
                 if replacement_id == '':
                     replacement_id = val
                 elif val != '':