Notebook development

[BSchilperoort]

Created this PR so the changes in the notebook_development branch are easier to see

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[BSchilperoort]

I merged main into this branch, to make the SRC link etc. work.

[BSchilperoort]

• GRPC4BMI, a 'translater' for BMI function calls between different programming languages. This library was build by the eWaterCycle team, but is available openly for anyone that can benefit from its functionality.

Might be good to mention that the main reason this is needed is to be able to talk across containers. There are other ways to communicate between languages if they're all locally installed, e.g. PyMT.

[BSchilperoort]

@RolfHut is this block needed? The names aren't used anywhere. Or just put them closer where they're used and don't put them in a code block, just markdown for explanation

[RolfHut]

these are a holdover from the calibration thing... will remove

[BSchilperoort]

# # option one: generate forcing data

# # option two: load data you generated yourself

# option three: load data someone else generated

I think we can merge option two and three for conciseness ("load data you or someone else generated") @RolfHut

[BSchilperoort]

The main reason why your calibration breaks is that netCDF4 and HDF5 don't support concurrent reads. You have multiple ensemble members all trying to read the same file at the same time.

There are libraries that support concurrent file IO for netCDF files e.g.; https://github.com/gauteh/hidefix
The alternative is to use a format such as Zarr, which can be accessed concurrently (and can be directly hosted online, for more info see https://excited-co2.github.io/workshop_tutorial/main/overview.html#introduction-to-zarr

Workaround is copying the forcing files and only providing unique nc files to each ensemble member.

[RolfHut]

I don't understand why this has become a problem? In the previous iteration we had the same situation: one netcdf forcing file and 100 ensemble members reading it.

I'm trying to get my head around what changed