Linopy integration for Power and Sector-Coupled Modelling #1172

ekatef · 2024-11-05T20:38:22Z

Closes #494

(and hopefully #968)

Changes proposed in this Pull Request

Integrated changes needed to update PyPSA version and use linopy.

To have a better overview, the PR has been implemented in three steps:

Update PyPSA version (follows Update PyPSA version #1065)
Integrate linopy into power part of the model (follows Linopy transition #796, the diffs can be checked in this local PR)
Integrate linopy into cross-sectoral part of the model (implemented currently as a dedicated PR to the Sec- repo)

Checklist

I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
I tested my contribution locally and it seems to work fine.
Code and workflow changes are sufficiently documented.
Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

* Add a zenodo link to natura.tiff * Update environment * Revise structure definition for lines * Remove get_aggregation_strategies * Fix typo aggregation_strategies * Replace aggregategenerators with aggregateoneport * Add aggregation strategies as a parameter * Re-define aggregation strategies * Update aggregation strategies * Update aggregation strategies for lines * Update aggregation strategies for buses * Fix typo * Put aggregation strategies into a variable * Parametrize the aggregation strategies * Refactor update of the aggregation strategies * Clean-up the code * Revert "Add a zenodo link to natura.tiff" This reverts commit 7700759. * Define an explicit clustering strategy for v_nom * Add a release note * Get glpk back * Specify v_nom for buses explicitly * Revert "Specify v_nom for buses explicitly" This reverts commit 20192e6. * Add a version restriction to the environment specification * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Adjust naming * Move the variable definition * Move the variable * Upgrade PyPSA version --------- Co-authored-by: Davide Fioriti <[email protected]> Co-authored-by: Davide Fioriti <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add a zenodo link to natura.tiff * Update environment * Revise structure definition for lines * Remove get_aggregation_strategies * Fix typo aggregation_strategies * Replace aggregategenerators with aggregateoneport * Add aggregation strategies as a parameter * Re-define aggregation strategies * Update aggregation strategies * Update aggregation strategies for lines * Update aggregation strategies for buses * Fix typo * Put aggregation strategies into a variable * Parametrize the aggregation strategies * Refactor update of the aggregation strategies * Clean-up the code * Revert "Add a zenodo link to natura.tiff" This reverts commit 7700759. * Define an explicit clustering strategy for v_nom * Add a release note * Get glpk back * Specify v_nom for buses explicitly * Revert "Specify v_nom for buses explicitly" This reverts commit 20192e6. * Add a version restriction to the environment specification * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Adjust naming * Move the variable definition * Move the variable * Upgrade PyPSA version * Update docstring * Fix imports duplication * Update imports * Update the carrier-capacity constraint * Add docstring * Update the equity constraint * Add docstring * Update BAU constraint * Update SAFE constraint * Add docstring * Update operational reserve margin constraint * Add docstring * Add an new argument to the RM constraint * Update the update of capacity constraints * Update adding an operational reserve margin constraint * Update docstring * Update battery constraint * Add docstring * Update a constraint related to a RES share * Fix usage of add_ERS_constraints * Update solving script * Update a solving run * Fix typos --------- Co-authored-by: Davide Fioriti <[email protected]> Co-authored-by: Davide Fioriti <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

ekatef · 2024-11-05T20:54:15Z

The PR has been tested against main, and the change in the objective function has been found to be less than 1%.

Codewise, the implementation is same as in #1065. Manual reimplementation has been needed because of #1170

ekatef · 2024-11-05T21:06:34Z

@GbotemiB As discussed, we can use a branch of this PR (pypsa_linopy_update located in upstream directly) to accommodate your linopy PR from PyPSA-Earth-Sec. What do you think?

GbotemiB · 2024-11-07T00:13:17Z

@GbotemiB As discussed, we can use a branch of this PR (pypsa_linopy_update located in upstream directly) to accommodate your linopy PR from PyPSA-Earth-Sec. What do you think?

Thanks @ekatef. I can start working on the linopy PR for the sec aspect.

davide-f

Great :)
Very minor things, basically ready to merge :)

davide-f · 2024-11-07T10:47:05Z

scripts/solve_network.py

    nodes = n.buses.index[n.buses.carrier == "battery"]
-    if nodes.empty or ("Link", "p_nom") not in n.variables.index:
+    # TODO Check if the second part of the condition can make sense


Does this still apply?

My understanding is that this second part addresses some special case when the network doesn't contain any batteries, despite of the fact that there are nodes with battery carrier. Not sure when exactly can it be the case and just leaving it as is for now, as it seems not to relate to interfacing with linopy. Does it make sense for you or am I missing something?

davide-f · 2024-11-07T10:47:37Z

scripts/solve_network.py

    lgrouper = n.loads.bus.map(n.buses.country)
+    # TODO drop load


Is this for a later stage?

Yes, my feeling is that the enclosing add_RES_constraints( ) will need some additional revisions which should go beyond this PR.

In our current upstream implementation, the function throws a warning stating that it's only a preliminary implementation. I think, it may be a good idea to open an issue to track further improvements

In the updates version of add_RES_constraints( ), this TODO is not needed anymore

davide-f · 2024-11-07T10:48:06Z

scripts/solve_network.py

    # Generators
+    # TODO restore grouping by countries un-commenting calls of groupby()


Does this not work? for later?

Yeah, linopy.groupby( ) has been not in the best shape when I have been debugging this part. So, I have left it out for future improvements of add_RES_constraints( )

Maybe we should add a warning&issue then?

Have fixed the functionality. It has been more straight-forward ;)

davide-f · 2024-11-07T10:50:09Z

scripts/solve_network.py

-        n.lines.loc[n.lines.index.str.contains("new"), "s_nom_min"] = (
-            snakemake.params.augmented_line_connection.get("min_expansion")
-        )
+    # TODO Double-check handling the augmented case


Why comment this? Does it trigger an issue?
It seems look it generates issues, what do you think?

A great catch! That is just an old comment needed to debug the main functionality. Now it must be lifted

Testing it

Have get this part back, and testing worked well

can we drop this then?

scripts/solve_network.py

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Add a zenodo link to natura.tiff * Update environment * Revise structure definition for lines * Remove get_aggregation_strategies * Fix typo aggregation_strategies * Replace aggregategenerators with aggregateoneport * Add aggregation strategies as a parameter * Re-define aggregation strategies * Update aggregation strategies * Update aggregation strategies for lines * Update aggregation strategies for buses * Fix typo * Put aggregation strategies into a variable * Parametrize the aggregation strategies * Refactor update of the aggregation strategies * Clean-up the code * Revert "Add a zenodo link to natura.tiff" This reverts commit 7700759. * Define an explicit clustering strategy for v_nom * Add a release note * Get glpk back * Specify v_nom for buses explicitly * Revert "Specify v_nom for buses explicitly" This reverts commit 20192e6. * Add a version restriction to the environment specification * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Adjust naming * Move the variable definition * Move the variable * Upgrade PyPSA version --------- Co-authored-by: Davide Fioriti <[email protected]> Co-authored-by: Davide Fioriti <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add a zenodo link to natura.tiff * Update environment * Revise structure definition for lines * Remove get_aggregation_strategies * Fix typo aggregation_strategies * Replace aggregategenerators with aggregateoneport * Add aggregation strategies as a parameter * Re-define aggregation strategies * Update aggregation strategies * Update aggregation strategies for lines * Update aggregation strategies for buses * Fix typo * Put aggregation strategies into a variable * Parametrize the aggregation strategies * Refactor update of the aggregation strategies * Clean-up the code * Revert "Add a zenodo link to natura.tiff" This reverts commit 7700759. * Define an explicit clustering strategy for v_nom * Add a release note * Get glpk back * Specify v_nom for buses explicitly * Revert "Specify v_nom for buses explicitly" This reverts commit 20192e6. * Add a version restriction to the environment specification * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Adjust naming * Move the variable definition * Move the variable * Upgrade PyPSA version * Update docstring * Fix imports duplication * Update imports * Update the carrier-capacity constraint * Add docstring * Update the equity constraint * Add docstring * Update BAU constraint * Update SAFE constraint * Add docstring * Update operational reserve margin constraint * Add docstring * Add an new argument to the RM constraint * Update the update of capacity constraints * Update adding an operational reserve margin constraint * Update docstring * Update battery constraint * Add docstring * Update a constraint related to a RES share * Fix usage of add_ERS_constraints * Update solving script * Update a solving run * Fix typos --------- Co-authored-by: Davide Fioriti <[email protected]> Co-authored-by: Davide Fioriti <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

for more information, see https://pre-commit.ci

ekatef · 2024-12-23T10:05:44Z

Update: re-testing of the objective function is working well in some testing cases, but leads to the differences in others. In particular, the tutorial-based test is not working properly. I'm able to reproduce the issue locally, and it appears to be linked with the changes introduced before solve_network script.

Basically, the value of the objective function does not depend on the solving script but on the procedure before: solving the network prepared by main workflow with the solve_network of this PR leads to the same objective value as in this PR. Visa versa is true as well: solving the prepared network of this PR with solve_network of main gives the same objective value as in this PR.

A similar issue has been observed before for power-only model, and has been resolved by setting v_nom needed to resolve the negative objective issue #961. It looks like a similar pitfall has appeared due to the changes introduced by some cross-sectoral related changes.

davide-f · 2024-12-23T17:26:55Z

Update: re-testing of the objective function is working well in some testing cases, but leads to the differences in others. In particular, the tutorial-based test is not working properly. I'm able to reproduce the issue locally, and it appears to be linked with the changes introduced before solve_network script.

Basically, the value of the objective function does not depend on the solving script but on the procedure before: solving the network prepared by main workflow with the solve_network of this PR leads to the same objective value as in this PR. Visa versa is true as well: solving the prepared network of this PR with solve_network of main gives the same objective value as in this PR.

A similar issue has been observed before for power-only model, and has been resolved by setting v_nom needed to resolve the negative objective issue #961. It looks like a similar pitfall has appeared due to the changes introduced by some cross-sectoral related changes.

Many thanks for investigating. What is your insight on this issue and status of the PR therefore?

can we track what has been merged that may have led to the issue?

davide-f · 2024-12-23T18:25:16Z

Let's remind to change the dependencies also in the file doc/requirements.txt.
The API reference was not compiling because of that

davide-f · 2024-12-24T10:39:53Z

I've been testing the model and to me, it seems like the change in the config.tutorial.yaml are not due to changes in solve_network but something else.
In particular, when debugging the solve_network in this PR and in the main, only 3 functions are actually activated add_battery_constraints, add_chp_constraints (but irrelevant and skipped) and add_co2_sequestration_limit.
Only add_battery_constraint actually changes the model but they seem fine

The checks on the network seem to suggest that the input networks match but something changes for some reason.

At deeper analysis, it seems like the onwind (and marginally offwind) are different.
In particular, the new formulation has half onwind than the base case, which may lead to different outputs

Update: the onwind p_max_pu is expected to be the same after elec.nc [add_electricity]; probably there is something going on in the clustering steps that is different

ekatef · 2024-12-24T13:57:44Z

@davide-f many thanks for testing (and providing the fantastic infrastructure for inter-comparing the networks!). The issues indeed are quite weird and it has been precious to have your independent workflow to be sure that they are reproducible. Writing down the results of the testing below to possibly improve further both implementations 🙂

It has been found that the network structure and parameters are exactly the same for both implementations. The reason of the differences in the objective function are linked with the difference in p_max_pu time series for wind generators (a quick way to reproduce: n_gen_onwind = n.generators[n.generators.carrier == "onwind"].index; n.generators_t.p_max_pu[n_gen_onwind].sum(axis=1).sum()). This difference stems from the fact that aggregation_strategies are defined in the different way for the main and the updated implementation.

In the older version of PyPSA, a default strategy have has been used for p_max_pu time-series, while now we must use custom definitions for aggregation strategies. If we change aggregation_strategies:generators:p_max_pu between mean and max, the objective value changes between ~4.5 [financial units] and ~1.7 [financial units] for the updated implementation with ~2.5 [financial units] for the implementation in main.

Points of action
Given a dramatic change in the underlaying computational approaches, it doesn't make any sense trying to keep the objective. What must be done instead is tracing across the code values affected by the aggregation in simplify_network and cluster_network and check that conservation laws are respected. Potentially, it means that checks or improvements can be also needed to be added into the current main implementation which will remain as a stable backward compatible version after this PR will be merged.

@davide-f great that we have been able to discover the reason, as it appears to be a potentially major issue also for the current implementation 🎉 🎉 🎉 Thank you so much for that. I think it's a great result, and we are very close to both finalise this PR and make the current implementation really stable 😄

ekatef · 2024-12-28T00:18:20Z

It appears, the trick was to use an explicit definition of an aggregation strategy as p_max_pu: weighted_average. That brings the objective function to 2512675751.0 which is quite close to 2512675455.0 from main.

The full testing results looks like follows:

main

INFO:main:Objective function: 2512675455.0
INFO:main:Objective function: 2341159599.0
INFO:main:Objective function: 2321096575.0
INFO:main:Objective function: 336330443.3
INFO:main:Objective function: 4248565071.0
INFO:main:Objective function: 4602716893.0
INFO:main:Objective function: 5723516039.0
INFO:main:Objective function: 476366499.4
INFO:main:Objective function: 10926432310.0
INFO:main:Objective function: 1461452593.0
INFO:main:Objective function: 0.211751626
INFO:main:Objective function: 4376884296.0
INFO:main:Objective function: -0.006962630898
INFO:main:Objective function: 722220262793.1165

this PR

INFO:main:Objective function: 2512675751.0
INFO:main:Objective function: 2341214199.0
INFO:main:Objective function: 2321096682.0
INFO:main:Objective function: 476365695.9
INFO:main:Objective function: 10747113970.0
INFO:main:Objective function: 1461452491.0
INFO:main:Objective function: 4324387954.0
INFO:main:Objective function: 0.001945018768
INFO:main:Objective function: 5723515621.0
INFO:main:Objective function: 336330259.2
INFO:main:Objective function: 4503925925.0
INFO:main:Objective function: 4186016571.0
INFO:main:Objective function: 5.44154318e-07
INFO:main:Objective function: 722265045983.7013

That gives the following differences:

config.custom.yaml

1.178027e-07

config.landlock.yaml

2.332123e-05

config.sector.yaml

4.609890e-08

config.test_myopic.yaml

6.200382e-05

For Monte-Carlo test, the differences may be very high (2.939659e-01 6.046785e-01 -2.149413e+00 -3.235436e-01 -2.449161e+11 -9.090421e-01 [10] -3.345290e+00 1.000000e+00 -4.559650e-02) but probably it's intentional.

ekatef · 2024-12-28T00:20:03Z

@davide-f I think we have done it 😄

davide-f

Great achievement @ekatef ! we have narrowed down the issue!
We have some large differences in few iterations.
A sorted comparison is better than it looks at first though; differences within 1-2% generally or limited in absolute terms.

Do you have 2 folders with the results of the two monte carlo executions? it would be amazing to compare if you have availability; I'm trying them locally in case with #1276

The monte carlo should have the seed implemented by looking at the code; @GbotemiB can you confirm?

GbotemiB · 2025-01-03T18:33:44Z

@davide-f, Yes I can confirm that the monte carlo has a seed option implemented.

davide-f · 2025-01-03T22:14:05Z

Update: I've been investigating the differences and the "sorted comparison" is confirmed, thus differences shown in pu in the diff column are actually small.

The larger difference is about 2.2% related to the "m3" case, the reason being a different resourse allocation between offwind and solar induced by noisy costs on node mapping that is not identical across the two iterative execution.

An "issue" identified is that the capital_cost and marginal_costs of offshore-dc technology are not to be fully reproducible with differences in the order of 1e-3-1e-2.
The root cause of this is that the order of the nodes of the file base.nc is not fully reproducibile. As the order of the nodes is not univoque, the final result of the clustering is not fully exactly repeatible, that summed with the use of noisy costs make slight deviations in the objective function.

In summary, this PR for me is fine as is and experienced changes are not due to the changes proposed here.
Worth tracking this point. An interesting option would be to potentially sort the rows to improve repeatability.

ekatef · 2025-01-04T13:05:14Z

Fantastic @davide-f! 😄 A really brilliant investigation. Totally agree that it would be a great feature to maintain the network structure after clustering.

Just out of curiosity: I see that the objective values for m6 are very low, and the relative difference in the objective are high which may be not meaningful, though. By any chance, do you have an explanation for that?

ekatef · 2025-01-05T00:11:19Z

I think this PR should be closed in favour of #1276.

For the full transparency, the issues we have found along the way are summarised in this comment. For now, point 1 seems to be resolved thanks to some environment updates (as CI also tests fetching isolated networks), points 2 and 3 are tracked in the issues #1283 and #1245, while point 4 addressed by adding a comment.

Eddy-JV · 2025-01-15T12:51:52Z

Please adapt the new constraint in PR #1192 before merging.

github-actions bot and others added 3 commits October 30, 2024 14:47

docs(contributor): contrib-readme-action has updated readme

747441f

ekatef mentioned this pull request Nov 5, 2024

Fix linopy updates #1173

Closed

8 tasks

ekatef added 2 commits November 5, 2024 22:28

Get back CI for Windows

5c48d60

Update environment

35517e9

davide-f reviewed Nov 7, 2024

View reviewed changes

yerbol-akhmetov and others added 13 commits November 8, 2024 20:47

Add H2 carrier if not exists

c71c538

Add battery carrier if not exists only

b40e643

use H2 in both AC and DC buses in h2_hc_conversions function

fc11c02

use AC and DC based H2 to make parameters of m.add consistent shape

38f6090

add release notes

8c9077e

docs(contributor): contrib-readme-action has updated readme (#1162)

5f52c50

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Get back CI for Windows

0d3ebdf

Update environment

34b6168

revise implementation to accomodate linopy for sec

268851b

Merge branch 'pypsa_linopy_update' into linopy-sec

82d9f10

[pre-commit.ci] auto fixes from pre-commit.com hooks

9fcee64

for more information, see https://pre-commit.ci

ekatef changed the title ~~Integrate PyPSA and linopy update for power and cross-sectoral parts of the model~~ Linopy integration for power Nov 14, 2024

ekatef changed the title ~~Linopy integration for power~~ Linopy integration for Power Nov 14, 2024

ekatef and others added 5 commits November 15, 2024 23:37

Update PyPSA version

08bd2bd

Merge remote-tracking branch 'upstream/main' into pypsa_linopy_update

fe025fc

Merge branch 'pypsa_linopy_update' into linopy-sec

6e892ea

Lift a version restriction

5264e62

Enable augmented network

cedfda8

Merge branch 'main' into pypsa_linopy_update

1d507dc

ekatef added 5 commits December 24, 2024 09:07

Minor refactoring

9dcfd1e

Replace merge functions

82e431f

Merge main into pypsa_linopy_update

5a7efce

Add a TODO

d6b3659

Fix typo

e4386fc

davide-f mentioned this pull request Dec 24, 2024

Update workflow to latest geopandas and fiona #1262

Closed

8 tasks

ekatef added 2 commits December 27, 2024 23:36

Merge pypsa-earth/main into pypsa_linopy_update

2ee49f3

Update configuration strategies

4b05b02

davide-f mentioned this pull request Jan 3, 2025

Update environment after linopy PR #1276

Open

8 tasks

davide-f reviewed Jan 3, 2025

View reviewed changes

davide-f approved these changes Jan 3, 2025

View reviewed changes

This was referenced Jan 3, 2025

Enhance repeatability of the workflow #1277

Open

Support shapes #1280

Open

Update the arguments list for a special case

2a5c930

ekatef mentioned this pull request Jan 9, 2025

Finalize duplication of retrieve_data #1249

Merged

8 tasks

Eddy-JV mentioned this pull request Jan 14, 2025

Lossy Bidirectional Links #1192

Merged

8 tasks

ekatef mentioned this pull request Jan 14, 2025

Validator #1164

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Linopy integration for Power and Sector-Coupled Modelling #1172

Linopy integration for Power and Sector-Coupled Modelling #1172

ekatef commented Nov 5, 2024 •

edited

Loading

ekatef commented Nov 5, 2024

ekatef commented Nov 5, 2024

GbotemiB commented Nov 7, 2024

davide-f left a comment

davide-f Nov 7, 2024

ekatef Nov 19, 2024

davide-f Nov 7, 2024

ekatef Nov 11, 2024

ekatef Nov 20, 2024

davide-f Nov 7, 2024

ekatef Nov 11, 2024

davide-f Nov 12, 2024

ekatef Nov 20, 2024

davide-f Nov 7, 2024

ekatef Nov 11, 2024

ekatef Nov 19, 2024

davide-f Dec 19, 2024

ekatef commented Dec 23, 2024

davide-f commented Dec 23, 2024

davide-f commented Dec 23, 2024

davide-f commented Dec 24, 2024 •

edited

Loading

ekatef commented Dec 24, 2024

ekatef commented Dec 28, 2024 •

edited

Loading

ekatef commented Dec 28, 2024

davide-f left a comment •

edited

Loading

GbotemiB commented Jan 3, 2025

davide-f commented Jan 3, 2025

ekatef commented Jan 4, 2025

ekatef commented Jan 5, 2025

Eddy-JV commented Jan 15, 2025

		# Generators
		# TODO restore grouping by countries un-commenting calls of groupby()

Linopy integration for Power and Sector-Coupled Modelling #1172

Are you sure you want to change the base?

Linopy integration for Power and Sector-Coupled Modelling #1172

Conversation

ekatef commented Nov 5, 2024 • edited Loading

Closes #494

Changes proposed in this Pull Request

Checklist

ekatef commented Nov 5, 2024

ekatef commented Nov 5, 2024

GbotemiB commented Nov 7, 2024

davide-f left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekatef commented Dec 23, 2024

davide-f commented Dec 23, 2024

davide-f commented Dec 23, 2024

davide-f commented Dec 24, 2024 • edited Loading

ekatef commented Dec 24, 2024

ekatef commented Dec 28, 2024 • edited Loading

main

this PR

config.custom.yaml

config.landlock.yaml

config.sector.yaml

config.test_myopic.yaml

ekatef commented Dec 28, 2024

davide-f left a comment • edited Loading

Choose a reason for hiding this comment

GbotemiB commented Jan 3, 2025

davide-f commented Jan 3, 2025

ekatef commented Jan 4, 2025

ekatef commented Jan 5, 2025

Eddy-JV commented Jan 15, 2025

ekatef commented Nov 5, 2024 •

edited

Loading

davide-f commented Dec 24, 2024 •

edited

Loading

ekatef commented Dec 28, 2024 •

edited

Loading

davide-f left a comment •

edited

Loading