Refactoring changes and Pytest Integration

[ekatef]

Restores #1083 with the updated repo structure.

Checklist

• I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[davide-f]

I understand the previous PR was kept in Draft mode, opened to keep an eye on the working process (as stated in the text), with no request to review nor changes into review mode.
As such, no review was originally intended because of both points. However, we still decided to still give a revision of the PR with priority

I understand this PR combines:

1. pytest: very interesting feature that we love to have. I quickly scanned them and a large part is likely well done, some tests may not be needed but are already available and that may not harm. I am aware you also reviewed them with Fabian too.
2. revision of pathlib implementation: great feature as well to have
3. architectural changes: such as the heavy revision of build_shapes and powerplantmatching
4. lots of minor revision to include and improve the code

Features 1&2 are great and nearly ready to improve.

Feature 4 is also greatly appreciated; given the quantity hard to summarize them but by scanning them some are interesting but others may need some attention. Some comments and TODOs are dropped without explanation and they may be instead useful for developers. Naming conventions are changed sometimes improving the code, but that's not always the case.

Feature 3 is instead very delicate:

1. Personally, I would not recomment moving the gadm stuff of build_shapes script into other scripts. The reason is that we want helpers to be light and general features, population filling is quite a specific task of build_shape and it would be better placed there. Moreover, the -sec model had a different standard use of gadm and the change should verify limited modelling differences. I believe and trust the approach but good to have that openly.
2. There are heavy changes into build_powerplants that are not clear; it would be better to have a justification on why that is recommended.

It would be great to align on how to best proceed.

Having at least 2/3 PRs on specific tasks would be the standard approach. For example:

1. for pathlib
2. for pytest
3. other improvements

Other approaches, such as reviewing this can be discussed also according to needs. It would take a larger toll but can be discussed.
It is of importance to solve this pending topic

[davide-f]

Hello :D
As we are back of holidays, I'm writing the todos we agreed before going on holiday during last meeting with @FabianHofmann and @ekatef .

In the coming period, we aim to investigate the contributions and test them to merge them.
Contributions can be divided in 4 types:

1. pytest contributions [basically straightforward to merge]
2. pathlib implementation [nearly staightforward to merge, but little check may be helpful as it changes a lot of stuff]
3. architectural changes (e.g. GADM revision and powerplants implementat), which need attention to ensure functionalities and reproducibility of results
4. minor revisions that are generally fine

We investigate the best approach to merge the contributions:
a) review the PR as is
b) disentable the PR into simpler pieces easier to review and test

b) is generally preferrable if overhead is limited.

Plan forward:

1. as the pytest implementations are generally stand-alone, creating a PR with them would be a good way forward. Fabian proposed to investigate this
2. Katia and Davide will investigate the overlap of the other commits, aiming to define how to merge the other contributions, according to strategy a or b

Feel free to comment and revise :)

[davide-f]

At first check, the major architectural changes (GADM and changes in powerplants) are added in this two contributions that are squash merges:

• (this integrates tests and changes into build_powerplants) Generator allocation #10 with fixes in 73ca8d2
• (GADM revision) Unify gadm download and layer open-energy-transition/pypsa-earth#3

Therefore, the testing of the contributions separately should not add much overhead [in terms of conflicts while cherry-picking]