Add support for Anemoi Datasets

[jacobbieker]

ECMWF has now started doing an idea similar to the one in #3
Blog post: https://www.ecmwf.int/en/about/media-centre/aifs-blog/2024/data-driven-regional-modelling

Detailed Description

https://anemoi-datasets.readthedocs.io/en/latest/using/combining.html

Context

Using a high-resolution subset within a global model is quite interesting to get the higher resolution required. Anemoi datasets works to do that as well, making the cutouts.

Possible Implementation

Import the package and use that? Copy over the cutout functionality? As otherwise, its a light wrapper around Zarr stores, which we already support.

[0xFrama]

Hi @jacobbieker, could you give me more details about this?

[jacobbieker]

Hi, we just want to make it so we can train with the anemoi datasets package. It has some nice functionality, so would want to add it to the different ways of loading weather datasets. Probably as another option in the data/ module here, somewhat like the IFS dataloader.

[0xFrama]

Have you a specific weather dataset in mind to start with?

[jacobbieker]

The easiest would probably be either WeatherBench, or get it working with the ICON archive that we've made on Hugging Face.

With the ICON EU and ICON Global datasets on HF, we could get the loading of a low resolution global model + high resolution regional model working together, but would take more work to get up and running with this.

First step would probably try WeatherBench, and just get it to output training examples

[glitch401]

hey, is this issue still being worked on?

[jacobbieker]

I'm not sure, but if you want to tackle it, that would be great!

[glitch401]

alrighty, taking a jab at it :D

[FilippoContessa]

@jacobbieker could i work on this?

[jacobbieker]

@FilippoContessa yeah! Go right ahead

[FilippoContessa]

I'll start from this issue. If I understood correctly, we aim to integrate the anemoi datasets by creating another dataloader specifically for this type of ds.

Is that right?

[jacobbieker]

Yes, so we want to be ideally able to load any dataset in the anemoi datasets