Feedback Request: Late-Fusion Model Worsens with Satellite.

[kwon-encored]

Hi @Sukh-P
I hope this message finds you well.

With the diagram I made from our previous discussion, I have tried the late-fusion model with Korean Satellite Images and Korean weather forecasting (NWPs) on Jeju Island for only one month of this year. My current input settings are:

• GSP: Divided equally into 0.1 x 0.1 degrees latitude and longitude.
• Inputs: Hourly data (2024-03-01-00:00, 01:00, 02:00...) for the entire month.
• Window dimensions: Using exactly the same as in your paper (calculated based on the provided km dimensions).
• Output: setting t0 as hour 0, we predict t0 to t0+7 (total 8 hours)

However, when I try this and calculate my NMAE, it comes out as high as 0.5. Ironically, the result with satellite images is worse than without them (2 NWP > 2 NWP + Satellite).

May I ask for any quick tips or feedback on this?
I was wondering if using just Jeju Island, which is small and might have too much overlap, could be affecting the results.

Thank you for your time and consideration!

[AUdaltsova]

Hi @kwon-encored! Great that you were able to use our code, that's so exciting! And I'm sorry it's not going great so far, that's always frustrating. A couple of things off the top of my head, and I'm sure you've done some of that already but just in case anyway:

• Always good to check that all of your data sources are aligned. I you're using our GSP code which saves numpy tensors it can be a bit tricky, but there is this visualisation script that might be helpful. Generally you want to be sure they all cover the space you expect and that the time matches up.
• Speaking of which, timezones are always a big pain, and I'm sure you've done that, but just to make sure everything is in the same timezone would be good, too.
• It's strange that satellite makes your model worse. One thing to check would be that everything is normalised correctly, so for example you don't get a situation when everything is in [0,1] and one source is up to 10, as it can confuse the model
• The set up you describe seems reasonable to me. I wouldn't expect the size of the island itself to be an issue unless there is not enough resolution in the data you are using, so if the whole island is covered by one pixel of data or something, the model will struggle to find what causes different readings at different GSPs. In that case, forecasting some sort of aggregate might be a better option.
• If the resolution is high enough, it's still good to check that the window size makes sense for your set-up, i e that it covers enough relevant territory but not too much that it will get chunks of other GSPs/a lot of noise in it. You might have to experiment with that a bit, as far as I know we've not developed a hard and fast rule on that, but maybe others might provide a better intuition here - thoughts @Sukh-P?

Hope this is helpful, do get in touch if you have any other questions!

[Sukh-P]

@kwon-encored it would also be good if you were able to provide a bit more detail on what your target data is and the sources for the different input data you used were i.e is it generation data from the whole island split into different regions or is it generation data from one site on the island?

Also are you using the history of the PV generation data as an input to the model?

As @AUdaltsova mentioned also worth checking all of your data inputs, especially satellite to see there is decent data quality there. Additionally what was the satellite data source that you used and what satellite channels were used? Also how much satellite history was passed to the model? Hopefully with some additional detail we can provide some assistance, thanks

[kwon-encored]

@AUdaltsova @Sukh-P
Thank you both sincerely for your thoughtful feedback!

To clarify, I have divided Jeju Island into a grid of 0.1° latitude x 0.1° longitude. Out of approximately 30 regions, we currently have data for only 19 sites, each with separate PV generation data. We’ve normalized the PV generation rate relative to the total generation over the total nameplate capacity and are using this as our target Y value.

In the paper, I couldn’t find details regarding the PV generation history, so I’ve omitted that part for now. Would this exclusion significantly impact the results?

I'm currently using satellite data from the Korean GK2A-KMA satellite, which conveniently includes the same 11 channels referenced in your paper. However, since it provides data every 2 minutes, to align with your study, I am opting to use the data at 4-minute intervals.

Lastly, I was thinking if the problem is from the potential overlap between the 19 regions. Currently, my window size is identical to yours—for example, my ECMWF size is 6 by 6 (equivalent to 0.6° x 0.6°)—which means the data overlaps with approximately 10 neighboring sites. Furthermore, since I am training the data with an hourly interval and each input spans from t-2 to t+8 hours, I worry that such overlaps might introduce issues. However, I believe this concern is unrelated to the satellite data affecting the results.

[kwon-encored]

@kwon-encored @AUdaltsova
Apologies for the frequent inquiries.

I believe I may have found a difference or mistake during my code modification.
Currently, I use the entire dataset of 19 Jeju GSPs as a single input dataset for training.
However, from your paper, it seems that a different single-GSP dataset is used at each batch sample.
Is my understanding correct?

---

Follow-up (if my understanding is correct):

1. The paper mentions training with 20 epochs and a batch size of 128.
2. For the 2020–2022 large dataset:
   • For each 128-batch sample, are the samples strictly separated by GSP?

[AUdaltsova]

Hi @kwon-encored,

so, in terms of your setup, it looks like you maybe have data on 19 individual plants as opposed to regional aggregates? It shouldn't necessarily cause issues but our site pipeline might be better suited for your data. Generally, if you are working with individual sites I'd select their positions as points of interest instead of creating a grid over the island if that's what you're doing-but I might've misunderstood you.

We do not use GSP generation history in the model, so that shouldn't affect your performance, but generally, if it's available to you and is of good enough quality you can feed in a couple of hours (or more) of history, and it will likely help with performance.

You're right that neither of those things would explain why including satellite data makes it worse for you though. However, NMAE of 0.5 is really, really high, and it might be worth trying to debug with less data sources, because it doesn't look like the model is functioning correctly. I'd still perform some checks that you're feeding in the right crops, aligned by time, correctly normalised etc. Might also be worth looking at the correlation of some of your data and your targets, though if you're feeding in relevant weather data and your pv doesn't look completely unreasonable I can't imagine it being the problem.

Re: your follow-up, someone might correct me on this, but as far as I know each sample will contain data from just one GSP. It will look something like this:

• GSP data: single time series [t, t+8], GSP ID for embedding (so the model knows which one it's working with)
• IFS data: for each channel, a series of square spatial crops, each corresponding to a time in [t-2, t+8] (I am writing sizes from memory so if the paper says different hours believe the paper)
• EUMETSAT: a series of square crops of satellite images
• UKV: same as IFS

A batch of size 128 will contain 128 samples like that, with some random mix of GSPs used in it for targets and IDs. If you're trying to predict some sort of mixture and do not distinguish between different IDs it might be causing your issues!

I hope this helps :)

[kwon-encored]

Hi @AUdaltsova
Thank you so much for your time and effort in providing such a valuable response—I truly appreciate it.

I’ve attempted restructuring the dataset by organizing the input data based on each GSP location, which has slightly improved accuracy to 70%. However, the underlying mystery persists :(

To address your question: within the 19 divided sectors (which I call it my own GSPs), I calculate the generation rate based on all existing generation sites relative to the total nameplate capacity, similar to the approach in your paper.

My concern is whether overlapping hourly input data might be contributing to this issue. Since the input NWPs account for t+8 hours yet, my inputs are every hour of each month.

Would you recommend separating the input by every 4 hours instead? Do you think this adjustment might yield better results?

At this stage, I’m experimenting with different window sizes and hyperparameters to see what works best.

(Happy Merry Early Christmas :))

[AUdaltsova]

Hi @kwon-encored! Happy New Year :) Sorry for taking a while to get back to you.

I'm not sure what you mean by 70% accuracy, what metric are you using?

Normalising sites by capacity sounds reasonable; if you have specific sites you might still be better off treating them separately, but it shouldn't necessarily cause poor performance.

Re: NWP overlap, I'm afraid I don't understand what you mean by overlapping data, and every hour of each month. Do you mean your spatial crops overlap, or that the forecast re-initialises before the previous one is out of date, or something? Can you elaborate, please?

[kwon-encored]

@AUdaltsova
Happy New Year!
Please disregard my earlier comment about accuracy—it was a misinterpretation on my part.
Could you kindly provide either the equation or the code you used to calculate your NMAE please?

Answer your question about Overlap

As shown by the illustration (I tried my best)
Since I am including hourly data for every GSP region in the training process, I was concerned about potential overlaps between consecutive hourly inputs. For example, as shown in the image, the data between t0=2AM and t0=3 AM contains high overlaps from 4 AM to 10 AM.

Additionally, regarding the regional overlap, I created an example using Alaska. In the chosen region, I divided the GSP into equally sized rectangular regions. To match the dimensions (in terms of coverage area of NWPs and Satellites) used in your study, this results in, for instance, a 4x4 grid of NWP data covering a total of 16 smaller squares from neighboring GSP regions. If you imagine placing the GreenBox over the map of Alaska for every blue boxes, you can see that there is a substantial amount of overlap between the BlueBoxes.

Given this, I was wondering if such overlaps might cause issues in the analysis or model performance.

[AUdaltsova]

Hi @kwon-encored!

You can find our error calculations in the model here, or here for forecast horizon specific ones. Note that these calculate just MAE, because the generation data goes into the model already normalised by capacity.

Re: your overlap concerns, fist of all, thank you for the plot, it's really clear now!

In terms of time overlaps, we use the same procedure, so that shouldn't be affecting your model.

In terms of the spatial overlap, I can't say for sure if we had the same situation in our training data, but if I had to guess given the area and position of GSPs there must've been at least some crops that intersected, and also just generally I wouldn't expect it to be an issue, as long as the data itself is fine.

You've mentioned a couple of times, both in the case of Jeju and Alaska, that you have divided them into a grid that you then treat like GSPs. I am still unsure about this; could you perhaps tell me more about your generation data source?

GSPs are a bit tricky: they are not a single PV farm, and their generation is not even a precise measurement, because each one represents multiple PV systems within it. The generation we end up using is an estimate made by upscaling the data from some of the plants a GSP represents, done by Sheffield Solar. If your data situation is different, e g you have generation data of specific plants and not a regional aggregate/estimate, regridding like that might mean the crops you are feeding in are less relevant to the sites you are forecasting for, which would negatively affect the performance of the model.

Also, maybe you're not doing that, but just in case: replicating our crops exactly in terms of pixels might not serve you, as Jeju is a lot smaller than the UK and Alaska is a lot larger; for best results, the crops should be done according to your specific data situation. Unfortunately, we don't have a rule of thumb for that and you might have to experiment a bit to find the right size, but generally, you want to give it a bit of a margin to capture relevant weather events nearby that might affect the site soon, but not too much so it doesn't capture irrelevant information.

I hope that's helpful!