Memory leak issue running GCHP14.3.1 on AWS

[YanshunLi-washu]

Your name

Yanshun Li

Your affiliation

WashU

Please provide a clear and concise description of your question or discussion topic.

Hi Team,

I'm running GCHP 14.3.1 C180 simulation on AWS and got a memory leak problem. The memory usage fill up to abortion within 24 hours' model run:

AGCM Date: 2020/12/31 Time: 23:10:00 Throughput(days/day)[Avg Tot Run]: 0.7 0.7 23.7 TimeRemaining(Est) 135:06:43 73.9% : 64.3% Mem Comm:Used
Mem/Swap Used (MB) at MAPL_Cap:TimeLoop= 1.248E+05 0.000E+00
...
AGCM Date: 2021/01/01 Time: 23:00:00 Throughput(days/day)[Avg Tot Run]: 40.6 83.2 137.2 TimeRemaining(Est) 001:46:22 108.3% : 98.1% Mem Comm:Used
Mem/Swap Used (MB) at MAPL_Cap:TimeLoop= 1.848E+05 0.000E+00

I was using hourly anthropogenic emission inventories and take the option to output check points. When using only monthly emission inventories and opt out for writing check points, the simulation can last for one month.

My intuition is that the memory leak is related to netcdf reading and writing, but I got no such issue when running on NASA pleiades or WashU compute1. I attached the log of building gchp executable so that more info about the linux environment can be provided.

Appreciate if there could be any suggestions and solutions.
ecbuild.log

Yanshun

[lizziel]

Hi @YanshunLi-washu, have you checked in with other GCHP users at WashU who use AWS? I think @BettyCroft has been doing runs with at least that version. If she has not seen this memory leak then I wonder if it has something to do with your settings. You can try to narrow down the issue by systematically reverting your changes back to default, such as turning off mid-run checkpoints, and then turning off hourly emissions.

[YanshunLi-washu]

Hi @lizziel I've identified that using hourly emission and write mid-run checkpoints are both factors. Optioning out for these two can make the C180 simulation last for one month.