Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for a time series storage file that can be shared across systems. #412

Open
daniel-thom opened this issue Nov 6, 2024 · 0 comments

Comments

@daniel-thom
Copy link
Contributor

The current system design creates a time series storage file (HDF5) that contains all time series arrays used by components in the system. Whenever components are removed from the system, associated time series arrays are also removed.

This has a downside: when people create derived systems, much of the time series data is copied. We can instead create a time series storage file that is shared across systems. We wouldn't want to track references across systems, and so deleting time series arrays would not likely be supported.

This would be very beneficial in an HPC environment where all compute nodes have access to a shared filesystem. For non-HPC environments, we could expand the feature by providing a way to store the data in a database server, such as postgres. That may present performance challenges when reading the data. Our existing data prefetching behavior may or may not be sufficient, and would require benchmarking. It would also require some thought about how to store multi-dimensional data.

Also, the current code generates UUIDs as unique identifiers for time series arrays. We probably want to change the identifiers to be hashes of the data. That would allow us to prevent storing the same data multiple times. We would have to account for two arrays with different dimensions that create the same hash (which could easily occur in dummy data used in tests).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant