[discussion]: Add functionality for Parquet to Hive-partitioned Parquet?

[signekb]

Out of scope for now, but just had a talk with a Data Manager at Steno about this; They had transformed the register files with StatTransfer (resulting in e.g., bef2020.parquet, bef2021.parquet) and deleted the original SAS files. Now, to get the register data into the fastreg (year-partitioning), they'll redo the SAS files to be able to convert them again to Parquet.

Maybe it could make sense - somewhere down the line - to include functionality for transforming non-partitioned Parquets into the same format as fastreg produces.

[lwjohnst86]

Very far down the line, so far down the line that DST actually implements Hive partitioned Parquet files instead of fastreg 😛