Convert MARC Export to use Celery (PP-1472)

[jonathangreen]

Description

Convert the MARC export script to use Celery.

Motivation and Context

This ended up being rather involved, since the export can take quite a long time. Even getting a large enough chunk of data for a single s3 multipart upload can take longer then I was comfortable with.

This PR takes the approach of processing batch_size (default: 500) records in one task, then saving the output to redis and re-queuing the task to process the next batch_size of records. Once the data in redis is large enough, a multipart upload is started in S3, and the multipart data is cached in redis. This continues until the file is completely generated.

How Has This Been Tested?

• Tested by generating marc files locally with some production DB data
• Unit tests

Checklist

• I have updated the documentation accordingly.
• All new and existing tests passed.

[codecov]

Codecov Report

Attention: Patch coverage is 98.80342% with 7 lines in your changes missing coverage. Please review.

Project coverage is 90.67%. Comparing base (b190c49) to head (c1098b0).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/palace/manager/service/redis/models/marc.py	97.63%	2 Missing and 2 partials ⚠️
src/palace/manager/marc/exporter.py	97.93%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2017      +/-   ##
==========================================
+ Coverage   90.59%   90.67%   +0.08%     
==========================================
  Files         338      342       +4     
  Lines       40135    40502     +367     
  Branches     8681     8777      +96     
==========================================
+ Hits        36360    36726     +366     
- Misses       2509     2510       +1     
  Partials     1266     1266              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dbernstein]

That is an absolute beast. I like your batch + re-queueing approach. I think that pattern will likely come in handy in other places.