Reindex after relation embedder to lambda implementation and infra complete

[kenoir]

Reindex to serve the dual purpose of populating aggregations for concept work and testing new relation embedder implementation.

Related:

• Update catalogue-api aggregations to use concept ID filters for subjects & contributors #5825
• Relation-embedder ECS to lambda infra catalogue-pipeline#2770
• Restructure Relation Embedder to allow different calling conventions catalogue-pipeline#2772

Checklist to completion:

• Merge @agnes's infra change: Relation embedder lambda infra build catalogue-pipeline#2771
• Run terraform apply on main in 2024-11-18
• Merge @paul Butcher's embedder change: Embedder reshuffle catalogue-pipeline#2777
• Merge @robert Kenny rules for requesting change: Update rules for requesting (03/11/24) catalogue-pipeline#2778
• Determine and set queue configuration for new relation embedder
• Allow reading of secrets to environment variable for containerised lambdas
• Clear denormalised and works/images indexed indices
  The ES delete by query API might be useful here.
• Reinstate 2024-11-18 pipeline #5869
  • Deploy relation embedder lambda subsystem #5840
• Ensure deployment to 2024-11-18 pipeline works as expected #5870
• Run a partial reindex to 2024-11-18 of a few thousand documents #5871
  • Check that image vector changes worked as expected (completed in previous reindex)
  • Check that nested aggregations changes worked as expected (completed in previous reindex)
• Checkpoint - are we happy with everything so far?
• Run a full reindex to 2024-11-18 pipeline #5872
• Test and deploy catalogue API change: Aggregation updates catalogue-api#832