Reimplemented email analytics prioritizing email opens

[9larsons]

ref #20835

• reimplemented email analytics changes that prioritized opened events over other events in order to speed up open analytics
• added db persistence to fetch missing job to ensure we re-fetch every window of events, especially important if we restart following a large email batch

We learned a few things with the previous trial run of this. Namely, that event throughput is not as high as we initially saw in the data for particularly large databases. This set of changes is more conservative, while a touch more complicated, in ensuring we capture edge cases for really large newsletter sends (100k+ members).

In general, we want to make sure we're fetching new open events at least every 5 mins, and often much faster than that, unless it's a quiet period (suggesting we haven't had a newsletter send or much outstanding event data).

[github-actions]

It looks like this PR contains a migration 👀
Here's the checklist for reviewing migrations:

General requirements

• Satisfies idempotency requirement (both up() and down())
• Does not reference models
• Filename is in the correct format (and correctly ordered)
• Targets the next minor version
• All code paths have appropriate log messages
• Uses the correct utils
• Contains a minimal changeset
• Does not mix DDL/DML operations
• Tested in MySQL and SQLite

Schema changes

• Both schema change and related migration have been implemented
• For index changes: has been performance tested for large tables
• For new tables/columns: fields use the appropriate predefined field lengths
• For new tables/columns: field names follow the appropriate conventions
• Does not drop a non-alpha table outside of a major version

Data changes

• Mass updates/inserts are batched appropriately
• Does not loop over large tables/datasets
• Defends against missing or invalid data
• For settings updates: follows the appropriate guidelines

[9larsons]

I'm thinking we may want to include some sort of backstop for the jobs table data in case it isn't cleared during this migration, the migration isn't run, etc. I don't know what would be reasonable and I also don't want to obscure the existing code with weird handling of cases like this. Ideas?

[cmraible]

I don't know what would be reasonable and I also don't want to obscure the existing code with weird handling of cases like this. Ideas?

Maybe a silly idea, but could we use a new name for the job, so it wouldn't even look at the same rows if the migration doesn't run/fails for some reason? Maybe we could introduce a concept of job version, so the rows would become e.g. email-analytics-latest-opened-v2 or something along those lines? And if we ever come across a similar challenge in the future where we change the functionality of the job, we apply the same pattern and increment to v3?

[9larsons]

While that does work, I don't see a need to version and that makes future migrations to try and clear this out extra messy. I'd personally rather avoid that.

[cmraible]

I think I'm missing something here, maybe you can help me understand. I'm not sure I see how this avoids us missing events with fetchMissing. I see the main changes here are that we're fetching a max of 10k opened events, rather than infinity.

As I understand this logic, if we fetch 10k opened events, we'll stop there and wait for the job to kick off again. If we end up fetching 10k opened events for a couple iterations in a row, couldn't we fall behind on fetchMissing again?

Again, I'm probably just confused! Maybe we can get on a call tomorrow AM to go over it 😃