Implement queuing priority #56

[patrick-austin]

Also includes changes from #45 #51, due to dependency of priority settings on submitting to and polling the queue. See those PRs for those changes in isolation.

• Add queue priority settings to run.properties.
  • This can be a single number, for all users, authenticated users, investigation users or instrument scientists.
  • For the latter two, it can also be a mapping of Instrument.name or role to a priority level
• Class to handle loading and formatting priority levels in JPQL
• Added functions to IcatClient to perform these queries for a given user to identify their level of priority
• Modified StatusCheck to get all queued Downloads, then when iterating only submit those with priority 1. If there is still space, start submitting from priority 2 and so on.
• For queue submission endpoints, now only submit if user has a positive priority

Closes #56

[kevinphippsstfc]

This looks great! I love the concept of being able to prioritise different types of request based on who the user is and this is exactly what we will need once open data becomes a thing.

Unless I'm missing something the prioritisation is only applied to downloads that are put in the queue via the new Queue Visit and Queue Files endpoints. What I think we probably need to move to is a system where ALL downloads go via the queue and are prioritised but that will need a bit more thinking about. For example, the new recall methods need to have less priority as they are likely to be larger requests even though they are likely to be submitted by Instrument Scientists who may have higher priority. And where do we prioritise those against recalls from open data users? However, I think this is probably changes that we can make in the future, depending on what ideas you have on this.

[kevinphippsstfc]

These last 3 lines in both the if and the else could go below the if/else as they are common.
Although you would also need to initialise msg above the if/else, so that line is borderline worth/not worth it.
However, the final two lines are probably worth moving.

[patrick-austin]

Have refactored the last three lines, this also means introducing an else clause so we can return early don't warn/set default when we don't meet either criteria (corresponding to a "normal" value for authenticatedPriority).

[patrick-austin]

Unless I'm missing something the prioritisation is only applied to downloads that are put in the queue via the new Queue Visit and Queue Files endpoints.

Yes, this is currently unique to the polling which moves from PAUSED with no preparedId (AKA queued) to PREPARING. I didn't want to change/break existing methods that are used by DataGateway currently, just add this as a new feature.

What I think we probably need to move to is a system where ALL downloads go via the queue and are prioritised but that will need a bit more thinking about.

Agreed - I wouldn't want to start changing the existing workflow (other than e.g. for checking feasibility) without getting some buy in from the others (frontend/service management/DLS themselves).

For example, the new recall methods need to have less priority as they are likely to be larger requests even though they are likely to be submitted by Instrument Scientists who may have higher priority. And where do we prioritise those against recalls from open data users? However, I think this is probably changes that we can make in the future, depending on what ideas you have on this.

Yeah it's not obvious to me what we want to enforce. Practically, I think the current system could be extended to cover some of this. For example, the reason for having a different authenticatedPriority and defaultPriority is to allow us to discriminate the open data users. To combine priorities, I think we can add multiple factors and evaluate the result. For example, I might be instrument scientist and have a "user" priority of 1, but requesting an entire visit might give me a "content" priority of say 3 for a total of 4, which then might be higher/equal/lower to whatever priority is given to an anonymous user requesting a Dataset. The challenge with the current approach will be mapping the "content" or "source" of the Download. Once something is in the queue, how do we know where it came from? If we base this on e.g. just the size of the Download we have that in the DB, but it might be trickier to work out whether something came by the API or frontend without either piggybacking an existing Download field or just creating a new dedicated field for priority. At any rate, what we do would depend on requirements, so definitely something for the future I think.

[kevinphippsstfc]

Apologies I didn't take in to account the fact that an else would be needed to cater for a direct return from the method. Having seen the updated method side by side with the old one, I prefer the original! Although it has the repeated lines of code it is simpler and easier to follow. So please just revert this commit and I'll approve the PR.