Dashboards: Worker Filtering

[chrispdriscoll]

Description

Investigate expanding worker filtering options for potentially user-facing dashboards

Background (user request)

NOTE all references are internal to Target

We are looking at some additions and changes to our pipeline that will increase our build volume. It's difficult to get a sense on how frequently the workers for our label are busy or if builds are queued (or for how long)
As were making changes we want to be able to observe what kind of impact this is having so we can try to make adjustments or request additional workers

Count of busy instances from this dashboard looks like we want - although we can only select single host or all host. There's no option to use a "label" or select multiple machines that could be behind that label https://visualize.prod.target.com/d/JCVjfY1Zz/vela-workers?orgId=5&var-vela_version=0.25-prod&var-vela_host=All&from=now-2d&to=now

Pending build and build times from this dashboard look like what we want but it doesn't have the ability to filter down by host or label https://visualize.target.com/d/yRslXr7nz/vela-consumption?orgId=5&refresh=1m

Target Slack threads: user vela engineer

Definition of Done

dashboards have ability to filter to multi-selected workers and/or a flavor of workers