[Feature Request] Record metrics on whether the worker pool was sufficient #4746
Description
Metrics on Pool Sufficiency
Concept: I want to know how often my dev team is delayed in their CI runs because I didn't have enough hosts available in the worker pool. We have some large pools, but we also have a lot of PRs with a CI step that takes a lot of hosts.
If I'm sufficient 100% of the time during certain hours of the day, perhaps I'm overscaling the pool size. If I'm insufficient 70% of the time during other hours of the day, clearly my pool isn't large enough at that time. This data will let me modify those pool size configs as needed to ensure I'm neither overspending nor under-provisioning.
How?
My naive approach is to just modify scale-up.ts such that if (scaleUp) is true, we record the metric as a 0.0 value, and if it's false, we record the metric as a 1.0 value. Eg: If you didn't need to scale up, then you had enough hosts.
I've got a code change written up that I'll follow-up with shortly. Please tell me if this is totally the wrong direction or you see reasons not to do this. Happy to try other things. For me, this was a fun dive into how this repo works.