Default unlimited queue upon first installation

[romanbaron]

What you would like to be added?

I’ve observed that new users of KAI Scheduler often perceive queue creation and configuration as a complex process. Currently, we direct them to the following guide:
https://github.com/NVIDIA/KAI-Scheduler/blob/main/docs/queues/README.md

The guide recommends starting with an unlimited queue that can later be tailored to specific requirements. However, the onboarding experience could be significantly simplified if such a default queue were automatically provisioned during the initial installation.

Why is this needed?

Propose a straightforward design in which the initial installation automatically provisions an unlimited queue set for scheduling workloads with KAI Scheduler. Clarify how users can subsequently evolve these defaults into more advanced queue configurations with minimal complexity. Update the KAI examples to leverage these queues.

[singh1203]

Hi @romanbaron, I'm taking this task on; let's see how it goes. If I have trouble understanding, I will reach out to the team.
Also, I will be obliged by any guidance and support.
Thank you! 🙇

[romanbaron]

Thanks @singh1203 — this looks very promising 🙌

I was considering the following approach:

1. Default Queues on New Installations: On a fresh installation (but not on upgrades), KAI should deploy a set of default scheduling queues. These can be leveraged for scheduling instead of requiring users to create them manually, as outlined here: https://github.com/NVIDIA/KAI-Scheduler/tree/main/docs/quickstart#scheduling-queues
2. PodGrouper Integration: If a pod is created without an explicit scheduling queue, these defaults should be used in the PodGrouper code: https://github.com/NVIDIA/KAI-Scheduler/blob/main/pkg/podgrouper/podgrouper/plugins/defaultgrouper/default_grouper.go#L134
3. Quotas and Resource Isolation: These queues should enable workloads to be scheduled immediately by simply changing the schedulerName in the pod spec. At the same time, they should not reserve resources away from user-defined queues created for other purposes. This can be achieved by setting their quota to 0.
4. Opt-Out Mechanism: This behavior should be enabled by default, but we should also provide a feature flag in Helm values to allow users to disable it if needed.

Previously, I would have suggested implementing this in a post-install Helm hook. However, with @enoodle’s recent work on the new operator for KAI—which manages deployment, lifecycle, and configuration of KAI services—this seems like a more suitable place. @enoodle, could you share your thoughts on where exactly this feature should be implemented in the operator code?

[singh1203]

Thanks @singh1203 — this looks very promising 🙌

I was considering the following approach:

1. Default Queues on New Installations: On a fresh installation (but not on upgrades), KAI should deploy a set of default scheduling queues. These can be leveraged for scheduling instead of requiring users to create them manually, as outlined here: https://github.com/NVIDIA/KAI-Scheduler/tree/main/docs/quickstart#scheduling-queues
2. PodGrouper Integration: If a pod is created without an explicit scheduling queue, these defaults should be used in the PodGrouper code: https://github.com/NVIDIA/KAI-Scheduler/blob/main/pkg/podgrouper/podgrouper/plugins/defaultgrouper/default_grouper.go#L134
3. Quotas and Resource Isolation: These queues should enable workloads to be scheduled immediately by simply changing the schedulerName in the pod spec. At the same time, they should not reserve resources away from user-defined queues created for other purposes. This can be achieved by setting their quota to 0.
4. Opt-Out Mechanism: This behavior should be enabled by default, but we should also provide a feature flag in Helm values to allow users to disable it if needed.

Previously, I would have suggested implementing this in a post-install Helm hook. However, with @enoodle’s recent work on the new operator for KAI—which manages deployment, lifecycle, and configuration of KAI services—this seems like a more suitable place. @enoodle, could you share your thoughts on where exactly this feature should be implemented in the operator code?

Thank you, @romanbaron, for the guidance here. 👍
I will follow your instructions, and I think the best place to implement the feature is KAI-Scheduler/pkg/operator/operands/queue_controller/.
And looking for more info from @enoodle.

[enoodle]

@romanbaron :

Quotas and Resource Isolation: These queues should enable workloads to be scheduled immediately by simply changing the schedulerName in the pod spec.

doesn't it mean that all pods in the cluster will use Run:AI ? I think for most it will be undesirable .

@singh1203 : I am not sure that queue controller is directly involved here.

1. Installation can happen in the helm chart / kai operator. The operator is more about ensuring that objects exist, so maybe we can do it in helm with {{- if .Release.IsInstall }} guard rails to make sure it is only installed once.
2. Pod grouper already sets a default queue name if no label is found. (link mentioned above)
3. If we want to do point 3 it will probably be in the admission webhook for pods that we have in the admission service.
4. There can be a flag in the helm values to configure this which will default to true.

The queue controller is more about handling existing queues, I think it is a bit of an anti pattern to let a controller create the objects that it "controls".

[romanbaron]

Quotas and Resource Isolation: These queues should enable workloads to be scheduled immediately by simply changing the schedulerName in the pod spec.

doesn't it mean that all pods in the cluster will use Run:AI ? I think for most it will be undesirable .

No, I meant that users who want to schedule workloads with KAI will be able to do it by changing the schedulerName and by using the default queue. It should address the use case of someone who is just starting using KAI and wants to try something very simple before diving into resource division parameters.

[romanbaron]

Another example why this feature is so important: a user tried to schedule a pod with KAI and when the pod remained pending, the only visible error was:

Warning  Unschedulable      3m42s (x25 over 4m6s)  kai-scheduler  Unable to schedule pod

After a long investigation the root cause turned to be that the pod referenced a non-existing queue. The feedback I got from the user:
"I'd recommend increasing the amount diagnostic information though. Even increasing log verbosity and reading through logs didn't help. Ideally it should be on the pending pod event scheduler events... Full explanation: I was only creating a default queue from the quick start queues.yaml thinking that would be enough since my workloads targeted this default queue. Not even a simple test pod would start with this though. I re-read the quickstart documentation which mentions the 'test' queue: Workloads should reference this queue."

I am certain that queues that are deployed by default will definitely help but maybe we should also post a proper message on the pod/podgroup indicating that it references a non existing queue. Any thoughts?

[singh1203]

Another example why this feature is so important: a user tried to schedule a pod with KAI and when the pod remained pending, the only visible error was:

Warning  Unschedulable      3m42s (x25 over 4m6s)  kai-scheduler  Unable to schedule pod

After a long investigation the root cause turned to be that the pod referenced a non-existing queue. The feedback I got from the user: "I'd recommend increasing the amount diagnostic information though. Even increasing log verbosity and reading through logs didn't help. Ideally it should be on the pending pod event scheduler events... Full explanation: I was only creating a default queue from the quick start queues.yaml thinking that would be enough since my workloads targeted this default queue. Not even a simple test pod would start with this though. I re-read the quickstart documentation which mentions the 'test' queue: Workloads should reference this queue."

I am certain that queues that are deployed by default will definitely help but maybe we should also post a proper message on the pod/podgroup indicating that it references a non existing queue. Any thoughts?

Certainly, the default queue installation would help here, and I assume whenever the pod/podgroup references another queue apart from the default queue one, correct?
Also, do we need to increase the log verbosity in the event section? If yes, how can we?
Sorry for asking such a naive question. 🙇

[romanbaron]

A good candidate could be an event in the scheduler's status-updater that can check if pending podgruop references an un-existing queue.