Debugging Kubernetes FailedScheduling

Background

While deploying Druid on Kubernetes, I noticed that after changing the replica count of the StatefulSet, the pod remained stuck in Pending. Running kubectl describe on the pod surfaced the following logs.

nodeSelector_new:
  dedicated: druid-middlemanager

tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "druid"
    effect: "NoSchedule"

Node-Selectors:              dedicated=druid-middlemanager
Tolerations:                 dedicated=druid:NoSchedule

Events:
  Type     Reason             Age                 From                Message
  ----     ------             ----                ----                -------
  Normal   NotTriggerScaleUp  117s                cluster-autoscaler  pod didn't trigger scale-up: 1 node(s) had untolerated taint {dedicated: ***} .......
  Warning  FailedScheduling   10s (x9 over 2m8s)  default-scheduler   0/84 nodes are available: 1 Insufficient memory, 1 node(s) had untolerated taint {dedicated: ****} ...

Debug

My first instincts pointed to one of the following bugs:

1. Incorrect labels on the nodegroup resource
2. Incorrect labels on the StatefulSet resource
3. Auto-scaling on the nodegroup failing to trigger

After inspecting the dedicated label and tolerations on both the StatefulSet and the nodegroup, nothing looked wrong, which ruled out Bug 1 and Bug 2. That left Bug 3 as the focus.

K8s Labels: foc/bottlerocket=true dedicated=druid-middlemanager
K8s Taints: dedicated=druid:NoSchedule

The nodegroup already contained a node. If capacity were the issue, there should be scale-up attempt logs in the nodegroup’s activities — but there weren’t any. So scaling out would not have solved this either, which ruled out Bug 3 as well.

Conclusion

The root cause turned out to be that the nodegroup’s instance type had too little memory: even if the autoscaler did scale up, no node in that group could offer enough memory for a single pod, which triggered the errors above.
These logs easily mislead you into thinking the problem is a label mismatch or a broken autoscaler — a very sneaky bug.

resources:
  limits:
    memory: 110Gi
  requests:
    memory: 110Gi