An Introduction to Kubernetes nodeAffinity & nodeSelector

Affinity-based Scheduling in Kubernetes

K8s POD is automatically scheduled to a NODE. By default, the scheduler considers whether resources are sufficient and tries to keep the load as evenly distributed as possible. However, sometimes we need finer-grained control over POD scheduling, for example, controlling whether PODs run on the same NODE. This is where a Kubernetes concept comes in: affinity. Affinity is mainly divided into two categories: nodeAffinity and podAffinity.

nodeSelector

Users can flexibly leverage label to manage resources in a cluster. The most common example is that a service selects PODs by matching label. Likewise, POD scheduling can also be targeted to specific deployments based on node labels.

We can view the labels of our nodes with the following command:

1
2
3
4
5
6
7
$ kubectl get nodes --show-labels
NAME            STATUS    ROLES     AGE       VERSION   LABELS
192.168.1.140   Ready     <none>    42d       v1.8.1    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=192.168.1.140
192.168.1.161   Ready     <none>    118d      v1.8.1    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/cluster-service=true,kubernetes.io/hostname=192.168.1.161
192.168.1.170   Ready     <none>    118d      v1.8.1    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/cluster-service=true,kubernetes.io/hostname=192.168.1.170
192.168.1.172   Ready     <none>    114d      v1.8.1    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/cluster-service=true,kubernetes.io/hostname=192.168.1.172

Now let’s first add a source=qikqiak label to node 192.168.1.140, with the following command:

1
2
3
$ kubectl label nodes 192.168.1.140 source=qikqiak
node "192.168.1.140" labeled

We can use the --show-labels parameter above to check whether the label took effect. Once a node has been labeled, these labels can be used during scheduling. You simply add a nodeSelector field in the POD’s spec, containing the labels of the node you want the POD scheduled onto. For example, here is a default busybox POD YAML file from earlier:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: busybox-pod
  name: test-busybox
spec:
  containers:
  - command:
    - sleep
    - "3600"
    image: busybox
    imagePullPolicy: Always
    name: test-busybox

Now if I want the above POD to be scheduled onto node 140, the simplest approach is to match the label on 140, as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: busybox-pod
  name: test-busybox
spec:
  containers:
  - command:
    - sleep
    - "3600"
    image: busybox
    imagePullPolicy: Always
    name: test-busybox
  nodeSelector:
    source: qikqiak

Then we can check the scheduling result via the describe command:

1
2
3
4
5
6
7
8
9
10
11
12
$ kubectl describe pod test-busybox
......
Events:
  Type    Reason                 Age   From                    Message
  ----    ------                 ----  ----                    -------
  Normal  Scheduled              49s   default-scheduler       Successfully assigned test-busybox to 192.168.1.140
  Normal  SuccessfulMountVolume  49s   kubelet, 192.168.1.140  MountVolume.SetUp succeeded for volume "default-token-hmpbz"
  Normal  Pulling                49s   kubelet, 192.168.1.140  pulling image "busybox"
  Normal  Pulled                 41s   kubelet, 192.168.1.140  Successfully pulled image "busybox"
  Normal  Created                41s   kubelet, 192.168.1.140  Created container
  Normal  Started                41s   kubelet, 192.168.1.140  Started container

Looking at the information under Events, we can see that the POD above was correctly scheduled to node 140. From the example above, we can feel that the nodeSelector approach is fairly intuitive, but not flexible enough and too coarse-grained. Next we’ll look at another, more flexible approach: nodeAffinity.

nodeAffinity

nodeAffinity refers to node affinity, with its counterpart being Anti-Affinity. This approach is more flexible than the nodeSelector above, allowing for some simple logical combinations rather than just simple equality matching. Scheduling can be divided into soft and hard policies. A soft policy means that if there is no node that satisfies your scheduling requirements, the POD will ignore that rule and continue the scheduling process. In other words, it’s the “it’s best if it’s satisfied, but it doesn’t really matter if not“ policy. A hard policy is much stricter: if no node satisfies the conditions, it will keep retrying until the conditions are met. In short, it’s the “you must meet my requirements, otherwise I won’t do it“ policy.

nodeAffinity is divided into the following two policies:

  1. preferredDuringSchedulingIgnoredDuringExecution — soft policy
  2. requiredDuringSchedulingIgnoredDuringExecution — hard policy.

Here is an example: (test-node-affinity.yaml)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
apiVersion: v1
kind: Pod
metadata:
  name: with-node-affinity
  labels:
    app: node-affinity-pod
spec:
  containers:
  - name: with-node-affinity
    image: nginx
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: NotIn
            values:
            - 192.168.1.140
            - 192.168.1.161
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: source
            operator: In
            values:
            - qikqiak

The above POD first requires that the POD cannot run on nodes 140 and 161, and if there is a node that satisfies source=qikqiak, it will be preferentially scheduled onto that node. Likewise, we can use the descirbe command to check whether the specific scheduling result meets our expectations. The matching logic here is that a label’s value is within a given list. The operators currently provided by Kubernetes are as follows:

  • In: the label’s value is in a given list
  • NotIn: the label’s value is not in a given list
  • Gt: the label’s value is greater than some value
  • Lt: the label’s value is less than some value
  • Exists: a label exists
  • DoesNotExist: a label does not exist

If there are multiple options under nodeSelectorTerms, satisfying any one of the conditions is enough; if there are multiple options in matchExpressions, all of these conditions must be satisfied simultaneously for the POD to be scheduled normally.

podAffinity

Both of the approaches above let a POD select a node. Sometimes we also want to perform scheduling based on relationships between PODs. The podAffinity concept introduced in Kubernetes 1.4 can fulfill this need.

Similar to nodeAffinity, podAffinity also has two scheduling policies: requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution. The only difference is that if we want to use mutual exclusion, we need to use the podAntiAffinity field. In the following example, we want with-pod-affinity and busybox-pod to be co-located, but we don’t want it to be deployed in the same topology domain as node-affinity-pod: (test-pod-affinity.yaml)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
apiVersion: v1
kind: Pod
metadata:
  name: with-pod-affinity
  labels:
    app: pod-affinity-pod
spec:
  containers:
  - name: with-pod-affinity
    image: nginx
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - busybox-pod
        topologyKey: kubernetes.io/hostname
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - node-affinity-pod
          topologyKey: kubernetes.io/hostname

The POD in the example above needs to be scheduled onto a specific host on which at least one node is running a POD with an app=busybox-pod label. podAntiAffinity, on the other hand, means we’d prefer not to schedule onto a node that is running a POD with an app=node-affinity-pod label. Based on the definitions of the previous two PODs, we can predict that the POD above should be scheduled onto node 140, because busybox-pod was scheduled to node 140 while node-affinity-pod was scheduled to a node other than 140, which exactly satisfies the requirements above. Check with describe:

1
2
3
4
5
6
7
8
9
$ kubectl describe pod with-pod-affinity
......
Events:
  Type    Reason                 Age   From                    Message
  ----    ------                 ----  ----                    -------
  Normal  Scheduled              8s    default-scheduler       Successfully assigned with-pod-affinity to 192.168.1.140
  Normal  SuccessfulMountVolume  7s    kubelet, 192.168.1.140  MountVolume.SetUp succeeded for volume "default-token-lcl77"
  Normal  Pulling                7s    kubelet, 192.168.1.140  pulling image "nginx"

The event information above also confirms our reasoning.

At the same level as labelSelector and topologyKey, you can also define a namespaces list, indicating which namespaces’ pods to match. By default, it matches the namespace of the defined pod; if this field is defined but its value is empty, it matches all namespaces.

Check the results of the 3 PODs we defined above:

1
2
3
4
5
6
$ kubectl get po -o wide
NAME                 READY     STATUS    RESTARTS   AGE       IP             NODE
test-busybox         1/1       Running   0          8m        172.30.95.18   192.168.1.140
with-node-affinity   1/1       Running   0          10m       172.30.81.25   192.168.1.172
with-pod-affinity    1/1       Running   0          8m        172.30.95.17   192.168.1.140

Comparison of affinity/anti-affinity scheduling policies:

Scheduling policy Matching label Operators Topology domain support Scheduling target
nodeAffinity Host In, NotIn, Exists, DoesNotExist, Gt, Lt No Specified host
podAffinity POD In, NotIn, Exists, DoesNotExist Yes POD in the same topology domain as the specified POD
podAnitAffinity POD In, NotIn, Exists, DoesNotExist Yes POD not in the same topology domain as the specified POD

Taints and Tolerations

For nodeAffinity, whether hard or soft policy, the goal is to schedule the POD onto the expected node. Taints does the exact opposite: if a node is marked with a Taint, unless the POD is also marked as able to tolerate tainted nodes, no PODs will be scheduled to that tainted node.

For example, if users want to reserve the Master node for Kubernetes system components, or reserve a group of nodes with special resources for certain PODs, taints become very useful—PODs will no longer be scheduled onto nodes marked with a taint. Here is an example of tainting a node:

1
2
3
$ kubectl taint nodes 192.168.1.40 key=value:NoSchedule
node "192.168.1.40" tainted

If you still want a POD to be scheduled onto a tainted node, you must define a Toleration in its Spec before it can be scheduled onto that node, as shown below:

1
2
3
4
5
6
tolerations:
- key: "key"
operator: "Equal"
value: "value"
effect: "NoSchedule"

There are three available options for effect, which can be set according to actual needs:

  1. NoSchedule: Unless there is a matching toleration specification, new Pods will not be scheduled onto the tainted node. Pods currently running on the node will not be evicted.
  2. PreferNoSchedule: PreferNoSchedule is the “preferred” or “soft” version of NoSchedule. The control plane will try to avoid scheduling Pods that cannot tolerate the taint onto the node, but this is not guaranteed to be completely avoided.
  3. NoExecute: This option means that once the Taint takes effect, any POD running on the node that does not have a matching Tolerate setting will be evicted directly.

Refer

Cloud Native > Kubernetes
#Kubernetes
An Introduction to Kubernetes nodeAffinity & nodeSelector
https://garychow-lgtm.github.io/en/2024/07/31/Kubernetes-nodeAffinity-And-Selector/
Author
Gary Chow (Yimin Cao)
Posted on
July 31, 2024
Licensed under