Patching Kubernetes resources with kubectl
In this blog we’re going to learn how we can use kubectl’s patch command to modify the configuration of Kubernetes-managed resources via the command-line. Before we do that though, we’ll go through a quick primer on how you can display Kubernetes resources so that you know what and where to patch.
Getting Kuberenetes resources
One of the first kubectl commands a Kubernetes beginner will become intimately acquainted with is the get command. It displays all manner of different resources that Kubernetes is aware of, such as pods, deployments and secrets.
1# kubectl get pods
2NAME READY STATUS RESTARTS AGE
3appuio-php-docker-ex-1-build 1/1 Running 0 11s
In the example above, we can see that the default behaviour of the get command is to produce human-readable output. We can customize that behaviour with the use of the --output (or -o) option to select from parseable formats such as yaml and json. These formats are useful to view in order to understand how we can subsequently patch them. In the example below we are retrieving the list of Kubernetes projects, but in YAML form:
1# kubectl get projects -o yaml
2apiVersion: v1
3items:
4- apiVersion: project.openshift.io/v1
5 kind: Project
6 metadata:
7 name: default
8 namespace: ""
9 resourceVersion: "1205"
10 spec:
11 finalizers:
12 - kubernetes
We can even define our own output formats via JSONPath or Golang templates. These grant us the flexibility to specify the fields we want to see and how we wish to see them, making them useful for scenarios where we wish to feed the state of Kubernetes into other scripts. This is a useful technique to have at our disposal, so let’s look at JSONPath in particular.
JSONPath is similar in concept to jq, a well-known CLI tool for interacting with JSON. JSONPath has a different syntax for querying the JSON, but largely achieves the same goals. In the simple exmaple below, we’ll use JSONPath to pull back the project name for a project that happens to have a matching UID value.
1# kubectl get projects --output \
2# jsonpath='{.items[?(@.metadata.uid=="054dda83-4e53-11ea-aed8-000c29eb7917")].metadata.name}'
3hello
However, if you’re already quite familiar with jq, you can of course pipe your json output to it instead and achieve the same goal:
1# kubectl get projects --output json | \
2# jq '.items[] | select(.metadata.uid=="054dda83-4e53-11ea-aed8-000c29eb7917") | .metadata.name'
3"hello"
There is an excellent and steadily-growing resource of creative ways to use kubectl and jsonpath to produce a concise summary of information over in this github gist.
Patching Kubernetes resources
Now that we know how to display Kubernetes resources, let’s learn how we can modify them. kubectl provides the patch command for exactly this purpose. What’s more, it provides three different techniques for applying patches, each of which has advantages (or disadvantages) in certain situations. Whilst you might be able to get away with its default behaviour (the “strategic merge”) for most scenarios, being aware of the other techniques is helpful. We’re going to look at those alternate methods first, and then see how the default behaviour differs.
Patching the JSON Merge way
JSON Merge is the first technique we’ll explore, and - as the name would suggest - lets you merge new JSON into existing JSON document. To illustrate this with an example, let’s first create a configmap:
1# kubectl create configmap hello-config \
2# --from-literal=foo=bar --from-literal=beep=boop \
3# -o json
4configmap/hello-config created
5{
6 "apiVersion": "v1",
7 "data": {
8 "beep": "boop",
9 "foo": "bar"
10 },
11 "kind": "ConfigMap",
12 "metadata": {
13 ... output trimmed ...
14 }
15}
To use the JSON Merge technique when patching, we must provide the --type=merge argument, and then supply the JSON to merge in. If we want to change the value of the foo key, we can simply supply that portion of the JSON:
1# kubectl patch configmap hello-config \
2# --type=merge -p '{"data": {"foo": "baz"}}' \
3# -o json
4configmap/hello-config patched
5{
6 "apiVersion": "v1",
7 "data": {
8 "beep": "boop",
9 "foo": "baz"
10 },
11 ... output trimmed ...
If we supply previously non-existent JSON, it will just get merged straight in. The example below illustrates this, by adding an additional key-value pair:
1# kubectl patch configmap hello-config \
2# --type=merge -p '{"data": {"new": "yes"}}'
3# -o json
4configmap/hello-config patched
5{
6 "apiVersion": "v1",
7 "data": {
8 "beep": "boop",
9 "foo": "baz",
10 "new": "yes"
11 },
12 ... output trimmed ...
There is a catch, however! When you use this technique on JSON arrays it will replace the entire array, rather than merge a new item into it. Let’s see that with an example.
We’ll start with a limitrange definition for pods:
1# echo """apiVersion: v1
2# kind: LimitRange
3# metadata:
4# creationTimestamp: 2020-02-16T11:51:35Z
5# name: core-resource-limits
6# namespace: myproject
7# spec:
8# limits:
9# - max:
10# memory: 1Gi
11# min:
12# memory: 6Mi
13# type: Pod""" | oc create -f -
14limitrange/core-resource-limits created
We’ll now attempt to add a new array element for containers with the merge technique:
1# kubectl patch limitrange core-resource-limits \
2# --type=merge -p '''
3# { "spec":
4# { "limits": [
5# { "default" : { "memory": "200Mi"},
6# "defaultRequest" : { "memory": "200Mi"},
7# "type": "Container"}
8# ]
9# }
10# }''' \
11# -o yaml
12"core-resource-limits" patched
13spec:
14 limits:
15 - default:
16 memory: 200Mi
17 defaultRequest:
18 memory: 200Mi
19 type: Container
Our entire array has been replaced by the new item, which we clearly don’t want. How can we perform patches on arrays non-destructively? Our remaining two kubectl patching methods cater for that.
More information on the JSON Merge technique can be read about in RFC7386.
Patching, the JSON Patch way
The next technique we’ll look at is known as the JSON Patch. This allows for a more sophisticated ability to manipulate the JSON by allowing us to be explicit about the technique we wish to perform (add, remove, replace, move and copy) and what we wish to see affected.
To apply the JSON Patch technique, we need to:
- inform
kubectlvia the--type=jsonargument. - specify the operation taking place (
replace,add,move, and so on) - specify the
pathto the JSON where the patch is applied - specify the
valueto be used in the patch
Here’s an example below:
1[{ "op": "add",
2 "path": "/spec/containers/-",
3 "value": {
4 "name": "php-hello-world",
5 "image": "php-hello-world-1.0"
6 }
7}]
As we’ve seen earlier, the JSON Merge technique is poor for array operations, but this is a place in which JSON Patch excels. Using array-like indexing, we can be surgical about where in the array an item should be patched. Let’s revisit our earlier limitrange patching example with a JSON Patch approach.
As you’ll recall, our JSON looks like the example below.
1 limits:
2 - max:
3 memory: 1Gi
4 min:
5 memory: 6Mi
6 type: Pod
We wish to add a new array element. We’ll do so at the end of the array, indicated using the - symbol.
1# kubectl patch --type=json \
2# limitrange core-resource-limits -p '''
3# [{ "op": "add",
4# "path": "/spec/limits/-",
5# "value": {
6# "defaultRequest" : {
7# "memory": "200Mi"
8# },
9# "type": "Container"
10# }}]''' \
11# -o yaml
12spec:
13 limits:
14 - max:
15 cpu: "2"
16 memory: 1Gi
17 min:
18 cpu: 200m
19 memory: 6Mi
20 type: Pod
21 - defaultRequest:
22 memory: 200Mi
23 type: Container
As you can see, both the existing and the new element are present within the array.
There is one weakness with JSON Patch and array operations, however. We must know beforehand precisely where we want to operate on the array. We cannot, for example, search for the correct position to apply a patch based on the array’s sub-elements.
If we had a pod spec as follows:
1spec:
2 containers:
3 - name: httpd
4 image: httpd-1.0
5 - name: sidecar
6 image: my-sidecar-1.0
and we wanted to patch the image value for the sidecar container, can we do so? If we were confident that it always appeared in the second position of the array, maybe, but we can’t always make that assumption. The last technique we’ll look at, the Strategic Merge, goes some way towards alleviating this issue.
By the way, if you’re starting to get confused about JSON Patch and JSON Merge.. well, I don’t blame you! But there is a great resource that summarizes the differences between the two.
Patching strategically
The Strategic Merge is the default behaviour of kubectl. That is to say, if you don’t specify a --type at all, it’ll be interpreted as a strategic merge.
Strategic Merges look similar to JSON Merges, but are more array-aware through the application of a patch strategy. The patch strategy dictates what happens when a patch is applied to an array and is
mergereplacedelete
You can look up whether an array resource definition has a patch strategy defined in the Kubernetes API Documentation. If there isn’t one defined, the default strategy is to replace the array.
Look at the documentation for the DaemonSet resource definition as an example. We can see that:
- The
specsub-field is a DaemonSetSpec, which contains atemplatestruct of type PodTemplateSpec.. - ..which contains a
specstruct of type PodSpec.. - ..which contains a
containersarray of type Container, which defines a patch strategy ofmergeand amerge keyofname.
The merge key is used as the distinguishing element between array field elements. Let’s look at this in practice by attempting to patch a DaemonSet example:
1apiVersion: apps/v1
2kind: DaemonSet
3metadata:
4 name: daemonset-example
5spec:
6 selector:
7 matchLabels:
8 app: daemonset-example
9 template:
10 metadata:
11 labels:
12 app: daemonset-example
13 spec:
14 containers:
15 - name: daemonset-example
16 image: ubuntu:trusty
17 command:
18 - /bin/sh
19 args:
20 - -c
21 - >-
22 while [ true ]; do
23 echo "DaemonSet running on $(hostname)" ;
24 sleep 10 ;
25 done```
The name field in the containers array is our merge key in this instance, uniquely identifying that particular array entry. We must therefore supply the name in our patch, alongside the value of daemonset-example. If we don’t do that, our patch will be added in as a new array element instead of merging with the existing one.
Let’s patch the array entry to change the command to bash. Our patch will look like the following:
1{ "spec":
2 { "template":
3 { "spec":
4 { "containers":
5 [ { "name": "daemonset-example", "command": [ "/bin/bash" ] } ]
6 }
7 }
8 }
9}
We can apply it as follows:
1kubectl patch daemonset daemonset-example -p "$(cat patch.json)" -o yaml
and can subsequently observe that the command portion of the array entry has changed to:
17 command:
18 - /bin/bash
If we provided a different value of name, for instance new-daemonset-example, our patch would instead try to create a second list element using our patch. In our example, this would actually fail because our patch would be missing a required field:
The DaemonSet "daemonset-example" is invalid:
spec.template.spec.containers[0].image: Required value
You might be wondering: what happens if you’re operating on an array that doesn’t seem to have a patch strategy defined? The default behaviour is to replace the list - in essence, the same behaviour that we observe with JSON Merge. Therefore, it is prudent to check what patch strategy might exist before rushing into performing a strategic merge operation on a list.
To fully discuss the strategic technique is well beyond the scope of this article. However, the best documentation I have seen on the topic is available here.
The fine print
Look before you leap
Patching a Kubernetes resource with patch is a venture that’s risky. There’s no backups kept, and no preserved history of your previous state. Whilst kubectl will protect you from outright corruption of the structural integrity of the resource (for example, by supplying improperly-formatted JSON), the changes are immediate and may impact running workloads.
It may be wise to employ use of the --dry-run and --output parameters to see what your requested change would have done without actually carrying it out.
YAML is fine too
In the examples above, we’ve supplied JSON representations of the various patches we’ve wanted to apply. But if you’re more of a YAML person, you’re in luck! YAML will work just as well.
Because YAML is a newline-oriented format, the easiest way to supply it is by writing the patch to a file and then passing it to kubectl via a subshell:
# kubectl patch dc/hello-world -p "$(cat mypatch.yaml)"
OpenShift/OKD and patching
We’ve been referring to kubectl above, but if you’re a user of Red Hat OpenShift or OKD, do note that the oc client behaves in the same manner and that all examples are cross-compatible.
Why are there so many different ways to mess with JSON data?
Sorry friend, you’re asking the wrong person there.
