Kubernetes and Python

Posted:   |  More posts about kubernetes python

While Go is the language-of-choice in the cloud-native world, Python has a huge community and makes it really easy to extend Kubernetes in only a few lines of code. This post is a follow-up to a talk I gave last week in Prague.

Marek invited me to give a talk at the Cloud Native meetup in Prague. I chose "Kubernetes + Python = ❤" as a subject because I realized that I never talked about my usage of Python for Kubernetes controllers/operators/apps. You can find the slides of my talk on Slideshare and relevant slides are linked in the text below.

Cloud Native Prague meetup

Running Python on Kubernetes

Let's start simple: how to run a Python web service on Kubernetes? I tend to use the new asyncio AIOHTTP framework, so the code is less than 10 lines of Python and the Dockerfile is straightforward:

FROM python:3.7-alpine

RUN pip install aiohttp


ENTRYPOINT ["python", ""]

Everybody "knows" that "Python is slow" due to its GIL, but a non-scientific test (vegeta) shows that it's fast enough and can sustain 200 rps with 4.8ms p99 latency (slide 12).

There is one detail which makes the default behavior of aiohttp.web.run_app not safe to use: Kubernetes' "graceful shutdown" behavior (slide 14). AIOHTTP will by default handle the SIGTERM signal which leads to failing requests during Pod termination / rolling deployments. Disabling the signal handling (thus relying on the KILL signal after 30 seconds) with web.run_app(app, handle_signals=False) will remove the failing requests as this test shows (slide 15):

Elapsed:        41.60 seconds
Successes:      2056
Errors:         0

Elapsed:        15.16 seconds
Successes:      816
Errors:         2

An alternative which works for any app is the prestop trick: add a preStop lifecycle command with "sleep 20" to your container (slide 16).

You can find the complete "hello world" aiohttp example in the repo.

Accessing Kubernetes from Python

client-go isn't for mortals - Bryan Liles

There are (at least) two Kubernetes clients for Python (slide 24):

Both do their job, but (no surprise) I prefer Pykube-NG as it is more lightweight and feels more "Pythonic", compare yourself:

# Official Kubernetes Python client
from kubernetes import client, config

v1 = client.CoreV1Api()
ret = v1.list_namespaced_pod("foobar")
for pod in ret.items:
    images = [c.image for c in pod.spec.containers]
    print(, ", ".join(images))
# Pykube-NG
from pykube import HTTPClient, KubeConfig, Pod

api = HTTPClient(KubeConfig.from_file())

for pod in Pod.objects(api).filter(namespace="foobar"):
    images = [c["image"] for c in pod.obj["spec"]["containers"]]
    print(, ", ".join(images))

Pykube is more lightweight (160 KiB vs 23 MiB for the official client) and provides generic methods like obj.create() for Pods, Deployments, etc (the official client has methods for each OpenAPI operation). This is helpful when writing generic controllers which have to deal with different kind of Kubernetes objects.

Writing a simple Kubernetes controller takes only a few lines of Python code (slide 26) and I created a bunch of controllers this way:

  • kube-downscaler to scale down deployments after work hours, e.g. to save cloud costs

  • kube-janitor to clean up (delete) Kubernetes resources after a configured TTL (time to live), e.g. to automatically remove PR/preview deployments in test clusters

Writing Operators

Kubernetes Operators can provide important functionality to integrate Kubernetes with other infrastructure or to run stateful services on Kubernetes. Zalando open sourced two operators which are successfully used in critical production scenarios:

Both operators are written in Go and evolved over time. What is the best way to write operators in Python without boilerplate? Zalando's answer is the Kubernetes Operator Pythonic Framework (Kopf)! Kopf allows writing simple handler functions which are called automatically on create/update/delete for your CRD objects (slide 40):

import kopf

@kopf.on.create("", "v1", "helloworlds")
def on_create(spec, **kwargs):
    print(f"Create handler is called with spec: {spec}")
    return {"message": f"Hello {spec['name']}!"}

@kopf.on.update("", "v1", "helloworlds")
def on_update(body, **kwargs):
    print(f"Update handler is called with body: {body}")

That's all we need for handling creates and updates of "HelloWorld" CRD objects. You can find the complete Kopf example in the repo.

Michael Gasch did a more elaborate demo (creating vSphere VMs from Kubernetes) with Kopf at VMWorld, there is a Medium article about Building a Kubernetes Operator in Python with Zalando's Kopf, and you can check out Kopf's extensive documentation.

Testing with Pytest and Kind

Writing apps/controllers/operators for Kubernetes obviously requires some testing! The tool of choice for this is kind (Kubernetes IN Docker) as it has no dependencies (except Docker) and provides a tight feedback loop (cluster bootstrap is fast).

I created pytest-kind, a small plugin for pytest to integrate with kind (slide 19). It will download kind, create a cluster, and provide access via a pytest fixture. A complete test for our "hello world" Kopf operator does not require much:

import requests
import time

from pykube.objects import NamespacedAPIObject

class HelloWorld(NamespacedAPIObject):
    version = ""
    kind = "HelloWorld"
    endpoint = "helloworlds"

def test_web_hello_world(kind_cluster):
    kind_cluster.kubectl("apply", "-f", "deploy/")
    kind_cluster.kubectl("rollout", "status", "deployment/kopfexample-operator")
    kind_cluster.kubectl("apply", "-f", "example.yaml")

    for i in range(10):
        obj = HelloWorld.objects(kind_cluster.api).get(name="test-1")
        if "status" in obj.obj:

    assert obj.obj["status"]["on_create"]["message"] == "Hello Prague!"

Even my recently introduced frontend project Kubernetes Web View (slide 47+) is easy to test as it relies on plain HTML rendering (using Jinja2) and therefore does not require sophisticated browser-based testing (slide 67).

Kubernetes Web View: Label and Custom Columns

Kubernetes + Python = ❤

It's pretty clear that I like Kubernetes and Python, a quick look at my open source projects is enough:

I feel rather productive with my personal Python+Kubernetes setup, what are your experiences?

Do you write any Kubernetes applications/controllers/operators in Python? Do you use any of the mentioned open source projects? Let me know on Twitter!