Running Python on Kubernetes

Let's start simple: how to run a Python web service on Kubernetes? I tend to use the new asyncio AIOHTTP framework, so the code is less than 10 lines of Python and the Dockerfile is straightforward:

FROM python:3.7-alpine

RUN pip install aiohttp

COPY web.py /

ENTRYPOINT ["python", "web.py"]

Everybody "knows" that "Python is slow" due to its GIL, but a non-scientific test (vegeta) shows that it's fast enough and can sustain 200 rps with 4.8ms p99 latency (slide 12).

There is one detail which makes the default behavior of aiohttp.web.run_app not safe to use: Kubernetes' "graceful shutdown" behavior (slide 14). AIOHTTP will by default handle the SIGTERM signal which leads to failing requests during Pod termination / rolling deployments. Disabling the signal handling (thus relying on the KILL signal after 30 seconds) with web.run_app(app, handle_signals=False) will remove the failing requests as this test shows (slide 15):

tests/e2e/test_update.py::test_rolling_update_no_signal_handling
Elapsed:        41.60 seconds
Successes:      2056
Errors:         0

tests/e2e/test_update.py::test_rolling_update_with_signal_handling
Elapsed:        15.16 seconds
Successes:      816
Errors:         2

An alternative which works for any app is the prestop trick: add a preStop lifecycle command with "sleep 20" to your container (slide 16).

You can find the complete "hello world" aiohttp example in the repo.

Accessing Kubernetes from Python

There are (at least) two Kubernetes clients for Python (slide 24):

• Official Kubernetes Python client: pip install kubernetes

• Pykube-NG: pip install pykube-ng

Both do their job, but (no surprise) I prefer Pykube-NG as it is more lightweight and feels more "Pythonic", compare yourself:

# Official Kubernetes Python client
from kubernetes import client, config

config.load_kube_config()
v1 = client.CoreV1Api()
ret = v1.list_namespaced_pod("foobar")
for pod in ret.items:
    images = [c.image for c in pod.spec.containers]
    print(pod.metadata.name, ", ".join(images))

# Pykube-NG
from pykube import HTTPClient, KubeConfig, Pod

api = HTTPClient(KubeConfig.from_file())

for pod in Pod.objects(api).filter(namespace="foobar"):
    images = [c["image"] for c in pod.obj["spec"]["containers"]]
    print(pod.name, ", ".join(images))

Pykube is more lightweight (160 KiB vs 23 MiB for the official client) and provides generic methods like obj.create() for Pods, Deployments, etc (the official client has methods for each OpenAPI operation). This is helpful when writing generic controllers which have to deal with different kind of Kubernetes objects.

Writing a simple Kubernetes controller takes only a few lines of Python code (slide 26) and I created a bunch of controllers this way:

• kube-downscaler to scale down deployments after work hours, e.g. to save cloud costs

• kube-janitor to clean up (delete) Kubernetes resources after a configured TTL (time to live), e.g. to automatically remove PR/preview deployments in test clusters

Writing Operators

Kubernetes Operators can provide important functionality to integrate Kubernetes with other infrastructure or to run stateful services on Kubernetes. Zalando open sourced two operators which are successfully used in critical production scenarios:

• Postgres Operator which creates and manages PostgreSQL clusters on Kubernetes using our battle-tested Patroni

• Elasticsearch Operator which safely-drains and auto-scales Elasticsearch on Kubernetes

Both operators are written in Go and evolved over time. What is the best way to write operators in Python without boilerplate? Zalando's answer is the Kubernetes Operator Pythonic Framework (Kopf)! Kopf allows writing simple handler functions which are called automatically on create/update/delete for your CRD objects (slide 40):

import kopf


@kopf.on.create("example.org", "v1", "helloworlds")
def on_create(spec, **kwargs):
    print(f"Create handler is called with spec: {spec}")
    return {"message": f"Hello {spec['name']}!"}


@kopf.on.update("example.org", "v1", "helloworlds")
def on_update(body, **kwargs):
    print(f"Update handler is called with body: {body}")

That's all we need for handling creates and updates of "HelloWorld" CRD objects. You can find the complete Kopf example in the repo.

Michael Gasch did a more elaborate demo (creating vSphere VMs from Kubernetes) with Kopf at VMWorld, there is a Medium article about Building a Kubernetes Operator in Python with Zalando's Kopf, and you can check out Kopf's extensive documentation.

Testing with Pytest and Kind

Writing apps/controllers/operators for Kubernetes obviously requires some testing! The tool of choice for this is kind (Kubernetes IN Docker) as it has no dependencies (except Docker) and provides a tight feedback loop (cluster bootstrap is fast).

I created pytest-kind, a small plugin for pytest to integrate with kind (slide 19). It will download kind, create a cluster, and provide access via a pytest fixture. A complete test for our "hello world" Kopf operator does not require much:

import requests
import time

from pykube.objects import NamespacedAPIObject


class HelloWorld(NamespacedAPIObject):
    version = "example.org/v1"
    kind = "HelloWorld"
    endpoint = "helloworlds"


def test_web_hello_world(kind_cluster):
    kind_cluster.load_docker_image("kopf-example")
    kind_cluster.kubectl("apply", "-f", "deploy/")
    kind_cluster.kubectl("rollout", "status", "deployment/kopfexample-operator")
    kind_cluster.kubectl("apply", "-f", "example.yaml")

    for i in range(10):
        obj = HelloWorld.objects(kind_cluster.api).get(name="test-1")
        if "status" in obj.obj:
            break
        time.sleep(2)

    assert obj.obj["status"]["on_create"]["message"] == "Hello Prague!"

Even my recently introduced frontend project Kubernetes Web View (slide 47+) is easy to test as it relies on plain HTML rendering (using Jinja2) and therefore does not require sophisticated browser-based testing (slide 67).

Kubernetes + Python = ❤

It's pretty clear that I like Kubernetes and Python, a quick look at my open source projects is enough:

• kube-downscaler: uses pykube-ng to downscale deployments after working hours

• kube-janitor: uses pykube-ng to clean up resources with a given TTL

• kube-ops-view: uses pykube-ng to query the Kubernetes API and render a WebGL view

• kube-resource-report: uses pykube-ng to collect resource requests/usage and render HTML pages with Jinja2

• kube-web-view: uses pykube-ng and AIOHTTP to provide a "kubectl for the web" experience

• pykube-ng: a lightweight Python client for Kubernetes

• pytest-kind: pytest plugin to test Python apps/operators with KIND

I feel rather productive with my personal Python+Kubernetes setup, what are your experiences?

Do you write any Kubernetes applications/controllers/operators in Python? Do you use any of the mentioned open source projects? Let me know on Twitter!