Kubernetes Production Isolation

[JakobStadlhuber]

In addition to the described security mechanism in the llm-sandbox documentation, there are necessary mechanisms to have a secure environment when running workloads.

Disable all network access (reduce the risk of DDoS or connecting to a local db for example):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: isolate-sandbox
  namespace: llm-sandbox
spec:
  podSelector:
    matchLabels:
      app: sandbox-executor
  policyTypes:
  - Ingress
  - Egress
  ingress: []
  egress: []

Having a Service Account with Role and/ or ClusterRoles with Bindings to reduce what llm-sandbox can do with the Kubernetes API:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: llm-sandbox-sa-role
  namespace: llm-sandbox
rules:
  # Minimal permissions to run Pods and view logs in this namespace
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["pods/status"]
    verbs: ["get", "list", "watch"]
  # Allow exec/attach into pods (required for kubernetes exec websocket)
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]
  - apiGroups: [""]
    resources: ["pods/attach"]
    verbs: ["create"]

And one of the most important one is to protect the kernel with an issolation like gVisor or Kata

from kubernetes import client, config
from llm_sandbox import SandboxSession, SandboxBackend

# Load custom kubeconfig
config.load_kube_config(config_file="~/.kube/config")

# Or use in-cluster config
# config.load_incluster_config()

k8s_client = client.CoreV1Api()

# Pod manifest that ensures writable volumes for /sandbox and /tmp,
# keeps namespace name, runtimeClassName, and tolerations
pod_manifest = {
    "metadata": {
        "namespace": "llm-sandbox",
    },
    "spec": {
        "runtimeClassName": "kata-mshv-vm-isolation",
        "containers": [{
            "name": "sandbox",
            "image": "python:3.11-slim",
            "securityContext": {
                "runAsNonRoot": True,
                "runAsUser": 1000,
                "readOnlyRootFilesystem": True,
                "allowPrivilegeEscalation": False,
            },
            "volumeMounts": [
                {"name": "sandbox-writable", "mountPath": "/sandbox"},
                {"name": "tmp-writable", "mountPath": "/tmp"},
            ],
        }],
        "securityContext": {
            "runAsNonRoot": True,
            "fsGroup": 2000,  # Ensures volumes are writable by the group
        },
        "volumes": [
            {"name": "sandbox-writable", "emptyDir": {}},
            {"name": "tmp-writable", "emptyDir": {}},
        ],
        "tolerations": [{
            "key": "kata-only",
            "operator": "Equal",
            "value": "true",
            "effect": "NoSchedule",
        }],
    }
}

with SandboxSession(
    backend=SandboxBackend.KUBERNETES,
    client=k8s_client,
    lang="python",
    kube_namespace="llm-sandbox",
    pod_manifest=pod_manifest,
) as session:
    result = session.run("print('Hello from Kubernetes!')")
    print(result.stdout)

Do you have any other security or hardining mechanism or is this setup good? should we also document this in the project?