keep_template does not seems to be working when using k8s backend

[aagarwal-amit]

I am creating a session like this -

with SandboxSession( backend=SandboxBackend.KUBERNETES, lang="python", verbose=True, keep_template=True, pod_manifest=generate_pod_manifest(podname, namespace=namespace), client=K8sClient(), workdir=working_dir ) as session:

still it deletes the pod while exiting. Is this the expected behavior?

[vndee]

Hey @aagarwal-amit, for the current implementation, this is the expected behavior. keep_template is used to retain only the images in order to reduce image building time. If you want to minimize pod/container startup time, you could try opening only one session for your recurring sandbox runs. I will think about adding an option to just stop containers rather than stop and remove them.

[avivkarbolk]

@vndee I’m wondering about an approach for improving both startup latency and security isolation when running code with llm-sandbox on Kubernetes. Specifically, what do you think about using a Deployment to maintain a pool of always-running pods (pre-warmed), and then having llm-sandbox connect to one of these pods for a single code execution, after which the pod is deleted, This way, the Deployment’s replica set would automatically create a new pod to maintain the desired pool size, ensuring that each run still happens in a fresh environment.
Do you see any challenges or drawbacks with this pattern, or suggestions for integrating it more smoothly with llm-sandbox? I’d love to hear your thoughts

[vndee]

@avivkarbolk Great question! I actually recently added a new feature that enables exactly what you're describing - the ability to connect to existing containers/pods. This makes it much easier to build a pre-warmed pod pool management system for llm-sandbox.

New Feature: Existing Container/Pod Support

You can now connect to existing running pods using the container_id parameter (which works for both Docker containers and Kubernetes pods):

from llm_sandbox import SandboxSession, SandboxBackend

# Connect to an existing pod by name/ID
with SandboxSession(
    backend=SandboxBackend.KUBERNETES,
    container_id="your-pod-name-here", 
    lang="python",
    client=your_k8s_client
) as session:
    result = session.run("print('Hello from existing pod!')")
    # Pod stays running after session ends

Pre-warmed Pod Pool Implementation

Here's how you can implement the exact pattern you described:

import time
from kubernetes import client, config
from llm_sandbox import SandboxSession, SandboxBackend

class PodPoolManager:
    def __init__(self, namespace="default", pool_size=3):
        config.load_kube_config()  # or load_incluster_config()
        self.k8s_client = client.CoreV1Api()
        self.namespace = namespace
        self.pool_size = pool_size
        self.deployment_name = "llm-sandbox-pool"
        
    def setup_pool_deployment(self):
        """Create a Deployment to maintain pre-warmed pods"""
        deployment_manifest = {
            "apiVersion": "apps/v1",
            "kind": "Deployment", 
            "metadata": {"name": self.deployment_name, "namespace": self.namespace},
            "spec": {
                "replicas": self.pool_size,
                "selector": {"matchLabels": {"app": "llm-sandbox-pool"}},
                "template": {
                    "metadata": {"labels": {"app": "llm-sandbox-pool"}},
                    "spec": {
                        "containers": [{
                            "name": "sandbox",
                            "image": "ghcr.io/vndee/sandbox-python-311-bullseye",
                            "command": ["tail", "-f", "/dev/null"],  # Keep running
                            "resources": {
                                "requests": {"cpu": "100m", "memory": "256Mi"},
                                "limits": {"cpu": "1", "memory": "1Gi"}
                            }
                        }]
                    }
                }
            }
        }
        
        apps_v1 = client.AppsV1Api()
        apps_v1.create_namespaced_deployment(
            namespace=self.namespace, 
            body=deployment_manifest
        )
        
    def get_available_pod(self):
        """Get an available pod from the pool"""
        pods = self.k8s_client.list_namespaced_pod(
            namespace=self.namespace,
            label_selector="app=llm-sandbox-pool"
        )
        
        for pod in pods.items:
            if pod.status.phase == "Running":
                return pod.metadata.name
        
        # If no pods available, wait for one to be ready
        print("No available pods, waiting...")
        time.sleep(2)
        return self.get_available_pod()
    
    def execute_code(self, code, libraries=None):
        """Execute code in a fresh pod from the pool"""
        pod_name = self.get_available_pod()
        
        try:
            # Connect to existing pod
            with SandboxSession(
                backend=SandboxBackend.KUBERNETES,
                container_id=pod_name,
                lang="python",
                client=self.k8s_client
            ) as session:
                result = session.run(code, libraries=libraries)
                return result
                
        finally:
            # Delete the used pod - Deployment will create a new one
            self.k8s_client.delete_namespaced_pod(
                name=pod_name, 
                namespace=self.namespace
            )

# Usage example
pool = PodPoolManager(pool_size=5)
pool.setup_pool_deployment()

# Execute code with instant startup + fresh isolation
result = pool.execute_code("""
import pandas as pd
print("Fast execution with fresh pod!")
""", libraries=["pandas"])

Benefits of This Approach

• Fast startup: Pods are pre-warmed and ready
• Fresh isolation: Each execution gets a clean pod
• Auto-scaling: Deployment maintains desired pool size
• Resource efficiency: Only pay for running pods, not creating them

Future Integration Considerations

I'm definitely thinking about adding this functionality directly into llm-sandbox, but there are some architectural design considerations. Your use case is a perfect validation of this pattern! The existing container support gives you all the building blocks to implement this today, and your feedback will help shape how we might integrate it natively.

Would love to hear how this works for your setup and any other requirements you discover!

[CoreJa]

@vndee Hi, any new update on your integration of this feature directly into llm-sandbox? It is a great feature, and it would just greater if it comes out of the box by via just a parameter or likewise. BTW I think your code above is good enough.

[vndee]

Hey @CoreJa, I'll work on this feature this weekend.

[vndee]

Hey @CoreJa, I have just released it in version 0.3.26 with this PR #115
https://vndee.github.io/llm-sandbox/getting-started/#performance-optimization-with-container-pooling

[CoreJa]

@vndee This is great! I'll give it a try! Thank you for the support!

[CoreJa]

@vndee I had a little try out there, but it wasn't behaving as I expected.(e.g. namespace was using default, rather than a explicted namespace; session using that pool was looping;)

I guess I might have mis-configured something(pod_manifest was using pod instead of deployment, tests are run in a jupyter kernel) I'll have a deeper look into your code and figure out what the correct config is.