Build-packs Implementation

[ehsaniara]

Basic Structure

Every job now requires a buildpack, but all existing resource management and scheduling features remain:

rnx run --buildpack=<buildpack> [resource-flags] [schedule-flags] [upload-flags] <command> [args...]

Resource Management (Unchanged)

CPU and Core Management

# CPU percentage limiting with buildpack
rnx run --buildpack=base-system --max-cpu=50 stress --cpu 4

# CPU core binding with buildpack  
rnx run --buildpack=java-17 --cpu-cores="0-3" java -jar app.jar

# Single core allocation
rnx run --buildpack=python-3.11 --cpu-cores="2" python compute.py

# Complex core patterns
rnx run --buildpack=base-system --cpu-cores="1,3,5,7" simulation_app
rnx run --buildpack=node-18 --cpu-cores="0-1,4-5" node server.js

# Combine CPU percentage + core binding
rnx run --buildpack=java-17 --cpu-cores="2-3" --max-cpu=150 java -jar app.jar

Memory and I/O Limits

# Memory limiting with buildpack
rnx run --buildpack=python-3.11 --max-memory=1024 python data_processor.py

# I/O bandwidth limiting
rnx run --buildpack=base-system --max-iobps=10485760 dd if=/dev/zero of=/tmp/test bs=1M count=100

# All resource limits combined
rnx run --buildpack=java-17 \
    --max-cpu=75 \
    --cpu-cores="0-1" \
    --max-memory=2048 \
    --max-iobps=5242880 \
    java -Xmx1g -jar microservice.jar

Scheduling (Unchanged)

Immediate vs Scheduled Execution

# Immediate execution (default)
rnx run --buildpack=base-system echo "Hello World"

# Scheduled execution - relative time
rnx run --buildpack=python-3.11 --schedule="1hour" python batch_job.py
rnx run --buildpack=java-17 --schedule="30min" java -jar report_generator.jar
rnx run --buildpack=node-18 --schedule="2h30m" node cron_task.js

# Scheduled execution - absolute time
rnx run --buildpack=python-3.11 --schedule="2025-07-18T20:02:48" python nightly_backup.py
rnx run --buildpack=java-17 --schedule="2025-07-18T20:02:48Z" java -jar scheduled_task.jar

Scheduled Jobs with Resource Limits

# Schedule a resource-intensive job for later
rnx run --buildpack=python-3.11-data \
    --schedule="tonight at 2am" \
    --max-memory=8192 \
    --cpu-cores="0-7" \
    python ml_training.py

# Schedule Java microservice with specific resources
rnx run --buildpack=java-17-spring \
    --schedule="1hour" \
    --max-cpu=50 \
    --cpu-cores="2-3" \
    --max-memory=1024 \
    java -jar microservice.jar

File Uploads (Unchanged)

Single File Uploads

# Upload single file with buildpack
rnx run --buildpack=python-3.11 --upload=script.py python script.py
rnx run --buildpack=java-17 --upload=app.jar java -jar app.jar

# Multiple file uploads
rnx run --buildpack=python-3.11 \
    --upload=data.csv \
    --upload=process.py \
    python process.py

Directory Uploads

# Upload entire directory
rnx run --buildpack=node-18 --upload-dir=. npm start
rnx run --buildpack=python-3.11 --upload-dir=ml-project python train.py

# Upload with resource limits
rnx run --buildpack=java-17 \
    --upload-dir=./target \
    --max-memory=2048 \
    --cpu-cores="0-1" \
    java -jar target/app.jar

Upload + Scheduling Combined

# Schedule job with file uploads
rnx run --buildpack=python-3.11-data \
    --schedule="2hour" \
    --upload-dir=dataset \
    --max-memory=4096 \
    --cpu-cores="2-5" \
    python analyze_data.py

# Schedule with single file upload and limits
rnx run --buildpack=java-17 \
    --schedule="30min" \
    --upload=config.json \
    --max-cpu=25 \
    --cpu-cores="6" \
    java -jar -Dconfig=config.json app.jar

Complete Real-World Examples

Data Science Workload

# Large dataset processing - scheduled for off-hours
rnx run --buildpack=python-3.11-data \
    --schedule="today at 11pm" \
    --upload-dir=datasets \
    --max-memory=16384 \
    --cpu-cores="0-7" \
    --max-iobps=104857600 \
    python process_big_data.py

Java Microservice Deployment

# Production microservice with precise resource allocation
rnx run --buildpack=java-17-spring \
    --upload=microservice.jar \
    --upload=application.yml \
    --max-cpu=100 \
    --cpu-cores="2-3" \
    --max-memory=1024 \
    java -jar -Xmx512m microservice.jar

Machine Learning Training

# GPU-accelerated training scheduled for later
rnx run --buildpack=python-3.11-pytorch \
    --schedule="2hour" \
    --upload-dir=model \
    --upload-dir=data \
    --max-memory=8192 \
    --cpu-cores="0-15" \
    python train_model.py --epochs=100

CI/CD Pipeline Job

# Build and test job with time limits and resource constraints
rnx run --buildpack=node-18 \
    --upload-dir=. \
    --max-cpu=200 \
    --cpu-cores="4-7" \
    --max-memory=2048 \
    --max-iobps=52428800 \
    bash -c "npm install && npm test && npm run build"

High-Performance Computing

# Scientific simulation with dedicated cores
rnx run --buildpack=base-system \
    --upload=simulation.bin \
    --upload-dir=input_data \
    --cpu-cores="8-15" \
    --max-memory=32768 \
    ./simulation.bin --threads=8 --input=input_data/

Buildpack Bundle Examples

Bundle with All Features

# complex-workload-bundle.yml
apiVersion: joblet.io/v1
kind: JobBundle
metadata:
  name: "complex-ml-workload"
  
spec:
  buildpack:
    name: "python-3.11-pytorch"
    version: "2.0.0"
    
  # Same resource management as CLI flags
  resources:
    memory: "8192Mi"      # --max-memory=8192
    cpu: "400m"           # --max-cpu=400
    cores: "0-7"          # --cpu-cores="0-7"
    ioBPS: "104857600"    # --max-iobps=104857600
    
  # Scheduling support
  schedule:
    time: "2025-07-18T22:00:00"  # --schedule="2025-07-18T22:00:00"
    
  # File upload equivalent
  files:
    - "./model/"    # dest internally /work/model/
    - "./data/"       # dest internally /work/data/
    - "./train.py"   # dest internally /work/train.py
      
  application:
    entrypoint: ["python", "train.py"]
    args: ["--epochs=100", "--batch-size=32"]
    
  environment:
    CUDA_VISIBLE_DEVICES: "0"
    PYTORCH_CUDA_ALLOC_CONF: "max_split_size_mb:512"

Using Bundle with Additional Overrides

# Use bundle but override some settings
rnx run --buildpack-bundle=complex-workload-bundle.yml \
    --schedule="1hour" \           # Override bundle schedule
    --max-memory=16384 \           # Override bundle memory
    --upload=additional_data.csv   # Add extra files

# Immediate execution overriding scheduled bundle
rnx run --buildpack-bundle=complex-workload-bundle.yml \
    --schedule="" \                # Override to immediate execution
    --cpu-cores="0-3"              # Override to fewer cores

Buildpack Management Commands

Basic Buildpack Operations

# List available buildpacks
rnx buildpack list

# Install new buildpack
rnx buildpack install java-17:1.0.0
rnx buildpack install python-3.11-data:2.1.0 --from=registry.company.com

# Show buildpack details
rnx buildpack info java-17:1.0.0

# Remove unused buildpacks
rnx buildpack prune

Status and Monitoring (Unchanged)

# List all jobs (same as before)
rnx list

# Get job status with buildpack info
rnx status <job-id>

# Stream logs (unchanged)
rnx log <job-id>

# Stop running job (unchanged)
rnx stop <job-id>

Migration Examples

Before (Old Joblet)

# ❌ These won't work anymore
rnx run echo "hello"
rnx run --max-memory=1024 python script.py
rnx run --schedule="1hour" --upload=data.csv python process.py

After (Buildpack-First)

# ✅ Same functionality, explicit buildpack
rnx run --buildpack=base-system echo "hello"
rnx run --buildpack=python-3.11 --max-memory=1024 python script.py
rnx run --buildpack=python-3.11 --schedule="1hour" --upload=data.csv python process.py

# ✅ Or use convenience alias
rnx run --base echo "hello"  # --base is alias for --buildpack=base-system

Configuration Changes

Auto-install Essential Buildpacks

# /opt/joblet/config/joblet-config.yml
buildpacks:
  cacheDir: "/opt/joblet/buildpack-cache"
  autoInstall:
    - "base-system:1.0.0"      # Essential for basic commands
    - "java-17:1.0.0"          # Common runtime
    - "python-3.11:1.0.0"      # Common runtime
    - "node-18:1.0.0"          # Common runtime
  repositories:
    - name: "default"
      url: "https://buildpacks.company.com"

Summary

What Changes:

• All jobs must specify --buildpack=<name> or use --buildpack-bundle=<file>
• Buildpacks replace the old host bind mount system

What Stays the Same:

• All resource flags: --max-cpu, --max-memory, --max-iobps, --cpu-cores
• All scheduling: --schedule with relative/absolute time formats
• All file uploads: --upload and --upload-dir
• All monitoring: rnx list, rnx status, rnx log, rnx stop
• All cgroup and namespace isolation
• All security and process isolation features

Better Developer Experience:

• Predictable, versioned environments
• Better reproducibility across different hosts
• Clearer separation between environment (buildpack) and resources (flags)
• Same powerful resource management with more reliable execution environments