Skip to content

fix s3 setup #252

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ehsaniara merged 1 commit into from
Jan 6, 2026
Merged

fix s3 setup #252

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 75 additions & 19 deletions docs/AWS_DEPLOYMENT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,14 +45,16 @@ This interactive script creates:
- Local: DynamoDB + Secrets Manager only
- `joblet-jobs` DynamoDB table for job state persistence
- **DynamoDB VPC Endpoint** (required) for secure access to DynamoDB
- **S3 VPC Endpoint** (when using S3 storage) with bucket-specific policy
- **CA and client certificates** in AWS Secrets Manager (for horizontal scaling)

The script will:

1. Ask you to select a VPC for the EC2 instance
2. Check if a DynamoDB VPC Endpoint exists in that VPC
3. Let you select an existing endpoint or create a new one
4. Generate and store shared CA/client certificates in Secrets Manager
4. If using S3 storage: Configure S3 VPC Endpoint with bucket-specific policy
5. Generate and store shared CA/client certificates in Secrets Manager

**Note the VPC ID** shown at the end - you'll need it in Step 2.

Expand Down Expand Up @@ -103,7 +105,7 @@ The script will:

**Features**: Low cost, unlimited storage, lifecycle policies for archival

**Additional options**: `--s3-prefix=PREFIX` (default: joblet), `--s3-class=CLASS` (default: STANDARD)
**Additional options**: `--s3-prefix=PREFIX` (default: jobs/), `--s3-class=CLASS` (default: STANDARD)
</details>

<details>
Expand Down Expand Up @@ -250,6 +252,17 @@ ssh ubuntu@${PUBLIC_IP} "ls -la /opt/joblet/logs/"
- **No NAT Gateway required** for DynamoDB access
- **Created automatically** by the pre-setup script if not exists

### VPC Endpoint (S3 Gateway) - For S3 Storage

When using S3 storage backend, the pre-setup script also configures:

- **S3 VPC Endpoint** for EC2 instances without internet access
- **Bucket-specific policy** allowing only your Joblet S3 bucket
- **Actions allowed**: PutObject, GetObject, DeleteObject, ListBucket
- **Created/updated automatically** by `pre-setup.sh --storage=s3`

**Note**: If the S3 VPC Endpoint policy doesn't allow access to your bucket, you'll see `AccessDenied` errors in the persist service logs. The pre-setup script updates the endpoint policy automatically to allow access to the specified bucket.

### Secrets Manager (TLS Certificates)

- **Shared CA certificate** (`joblet/ca-cert`, `joblet/ca-key`) - Same across all instances
Expand Down Expand Up @@ -294,7 +307,7 @@ The install script supports command-line options for cleaner User Data:
Options:
--storage=TYPE cloudwatch (default), s3, or local
--s3-bucket=NAME S3 bucket name (required for s3)
--s3-prefix=PREFIX S3 key prefix (default: joblet)
--s3-prefix=PREFIX S3 key prefix (default: jobs/)
--s3-class=CLASS STANDARD, STANDARD_IA, GLACIER, etc.
--version=VERSION Joblet version (default: latest)
--port=PORT Server port (default: 443)
Expand Down Expand Up @@ -607,6 +620,46 @@ cat /opt/joblet/config/joblet-config.yml | grep -A 5 "persist:"
aws logs describe-log-groups --log-group-name-prefix /joblet
```

### S3 Storage Access Denied

If you see errors like:
```
AccessDenied: User is not authorized to perform: s3:PutObject on resource
because no VPC endpoint policy allows the s3:PutObject action
```

**Cause**: The S3 VPC Endpoint policy doesn't allow access to your bucket.

**Fix 1**: Re-run pre-setup with your bucket name:
```bash
./pre-setup.sh --storage=s3 --s3-bucket=your-bucket-name
```

**Fix 2**: Manually update the S3 VPC Endpoint policy:
```bash
# Find the S3 VPC Endpoint ID
aws ec2 describe-vpc-endpoints --filters "Name=service-name,Values=com.amazonaws.REGION.s3" \
--query 'VpcEndpoints[*].[VpcEndpointId,VpcId]' --output table

# Update the policy
aws ec2 modify-vpc-endpoint --vpc-endpoint-id vpce-xxxx \
--policy-document '{
"Version":"2012-10-17",
"Statement":[{
"Effect":"Allow",
"Principal":"*",
"Action":"s3:*",
"Resource":["arn:aws:s3:::your-bucket","arn:aws:s3:::your-bucket/*"]
}]
}'
```

**Fix 3**: Via AWS Console:
1. Go to **VPC → Endpoints**
2. Find the S3 Gateway endpoint for your VPC
3. Click **Actions → Manage policy**
4. Update the policy to allow access to your bucket

### Client Cannot Connect

**Check security group:**
Expand Down Expand Up @@ -668,22 +721,25 @@ cat /opt/joblet/config/joblet-config.yml | grep -A 20 "certificates:"
│ │ │ │ │ │ │
│ │ └────────────────────┼───────────────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌───────────────────────┐ │ │
│ │ │ VPC Endpoint │ │ │
│ │ │ (DynamoDB Gateway) │ │ │
│ │ └───────────┬───────────┘ │ │
│ │ │ │ │
│ └───────────────────────┼──────────────────────────────────────┘ │
│ │ │
│ ┌─────────────┴─────────────┐ │
│ ▼ ▼ │
│ ┌─────────────────┐ ┌──────────────────────┐ │
│ │ CloudWatch Logs │ │ DynamoDB │ │
│ │ OR S3 Bucket │ │ │ │
│ │ /joblet/... │ │ Table: joblet-jobs │ │
│ │ (job logs) │ │ (job state) │ │
│ └─────────────────┘ └──────────────────────┘ │
│ │ ┌───────────┴───────────┐ │ │
│ │ ▼ ▼ │ │
│ │ ┌───────────────────┐ ┌───────────────────┐ │ │
│ │ │ VPC Endpoint │ │ VPC Endpoint │ │ │
│ │ │ (DynamoDB Gateway)│ │ (S3 Gateway)* │ │ │
│ │ └─────────┬─────────┘ └─────────┬─────────┘ │ │
│ │ │ │ │ │
│ └────────────┼──────────────────────┼─────────────────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────────────┐ ┌─────────────────┐ │
│ │ DynamoDB │ │ CloudWatch Logs │ │
│ │ │ │ OR S3 Bucket │ │
│ │ Table: joblet-jobs │ │ /joblet/... │ │
│ │ (job state) │ │ (job logs) │ │
│ └──────────────────────┘ └─────────────────┘ │
│ │
│ * S3 VPC Endpoint created when using --storage=s3 │
│ │
└───────────────────────────────────────────────────────────────────┘
↑
Expand All @@ -701,7 +757,7 @@ cat /opt/joblet/config/joblet-config.yml | grep -A 20 "certificates:"
2. **Joblet → VPC Endpoint → DynamoDB**: Job state persistence (private, no internet)
3. **Joblet → CloudWatch OR S3**: Log/metrics storage
- CloudWatch: Real-time log streaming (PutLogEvents)
- S3: Batch uploads as gzipped JSONL files (PutObject)
- S3: Batch uploads via VPC Endpoint as gzipped JSONL files (PutObject)
4. **Client ← CloudWatch/S3**: Historical log queries via `rnx job log`

---
Expand Down
151 changes: 150 additions & 1 deletion scripts/aws/pre-setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,9 @@ echo " • IAM Role: JobletEC2Role"
echo " • Instance Profile: JobletEC2Role"
echo " • DynamoDB Table: joblet-jobs"
echo " • VPC Endpoint for DynamoDB (required)"
if [ "$STORAGE_BACKEND" = "s3" ]; then
echo " • VPC Endpoint for S3 (with bucket policy)"
fi
echo " • Secrets Manager: CA and client certificates (for horizontal scaling)"
echo ""
echo "Permissions granted:"
Expand Down Expand Up @@ -852,6 +855,141 @@ POLICY_EOF

rm -f /tmp/dynamodb-endpoint-policy.json
fi

# ============================================================================
# S3 VPC Endpoint Configuration (when using S3 storage)
# ============================================================================
if [ "$STORAGE_BACKEND" = "s3" ]; then
echo ""
echo "=========================================================================="
echo "S3 VPC Endpoint Configuration"
echo "=========================================================================="
echo ""
echo "Checking for S3 VPC Endpoint in $VPC_ID..."
echo "(Required for EC2 instances to access S3 without internet gateway)"
echo ""

# Check if S3 endpoint already exists in this VPC
S3_ENDPOINT_ID=$(aws ec2 describe-vpc-endpoints --region "$REGION" \
--filters "Name=vpc-id,Values=$VPC_ID" "Name=service-name,Values=com.amazonaws.$REGION.s3" \
--query 'VpcEndpoints[0].VpcEndpointId' --output text 2>/dev/null || echo "")

if [ -n "$S3_ENDPOINT_ID" ] && [ "$S3_ENDPOINT_ID" != "None" ]; then
echo "✅ Found existing S3 VPC Endpoint: $S3_ENDPOINT_ID"
S3_ENDPOINT_STATUS="existing"
else
echo "No S3 VPC Endpoint found. Creating one..."

# Get route tables for Gateway endpoint
if [ -z "$ROUTE_TABLE_IDS" ]; then
ROUTE_TABLE_IDS=$(aws ec2 describe-route-tables --region "$REGION" \
--filters "Name=vpc-id,Values=$VPC_ID" \
--query 'RouteTables[*].RouteTableId' --output text 2>/dev/null | tr '\t' ' ')
fi

if [ -z "$ROUTE_TABLE_IDS" ] || [ "$ROUTE_TABLE_IDS" = "None" ]; then
echo "❌ No route tables found for VPC $VPC_ID"
echo " Cannot create S3 VPC Endpoint."
else
echo " Route tables: $ROUTE_TABLE_IDS"

if S3_ENDPOINT_ID=$(aws ec2 create-vpc-endpoint --region "$REGION" \
--vpc-id "$VPC_ID" \
--service-name "com.amazonaws.$REGION.s3" \
--route-table-ids $ROUTE_TABLE_IDS \
--vpc-endpoint-type Gateway \
--tag-specifications "ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=joblet-s3-endpoint},{Key=ManagedBy,Value=Joblet}]" \
--query 'VpcEndpoint.VpcEndpointId' --output text 2>&1); then
echo "✅ S3 VPC Endpoint created: $S3_ENDPOINT_ID"
S3_ENDPOINT_STATUS="created"
elif echo "$S3_ENDPOINT_ID" | grep -q "RouteAlreadyExists"; then
echo "⚠️ S3 route already exists - endpoint may exist in different state"
S3_ENDPOINT_ID=$(aws ec2 describe-vpc-endpoints --region "$REGION" \
--filters "Name=vpc-id,Values=$VPC_ID" "Name=service-name,Values=com.amazonaws.$REGION.s3" \
--query 'VpcEndpoints[0].VpcEndpointId' --output text 2>/dev/null || echo "")
S3_ENDPOINT_STATUS="existing"
else
echo "❌ Failed to create S3 VPC Endpoint: $S3_ENDPOINT_ID"
S3_ENDPOINT_ID=""
fi
fi
fi

# Update S3 VPC Endpoint policy to allow access to the bucket
if [ -n "$S3_ENDPOINT_ID" ] && [ "$S3_ENDPOINT_ID" != "None" ]; then
echo ""
echo "Updating S3 VPC Endpoint policy for bucket: $S3_BUCKET..."

# Get AWS account ID
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text 2>/dev/null || echo "*")

# Create S3 endpoint policy that allows access to the joblet bucket
cat > /tmp/s3-endpoint-policy.json << S3_POLICY_EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowJobletS3Access",
"Effect": "Allow",
"Principal": "*",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::${S3_BUCKET}",
"arn:aws:s3:::${S3_BUCKET}/*"
]
},
{
"Sid": "AllowAllS3ForPackages",
"Effect": "Allow",
"Principal": "*",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"aws:PrincipalAccount": "${AWS_ACCOUNT_ID}"
}
}
}
]
}
S3_POLICY_EOF

if aws ec2 modify-vpc-endpoint --region "$REGION" \
--vpc-endpoint-id "$S3_ENDPOINT_ID" \
--policy-document file:///tmp/s3-endpoint-policy.json 2>/dev/null; then
echo "✅ S3 VPC Endpoint policy updated"
echo " - Bucket access: s3://$S3_BUCKET"
echo " - Actions allowed: PutObject, GetObject, DeleteObject, ListBucket"
else
echo "⚠️ Could not update S3 VPC Endpoint policy"
echo ""
echo " You may need to manually update the policy to allow S3 access."
echo " Run this command:"
echo ""
echo " aws ec2 modify-vpc-endpoint --region $REGION \\"
echo " --vpc-endpoint-id $S3_ENDPOINT_ID \\"
echo " --policy-document '{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"s3:*\",\"Resource\":[\"arn:aws:s3:::$S3_BUCKET\",\"arn:aws:s3:::$S3_BUCKET/*\"]}]}'"
echo ""
echo " Or via AWS Console: VPC → Endpoints → $S3_ENDPOINT_ID → Policy → Edit"
fi

rm -f /tmp/s3-endpoint-policy.json
else
echo ""
echo "⚠️ S3 VPC Endpoint not configured"
echo " EC2 instances may need internet access to reach S3."
echo " Alternatively, create an S3 Gateway endpoint manually."
fi
fi
fi

echo ""
Expand All @@ -872,11 +1010,22 @@ if [ -n "$VPC_ID" ]; then
else
echo " • VPC Endpoint: Configured (DynamoDB Gateway)"
fi
if [ "$STORAGE_BACKEND" = "s3" ]; then
if [ "$S3_ENDPOINT_STATUS" = "created" ]; then
echo " • VPC Endpoint: Created (S3 Gateway)"
elif [ -n "$S3_ENDPOINT_ID" ] && [ "$S3_ENDPOINT_ID" != "None" ]; then
echo " • VPC Endpoint: Configured (S3 Gateway)"
fi
fi
fi
echo ""
echo "Storage backend: $STORAGE_BACKEND"
if [ "$STORAGE_BACKEND" = "s3" ]; then
echo " S3 bucket: $S3_BUCKET (IAM permissions configured)"
echo " S3 bucket: $S3_BUCKET"
echo " IAM permissions: Configured"
if [ -n "$S3_ENDPOINT_ID" ] && [ "$S3_ENDPOINT_ID" != "None" ]; then
echo " VPC endpoint policy: Updated for bucket access"
fi
fi
echo ""
echo "=========================================================================="
Expand Down
Loading
Loading