|
| 1 | +<!-- |
| 2 | + Licensed to the Apache Software Foundation (ASF) under one or more |
| 3 | + contributor license agreements. See the NOTICE file distributed with |
| 4 | + this work for additional information regarding copyright ownership. |
| 5 | + The ASF licenses this file to You under the Apache License, Version 2.0 |
| 6 | + (the "License"); you may not use this file except in compliance with |
| 7 | + the License. You may obtain a copy of the License at |
| 8 | +
|
| 9 | + http://www.apache.org/licenses/LICENSE-2.0 |
| 10 | +
|
| 11 | + Unless required by applicable law or agreed to in writing, software |
| 12 | + distributed under the License is distributed on an "AS IS" BASIS, |
| 13 | + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 14 | + See the License for the specific language governing permissions and |
| 15 | + limitations under the License. |
| 16 | +--> |
| 17 | + |
| 18 | +# Apache Ozone Quick Start |
| 19 | + |
| 20 | +Get started with Apache Ozone in 5 minutes using a single-container setup with the AWS S3 CLI. |
| 21 | + |
| 22 | +## What is the Quickstart Image? |
| 23 | + |
| 24 | +The quickstart image is a single-container version of Apache Ozone that runs all services (SCM, OM, DataNode, S3 Gateway, Recon, HttpFS) in one container. It's designed for: |
| 25 | + |
| 26 | +- Quick development and testing |
| 27 | +- Learning Ozone functionality |
| 28 | +- Demos and POCs |
| 29 | +- Local S3-compatible storage |
| 30 | + |
| 31 | +> **Note**: For production deployments, use the [multi-container setup](docker-compose.yaml) with separate containers for each service. |
| 32 | +
|
| 33 | +## Prerequisites |
| 34 | + |
| 35 | +- Docker installed |
| 36 | +- AWS CLI installed (optional, for S3 testing) |
| 37 | + - macOS: `brew install awscli` |
| 38 | + - pip: `pip install awscli` |
| 39 | + |
| 40 | +## Quick Start |
| 41 | + |
| 42 | +### Step 1: Start Ozone |
| 43 | + |
| 44 | +```bash |
| 45 | +docker run -d \ |
| 46 | + --name ozone \ |
| 47 | + -p 9876:9876 \ |
| 48 | + -p 9874:9874 \ |
| 49 | + -p 19864:19864 \ |
| 50 | + -p 9878:9878 \ |
| 51 | + -p 9888:9888 \ |
| 52 | + -p 14000:14000 \ |
| 53 | + -v ozone-metadata:/data/metadata \ |
| 54 | + -v ozone-hdds:/data/hdds \ |
| 55 | + -v ozone-logs:/var/log/hadoop \ |
| 56 | + apache/ozone-quickstart:latest |
| 57 | + |
| 58 | +# Wait 10-30 seconds for services to start |
| 59 | +docker logs -f ozone |
| 60 | +``` |
| 61 | + |
| 62 | +### Step 2: Configure AWS CLI |
| 63 | + |
| 64 | +Set up AWS CLI to point to Ozone's S3 Gateway: |
| 65 | + |
| 66 | +```bash |
| 67 | +# Configure credentials (use any value for local testing) |
| 68 | +aws configure set aws_access_key_id ozone |
| 69 | +aws configure set aws_secret_access_key ozone |
| 70 | +aws configure set default.s3.signature_version s3v4 |
| 71 | +``` |
| 72 | + |
| 73 | +**Note**: For this single-node cluster, authentication is minimal. Use any credentials. |
| 74 | + |
| 75 | +### Step 3: Create a Bucket |
| 76 | + |
| 77 | +```bash |
| 78 | +# Create bucket |
| 79 | +aws s3api create-bucket \ |
| 80 | + --bucket mybucket \ |
| 81 | + --endpoint-url http://localhost:9878 |
| 82 | + |
| 83 | +# List all buckets |
| 84 | +aws s3 ls --endpoint-url http://localhost:9878 |
| 85 | +``` |
| 86 | + |
| 87 | +### Step 4: Upload and Download Files |
| 88 | + |
| 89 | +```bash |
| 90 | +# Create a test file |
| 91 | +echo "Hello, Ozone!" > test.txt |
| 92 | + |
| 93 | +# Upload file |
| 94 | +aws s3 cp test.txt s3://mybucket/mykey.txt \ |
| 95 | + --endpoint-url http://localhost:9878 |
| 96 | + |
| 97 | +# List keys in bucket |
| 98 | +aws s3 ls s3://mybucket/ --endpoint-url http://localhost:9878 |
| 99 | + |
| 100 | +# Download file |
| 101 | +aws s3 cp s3://mybucket/mykey.txt downloaded.txt \ |
| 102 | + --endpoint-url http://localhost:9878 |
| 103 | + |
| 104 | +# Verify content |
| 105 | +cat downloaded.txt |
| 106 | +``` |
| 107 | + |
| 108 | +## Common Operations |
| 109 | + |
| 110 | +### Upload Directory |
| 111 | +```bash |
| 112 | +aws s3 sync ./local-folder/ s3://mybucket/remote-folder/ \ |
| 113 | + --endpoint-url http://localhost:9878 |
| 114 | +``` |
| 115 | + |
| 116 | +### Download Directory |
| 117 | +```bash |
| 118 | +aws s3 sync s3://mybucket/remote-folder/ ./local-folder/ \ |
| 119 | + --endpoint-url http://localhost:9878 |
| 120 | +``` |
| 121 | + |
| 122 | +### Delete Objects |
| 123 | +```bash |
| 124 | +aws s3 rm s3://mybucket/ --recursive --endpoint-url http://localhost:9878 |
| 125 | +``` |
| 126 | + |
| 127 | +### Copy Between Buckets |
| 128 | +```bash |
| 129 | +aws s3 cp s3://source-bucket/file.txt s3://dest-bucket/file.txt \ |
| 130 | + --endpoint-url http://localhost:9878 |
| 131 | +``` |
| 132 | + |
| 133 | +## Using Ozone Native CLI |
| 134 | + |
| 135 | +If you prefer Ozone's native CLI instead of S3: |
| 136 | + |
| 137 | +```bash |
| 138 | +# Enter the container |
| 139 | +docker exec -it ozone bash |
| 140 | + |
| 141 | +# Create a volume |
| 142 | +ozone sh volume create /vol1 |
| 143 | + |
| 144 | +# Create a bucket |
| 145 | +ozone sh bucket create /vol1/bucket1 |
| 146 | + |
| 147 | +# Upload key |
| 148 | +echo "Hello Ozone" > /tmp/test.txt |
| 149 | +ozone sh key put /vol1/bucket1/key1 /tmp/test.txt |
| 150 | + |
| 151 | +# Download key |
| 152 | +ozone sh key get /vol1/bucket1/key1 /tmp/test-out.txt |
| 153 | + |
| 154 | +# List keys |
| 155 | +ozone sh key list /vol1/bucket1/ |
| 156 | +``` |
| 157 | + |
| 158 | +## Web UIs |
| 159 | + |
| 160 | +Access Ozone's web interfaces: |
| 161 | + |
| 162 | +- **Recon Dashboard**: http://localhost:9888 - Visual overview of volumes, buckets, and keys |
| 163 | +- **S3 Gateway**: http://localhost:9878 |
| 164 | +- **SCM Web UI**: http://localhost:9876 |
| 165 | +- **OM Web UI**: http://localhost:9874 |
| 166 | +- **DataNode Web UI**: http://localhost:19864 |
| 167 | +- **HttpFS API**: http://localhost:14000 |
| 168 | + |
| 169 | +## Managing the Container |
| 170 | + |
| 171 | +### Stop the Container |
| 172 | +```bash |
| 173 | +docker stop ozone |
| 174 | +``` |
| 175 | + |
| 176 | +### Restart the Container |
| 177 | +```bash |
| 178 | +docker start ozone |
| 179 | +``` |
| 180 | + |
| 181 | +### Remove the Container |
| 182 | +```bash |
| 183 | +docker rm ozone |
| 184 | +``` |
| 185 | + |
| 186 | +### Remove Volumes (Start Fresh) |
| 187 | +```bash |
| 188 | +docker volume rm ozone-metadata ozone-hdds ozone-logs |
| 189 | +``` |
| 190 | + |
| 191 | +### Check Logs |
| 192 | +```bash |
| 193 | +# All services |
| 194 | +docker logs ozone |
| 195 | + |
| 196 | +# Specific service |
| 197 | +docker exec ozone tail -f /var/log/hadoop/scm.log |
| 198 | +docker exec ozone tail -f /var/log/hadoop/om.log |
| 199 | +docker exec ozone tail -f /var/log/hadoop/datanode.log |
| 200 | +docker exec ozone tail -f /var/log/hadoop/s3g.log |
| 201 | +docker exec ozone tail -f /var/log/hadoop/recon.log |
| 202 | +docker exec ozone tail -f /var/log/hadoop/httpfs.log |
| 203 | +``` |
| 204 | + |
| 205 | +## Troubleshooting |
| 206 | + |
| 207 | +### Container Exits Immediately |
| 208 | + |
| 209 | +Check the logs: |
| 210 | +```bash |
| 211 | +docker logs ozone |
| 212 | +``` |
| 213 | + |
| 214 | +Ensure you have allocated enough resources to Docker (at least 4GB RAM recommended). |
| 215 | + |
| 216 | +### SCM Not Exiting Safe Mode |
| 217 | + |
| 218 | +Wait 10-30 seconds after startup. Check status: |
| 219 | +```bash |
| 220 | +docker exec ozone ozone admin safemode status --verbose |
| 221 | +``` |
| 222 | + |
| 223 | +### Permission Errors |
| 224 | + |
| 225 | +If using host-mounted directories instead of Docker volumes, ensure proper permissions: |
| 226 | +```bash |
| 227 | +mkdir -p ./ozone-metadata ./ozone-hdds ./ozone-logs |
| 228 | +chmod -R 777 ./ozone-metadata ./ozone-hdds ./ozone-logs |
| 229 | +``` |
| 230 | + |
| 231 | +### Reset and Start Fresh |
| 232 | + |
| 233 | +```bash |
| 234 | +# Stop and remove container |
| 235 | +docker stop ozone |
| 236 | +docker rm ozone |
| 237 | + |
| 238 | +# Remove all volumes |
| 239 | +docker volume rm ozone-metadata ozone-hdds ozone-logs |
| 240 | + |
| 241 | +# Start fresh |
| 242 | +docker run -d \ |
| 243 | + --name ozone \ |
| 244 | + -p 9876:9876 -p 9874:9874 -p 19864:19864 \ |
| 245 | + -p 9878:9878 -p 9888:9888 -p 14000:14000 \ |
| 246 | + -v ozone-metadata:/data/metadata \ |
| 247 | + -v ozone-hdds:/data/hdds \ |
| 248 | + -v ozone-logs:/var/log/hadoop \ |
| 249 | + apache/ozone-quickstart:latest |
| 250 | +``` |
| 251 | + |
| 252 | +## Runtime Configuration |
| 253 | + |
| 254 | +Override configuration at runtime using environment variables: |
| 255 | + |
| 256 | +```bash |
| 257 | +docker run -d \ |
| 258 | + --name ozone \ |
| 259 | + -p 9876:9876 -p 9874:9874 -p 19864:19864 \ |
| 260 | + -p 9878:9878 -p 9888:9888 -p 14000:14000 \ |
| 261 | + -e OZONE-SITE.XML_ozone.server.default.replication=3 \ |
| 262 | + -e OZONE-SITE.XML_hdds.scm.safemode.min.datanode=1 \ |
| 263 | + -v ozone-metadata:/data/metadata \ |
| 264 | + -v ozone-hdds:/data/hdds \ |
| 265 | + -v ozone-logs:/var/log/hadoop \ |
| 266 | + apache/ozone:dev-quickstart |
| 267 | +``` |
| 268 | + |
| 269 | +## Next Steps |
| 270 | + |
| 271 | +- Visit [Apache Ozone documentation](https://ozone.apache.org/docs/) |
| 272 | + |
| 273 | +--- |
| 274 | + |
| 275 | +**That's it!** You now have a running Ozone cluster with S3-compatible storage. 🎉 |
| 276 | + |
0 commit comments