T3 Gemstone Package Server

What is it?

This project is created for two purposes:

• Automatizing every step of APT repo creation from DEB packages.
• Serving DEB packages for APT clients, operating system images and SDK files over HTTP.

To achieve those, three projects are relied upon: Docker, Aptly, and NGINX.

1. The Aptly project is used for creating, updating, signing, and obtaining snapshots of APT repositories. Interaction with it is allowed through CLI commands and REST API. A Dockerfile is contained within this project to create a Debian-based Aptly image. Inspiration was drawn from docker-aptly and it was customized for this project.

2. NGINX is used to protect Aptly's REST API, as well as to serve static files and APT repository files.

3. To deploy these two projects, Docker Compose is used. Required volumes are created and mounted and an isolated execution environment and network are created to facilitate communication between containers.

1. Install Docker

# This will install Docker and Taskfile to your system if they are not installed.
$ ./setup.sh

Note: If you are installing Docker for the first time, you must log out and log in again after the installation is complete.

2. Create Keys

Three keys are needed:

1. Public/private key pair for aptly to sign Release file
2. SSL certificate and private key for NGINX
3. API key for aptly's REST API

All keys can be generated by running scripts/gen_passwords.sh script. It will show some prompts to proceed. When all keys are successfully generated, it creates an encrypted tar archive which contains them. Then encrypted archive must be extracted to /etc/gemstone/keys directory in server.

3. Run Containers

All commands to manage containers can be found in Taskfile.yml file.

$ task run

Note: This project uses 80 and 443 ports. If you encounter port errors while running command above, that's probably because one of those ports are used by another process. Stop other processes or change port mapping from docker-compose.yml file. Don't forget to add port number at the end of IP address for commands below if you change port mappings.

Server setup is done after command above. All of remaining parts can be done using HTTP server.

Publishing DEB Files on the Server

All needed keys are located on ~/Desktop/user-keys folder. For <api-password> fields put whole text inside api.htpasswd.txt file. For GPG sign passphrase put whole text inside repo_gpg_pass.txt file. For security reasons, you can interact with the API only using HTTPS.

# Upload DEB Files to Server
$ python3 scripts/aptly_client.py --server 'https://<server-ip>/apt/api' --api-auth '<api-password>' upload --folder '<folder-containing-debs>'

# Create a New APT Repository
$ python3 scripts/aptly_client.py --server 'https://<server-ip>/apt/api' --api-auth '<api-password>' create-repo --name '<name-of-repo>' --distro '<distro (eg. jammy)>' --component '<component (eg. main)>'

# Add DEB Files to APT Repository (this may take long time based on how many debs you put)
$ python3 scripts/aptly_client.py --server 'https://<server-ip>/apt/api' --api-auth '<api-password>' add-debs --name '<name-of-repo>'

# Get Snapshot of APT Repository
$ python3 scripts/aptly_client.py --server 'https://<server-ip>/apt/api' --api-auth '<api-password>' snapshot --name '<name-of-snapshot>' --repo-name '<name-of-repo>'

# Publish (Sign) APT Repository
$ python3 scripts/aptly_client.py --server 'https://<server-ip>/apt/api' --api-auth '<api-password>' publish --snapshot-name '<name-of-snapshot>' --gpg-pass '<sign-key-passphrase>'

APT repo should be ready to access by now. Check it on https://localhost. Browser will show a warning about SSL key. This is normal because the SSL key generated in above steps was a self-signed one. So it can safely ignored. If apt/ folder is visible can be browsed then you can proceed to next step.

Note: Packages can be installed to server machine or a different machine. If server is accessed from server machine, put localhost to every field. If not, find server's IP address using ip addr command and put that one instead.

Copying Public Key Into Target Machine(s)

In first step, a key pair for signing APT repo got created and actually used in Publish (Sign) APT Repository section to sign the repo with private key. Now public key must be put into the machines which will access newly created APT server. This way, APT will ensure that downloaded packages from server are not compromised.

Copy public key from ~/Desktop/user-keys/repo_signing.gpg directory to target machine first. Then execute the command below.

# gemstone name can be changed
$ sudo install -m 0644 <copied-path>/repo_signing.gpg /etc/apt/keyrings/gemstone.gpg

Register Your Server to Target Machine(s)

In this step, packages will be attempted to be installed on target machines from APT server. To achieve this, a file needs to be added under the /etc/apt/sources.list.d directory. HTTP is used in the URLs below because using HTTPS with self-signed certificates can cause issues, such as hostname mismatches. If a certificate signed by a CA is provided, then HTTPS can be used.

# gemstone name can be changed
$ sudo nano /etc/apt/sources.list.d/gemstone.list

Edit and copy this into newly created file:

deb [signed-by=/etc/apt/keyrings/<your_repo_key.gpg>] http://<server-ip-address>/apt <distro> <component>

Replace the fields with <> characters. Here is an example:

deb [signed-by=/etc/apt/keyrings/gemstone.gpg] http://10.0.0.2/apt jammy main

Pull the package index with the command below. If it executes without errors, that means you successfully created your own APT server.

sudo apt update