Skip to content

Add central database service with InfluxDB for time series data storage#441

Open
Copilot wants to merge 5 commits intomainfrom
copilot/add-central-db-service
Open

Add central database service with InfluxDB for time series data storage#441
Copilot wants to merge 5 commits intomainfrom
copilot/add-central-db-service

Conversation

Copy link

Copilot AI commented Oct 26, 2025

Ground stations need persistent time series data storage for telemetry and mission data. This adds a configurable database service based on InfluxDB v2.

Implementation

Database Writer Service (backend-deno/samples/database/)

  • Subscribes to NATS subjects, writes JSON messages to InfluxDB
  • Measurement names derived from subject hierarchy (e.g., data.telemetry.temperaturetelemetry.temperature)
  • Auto-extracts timestamps, tags, and typed fields (float/string/boolean)

Database Query Service (backend-deno/samples/database-query/)

  • Request/reply pattern over NATS for querying
  • Parameter-based queries (measurement, range, fields, filters) or custom Flux queries
  • Queue-grouped for horizontal scaling
  • Uses simple NATS subject (e.g., query.database) - no wildcard pattern needed

Configuration

  • INFLUXDB_URL, INFLUXDB_TOKEN, INFLUXDB_ORG, INFLUXDB_BUCKET
  • DATA_SUBJECT - NATS subject pattern with wildcard for data (e.g., data.>)
  • QUERY_SUBJECT - NATS subject for queries (e.g., query.database)

Usage

// Write data
messageBus.publish("data.telemetry.temperature", JSON.stringify({
  time: new Date().toISOString(),
  value: 23.5,
  tags: { sensor: "temp-01" }
}));

// Query data
const response = await messageBus.request("query.database", JSON.stringify({
  measurement: "telemetry.temperature",
  range: "-1h",
  fields: ["value"]
}));

Deployment

Pre-built Docker Images (recommended)

  • ghcr.io/wuespace/telestion-database:latest - Database Writer Service
  • ghcr.io/wuespace/telestion-database-query:latest - Database Query Service
  • Published automatically on releases via GitHub Actions

Also Includes

  • Docker Compose configuration with InfluxDB 2.7
  • Kubernetes Deployment examples with published images
  • Production security guidelines (token management, TLS, retention policies)
  • Test client for validation (database-test-client/)
  • Build-from-source option for customization

Full deployment guide at docs/docs/Deployment/database-service.md.

  • Fixes wuespace/telestion#440
Original prompt

This section details on the original issue you should resolve

<issue_title>Central Database Service</issue_title>
<issue_description>Every ground station needs some database to store (mostly time series) data surrounding the mission.

Please

  1. implement a central DB service based on the Telestion framework to handle these requirements
  2. make it configurable such that it can be used in a multitude of projects
  3. add documentation around deploying it in a typical Telestion deployment</issue_description>

Comments on the Issue (you are @copilot in this section)

  • Fixes wuespace/telestion#440

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 3 commits October 26, 2025 16:57
- Implement database writer service to store time series data
- Implement database query service for data retrieval
- Add comprehensive documentation for both services
- Update docker-compose.yml with database services
- Add deployment guide for typical Telestion deployments

Co-authored-by: pklaschka <25147704+pklaschka@users.noreply.github.com>
- Create test client to verify database services work correctly
- Add database-service.md to deployment documentation navigation
- Test client demonstrates write and query workflows

Co-authored-by: pklaschka <25147704+pklaschka@users.noreply.github.com>
- Make INFLUXDB_BUCKET configurable in test client
- Update README with configuration option
- Address code review feedback

Co-authored-by: pklaschka <25147704+pklaschka@users.noreply.github.com>
Copilot AI changed the title [WIP] Implement central database service for ground stations Add central database service with InfluxDB for time series data storage Oct 26, 2025
Copilot AI requested a review from pklaschka October 26, 2025 17:06
@pklaschka
Copy link
Member

@copilot So far, so good. Two points I'd like changed:

  1. Currently, the address for the query service uses a wildcard. However, it doesn't look like this gets used. Please either specify how it could be used or remove it, since this is confusing.
  2. Make it so that the service Docker containers get published in the package registry and reflect that publication in the deployment docs (using the published containers), to make it easy to deploy

- Remove wildcard requirement from QUERY_SUBJECT since it's not used
- Change QUERY_SUBJECT to simple subject (e.g., query.database)
- Create individual Dockerfiles for database services
- Add GitHub Actions workflow to publish images to ghcr.io
- Update all documentation to use published images
- Add Kubernetes deployment examples with published images
- Keep development docker-compose.yml using build for testing

Co-authored-by: pklaschka <25147704+pklaschka@users.noreply.github.com>
Copy link
Author

Copilot AI commented Oct 26, 2025

@copilot So far, so good. Two points I'd like changed:

  1. Currently, the address for the query service uses a wildcard. However, it doesn't look like this gets used. Please either specify how it co...

Both issues addressed in commit 20ed43c:

  1. QUERY_SUBJECT wildcard removed: The wildcard wasn't used - the query service doesn't extract information from the subject hierarchy like the database writer does. Changed to simple subject (e.g., query.database instead of query.>).

  2. Docker images published:

    • Created individual Dockerfiles for both services
    • Added GitHub Actions workflow (.github/workflows/database-services.yml) to publish to ghcr.io on releases
    • Updated deployment docs to use published images: ghcr.io/wuespace/telestion-database:latest and ghcr.io/wuespace/telestion-database-query:latest
    • Added complete Kubernetes deployment examples with published images

@pklaschka pklaschka marked this pull request as ready for review October 26, 2025 17:46
@pklaschka pklaschka requested a review from fussel178 as a code owner October 26, 2025 17:46
@pklaschka pklaschka added 📕 documentation Improvements or additions to documentation 🌷 enhancement New feature or request labels Oct 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

📕 documentation Improvements or additions to documentation 🌷 enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants