Skip to content

add protobuf #167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
justin-phxm wants to merge 4 commits into
base: main
Choose a base branch
from
Open

add protobuf #167

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5c2643a
add protobuf
justin-phxm Nov 10, 2024
731f86d
update proto files
justin-phxm Jul 21, 2025
4b61ad2
fix build
justin-phxm Jul 21, 2025
e83f457
update package.json
justin-phxm Dec 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 18 additions & 12 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Solar Car Helios Telemetry

Deployment available at: https://telemetry-beta.calgarysolarcar.ca
Deployment available at: <https://telemetry-beta.calgarysolarcar.ca>

## Table of contents

Expand Down Expand Up @@ -67,13 +67,13 @@ Deployment available at: https://telemetry-beta.calgarysolarcar.ca

- [Introduction](./docs/SERVER.md#server-older-documentation)
- [Socket.IO](./docs/SERVER.md#wtf-is-a-socketio)
- [MQTT](#what-is-mqtt)
- [Express.js](#what-do-we-use-expressjs-for)
- [Routes](#what-is-a-route)
- [Controllers](#controllers)
- [Backend Controller](#backend-controller)
- [Lap Controller](#lap-controller)
- [DynamoDB Controller](#dynamo-db-controller)
- [MQTT](./docs/SERVER.md#what-is-mqtt)
- [Express.js](./docs/SERVER.md#what-do-we-use-expressjs-for)
- [Routes](./docs/SERVER.md#what-is-a-route)
- [Controllers](./docs/SERVER.md#controllers)
- [Backend Controller](./docs/SERVER.md#backend-controller)
- [Lap Controller](./docs/SERVER.md#lap-controller)
- [DynamoDB Controller](./docs/SERVER.md#dynamo-db-controller)

### [Shared Folder](./docs/SHARED.md)

Expand Down Expand Up @@ -102,28 +102,34 @@ Deployment available at: https://telemetry-beta.calgarysolarcar.ca

### Start Web App + Backend (Dev Build)

```
```bash
yarn dev
```
Comment on lines +105 to 107
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Docs: types:generate section should mention prerequisites (protoc, plugin install).
Right now it reads like yarn types:generate “just works”; in practice devs may need protoc installed (or a pinned toolchain). Consider adding a one-liner noting required installs / versions.

Also applies to: 113-116, 120-125

🤖 Prompt for AI Agents 
In README.md around lines 105-107 (and similarly update sections at 113-116 and
120-125), the docs for the types:generate step omit prerequisites; add a concise
note stating developers must have protoc installed (include minimum or pinned
version) and the required plugin(s) (e.g., protoc-gen-ts or
grpc_tools_node_protoc) or provide a pinned toolchain (homebrew/apt/npm package)
as an alternative, with a one-line example of install commands and the version
constraint so the command works reliably for contributors.


Appending `:client` or `:server` behind the dev command will start the servers individually

### Build Web App + Backend (Production build)

```
```bash
yarn build
yarn start
```

Appending `:client` or `:server` behind the build command will build/start the production servers individually

### Generate Protobuf Typescript Definitions

```bash
yarn types:generate
```

### Secrets and environment variables

Specific functions of the website require you to have certain environment variables set.

In packages/server `.env`:

```
```bash
LAP_POSITION_PASSWORD=
LAP_TABLE_NAME=
PACKET_TABLE_NAME=
Expand All @@ -135,7 +141,7 @@ MQTT_PASSWORD=

In packages/client `.env`:

```
```bash
NEXT_PUBLIC_MAPSAPIKEY=
```

Expand Down
3 changes: 2 additions & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@
"migrate:revert": "lerna run migrate:revert --scope=db",
"migrate:reset": "lerna run migrate:reset --scope=db",
"schema:sync": "lerna run schema:sync --scope=db",
"schema:drop": "lerna run schema:drop --scope=db"
"schema:drop": "lerna run schema:drop --scope=db",
"types:generate": "npx protoc --plugin=protoc-gen-ts_proto=\".\\node_modules\\.bin\\protoc-gen-ts_proto.cmd\" --ts_proto_out=\"./.\" ./packages/shared/src/protoTypes.proto"
},
"dependencies": {
"@mantine/dates": "^8.0.1",
Expand Down
1,613 changes: 0 additions & 1,613 deletions packages/client/src/components/molecules/PlaybackMolecules/fakedata.ts
View file
Open in desktop

This file was deleted.

12 changes: 6 additions & 6 deletions packages/client/src/hooks/PIS/PIS.motor.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,11 @@ const Motor = (): I_PIS => {
name: "Active Motor",
},
{
data: [{ value: MotorDetails0?.BEMF_D }],
data: [{ value: MotorDetails0?.bemfD }],
name: "BEMF D",
},
{
data: [{ value: MotorDetails0?.BEMF_Q }],
data: [{ value: MotorDetails0?.bemfQ }],
name: "BEMF Q",
},
{
Expand All @@ -61,7 +61,7 @@ const Motor = (): I_PIS => {
name: "Bus Voltage",
},
{
data: [{ value: MotorDetails0?.DC_Bus_Ah }],
data: [{ value: MotorDetails0?.DCBusAh }],
name: "DC Bus Ah",
},
{
Expand Down Expand Up @@ -221,11 +221,11 @@ const Motor = (): I_PIS => {
name: "Active Motor",
},
{
data: [{ value: MotorDetails1?.BEMF_D }],
data: [{ value: MotorDetails1?.bemfD }],
name: "BEMF D",
},
{
data: [{ value: MotorDetails1?.BEMF_Q }],
data: [{ value: MotorDetails1?.bemfQ }],
name: "BEMF Q",
},
{
Expand All @@ -237,7 +237,7 @@ const Motor = (): I_PIS => {
name: "Bus Voltage",
},
{
data: [{ value: MotorDetails1?.DC_Bus_Ah }],
data: [{ value: MotorDetails1?.DCBusAh }],
name: "DC Bus Ah",
},
{
Expand Down
1 change: 0 additions & 1 deletion packages/db/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
"scripts": {
"build": "tsc",
"start": "tsx src/index.ts",
"test": "jest --runInBand",
"db:up": "docker-compose up -d",
"db:down": "docker-compose down",
"db:logs": "docker-compose logs -f timescaledb",
Expand Down
12 changes: 4 additions & 8 deletions packages/server/src/datasources/SolarMQTTClient/SolarMQTTClient.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import {

import { createLightweightApplicationLogger } from "@/utils/logger";

import { validateTelemetryData } from "@shared/helios-types";
import { ITelemetryData } from "@shared/helios-types";

const {
carConnect,
Expand Down Expand Up @@ -101,14 +101,10 @@ export class SolarMQTTClient implements SolarMQTTClientType {
this.backendController.handleTelemetryToCar(serverToCarLatency);
break;
case packetTopic:
const packet = JSON.parse(message.toString());
const packet = JSON.parse(message.toString()) as ITelemetryData;
try {
const validPacket = validateTelemetryData(packet);
this.backendController.handlePacketReceive(validPacket);

if (validPacket.Pi.Rfid) {
this.latestRfid = validPacket.Pi.Rfid.toString();
}
this.backendController.handlePacketReceive(packet);
this.latestRfid = packet.Pi.Rfid.toString();
} catch (error) {
logger.error(`Invalid packet format: ${error.message}`);
}
Expand Down
3 changes: 3 additions & 0 deletions packages/shared/.eslintrc.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
const config = {
extends: ["../../.eslintrc.cjs"],
root: true,
rules: {
"no-console": "off", //TODO: Delete this...
},
};

module.exports = config;
12 changes: 8 additions & 4 deletions packages/shared/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,17 @@
"version": "0.1.0",
"main": "./index.ts",
"types": "./index.ts",
"scripts": {
"test": "tsx ./src/test.ts"
},
"dependencies": {
"@aws-sdk/client-secrets-manager": "^3.658.1",
"fp-ts": "^2.16.9",
"io-ts": "^2.2.22",
"typescript": "^5.5.3"
"@bufbuild/protobuf": "^2.6.1",
"protobufjs": "^7.4.0"
},
"devDependencies": {
"@faker-js/faker": "^9.0.3"
"@faker-js/faker": "^9.0.3",
"ts-proto": "^2.2.5",
"typescript": "^5.6.3"
}
Comment on lines 9 to 18
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

ts-proto v2.2.5 generated code default runtime protobufjs @bufbuild/protobuf

💡 Result:

Short answer: ts-proto v2.x (including v2.2.5) uses @bufbuild/protobuf as the default low‑level runtime (moving away from protobufjs). That means generated encode/decode now depend on @bufbuild/protobuf (e.g. BinaryReader/BinaryWriter from "@bufbuild/protobuf/wire"); if you relied on protobufjs internals you must update code or pin ts‑proto 1.x. [1][2][3]

Useful links:

  • ts-proto README (notes about 2.x migration to @bufbuild/protobuf). [1]
  • @bufbuild/protobuf package / docs. [2]
  • npm/summary mentioning the migration and migration advice to pin 1.x if needed. [3]

If you want, I can:

  • show concrete code diffs for migrating Reader/Writer usage to @bufbuild/protobuf, or
  • show how to pin ts-proto to 1.x in package.json.

🏁 Script executed:

# Search for protoTypes.ts and related generated protobuf files
find . -name "*proto*" -type f | grep -E "\.(ts|js)$" | head -20

Repository: UCSolarCarTeam/Helios-Telemetry

Length of output: 110

🏁 Script executed:

# Check if either runtime is imported in the codebase
rg "from ['\"]@bufbuild/protobuf['\"]|from ['\"]protobufjs['\"]|import.*protobufjs|import.*@bufbuild" --type ts --type js

Repository: UCSolarCarTeam/Helios-Telemetry

Length of output: 178

🏁 Script executed:

# Check the actual dependencies in package.json to understand context
cat packages/shared/package.json

Repository: UCSolarCarTeam/Helios-Telemetry

Length of output: 435

Remove unused protobufjs dependency.
ts-proto v2.2.5 generates code that depends exclusively on @bufbuild/protobuf (notably BinaryReader and BinaryWriter from @bufbuild/protobuf/wire). The codebase in packages/shared/src/protoTypes.ts confirms this—protobufjs is not imported or used. Drop it to reduce the dependency surface area.

🤖 Prompt for AI Agents 
In packages/shared/package.json around lines 6 to 15, remove the unused
"protobufjs" dependency from the "dependencies" object because ts-proto output
and our code use @bufbuild/protobuf only; update package.json to delete the
"protobufjs" entry and then run the package manager (npm/yarn/pnpm) to update
lockfile and node_modules (install or prune) so the dependency is fully removed.

}
9 changes: 9 additions & 0 deletions packages/shared/src/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Instructions to test

Ensure you have the protobuf server running on port 8000. You can find additional instructions in the [protobuf server README](https://github.com/UCSolarCarTeam/ML-Telemetry/blob/setup_protobuf/README.md#generate-python-code-from-protobuf)

To run the tests for the shared package, you can use the following command in the root directory of the repository:

```bash
yarn test
```
Comment on lines 1 to 9
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Avoid linking to a moving branch for setup docs; pin to a commit or mirror essentials here.
Branch links tend to break; consider pointing at a stable branch/tag/commit (or duplicating the minimal “run server on :8000” steps).

🤖 Prompt for AI Agents 
In packages/shared/src/README.md around lines 1 to 10, the README currently
links to a moving branch for protobuf server setup which can break; update the
link to a specific stable tag/commit (or a release) or replace the external link
by copying the minimal required setup steps (how to generate/run the protobuf
server on port 8000) into this README so tests remain reproducible; ensure the
link, if kept, points to a commit hash or tag and add the essential port/command
instructions inline.

45 changes: 7 additions & 38 deletions packages/shared/src/functions.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
import { faker } from "@faker-js/faker";
import { Coords, ITelemetryData, ITelemetryDataType } from "./types";
import { isRight } from "fp-ts/Either";
import { type ValidationError } from "io-ts";
import { Coords, ITelemetryData } from "./types";

/**
* This file generates fake telemetry data for testing purposes on the demo mode
Expand Down Expand Up @@ -301,11 +299,9 @@ export function generateFakeTelemetryData(): ITelemetryData {
},
MotorDetails0: {
ActiveMotor: faker.number.int({ max: 100, min: 0 }),
BEMF_D: faker.number.int({ max: 100, min: 0 }),
BEMF_Q: faker.number.int({ max: 100, min: 0 }),
BusCurrent: faker.number.int({ max: 100, min: 0 }),
BusVoltage: faker.number.int({ max: 100, min: 0 }),
DC_Bus_Ah: faker.number.int({ max: 100, min: 0 }),
DCBusAh: faker.number.int({ max: 100, min: 0 }),
DspBoardTemperature: faker.number.int({ max: 100, min: 0 }),
ErrorFlags: faker.number.int({ max: 100, min: 0 }),
HeatsinkTemperature: faker.number.int({ max: 100, min: 0 }),
Expand All @@ -329,14 +325,14 @@ export function generateFakeTelemetryData(): ITelemetryData {
Vd: faker.number.int({ max: 100, min: 0 }),
VehicleVelocity: faker.number.int({ max: 100, min: 0 }),
Vq: faker.number.int({ max: 100, min: 0 }),
bemfD: faker.number.int({ max: 100, min: 0 }),
bemfQ: faker.number.int({ max: 100, min: 0 }),
},
MotorDetails1: {
ActiveMotor: faker.number.int({ max: 100, min: 0 }),
BEMF_D: faker.number.int({ max: 100, min: 0 }),
BEMF_Q: faker.number.int({ max: 100, min: 0 }),
BusCurrent: faker.number.int({ max: 100, min: 0 }),
BusVoltage: faker.number.int({ max: 100, min: 0 }),
DC_Bus_Ah: faker.number.int({ max: 100, min: 0 }),
DCBusAh: faker.number.int({ max: 100, min: 0 }),
DspBoardTemperature: faker.number.int({ max: 100, min: 0 }),
ErrorFlags: faker.number.int({ max: 100, min: 0 }),
HeatsinkTemperature: faker.number.int({ max: 100, min: 0 }),
Expand All @@ -360,6 +356,8 @@ export function generateFakeTelemetryData(): ITelemetryData {
Vd: faker.number.int({ max: 100, min: 0 }),
VehicleVelocity: faker.number.int({ max: 100, min: 0 }),
Vq: faker.number.int({ max: 100, min: 0 }),
bemfD: faker.number.int({ max: 100, min: 0 }),
bemfQ: faker.number.int({ max: 100, min: 0 }),
},
Pi: {
Rfid: faker.number.int({ max: 100, min: 0 }).toString(),
Expand Down Expand Up @@ -413,35 +411,6 @@ export function generateFakeTelemetryData(): ITelemetryData {
};
}

// this is a helper function to format the validation errors if there
// is an error in the telemetry data that is being sent
function formatValidationErrors(errors: ValidationError[]): string {
return errors
.map((error) => {
const path = error.context
.map(({ key }) => key)
.filter(Boolean)
.join(".");
const expectedType =
error.context[error.context.length - 1]?.type.name ??
"error getting expected type";
const actualValue = JSON.stringify(error.value);
return `An invalid value of ${actualValue} was supplied to ${path} (expected type: ${expectedType}). Hint: if you received a value of undefined, it is likely the field is missing from the packet`;
})
.join(", ");
}

// using the we match the structure of the telemetry data that is coming in from the car
// and we validate it against the ITelemetryDataType
export function validateTelemetryData(packet: unknown) {
const validationResult = ITelemetryDataType.decode(packet);
if (isRight(validationResult)) {
return validationResult.right;
}
const errorMessages = formatValidationErrors(validationResult.left);
throw new Error(errorMessages);
}

// calculateBearing calculates the bearing from start to end coordinates
export const calculateBearing = (start: Coords, end: Coords): number => {
//using the haversine formula from https://www.movable-type.co.uk/scripts/latlong.html
Expand Down
Loading