Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 92 additions & 0 deletions .github/workflows/benchmark-report.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
name: Generate Benchmark Report

on:
# Trigger on push when cost.json is changed (all branches)
push:
paths:
- 'benchmark/cost.json'

# Trigger on pull request when cost.json is changed
pull_request:
paths:
- 'benchmark/cost.json'

# Daily schedule at UTC 16:00 (Beijing Time 00:00) - only runs on default branch (main)
schedule:
- cron: '0 16 * * *'

# Allow manual trigger
workflow_dispatch:

permissions:
contents: write # Allow committing report to main branch
pull-requests: write # Allow commenting on PRs

jobs:
generate-report:
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'

- name: Install dependencies
run: npm ci

- name: Generate benchmark report
run: npm run benchmark:calculate

- name: Check for changes
id: git-check
run: |
git diff --exit-code benchmark/report.json benchmark/report.md || echo "changed=true" >> $GITHUB_OUTPUT

- name: Commit and push if changed
if: steps.git-check.outputs.changed == 'true' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/master')
run: |
git config --local user.email "github-actions[bot]@users.noreply.github.com"
git config --local user.name "github-actions[bot]"
git add benchmark/report.json benchmark/report.md
git commit -m "chore: update benchmark cost report [skip ci]"
git push
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Upload reports as artifacts
uses: actions/upload-artifact@v4
with:
name: benchmark-reports
path: |
benchmark/report.json
benchmark/report.md
retention-days: 30

- name: Comment on PR (if applicable)
# Security: Only comment on PRs from the same repository (not forks)
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const reportPath = 'benchmark/report.md';

if (fs.existsSync(reportPath)) {
const report = fs.readFileSync(reportPath, 'utf8');
const body = `## 📊 Benchmark Cost Report\n\n${report}`;

github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: body
});
}

98 changes: 98 additions & 0 deletions benchmark/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Benchmark Cost Calculation

This directory contains benchmark data and automated cost calculation reports for the AMACI operator.

## Files

- **`cost.json`**: Configuration file containing base costs for different machine types and scales
- **`report.json`**: Auto-generated JSON report with detailed cost calculations (generated by CI/CD)
- **`report.md`**: Auto-generated human-readable Markdown report (generated by CI/CD)
- **`bench-12c-128G-Bare-metal.md`**: Manual benchmark results for 12c-128G bare metal configuration

## Automated Cost Calculation

The cost calculation is automatically triggered by:

1. **On Commit**: When `benchmark/cost.json` is modified and pushed/PR'd
2. **Daily Schedule**: Every day at 00:00 Beijing Time (16:00 UTC)
3. **Manual Trigger**: Can be manually triggered via GitHub Actions workflow

### How It Works

The calculation script (`scripts/calculate-benchmark.ts`):

1. Reads cost configuration from `cost.json`
2. Fetches current DORA price from CoinGecko API (ID: `dora-factory-2`)
3. Calculates costs using the formula:
```
Final USD Cost = Base Cost × Idle Sharing Multiplier
DORA Amount = Final USD Cost ÷ DORA Price
```
4. Compares with previous report to show price changes
5. Generates both JSON and Markdown reports
6. Commits the reports back to the repository

### Cost Configuration Format

```json
{
"machineTypes": {
"12c-128G-Bare-metal": {
"currentTallyCostUsd": {
"2-1-1-5": 0.02,
"4-2-2-25": 1
},
"idleSharingMultiplier": 10
}
}
}
```

- **currentTallyCostUsd**: Base cost in USD for each scale
- **idleSharingMultiplier**: Multiplier to account for idle machine sharing

### Running Manually

To generate a report manually:

```bash
# No need to build, tsx runs TypeScript directly
npm run benchmark:calculate
```

The generated reports will be saved to:
- `benchmark/report.json`
- `benchmark/report.md`

### Report Format

The generated report includes:

- **Timestamp**: When the report was generated
- **DORA Price**: Current price and 24h change from CoinGecko
- **Cost Breakdown**: For each machine type and scale:
- Base cost in USD
- Multiplier applied
- Final cost in USD
- Equivalent DORA amount
- Change indicator (compared to previous report)
- **Price Comparison**: How DORA price changed since last report

### GitHub Actions Workflow

The workflow is defined in `.github/workflows/benchmark-report.yml` and:

1. Checks out the repository
2. Sets up Node.js environment
3. Installs dependencies (including `tsx` for running TypeScript)
4. Runs the cost calculation directly using `tsx`
5. Commits and pushes the reports if changed
6. On pull requests, posts the report as a comment

## Notes

- The DORA price is fetched in real-time from CoinGecko API
- If the API fails, it will retry 3 times, then fall back to cached price
- Reports are automatically committed with `[skip ci]` to avoid triggering infinite loops
- The daily scheduled run ensures reports stay up-to-date with DORA price fluctuations

18 changes: 9 additions & 9 deletions benchmark/cost.json
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
{
"machineTypes": {
"12c-128G-Bare-metal": {
"currentTallyCostUsd": {
"2-1-1-5": 0.02,
"4-2-2-25": 1
},
"idleSharingMultiplier": 10
}
"machineTypes": {
"12c-128G-Bare-metal": {
"currentTallyCostUsd": {
"2-1-1-5": 0.02,
"4-2-2-25": 1
},
"idleSharingMultiplier": 10
}
}
}
}
27 changes: 27 additions & 0 deletions benchmark/report.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{
"timestamp": "2025-12-28T06:21:57.646Z",
"doraPrice": 0.0088,
"machineTypes": {
"12c-128G-Bare-metal": {
"costs": {
"2-1-1-5": {
"baseUsd": 0.02,
"finalUsd": 0.2,
"doraAmount": 22.62,
"multiplier": 10
},
"4-2-2-25": {
"baseUsd": 1,
"finalUsd": 10,
"doraAmount": 1130.98,
"multiplier": 10
}
}
}
},
"dora24hrChange": -2.32,
"previousReport": {
"timestamp": "2025-12-28T06:21:10.952Z",
"doraPrice": 0.0088
}
}
36 changes: 36 additions & 0 deletions benchmark/report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Benchmark Cost Report

**Generated:** 12/28/2025, 14:21:57 (Beijing Time)

## DORA Price

**Current Price:** $0.0088
**24h Change:** -2.32% 📉

**Comparison with Previous Report:**
- Previous Price: $0.0088
- Previous Time: 12/28/2025, 14:21:10
- Price Change: 0.00%

## 12c-128G-Bare-metal

| Scale | Base Cost (USD) | Multiplier | Final Cost (USD) | DORA Amount | Change |
|-------|----------------|------------|------------------|-------------|--------|
| 2-1-1-5 | $0.02 | 10x | $0.20 | 22.62 DORA | |
| 4-2-2-25 | $1.00 | 10x | $10.00 | 1130.98 DORA | |

## Calculation Formula

```
Final USD Cost = Base Cost × Idle Sharing Multiplier
DORA Amount = Final USD Cost ÷ DORA Price
```

---

*This report is automatically generated by CI/CD pipeline.*

*Change indicators:*
- 🔴 Cost increased
- 🟢 Cost decreased
- ⚪ No significant change
Loading