Weighted scorers

[gabimoncha]

I think it would be great if we can have a weighted average between the scorers.

For example I have some 2 LLM judges, but the things they verify are of different importance, and i would like to do a weighted average on their score.

So, the columns and cards can show the normal score (and maybe the weight of that), but have an additional card that shows they weighted average which is also shown on the side bard

[mattpocock]

Agree that this is something we need. Some ideas needed on how the UI would look for this.

@claude Can you come up with 2-3 options, using wireframes in ASCII, for how the table should look in the evalite-ui to display weighted scorers?

[claude]

Claude finished @mattpocock's task —— View job

---

UI Wireframe Options for Weighted Scorers

Based on my exploration of the evalite-ui codebase, I've designed three different approaches for displaying weighted scorers. Each option builds on the current UI structure which includes:

• Score cards (lines 408-456 in eval.$name.tsx) - Grid of cards showing individual scorer averages
• Results table (lines 477-556) - Table with columns for Input/Output/Expected and scorer columns
• Sidebar score display (eval.$name.result.$resultIndex.tsx) - Individual score details with metadata

---

Option 1: Inline Weight Display with Summary Card

This option adds weight information directly to existing scorer cards and adds a new "Weighted Average" summary card.

┌─────────────────────────────────────────────────────────────┐
│ Scores                                                      │
├─────────────────────────────────────────────────────────────┤
│ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐  │
│ │ Accuracy       │ │ Relevance      │ │ Completeness   │  │
│ │ (weight: 2.0)  │ │ (weight: 1.0)  │ │ (weight: 1.5)  │  │
│ │                │ │                │ │                │  │
│ │ 87% ↑          │ │ 92% ↑          │ │ 78% →          │  │
│ └────────────────┘ └────────────────┘ └────────────────┘  │
│                                                             │
│ ┌────────────────────────────────────────────────────────┐ │
│ │ ⚖ Weighted Average                                     │ │
│ │                                                         │ │
│ │ 84% ↑                                                  │ │
│ │                                                         │ │
│ │ Calculation: (87×2.0 + 92×1.0 + 78×1.5) / 4.5         │ │
│ └────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

Table Headers:
┌───────┬────────┬──────────┬────────────┬───────────┬──────────────┬──────────────┐
│ Input │ Output │ Expected │ Accuracy   │ Relevance │ Completeness │ Weighted Avg │
│       │        │          │ (wt: 2.0)  │ (wt: 1.0) │ (wt: 1.5)    │ (auto calc)  │
├───────┼────────┼──────────┼────────────┼───────────┼──────────────┼──────────────┤
│ ...   │ ...    │ ...      │ 85% ↑      │ 90% →     │ 75% ↓        │ 82% →        │
└───────┴────────┴──────────┴────────────┴───────────┴──────────────┴──────────────┘

Pros:

• Weight visible at a glance in both cards and table
• Weighted average gets prominent visual treatment
• Minimal disruption to existing layout
• Clear calculation shown in summary card

Cons:

• Adds extra column to table (could get crowded)
• Weight text in headers might feel cramped
• Summary card formula might be too technical for some users

---

Option 2: Expandable Weight Configuration Panel

This option keeps the UI cleaner by default, with weights shown in a collapsible section.

┌─────────────────────────────────────────────────────────────┐
│ Scores                                     [⚙ Configure ▼] │
├─────────────────────────────────────────────────────────────┤
│ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐  │
│ │ ⭐ Weighted Avg│ │ Accuracy       │ │ Relevance      │  │
│ │                │ │                │ │                │  │
│ │ 84% ↑          │ │ 87% ↑          │ │ 92% ↑          │  │
│ └────────────────┘ └────────────────┘ └────────────────┘  │
│                                                             │
│ │ ┌────────────────┐                                       │
│ │ │ Completeness   │                                       │
│ │ │                │                                       │
│ │ │ 78% →          │                                       │
│ │ └────────────────┘                                       │
│                                                             │
│ ╔═════════════════════════════════════════════════════════╗ │
│ ║ Weight Configuration                                    ║ │
│ ╠═════════════════════════════════════════════════════════╣ │
│ ║ Accuracy:       ━━━━●━━━━━  2.0                        ║ │
│ ║ Relevance:      ━━━●━━━━━━  1.0                        ║ │
│ ║ Completeness:   ━━━━━●━━━━  1.5                        ║ │
│ ║                                                         ║ │
│ ║ Formula: (87×2.0 + 92×1.0 + 78×1.5) / 4.5 = 84%       ║ │
│ ╚═════════════════════════════════════════════════════════╝ │
└─────────────────────────────────────────────────────────────┘

Table Headers:
┌───────┬────────┬──────────┬──────────────┬──────────┬───────────┬──────────────┐
│ Input │ Output │ Expected │ Weighted Avg │ Accuracy │ Relevance │ Completeness │
│       │        │          │              │          │           │              │
├───────┼────────┼──────────┼──────────────┼──────────┼───────────┼──────────────┤
│ ...   │ ...    │ ...      │ 82% →        │ 85% ↑    │ 90% →     │ 75% ↓        │
└───────┴────────┴──────────┴──────────────┴──────────┴───────────┴──────────────┘

Pros:

• Weighted average gets first position (most important metric)
• Cleaner default view without weight clutter
• Weight configuration panel is educational
• Could potentially make weights interactive in future

Cons:

• Requires interaction to see weights (less discoverable)
• More complex to implement
• Panel might be overkill if weights don't change often

---

Option 3: Dual-Row Table Headers with Weight Subtext

This option uses a more compact inline approach with weights shown as subtle subtext.

┌─────────────────────────────────────────────────────────────┐
│ Scores                                                      │
├─────────────────────────────────────────────────────────────┤
│ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐  │
│ │ Accuracy       │ │ Relevance      │ │ Completeness   │  │
│ │ weight: 2.0×   │ │ weight: 1.0×   │ │ weight: 1.5×   │  │
│ │                │ │                │ │                │  │
│ │ 87% ↑          │ │ 92% ↑          │ │ 78% →          │  │
│ └────────────────┘ └────────────────┘ └────────────────┘  │
│                                                             │
│ Overall weighted average: 84% ↑                            │
└─────────────────────────────────────────────────────────────┘

Table (in main eval page top-right, next to duration and date):
┌──────────────────────────────────────┐
│ my-eval                              │
├──────────────────────────────────────┤
│ 84% ↑ (weighted) | 2.3s | Oct 3     │  ← Top-level summary
└──────────────────────────────────────┘

Table Headers:
┌───────┬────────┬──────────┬────────────────┬─────────────────┬────────────────┐
│ Input │ Output │ Expected │   Accuracy     │   Relevance     │  Completeness  │
│       │        │          │   (wt: 2.0×)   │   (wt: 1.0×)    │   (wt: 1.5×)   │
├───────┼────────┼──────────┼────────────────┼─────────────────┼────────────────┤
│ ...   │ ...    │ ...      │   85% ↑        │   90% →         │   75% ↓        │
└───────┴────────┴──────────┴────────────────┴─────────────────┴────────────────┘

Sidebar (Result Detail View):
┌──────────────────────────────────────┐
│ Trace                                │
│ 84% ↑ (weighted) | 1.2s | tokens    │  ← Shows weighted score
├──────────────────────────────────────┤
│ ...                                  │
│                                      │
│ Accuracy (weight: 2.0×)              │
│ 87% ↑                                │
│                                      │
│ Relevance (weight: 1.0×)             │
│ 92% ↑                                │
│                                      │
│ Completeness (weight: 1.5×)          │
│ 78% →                                │
└──────────────────────────────────────┘

Pros:

• Most compact approach - works well at all screen sizes
• Weights visible but not overwhelming
• Weighted average in top summary (lines 340-377) is very prominent
• No extra table columns needed
• Natural fit with existing UI patterns

Cons:

• Two-row headers add vertical height
• Weight notation might need explanation for new users
• Weighted calculation not shown explicitly

---

Recommendation

I recommend Option 3 for the following reasons:

1. Best mobile experience - No extra columns, compact design
2. Matches existing patterns - evalite already shows the overall score at the top (line 345-353), this extends it naturally
3. Progressive disclosure - Casual users see the weighted average; detailed users can see individual weights
4. Simpler implementation - Leverages existing score display logic

The weighted average would:

• Replace/augment the current overall average in the eval header (line 345)
• Show in table headers as subtext (line 499-508)
• Display in score cards with weight multiplier (line 413-454)
• Appear in sidebar breadcrumb (line 205-213)

---

[mattpocock]

@claude give me an ASCII diagram showing this option: When weights are present, before the score title in the header, you would show an icon displaying the weight. This icon would be a ring that's completely filled in if it's 1.0 or 50% filled in if it's 0.5. By filled in, I mean that there would be two strokes around the edge of the ring:

1. One would be partially transparent (the background one)
2. The foreground one would indicate how much it was filled in
   If it was over 1.0, so weighted more than 1, then it would be filled in with a new color on top of the filled color.

Hopefully I'm making myself clear enough.

This icon would show before on the table headers but also on the scores boxes at the top. On hover it would display the weight. We should also add another column to the tables and the scorers at the top, which is weighted average.

[claude]

Claude finished @mattpocock's task —— View job

---

Weighted Scorers UI Design with Ring Icons

Here's an ASCII diagram showing the proposed UI with weight indicator rings:

---

Score Cards Section (Top of Page)

┌─────────────────────────────────────────────────────────────────────────────┐
│ Scores                                                                      │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐        │
│  │ ◉◉ Accuracy      │  │ ◑  Relevance     │  │ ◉◉ Completeness  │        │
│  │ (hover: wt 2.0)  │  │ (hover: wt 1.0)  │  │ (hover: wt 1.5)  │        │
│  │                  │  │                  │  │                  │        │
│  │     87% ↑        │  │     92% ↑        │  │     78% →        │        │
│  └──────────────────┘  └──────────────────┘  └──────────────────┘        │
│                                                                             │
│  ┌──────────────────────────────────────────┐                             │
│  │ ⚖  Weighted Average                       │                             │
│  │                                            │                             │
│  │     84% ↑                                 │                             │
│  │                                            │                             │
│  │  (87×2.0 + 92×1.0 + 78×1.5) / 4.5        │                             │
│  └──────────────────────────────────────────┘                             │
└─────────────────────────────────────────────────────────────────────────────┘

Legend for Ring Icons:
  ◯  = Empty ring (weight 0.0)          - Two strokes, inner transparent
  ◔  = 25% filled (weight 0.25)         - Quarter of ring filled
  ◑  = 50% filled (weight 0.5)          - Half of ring filled
  ◕  = 75% filled (weight 0.75)         - Three quarters filled
  ◉  = 100% filled (weight 1.0)         - Completely filled
  ◉◉ = Over 100% (weight > 1.0)         - Filled + additional accent color overlay
  ◉◉◉= Heavily weighted (weight > 2.0)  - Multiple accent layers

---

Results Table with Weight Icons

┌─────┬────────┬──────────┬──────────────┬─────────────┬───────────────┬──────────────┐
│ In  │ Output │ Expected │ ◉◉ Accuracy  │ ◑ Relevance │ ◉◉Completeness│ ⚖ Weighted   │
│ put │        │          │              │             │               │    Average   │
│     │        │          │ (hover:      │ (hover:     │ (hover:       │              │
│     │        │          │  wt 2.0)     │  wt 1.0)    │  wt 1.5)      │              │
├─────┼────────┼──────────┼──────────────┼─────────────┼───────────────┼──────────────┤
│ ... │  ...   │   ...    │   85% ↑      │   90% →     │    75% ↓      │   82% →      │
├─────┼────────┼──────────┼──────────────┼─────────────┼───────────────┼──────────────┤
│ ... │  ...   │   ...    │   90% ↑      │   95% ↑     │    80% ↑      │   87% ↑      │
└─────┴────────┴──────────┴──────────────┴─────────────┴───────────────┴──────────────┘

---

Visual Breakdown of Ring Icon States

Weight Value     Icon     Visual Description
─────────────────────────────────────────────────────────────
0.0              ◯        ╭─────╮    Background stroke (transparent gray)
                          │     │    No fill
                          ╰─────╯

0.5              ◑        ╭──█──╮    Background stroke (transparent gray)
                          │  █  │    Foreground fills 50% (solid color)
                          ╰─────╯

1.0              ◉        ╭─███─╮    Background stroke (transparent gray)
                          │ ███ │    Foreground fills 100% (solid color)
                          ╰─███─╯

2.0              ◉◉       ╭─███─╮    Background stroke (transparent gray)
                          │▓███▓│    Foreground fills 100% (solid color)
                          ╰─███─╯    Accent overlay (different color on top)

Over 2.0         ◉◉◉      ╭▓███▓╮    Multiple accent layers for emphasis
                          ▓▓███▓▓    Shows high importance visually
                          ╰▓███▓╯

---

Hover State Example

When hovering over a ring icon:

Normal State:
┌──────────────────┐
│ ◉◉ Accuracy      │
│                  │
│     87% ↑        │
└──────────────────┘

Hover State:
┌──────────────────┐
│ ◉◉ Accuracy      │  ╭─────────────────╮
│                  │  │ Weight: 2.0×    │  ← Tooltip appears
│     87% ↑        │  ╰─────────────────╯
└──────────────────┘

---

Sidebar Result Detail View

┌──────────────────────────────────────┐
│ Result #1                            │
├──────────────────────────────────────┤
│ ⚖ Weighted Average: 84% ↑            │
│ Duration: 1.2s                       │
├──────────────────────────────────────┤
│                                      │
│ ◉◉ Accuracy (weight: 2.0×)           │
│ 87% ↑                                │
│ [score metadata...]                  │
│                                      │
│ ◑ Relevance (weight: 1.0×)           │
│ 92% ↑                                │
│ [score metadata...]                  │
│                                      │
│ ◉◉ Completeness (weight: 1.5×)       │
│ 78% →                                │
│ [score metadata...]                  │
└──────────────────────────────────────┘

---

Implementation Notes

Ring Icon Rendering:

• Use CSS with conic-gradient for the fill effect
• Two overlapping circles: outer transparent ring, inner filled ring
• Fill percentage: 360deg * weight (capped at 360deg for base fill)
• For weights > 1.0: add accent color overlay using additional gradient layer

Color Scheme Suggestion:

• Background ring: rgba(gray, 0.2) (subtle)
• Foreground fill (0-1.0): Primary theme color (e.g., blue)
• Accent overlay (>1.0): Secondary color (e.g., amber/gold) to indicate "extra importance"
• Multiple accents (>2.0): Could pulse or have shimmer effect

Weighted Average Column:

• Auto-calculated, not a separate scorer
• Always shows at the end of scorer columns (or at the start for prominence)
• Uses ⚖ balance scale icon instead of ring (semantic difference)
• Tooltip explains the calculation formula

Responsive Behavior:

• On mobile: Icons might shrink or be replaced with simple "×2.0" text badges
• Table headers could show icons only, with weight in tooltip
• Score cards always show both icon and name

---

Todo List:

• Analyze the design requirements
• Create ASCII diagram showing score cards with weight icons
• Create ASCII diagram showing table headers with weight icons
• Show hover state representation
• Show weighted average column/card

---