-
Notifications
You must be signed in to change notification settings - Fork 63
Open
Milestone
Description
Problem
Currently evalite doesn't track which git commit or branch an eval was run against. This makes it hard to:
- Correlate eval results with specific code versions
- Track performance across commits/branches
- Understand which code changes affected eval scores
Proposal
Store in eval run metadata:
- Commit SHA
- Branch name
- (Optional) dirty state flag
Implementation considerations
- Read git info at run start (similar to how we get
cwd) - Store in
runstable (addcommit&branchcolumns) - Display in UI (run history, comparison views)
- Handle non-git repos gracefully
- Handle dirty working trees
Scope
Likely post-v1 - fairly tasty chunk of work that could expose funny bugs.
Questions
- Should we block runs if working tree is dirty?
- Should we auto-compare across branches?
- Git info in filesystem mode vs server mode?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels