Skip to content

Git integration: track commit & branch in eval runs #236

@mattpocock

Description

@mattpocock

Problem

Currently evalite doesn't track which git commit or branch an eval was run against. This makes it hard to:

  • Correlate eval results with specific code versions
  • Track performance across commits/branches
  • Understand which code changes affected eval scores

Proposal

Store in eval run metadata:

  • Commit SHA
  • Branch name
  • (Optional) dirty state flag

Implementation considerations

  • Read git info at run start (similar to how we get cwd)
  • Store in runs table (add commit & branch columns)
  • Display in UI (run history, comparison views)
  • Handle non-git repos gracefully
  • Handle dirty working trees

Scope

Likely post-v1 - fairly tasty chunk of work that could expose funny bugs.

Questions

  • Should we block runs if working tree is dirty?
  • Should we auto-compare across branches?
  • Git info in filesystem mode vs server mode?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions