Feature: Allow scorer functions to return arrays of scores

[mihilmy]

Feature Request: Allow scorer functions to return arrays of scores

Problem

Currently, scorer functions can only return a single Score object. However, some use cases involve LLM graders that assess multiple criteria in a single call and return multiple pass/fail results.

Use Case

I have an LLM grader that performs a one-shot assessment of multiple criteria and returns a pass/fail for each criterion. For example, evaluating if a response is accurate, complete, and clear - all in a single LLM call for efficiency.

Currently, I need to either:

• Make separate LLM calls for each criterion (inefficient and costly)
• Write custom logic to split the results into multiple scorer calls
• Use only a single aggregate score

Proposed Solution

Allow scorers to return either a single Score or an array of Score[], which would be flattened into the final scores array.

This appears to be achievable with a simple .flat() here:

evalite/packages/evalite/src/evalite.ts

Line 89 in e04b48f

const scores = await Promise.all(

(plus type changes to allow Score | Score[]).

[mattpocock]

Yeah I literally just ran into this need 😂

I think this would require a new function called a multi-scorer. Since each score would need to be reported to Evalite separately and have a separate name, description, and config options like 'weight': #289

Then this would need docs etc.

But it's definitely required, and I'll put it in the post v1 milestone.