Synthetic Test Data Set Generation

[cantemizyurek]

Problem

LLM evaluations depend on abundant, diverse, and regularly refreshed test data, yet manual dataset creation is slow, costly, and fails to capture complex reasoning, multi‑hop queries, edge cases, and adversarial prompts. This leads to incomplete coverage, inconsistent regression testing, and limited confidence in real world performance.

Solution

I propose a synthetic test data pipeline that generates domain grounded evaluations using a knowledge graph to structure entities and relations, personas to vary intents and language styles, and both single‑hop and multi‑hop synthesizers to control reasoning depth. The pipeline produces balanced test sets across difficulty and scenarios for continuous regression, with traceable provenance from graph nodes, persona templates, and hop strategies.

Knowledge Graph

We ingest source content, use chunking strategies to split it into chunks, then run extractors to label entities and topics per chunk. Each chunk becomes a node, and we build edges with topic similarity and entity overlap to connect related nodes. This yields a compact, navigable graph that mirrors domain structure and preserves provenance to the original content.

Scenario Generation

We sample graph nodes to assemble context passages, then use single‑hop and multi‑hop synthesizers to generate queries that require shallow or compositional reasoning. Personas vary intent, tone, and language style, while configurable question lengths and formats ensure balanced coverage. Each scenario outputs the user query, the source context used, and a golden/reference answer with provenance to the contributing nodes.

[mattpocock]

@cantemizyurek I'm really excited to work on this. This is a great post-v1 goal.

One thing I'm thinking about in terms of an API for this is progressive disclosure of complexity. How do we make it so this is easy to start with and ramps up to a more powerful solution when you need it? It seems like this design you have can be chopped into pieces. I would love to see how those pieces can be composed in a modular way.

Also, what do other SDG solutions do? Evalite is basically lifted from Braintrust's evals API, would love to see prior art for SDG so we can design the best possible API.

[cantemizyurek]

I can provide examples from some frameworks, plus one from Open Evals. I’ve been working on a declarative version of the approach, though the API isn’t finalized yet. From what I’ve seen, Ragas currently offers the strongest SDG model, and I drew heavily from it while building Open Evals.

Open Evals

const documents = [new DocumentNode('typescript-guide.md', content, {})]

const knowledgeGraph = await transform(graph(documents))
  .pipe(summarize(openai.chat('gpt-4.1')))
  .pipe(embedProperty(openai.embedding('text-embedding-3-small'), {
    embedProperty: 'summary',
    propertyName: 'summaryEmbedding',
  }))
  .pipe(chunk(new RecursiveCharacterSplitter()))
  .pipe(embed(openai.embedding('text-embedding-3-small')))
  .pipe(relationship())
  .apply()

const personas = await generatePersonas(knowledgeGraph, openai.chat('gpt-4.1'), {
  count: 5,
})

const dataset = await synthesize({
  graph: knowledgeGraph,
  synthesizers: [
    [createSynthesizer(openai.chat('gpt-4.1'), 'single-hop-specific'), 1],
  ],
  personas,
  count: 10,
})

Ragas

Imperative

kg = KnowledgeGraph()

for doc in docs:
    kg.nodes.append(
        Node(
            type=NodeType.DOCUMENT,
            properties={"page_content": doc.page_content, "document_metadata": doc.metadata}
        )
    )

transformer_llm = generator_llm
embedding_model = generator_embeddings

trans = default_transforms(documents=docs, llm=transformer_llm, embedding_model=embedding_model)
apply_transforms(kg, trans)

generator = TestsetGenerator(llm=generator_llm, embedding_model=embedding_model, knowledge_graph=kg)
query_distribution = [
    (SingleHopSpecificQuerySynthesizer(llm=llm), 0.5),
    (MultiHopAbstractQuerySynthesizer(llm=llm), 0.25),
    (MultiHopSpecificQuerySynthesizer(llm=llm), 0.25),
]

testset = generator.generate(testset_size=10, query_distribution=query_distribution)

Declerative

generator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
openai_client = openai.OpenAI()
generator_embeddings = OpenAIEmbeddings(client=openai_client)

generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
dataset = generator.generate_with_langchain_docs(docs, testset_size=10)