Optional CI check for agent behavior?

[ZackMitchell910]

Hi! I'm building RunLedger, a small optional CI check for tool-using agents. You run it once locally to record the expected tool calls and outputs, then CI reuses that recording so runs are stable and don't call external APIs. It adds a small eval folder + workflow and is fully removable.

If you're open to it, what's the best existing agent/example entrypoint to wire to for a minimal demo?

[fengju0213]

@ZackMitchell910 this sounds really cool 👍 One question I had though: how do you handle tools whose outputs are inherently non-deterministic?

For example, a search tool might return different results on each call. In those cases, how does RunLedger deal with the variability? Do you rely on mocking, output normalization, or validating only the tool call structure rather than the exact outputs?

I’m curious what the design trade-offs look like for these kinds of tools.

[ZackMitchell910]

Thanks! For something like a search tool, CI shouldn’t be hitting the live search at all. The CI check uses a fixed response captured from a known‑good run, so it stays stable even if the real search results change day to day.

If you want more flexibility, you can also keep the checks focused on what should be stable: did the agent call the right tools in the right order with the right inputs, and did it return the expected output shape (required fields/schema), instead of matching every byte of the tool output.

For CAMEL, an easy first example is a small “search → summarize” flow. CI can flag regressions if a change makes it skip search, call it multiple times, change the query format, or break the structured summary output.

[fengju0213]

Thanks! For something like a search tool, CI shouldn’t be hitting the live search at all. The CI check uses a fixed response captured from a known‑good run, so it stays stable even if the real search results change day to day.

If you want more flexibility, you can also keep the checks focused on what should be stable: did the agent call the right tools in the right order with the right inputs, and did it return the expected output shape (required fields/schema), instead of matching every byte of the tool output.

For CAMEL, an easy first example is a small “search → summarize” flow. CI can flag regressions if a change makes it skip search, call it multiple times, change the query format, or break the structured summary output.

Thanks, that makes sense — appreciate the explanation.

One thing I’m still a bit unclear about: in the current CAMEL repo, https://github.com/camel-ai/camel/tree/master/test/toolkits we already have unit tests for each toolkit. From your perspective, what’s the key difference between those existing unit tests and the kind of CI/CD check you’re proposing with RunLedger?

In other words, what additional failure modes or regressions does this approach catch that wouldn’t already be covered by per-tool unit tests?

[ZackMitchell910]

The check I’m proposing is more of an integration/contract test for the agent’s tool‑using workflow.

It catches failure modes that unit tests usually won’t, like:

• the agent stops calling a tool at all after a prompt/planner change,
• the agent calls the wrong tool or sends subtly wrong args,
• tool call order/count drifts (extra calls, loops),
• output structure drifts even though each tool still works.

Example: SearchToolkit tests can pass, but if the chat agent starts skipping search and answering from the model, or changes the query format, unit tests won’t notice, this check will.

So it’s complementary: unit tests validate tool code; this validates the agent’s use of those tools in a full run.

[fengju0213]

The check I’m proposing is more of an integration/contract test for the agent’s tool‑using workflow.

It catches failure modes that unit tests usually won’t, like:

• the agent stops calling a tool at all after a prompt/planner change,
• the agent calls the wrong tool or sends subtly wrong args,
• tool call order/count drifts (extra calls, loops),
• output structure drifts even though each tool still works.

Example: SearchToolkit tests can pass, but if the chat agent starts skipping search and answering from the model, or changes the query format, unit tests won’t notice, this check will.

So it’s complementary: unit tests validate tool code; this validates the agent’s use of those tools in a full run.

sorry for late reply！@ZackMitchell910 sounds great！thanks for the explanation — framing this as an integration/contract test makes sense, and I agree those failure modes are very real.

I’m still trying to calibrate the expected stability though. Since agent tool use is inherently stochastic and often evolves with prompt/model changes, freezing a single tool-use trace feels a bit counter-intuitive at first.

I do see the value for catching unintended regressions (skipping tools, arg drift, call loops), but I’m curious how you draw the line between unwanted behavioral drift and acceptable evolution. In practice, do teams re-record frequently, or rely more on higher-level invariants rather than exact call traces?

[ZackMitchell910]

Totally fair concern. I don’t treat this as ‘freeze everything forever.’ I treat it like an integration/contract test for the parts you expect to stay stable unless you intentionally change them.

How I draw the line:

• Intentional change (new prompt/tool logic, different routing, updated behavior) → update the expected run.
• Unintentional drift (skipped tool, arg shape change, extra loops, missing fields) → CI should catch.

In practice teams keep the check small and stable (1–2 tool calls, a simple input), and focus on invariants rather than exact outputs:

• tool allowlist/order
• argument shape/required fields
• output schema/required fields
• budgets (wall time, tool call count/errors)

For volatility, you can strip/normalize fields like timestamps/IDs before comparing. So you’re not locking to every byte, you’re locking to the behavior that should not change without intent.