bug: Missing or delayed /solve auction requests

[xozzslip]

Problem

Missing or delayed /solve auction requests

Impact

Our solver misses up to 5–10% of auctions during certain hours. This results in lost opportunities to compete and missed rewards. We miss multiple auctions with potentially strong bids on a daily basis.

Expected behaviour

• Receive 100% of /solve requests for eligible auctions.
• For each request, receive it with ≥ 8 seconds remaining before the solve deadline.

Additional context

We are running our solver on the Base network. Over the last 24 hours we missed multiple auctions where we could have provided competitive bids but did not due to this issue.

Example auction IDs:
26617492 — not received at all
26619294 — not received at all
26620139 — received, but too late to participate effectively

For auction 26620139, the first log line emitted by our /solve handler is:
2026-01-19T04:07:34.211Z DEBUG driver::infra::api::routes::solve: new /solve request: {"id":"26620139","tokens":[{"address":"0x6cd905df2ed214b22e0d...

This log is written immediately upon request receipt; there is no prior processing in the handler before this line. Comparing timestamps:

2026-01-19T04:07:34.211Z— request received
2026-01-19T04:07:34.337221708Z — solve deadline

This leaves approximately 120 ms before the deadline, which is effectively insufficient time to compute and submit a solution.

[xozzslip]

One potential source of the delay is repeated JSON serialization per driver.

In request_response the request body is built like this:

let mut request = {
    let builder = self.client.post(url.clone()).headers(tracing_headers());
    tokio::task::spawn_blocking(move || builder.json(&request))
        .await
        .context("failed to build request")?
};

With ~30 drivers and a ~5–10 MB /solve payload, this results in ~30 large JSON serializations per auction. Even though it’s offloaded to spawn_blocking, those tasks can queue on the blocking pool and delay when each HTTP request is actually constructed and sent (and it also adds significant CPU/alloc pressure).

Suggestion: serialize the /solve request once per auction, and reuse the same serialized bytes for every driver request.

[squadgazzz]

@xozzslip thanks for reporting this!

Our solver

Could you clarify which solver you are referring to?

[xozzslip]

BRRRolver on Base

[MartinquaXD]

One potential source of the delay is repeated JSON serialization per driver.

This was indeed the case originally but was resolved with #3391 half a year ago. So for the /solve request the logic that you linked really just copies the raw bytes from 1 shared request JSON that was serialized once.
I'll investigate further.

[MartinquaXD]

For auction 26620139 all drivers running in our infra saw the auction within 200ms. While this does not disprove that the autopilot could have been overloaded it makes it seem unlikely because there is not much else going on while the /solve requests are being sent out. Additionally the base autopilot has a lot less going on than mainnet and we are not aware of such reports on mainnet.
Is it possible that the ingest on your end is flaky or overloaded?

[xozzslip]

Hey! Thanks for your reply @MartinquaXD — I really appreciate it.

So for the /solve request the logic that you linked really just copies the raw bytes from 1 shared request JSON that was serialized once.

Oh okay, that’s great — I missed that.

Is it possible that the ingest on your end is flaky or overloaded?

That’s definitely possible. The issue could be on our side, but so far we haven’t found anything obvious. We’re running on a fairly powerful AWS instance in a North America region. Latency from our Europe servers is around ~200 ms, but it’s quite stable with a low packet loss rate. Our iperf tests also look decent. That said, we’ll continue investigating on our side.

For auction 26620139 all drivers running in our infra saw the auction within 200ms

Do you happen to have a metric from your drivers (for example, p95 over a 1-hour window) for how early /solve requests are received relative to the solve deadline? Here is what our metric looks like during the last 24 hours:

We receive the vast majority of auctions on time, but during one hourly window the tail degraded significantly: roughly 5% of auctions arrived with <500 ms remaining before the solve deadline.

One hypothesis is that physical distance between autopilot and our driver contributes to the tail latency. If that’s the case, drivers running in your infra would naturally be less affected.

Would you be open to trying the following changes on your side?

1. Keep the TCP connection alive between auctions

client: Client::builder()
    .timeout(RESPONSE_TIME_LIMIT)
    .tcp_keepalive(Some(Duration::from_secs(60)))
    .build()
    .map_err(Error::FailedToBuildClient)?,

2. Upgrade the reqwest library. Autopilot currently uses a release from March 19, 2024, and there have been many improvements since then. For example, TCP keepalive is enabled by default (with retries) in fix: add default tcp keepalive and user_timeout values seanmonstar/reqwest#2780.

[MartinquaXD]

At the moment we only have heatmap metrics measuring how much time of the deadline is left when the driver forwards the request to the solver. Also this metric does not discriminate between solving actual auctions and quoting (hence the 2 distinct rows in the graph) so it's not completely apples to apples but while there are definitely a couple of outliers there does not appear to be a period with clearly degraded latency.

It's possible that the measurements of quote requests (lower row) hide measurements of extremely slow solve requests but I would expect more of a gradient rather than seeing only regularly fast requests or very slow requests.

I'll update the metrics to distinguish between solve and quote requests and also use a simple counter instead of a histogram to more easily get p95 metrics.

[xozzslip]

I'll update the metrics to distinguish between solve and quote requests and also use a simple counter instead of a histogram to more easily get p95 metrics.

Great, thank you. Btw, we use histograms for p95 metrics:

histogram_quantile(0.05, sum by (le) (rate(driver_remaining_time_after_bucket{event="Request"}[60m])))

[xozzslip]

Hey @MartinquaXD, have you had a chance to check the p95 (or ideally p99) “time until deadline left” metric for your infrastructure? TCP keepalive unfortunately didn’t result in a noticeable improvement.