Skip to content

Conversation

@neithanmo
Copy link
Collaborator

Problem

The Gateway routes queries to different indexers that may be synced to different blocks. When an indexer is significantly behind (sometimes months or years), it returns data that is correct for its block but stale from the client's perspective. This causes allocation counts to fluctuate wildly:

17:39:03  allocations: 502
17:40:02  allocations: 0    ← routed to 1.5-year-old indexer
17:43:03  allocations: 110  ← routed to months-old indexer
17:44:03  allocations: 512

Solution

Add a configurable staleness check that validates the _meta.block.timestamp from network subgraph responses. Data is accepted only if:

  • It's fresh (within the configured threshold), OR
  • It's stale but fresher than the current best data (progressive improvement)

This ensures:

  • Service always starts (any data is accepted on init)
  • Fresh data always wins
  • Stale data can't replace better data
  • Gradual improvement toward fresh data when all indexers are behind

Configuration

[subgraphs.network]
# Maximum allowed age of data in minutes. Set to 0 to disable.
# Default: 30
max_data_staleness_mins = 30

Testing

Added unit tests covering:

  • Fresh data acceptance
  • Stale data rejection (when not an improvement)
  • Stale data acceptance (when fresher than current)
  • Init behavior (accepts any valid data)
  • Edge cases (missing/invalid timestamps)

@coveralls
Copy link

coveralls commented Jan 30, 2026

Pull Request Test Coverage Report for Build 21525823621

Details

  • 112 of 171 (65.5%) changed or added relevant lines in 4 files are covered.
  • 3 unchanged lines in 2 files lost coverage.
  • Overall coverage increased (+0.03%) to 68.484%

Changes Missing Coverage Covered Lines Changed/Added Lines %
crates/service/src/service/router.rs 0 1 0.0%
crates/config/src/config.rs 0 2 0.0%
crates/tap-agent/src/agent.rs 0 2 0.0%
crates/monitor/src/allocations.rs 112 166 67.47%
Files with Coverage Reduction New Missed Lines %
crates/config/src/config.rs 1 79.67%
crates/monitor/src/allocations.rs 2 46.09%
Totals Coverage Status
Change from base Build 21485319913: 0.03%
Covered Lines: 10500
Relevant Lines: 15332

💛 - Coveralls

@neithanmo neithanmo marked this pull request as ready for review January 30, 2026 18:38
/// Fetches allocations from the network subgraph with metadata for freshness validation.
///
/// This function does NOT perform staleness validation - that is the caller's responsibility.
/// Use `validate_freshness()` to check if the response is fresh enough.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Use `validate_freshness()` to check if the response is fresh enough.
/// Use `check_and_update_freshness()` to check if the response is fresh enough.

Right?

Prevent stale network subgraph data from replacing fresher data when the
Gateway routes queries to indexers synced to different blocks.

Acceptance rules:
- Fresh data (within max_data_staleness_mins): always accepted
- Stale but fresher than current best: accepted (improvement)
- Stale and not fresher than current: rejected

This ensures:
- Service always starts (any data beats initial state)
- Stale-but-better data progressively improves state
- Fresh data always wins (even with 0 allocations)
- Bad data never replaces good data

Add AllocationQueryResponse struct with block metadata, AtomicI64 to
track best known timestamp, and check_and_update_freshness() for
validation logic.
@neithanmo neithanmo enabled auto-merge (squash) February 2, 2026 22:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants