Skip to content

Allow consumer lags to be calculated #55

@tobyclemson

Description

@tobyclemson

Seems like the EventConsumerStateStore could be used to keep track of consumer lag, since it knows about the consumer's current progress, often with more accuracy than the events in the state stream.

This implies there are two types of lag, one based on the latest persisted state and one based on the in-memory state. The in-memory state is more accurate but not yet durable, so a consumer could jump backwards if restarted.

We also need to work out what to do when we have multiple partitions and keep state per partition. Each consumer state store will have in-memory state only for the partitions the consumer instance has been allocated.

It would be simpler to calculate lags based only on persisted/durable state - maybe this is the best starting point.

Further, the state store doesn't know about the event source(s) that the consumer is consuming, so it can't work out what the latest sequence number per partition is. Should the state store store the event sources as part of the consumer state? If it did, would it know how to get the latest event in each source in a partitioned case? This probably indicates that it should be a different abstraction that can calculate the lags, which uses the EventConsumerStateStore to do so.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions