-
Notifications
You must be signed in to change notification settings - Fork 5
Description
Seems like the EventConsumerStateStore could be used to keep track of consumer lag, since it knows about the consumer's current progress, often with more accuracy than the events in the state stream.
This implies there are two types of lag, one based on the latest persisted state and one based on the in-memory state. The in-memory state is more accurate but not yet durable, so a consumer could jump backwards if restarted.
We also need to work out what to do when we have multiple partitions and keep state per partition. Each consumer state store will have in-memory state only for the partitions the consumer instance has been allocated.
It would be simpler to calculate lags based only on persisted/durable state - maybe this is the best starting point.
Further, the state store doesn't know about the event source(s) that the consumer is consuming, so it can't work out what the latest sequence number per partition is. Should the state store store the event sources as part of the consumer state? If it did, would it know how to get the latest event in each source in a partitioned case? This probably indicates that it should be a different abstraction that can calculate the lags, which uses the EventConsumerStateStore to do so.