You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm not fully sure if I'm understanding the end to end acknowledgement feature correctly. We are currently running vector on two instances, lets call them system A and system B. System A reads CSV lines from files and sends them via a http sink to a vector instance running on system B. In this vector instance the data is parsed and then sent to an Elasticsearch database. We want to ensure that even if system A or B crashes or vector restarts no data is being lost. We were thinking to enable disk buffers instead of in memory buffers on both systems. However if I understand it correctly, end to end acknowledgements would already ensure, that no data is lost. Meaning if we use in memory buffers and the system crashes, the CSV lines will be resent as the buffer was lost, but the events haven't been acknowledged yet.
Is this understanding of end to end acknowledgements correct, that we can use in conjunction with in memory buffer while ensuring that each CSV line has been sent at least once to the Elasticsearch database?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
Question
I'm not fully sure if I'm understanding the end to end acknowledgement feature correctly. We are currently running vector on two instances, lets call them system A and system B. System A reads CSV lines from files and sends them via a http sink to a vector instance running on system B. In this vector instance the data is parsed and then sent to an Elasticsearch database. We want to ensure that even if system A or B crashes or vector restarts no data is being lost. We were thinking to enable disk buffers instead of in memory buffers on both systems. However if I understand it correctly, end to end acknowledgements would already ensure, that no data is lost. Meaning if we use in memory buffers and the system crashes, the CSV lines will be resent as the buffer was lost, but the events haven't been acknowledged yet.
Is this understanding of end to end acknowledgements correct, that we can use in conjunction with in memory buffer while ensuring that each CSV line has been sent at least once to the Elasticsearch database?
Vector Config
No response
Vector Logs
No response
Beta Was this translation helpful? Give feedback.
All reactions