Trovilo crashes without helpful error message / cause

[frittentheke]

After running for hours without any issue, Trovilo sometimes crashes with a very short error message like:

"{"error":"EOF","level":"fatal","msg":"Kubernetes ConfigMap watcher encountered an error. Exit..","time":"2018-08-30T16:04:31Z"}"

Unfortunately there is no indication of what could have causes this and after the container is restarted it works again for a long time until it crashes in the same matter again. While I cannot rule out an influence from consumed configmaps, I am very certain this is not causes but a change in the configmaps. It also happens when Kubernetes resources are not changed at all over a longer period of time.

Maybe some timeout when talking to the API which is not handled well?

[johscheuer]

The log message comes from the part: https://github.com/inovex/trovilo/blob/master/cmd/trovilo/main.go#L82-L85 the EOF looks like API Server ist not available for period or maybe your API servers switch the leadership. In order to prevent this a new watch should be created (like it's described in the godocs: https://godoc.org/github.com/ericchiang/k8s#Watcher.Next).

From a perspective of simple code it's actually a total valid way to exit the process and let Kubernetes take care of the restart. Otherwise the code must be improved to be more resilient and handle the reconnect if the watch fails.

In the current status of trovilo I think we should at-least document why this (can) happen.

[frittentheke]

@johscheuer
I do not think Kubernetes is there to handle an application having to reconnect to some endpoint by going through thousands of container restarts. Sure this will cause a reconnect, but makes it very hard to distinguish between an actual issue with an application causing restarts or an expected thing happening.

The words "exception handling" come to mind ;-) If you do NOT know how to handle something, you better crash, but if it's totally expected, please handle it.

[hikhvar]

This should be fixed. Together with #14. However there is a known workaround. Thus there is no point in wasting energy by debating here.

[johscheuer]

@frittentheke did you see this effect: #21 ?