CRL validation in Thanos Receive

[4xoc]

This topic is about a general discussion about CRL support in Thanos, specifically the receive component. Since Golang doesn't validate CRLs by default and has no built-in "here's a cert, check if that's revoked" function the question is if and how CRL validation could be implemented. Going further OCSP(-Stapling) would be another question that comes up with the whole idea of revocation handling. Running a private PKI with Thanos receive is a major feature in my opinion (clear client identification, mutual authentication, state of the art security concept, allows for zero-trust approaches). There are certainly a lot of things to consider and running a private PKI is no simple task. However, we have the functionality right now, and with every certificate created, I believe there must be a way to revoke it.

Now, one suggestion has been for some time now to use short-lasting certificates (like Let's Encrypt does). That is however not practical in many cases, increases the PKI complexity even more and worst of all: it doesn't solve the problem with keypairs being "lost" and used for malicious intent. Considering that monitoring is a mayor component of any system and an attack on it would cause one to essentially fly blind. While (d)DoS is a different discussion, worse than no service is a service one cannot trust. Image an attacker get's hold of a keypair and sends incorrect metrics to Thanos. Suddenly the whole monitoring strategy goes out the window because one cannot trust metrics coming in. Alerts as such shouldn't be affected when Prometheus still takes care of that itself, but still, such attack could waste a lot of resources chasing after bad values that are not really there. Short lived certificates would just reduce the time frame such attack works in, but it doesn't prevent such attack and most certainly cannot mitigate it. Using some revocation strategy however, would allow to revoke a cert used by an attacker and be an effective mitigation.

So coming back to the original question of this discussion: What would be a good way to go forward?