Permission Denied 403 (Forbidden) error in Thanos Querier when using CNAME record URL to add Thanos Sidecar of another Cluster running Prometheus with Thanos

[satyadevareti]

We have 2 EKS Clusters deployed on AWS. One cluster which we are using as an Observer Cluster is running Thanos Querier. And the other cluster running as an Observee cluster, has Prometheus (with 2 replicas) along with Thanos as a Sidecar deployed inside an Istio Service Mesh.

On the Observee Cluster:
We are using Headless service (named prometheus-thanos-headless-service) in the monitoring namespace, to expose these Sidecars as specified below. Service Entries and Virtual Services for each of the Prometheus-Thanos replicas are created as mentioned below with hosts value as respective DNS names.

prometheus-thanos-0.prometheus-thanos-headless-service.monitoring.svc.cluster.local

For each sidecar the following Service Entry and Virtual Service are created.

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: prometheus-se-0
  namespace: monitoring
spec:
  hosts:
  - prometheus-thanos-0.prometheus-thanos-headless-service.monitoring.svc.cluster.local
  location: MESH_INTERNAL
  ports:
  - name: grpc-thanos-querier
    number: 10901
    protocol: GRPC
  resolution: DNS
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: prometheus-server-0
  namespace: monitoring
spec:
  gateways:
  - monitoring/https-gateway-tls
  hosts:
  - prometheus-thanos-0.prometheus-thanos-headless-service.monitoring.svc.cluster.local
  http:
  - route:
     - destination:
         host: prometheus-thanos-0.prometheus-thanos-headless-service.monitoring.svc.cluster.local
         port:
           number: 10901

On the Observer Cluster :
On the Observer cluster, Thanos Querier is deployed with envoy as its sidecar. Using envoy as sidecar is to enable TLS communication between the Observer and Observee clusters as mentioned in the blog below.

https://thanos.io/tip/operating/cross-cluster-tls-communication.md/

To add the Sidecars of the Observee Cluster as stores to the Thanos Querier, in the envoy-config.yaml, the following configuration was used.

static_resources:
   listeners:
   - name: lsnr-for-sidecar-0
     address:
        socket_address: { address: 0.0.0.0, port_value: <envoy-service's Port Number for sidecar 0> }
     filter_chains:
     - filters:
       - name: envoy.http_connection_manager
         typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            codec_type: AUTO
            stat_prefix: ingress_http
            http_filters:
            - name: envoy.filters.http.router
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
            route_config:
               name: local_route
               virtual_hosts:
               - name: local_service
                 domains: ["*"]
                 routes:
                 - match:
                    prefix: "/"
                    grpc: {}
                   route:
                     host_rewrite_literal: prometheus-thanos-0.prometheus-thanos-headless-service.monitoring.svc.cluster.local
                     cluster: cluster-sidecar-1
   - name: lsnr-for-sidecar-1
     address:
        socket_address: { address: 0.0.0.0, port_value: <envoy-service's Port Number for sidecar 1> }
     filter_chains:
     - filters:
       - name: envoy.http_connection_manager
         typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            codec_type: AUTO
            stat_prefix: ingress_http
            http_filters:
            - name: envoy.filters.http.router
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
            route_config:
               name: local_route
               virtual_hosts:
               - name: local_service
                 domains: ["*"]
                 routes:
                 - match:
                    prefix: "/"
                    grpc: {}
                   route:
                     host_rewrite_literal: prometheus-thanos-1.prometheus-thanos-headless-service.monitoring.svc.cluster.local
                     cluster: cluster-sidecar-1                         
   clusters:
   - name: cluster-sidecar-1
     connect_timeout: 30s
     type: STRICT_DNS
     http2_protocol_options: {}
     dns_lookup_family: V4_ONLY
     load_assignment:
       cluster_name: cluster-sidecar-1
       endpoints:
       - lb_endpoints:
         - endpoint:
             address:
              socket_address:
                address: <Observee Cluster's IngressGateway Service's Load Balancer URL>
                port_value: 443
     transport_socket:
       name: envoy.transport_sockets.tls
       typed_config:
         "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
         sni: <Observee Cluster's IngressGateway Service's Load Balancer URL>
         common_tls_context:
           tls_certificates:
             - certificate_chain:
                 filename: /certs/tls.crt
               private_key:
                 filename: /certs/tls.key
           alpn_protocols:
           - "h2"
           - "http/1.1"

When the above configuration is being used, with the Load Balancer URL the both the sidecars of the Observee cluster are getting added to the Thanos Querier as expected.

The issue we are facing is when replacing the Load Balancer URL specified in the above config, with the CNAME Record URL which is mapped to the same Load Balancer URL. The sidecars are not getting added to the Thanos Querier and the logs of the Thanos Querier container show the error below.

rpc error: code = PermissionDenied desc = unexpected HTTP status code received from server: 403 (Forbidden); transport: received unexpected content-type "text/html"" address=172.20.87.222:10001

The corresponding logs of the envoy-sidecar container show the error below,

[2022-06-14 03:42:34.174][22][debug][http] [source/common/http/conn_manager_impl.cc:1435] [C22181][S4225530000374947114] encoding headers via codec (end_stream=false):
':status', '403'
'server', 'envoy'
'date', 'Tue, 14 Jun 2022 03:42:34 GMT'
'content-type', 'text/html'
'content-length', '151'
'cf-ray', '71b00e87992e8558-BOM'
'x-envoy-upstream-service-time', '1'