pod in crashloop back

[julienroussel00]

I tried installing imageswap on one of the k8s cluster im in charge with and im getting the following error message (see logs below)...
I can see the certificat request status is approved.

NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
imageswap.imageswap-system.cert-request 4m12s kubernetes.io/kubelet-serving system:serviceaccount:imageswap-system:imageswap-sa Approved

[2022-11-11 19:38:49,691] INFO: ImageSwap Init
[2022-11-11 19:38:49,691] INFO: Starting TLS init process
[2022-11-11 19:38:49,715] INFO: Did not find secret "imageswap-tls" in the "imageswap-system" namespace
[2022-11-11 19:38:49,715] INFO: Generating new cert/key pair for TLS
[2022-11-11 19:38:49,764] INFO: Building K8s CSR
[2022-11-11 19:38:49,767] INFO: Looking for existing CSR
[2022-11-11 19:38:49,788] INFO: Deleting k8s csr
[2022-11-11 19:38:49,808] INFO: Existing certificate request deleted
[2022-11-11 19:38:49,809] INFO: Create k8s CSR
[2022-11-11 19:38:49,831] INFO: Certificate signing request "imageswap.imageswap-system.cert-request" has been created
[2022-11-11 19:38:49,846] INFO: Patch k8s CSR: imageswap.imageswap-system.cert-request
[2022-11-11 19:38:49,864] INFO: Certificate signing request "imageswap.imageswap-system.cert-request" is approved
[2022-11-11 19:38:49,879] INFO: Waiting for certificate approval
[2022-11-11 19:38:49,887] INFO: Waiting for certificate approval
......
[2022-11-11 19:38:54,832] INFO: Waiting for certificate approval
[2022-11-11 19:38:54,839] INFO: Waiting for certificate approval
[2022-11-11 19:38:54,849] INFO: Waiting for certificate approval
[2022-11-11 19:38:54,858] INFO: Waiting for certificate approval
[2022-11-11 19:38:54,868] INFO: Waiting for certificate approval
[2022-11-11 19:38:54,868] INFO: Timed out reading certificate request "imageswap.imageswap-system.cert-request"
Traceback (most recent call last):
File "/app/imageswap-init.py", line 1201, in
main()
File "/app/imageswap-init.py", line 1190, in main
init_tls_pair(imageswap_namespace_name)
File "/app/imageswap-init.py", line 677, in init_tls_pair
tls_pair = build_tls_pair(
File "/app/imageswap-init.py", line 347, in build_tls_pair
tls_cert_pem = get_tls_cert_from_request(
File "/app/imageswap-init.py", line 319, in get_tls_cert_from_request
tls_cert = base64.b64decode(k8s_csr.status.certificate)
File "/usr/local/lib/python3.10/base64.py", line 80, in b64decode
s = _bytes_from_decode_data(s)
File "/usr/local/lib/python3.10/base64.py", line 45, in _bytes_from_decode_data
raise TypeError("argument should be a bytes-like object or ASCII "
TypeError: argument should be a bytes-like object or ASCII string, not 'NoneType'

[phenixblue]

Can you set logging to debug and post the logs from that? Also, did you check the contents of the certificate?

[dex4er]

I have the same error I believe EKS cannot issue the certificate if CertificateSigningRequest does not start with csr- prefix. Unfortunately its name is imageswap.imageswap-system.cert-request so it stuck on Approved.

[phenixblue]

This issue seems to indicate the opposite...that containing the csr- prefix will prevent the certificate from being issued.

aws/containers-roadmap#2079

[dex4er]

Ah so as https://docs.aws.amazon.com/eks/latest/userguide/cert-signing.html explains, application cannot use signerName: kubernetes.io/kubelet-serving. There is however additional signerName: beta.eks.amazonaws.com/app-serving

I can confirm that after installing this kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - https://raw.githubusercontent.com/phenixblue/imageswap-webhook/master/deploy/install.yaml
patches:
  - patch: |-
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: imageswap-env
        namespace: imageswap-system
      data:
        IMAGESWAP_CSR_SIGNER_NAME: beta.eks.amazonaws.com/app-serving

the app starts and certificate is approved and issued.