infinite redeployment...

[f18m]

Description

Hi,
I think I'm running again into the same issue described in #577
I.e. in the last 8hours or so, doco-cd is re-deploying the same stack over and over.

Steps to reproduce the issue

I entered in this loop just a few hours ago with the following steps:

1. I added to my .doco-cd.yaml a new docker-compose stack; see below for the Docker Compose and Deployment Config
2. I committed a few times on the file "config.yml" referenced by the docker-compose
3. See logs of infinite redeployment cycle

Operating System

Linux

Docker Compose File

services:
  homer:
    image: b4bz/homer:v25.11.1
    volumes:
      #- /home/landing-page:/www/assets # Make sure your local config directory exists
      - ./config.yml:/www/assets/config.yml # Make sure your local config directory exists
    ports:
      - 8080:8080
    user: 1000:1000 # default
    environment:
      - INIT_ASSETS=1 # default, requires the config directory to be writable for the container user (see user option)
    restart: unless-stopped

Deployment Config

name: landing-page
working_dir: config_files/MASTERMIND/landing-page
compose_files:
  - docker-compose.yml

Logs

{"time":"2025-11-20T00:19:05.755316522+01:00","level":"info","msg":"polling repository","repository":"github.com/f18m/my-ha-setup","job_id":"019a9e6a-1d14-78e7-b916-0c8cc3d05034","trigger":
{"event":"poll","config":{"CloneUrl":"https://github.com/f18m/my-ha-setup.git","Reference":"main","Interval":60,"CustomTarget":"","RunOnce":false,"Deployments":[]}}}
{"time":"2025-11-20T00:19:15.412829889+01:00","level":"info","msg":"deploying stack","repository":"github.com/f18m/my-ha-setup","job_id":"019a9e6a-1d14-78e7-b916-0c8cc3d05034","stack":"landing-page","reference":"main","repository":"github.com/f18m/my-ha-setup","stack":"landing-page","forced":false}
{"time":"2025-11-20T00:19:20.413477324+01:00","level":"info","msg":"deployment in progress","repository":"github.com/f18m/my-ha-setup","job_id":"019a9e6a-1d14-78e7-b916-0c8cc3d05034","stack":"landing-page","reference":"main","repository":"github.com/f18m/my-ha-setup","stack":"landing-page"}
{"time":"2025-11-20T00:19:21.38162942+01:00","level":"info","msg":"job completed successfully","repository":"github.com/f18m/my-ha-setup","job_id":"019a9e6a-1d14-78e7-b916-0c8cc3d05034","elapsed_time":"15.633s","next_run":"2025-11-20T00:20:21+01:00"}
{"time":"2025-11-20T00:20:21.409677382+01:00","level":"info","msg":"polling repository","repository":"github.com/f18m/my-ha-setup","job_id":"019a9e6b-4496-7e78-b71f-5426da1421c1","trigger":{"event":"poll","config":{"CloneUrl":"https://github.com/f18m/my-ha-setup.git","Reference":"main","Interval":60,"CustomTarget":"","RunOnce":false,"Deployments":[]}}}
{"time":"2025-11-20T00:20:30.254152938+01:00","level":"info","msg":"job completed successfully","repository":"github.com/f18m/my-ha-setup","job_id":"019a9e6b-4496-7e78-b71f-5426da1421c1","elapsed_time":"8.855s","next_run":"2025-11-20T00:21:30+01:00"}
{"time":"2025-11-20T00:21:30.293130225+01:00","level":"info","msg":"polling repository","repository":"github.com/f18m/my-ha-setup","job_id":"019a9e6c-51a7-770f-8cf2-39cb5d1b8f25","trigger":{"event":"poll","config":{"CloneUrl":"https://github.com/f18m/my-ha-setup.git","Reference":"main","Interval":60,"CustomTarget":"","RunOnce":false,"Deployments":[]}}}
{"time":"2025-11-20T00:21:39.097116873+01:00","level":"info","msg":"job completed successfully","repository":"github.com/f18m/my-ha-setup","job_id":"019a9e6c-51a7-770f-8cf2-39cb5d1b8f25","elapsed_time":"8.817s","next_run":"2025-11-20T00:22:39+01:00"}
{"time":"2025-11-20T00:22:39.110812875+01:00","level":"info","msg":"polling repository","repository":"github.com/f18m/my-ha-setup","job_id":"019a9e6d-5e7a-7e71-a472-16acd810d088","trigger":{"event":"poll","config":{"CloneUrl":"https://github.com/f18m/my-ha-setup.git","Reference":"main","Interval":60,"CustomTarget":"","RunOnce":false,"Deployments":[]}}}
{"time":"2025-11-20T00:22:47.791500956+01:00","level":"info","msg":"job completed successfully","repository":"github.com/f18m/my-ha-setup","job_id":"019a9e6d-5e7a-7e71-a472-16acd810d088","elapsed_time":"8.692s","next_run":"2025-11-20T00:23:47+01:00"}

[f18m]

Additional infos: docker describe of the app container is:

            "Labels": {
                "cd.doco.deployment.external_secrets_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
                "cd.doco.deployment.name": "landing-page",
                "cd.doco.deployment.target.ref": "main",
                "cd.doco.deployment.target.sha": "8e58d71ffbc83bf6563b311a392ef0d826c2e3d1",
                "cd.doco.deployment.timestamp": "2025-11-19T23:38:51Z",
                "cd.doco.deployment.trigger": "poll",
                "cd.doco.deployment.working_dir": "/home/doco-cd/data/github.com/f18m/my-ha-setup/config_files/MASTERMIND/landing-page",
                "cd.doco.metadata.manager": "doco-cd",
                "cd.doco.metadata.version": "v0.49.0",
                "cd.doco.repository.name": "f18m/my-ha-setup",
                "cd.doco.repository.url": "https://github.com/f18m/my-ha-setup.git",

Another consideration to reproduce the bug:

• doco-cd is using polling on the repo with docker-compose configs, not webhooks
• using doco-cd REST API "/v1/api/project/landing-page/stop" endpoint has the effect of temporarily stopping the infinite redeployment cycle and then when "/v1/api/project/landing-page/start" is used, the infinite redeployment restarts

I will leave my setup unaltered for now, so that if @kimdre you want me to grab any additional debug info, I can do that.

[kimdre]

Is doco-cd actually redeploying the container or does the container stay untouched (like does it keep running without restarts or anything)?
Can you show me the debug log for this situation LOG_LEVEL: debug?

[kimdre]

If the logs you provided I can see only one poll job that actually triggered a deployment, other than that all poll jobs ended before that.

[f18m]

Is doco-cd actually redeploying the container or does the container stay untouched (like does it keep running without restarts or anything)?

actually the container stays untouched

Can you show me the debug log for this situation LOG_LEVEL: debug?

here it is:

{"time":"2025-11-20T10:18:37.522959812+01:00","level":"debug","msg":"Start poll job","repository":"github.com/f18m/my-ha-setup"}
{"time":"2025-11-20T10:18:37.535724445+01:00","level":"info","msg":"polling repository","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0","trigger":{"event":"poll","config":{"CloneUrl":"https://github.com/f18m/my-ha-setup.git","Reference":"main","Interval":60,"CustomTarget":"","RunOnce":false,"Deployments":[]}}}
{"time":"2025-11-20T10:18:37.535756837+01:00","level":"debug","msg":"get repository","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0","url":"https://github.com/f18m/my-ha-setup.git"}
{"time":"2025-11-20T10:18:37.535790956+01:00","level":"debug","msg":"cloning repository","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0","container_path":"/data/github.com/f18m/my-ha-setup","host_path":"/data/github.com/f18m/my-ha-setup"}
{"time":"2025-11-20T10:18:37.536036416+01:00","level":"debug","msg":"repository already exists, checking out reference main","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0","host_path":"/data/github.com/f18m/my-ha-setup"}
{"time":"2025-11-20T10:18:38.338670354+01:00","level":"debug","msg":"retrieving deployment configuration","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0"}
{"time":"2025-11-20T10:18:38.339186372+01:00","level":"debug","msg":"deployment configuration retrieved","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0","stack":"psmqtt","reference":"main","repository":"github.com/f18m/my-ha-setup","config":{"Name":"psmqtt","RepositoryUrl":"","WebhookEventFilter":"","Reference":"main","WorkingDirectory":"config_files/MASTERMIND/psmqtt","ComposeFiles":["docker-compose.yml"],"EnvFiles":[".env"],"RemoveOrphans":true,"ForceRecreate":true,"ForceImagePull":false,"Timeout":180,"BuildOpts":{"ForceImagePull":false,"Quiet":false,"Args":null,"NoCache":false},"Destroy":false,"DestroyOpts":{"RemoveVolumes":true,"RemoveImages":true,"RemoveRepoDir":true},"Profiles":[],"ExternalSecrets":null}}
{"time":"2025-11-20T10:18:38.339213236+01:00","level":"debug","msg":"checking out reference main","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0","stack":"psmqtt","reference":"main","repository":"github.com/f18m/my-ha-setup","host_path":"/home/doco-cd/data/github.com/f18m/my-ha-setup"}
{"time":"2025-11-20T10:18:39.086098863+01:00","level":"debug","msg":"comparing commits","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0","stack":"psmqtt","reference":"main","repository":"github.com/f18m/my-ha-setup","deployed_commit":"9144b4d45d81343b5287f697fc4bac2bdfe4ddda","latest_commit":"65c8ee00542e3fc032e7174c5a8d38b5f5c97929"}
{"time":"2025-11-20T10:18:39.236265254+01:00","level":"debug","msg":"no changes detected in subdirectory, skipping deployment","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0","directory":"config_files/MASTERMIND/psmqtt","last_commit":"65c8ee00542e3fc032e7174c5a8d38b5f5c97929","deployed_commit":"9144b4d45d81343b5287f697fc4bac2bdfe4ddda"}
.... <other stacks monitoring > ...
{"time":"2025-11-20T10:18:45.623812395+01:00","level":"debug","msg":"deployment configuration retrieved","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0","stack":"landing-page","reference":"main","repository":"github.com/f18m/my-ha-setup","config":{"Name":"landing-page","RepositoryUrl":"","WebhookEventFilter":"","Reference":"main","WorkingDirectory":"config_files/MASTERMIND/landing-page","ComposeFiles":["docker-compose.yml"],"EnvFiles":[".env"],"RemoveOrphans":true,"ForceRecreate":false,"ForceImagePull":false,"Timeout":180,"BuildOpts":{"ForceImagePull":false,"Quiet":false,"Args":null,"NoCache":false},"Destroy":false,"DestroyOpts":{"RemoveVolumes":true,"RemoveImages":true,"RemoveRepoDir":true},"Profiles":[],"ExternalSecrets":null}}
{"time":"2025-11-20T10:18:45.623848438+01:00","level":"debug","msg":"checking out reference main","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0","stack":"landing-page","reference":"main","repository":"github.com/f18m/my-ha-setup","host_path":"/home/doco-cd/data/github.com/f18m/my-ha-setup"}
{"time":"2025-11-20T10:18:46.337923229+01:00","level":"debug","msg":"comparing commits","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0","stack":"landing-page","reference":"main","repository":"github.com/f18m/my-ha-setup","deployed_commit":"8e58d71ffbc83bf6563b311a392ef0d826c2e3d1","latest_commit":"65c8ee00542e3fc032e7174c5a8d38b5f5c97929"}
{"time":"2025-11-20T10:18:46.343431516+01:00","level":"debug","msg":"changes detected in subdirectory, proceeding with deployment","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0","stack":"landing-page","reference":"main","repository":"github.com/f18m/my-ha-setup","directory":"config_files/MASTERMIND/landing-page","last_commit":"65c8ee00542e3fc032e7174c5a8d38b5f5c97929","deployed_commit":"8e58d71ffbc83bf6563b311a392ef0d826c2e3d1"}
{"time":"2025-11-20T10:18:46.354596881+01:00","level":"info","msg":"deploying stack","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0","stack":"landing-page","reference":"main","repository":"github.com/f18m/my-ha-setup","stack":"landing-page","forced":false}
{"time":"2025-11-20T10:18:47.01279788+01:00","level":"info","msg":"job completed successfully","repository":"github.com/f18m/my-ha-setup","job_id":"019aa08e-ffd2-7e74-9644-41c59d15e9b0","elapsed_time":"9.489s","next_run":"2025-11-20T10:19:47+01:00"}

[f18m]

oh and now I realized that for my other docker compose stacks I adopted the

force_recreate: true

setting as workaround for #577 but I forgot to put the same workaround in place for this new "landing-page" stack...

But OTOH I think the bug is there and probably many other users will need to discover how to workaround it... :(

[kimdre]

Maybe I need to implement some sort of detection for this that detects when doco-cd tries to redeploy without success. I could imagine that this is caused by the commit sha label of the container that does not match the current commit sha of the repository while everything else is already up-to-date.

[f18m]

Maybe I need to implement some sort of detection for this that detects when doco-cd tries to redeploy without success.

from my reading of the logs the redeploy actually "succeeded" but the docker compose layer decided that there was no need to recreate the container at all right?
So on the next cycle doco-cd still thinks there is a drift and the need to re-deploy again.
Maybe doco-cd could just create an (hidden?) file next to the docker-compose and store there the latest-applied commit sha that will override the commit-sha read from the container's label (only when the file is there?).
Because the problem (to my understanding) is that Docker compose will not update a label of a container that is not recreated and, at the same time, does not provide any other way to attach some metadata to a container... so using the disk to store this info might be a way to work around it. Or maybe a sqlite DB.
Just my 2 cents :)

[kimdre]

from my reading of the logs the redeploy actually "succeeded" but the docker compose layer decided that there was no need to recreate the container at all right?

Yes that is correct. I use the container labels to store the current deployment state because I want to avoid using a file or database for this. Usually, if no redeployment is required, even if the commit sha does not align with the container label, doco-cd will skip the deployment logic. But in your case some files have changed, which causes doco-cd to trigger the deployment but the container itself did not change. So this might by the reason why it's now looping like this.