Docker stack: no suitable node (scheduling constraints not satisfied on 14 nodes; host-m… #6604
Description
Description
Docker stack cannot immediately update the created service.
no suitable node (scheduling constraints not satisfied on 14 nodes; host-m…
Reproduce
docker-compose.yml
services:
edge-de1:
image: test-private/test:1.1.3
ports:
- target: 80
published: 9090
mode: host
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.labels.location == de1
environment:
NODE_ID: 123
NODE_KEY: "123123123123123123123"
TEST: V1
configs:
- source: test_index_v1
target: /usr/share/nginx/html/index.html
- source: base_config_v1
target: /etc/xxxx/config.yml
edge-nl1:
image: test-private/test:1.1.3
ports:
- target: 80
published: 9090
mode: host
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.labels.location == nl1
environment:
NODE_ID: 1234
NODE_KEY: "1234123412341234123412341234"
TEST: V1
configs:
- source: test_index_v1
target: /usr/share/nginx/html/index.html
- source: base_config_v1
target: /etc/xxxx/config.yml
# ...
- You can imagine this as an edge service (cdn) that must listen on the port of the host machine it resides on.
- This is a configuration file used for testing; my private information has been hidden.
- They must all listen on the same port, such as 80 or 443; 9090 is used as an example here.
Create or update configuration
docker stack deploy test-edge -c docker-compose.yml
Observe a service that is experiencing problems
# docker service ps test-edge_edge-de1 --no-trunc
# Case 1 👇
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
3h5zqqyvs1qa1rstoy2x3x0ld test-edge_edge-de1.1 test-private/test:1.1.3@sha256:123123123 de1 Running Preparing about a minute ago
kx0noxsf3agizryq2qm6xrsqo \_ test-edge_edge-de1.1 nginx:latest@sha256:321321321 de1 Shutdown Shutdown about a minute ago
# docker service ps test-edge_edge-de1 --no-trunc
# Case 2 👇
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
vqf51hvogi7u5ng8mgas7f70n test-edge_edge-de1.1 test-private/test:1.1.3@sha256:123123123 Ready Pending 11 seconds ago "no suitable node (scheduling constraints not satisfied on 14 nodes; host-mode port already in use on 1 node)"
1vynsf8r1mnp3ojal1c44f8zi \_ test-edge_edge-de1.1 test-private/test:1.1.3@sha256:123123123 de1 Shutdown Preparing about a minute ago
It will not create an update service immediately.
I've confirmed there are no port conflicts.
The node status is fine.
# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
24o5yjnyh1kzmsm2f64w0hga3 de1 Ready Active 28.5.1
...
Try force
# docker service update --force test-edge_edge-de1
test-edge_edge-de1
overall progress: 0 out of 1 tasks
1/1: no suitable node (scheduling constraints not satisfied on 14 nodes; host-m…
^COperation continuing in background.
Use `docker service ps test-edge_edge-de1` to check progress.
Waiting for several minutes didn't help.
At this moment, I might realize that the old container has not been deleted, which could prevent the creation of a new container.
# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5f64ec2ced6c test-private/test:1.1.3 "zzz --config /etc…" 21 seconds ago Exited (2) 19 seconds ago test-edge_edge-de1.1.q6bnbhvdd0nbk25sjizhci4ui
362a7b42c2f2 test-private/test:1.1.3 "zzz --config /etc…" 2 minutes ago Exited (2) 2 minutes ago test-edge_edge-de1.1.aix9ldrqpdh4py4jiodbczpof
2d7635cdfb1b test-private/test:1.1.3 "zzz --config /etc…" 4 minutes ago Exited (2) 4 minutes ago test-edge_edge-de1.1.ihqj135vir56v00cp1s69jcuu
696a145c00e4 test-private/test:1.1.3 "zzz --config /etc…" 6 minutes ago Exited (2) 6 minutes ago test-edge_edge-de1.1.mrkuxvyooz1bw62ktqzw2v81f
285678652dea test-private/test:1.1.3 "zzz --config /etc…" 8 minutes ago Exited (2) 8 minutes ago test-edge_edge-de1.1.32setkw256zb37o6jbgcr7t15
# docker container prune
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Deleted Containers:
64d927e661793f3048476a1852e5448c10c731862b280b6d7e4b2ac335487135
576748781c0b0aa8d362f2bf5ce9ddb9c212b8b20ef8b9dbfc12225078c966b0
c08169e83beb97d63c1762992be0462f0ee9303934cb6fb349d672e824b84a90
5f64ec2ced6c85d23bafaac4693d9e71eae5167505a1e5a1c2a02ccca4bc72be
362a7b42c2f280cf08e120e0555900e71d35d7e9c48bc062e2af51b23c860f1c
2d7635cdfb1bdfeee3be40dceed94f8daad15054fe38b09d50611b555a32af81
8511e4f0f45cde3dfd0fc8dbeb59067ccb9d3dfca214c631121e61e01ca54311
Total reclaimed space: 0B
# docker service update --force test-edge_edge-de1
test-edge_edge-de1
overall progress: 0 out of 1 tasks
overall progress: 0 out of 1 tasks
overall progress: 0 out of 1 tasks
1/1: no suitable node (scheduling constraints not satisfied on 14 nodes; host-m…
^COperation continuing in background.
The docker service update command failed to create the service, It may never be created, or it may take a few minutes.
Therefore, I had to redeploy using stack.
# docker stack deploy test-edge -c docker-compose.yml --prune
Updating service test-edge_edge-de1 (id: w9gvl4iviv0lym2zqv1agvtu3)
...
# docker service ps test-edge_edge-de1 --no-trunc
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
gnzg47vc2o9x9zzmizketa5qb test-edge_edge-de1.1 test-private/test:1.1.3@sha256:123 de1 Running Preparing about a minute ago
itcdovxnw9sdpbkpfhgyk8729 \_ test-edge_edge-de1.1 test-private/test:1.1.3@sha256:123 de1 Shutdown Shutdown about a minute ago
- Use
--pruneto make it actively delete old containers.
It was successfully created 1.5 to 2 minutes later.
Strangely, I can create the service immediately using docker service create without waiting.
# docker service rm test-edge_edge-de1
test-edge_edge-de1
# docker service create \
--name edge-de1 \
--publish mode=host,target=80,published=9090 \
--constraint 'node.labels.location == de1' \
--env NODE_ID=123 \
--env NODE_KEY=123123123123123123123 \
--env TEST=V4 \
--config source=test-edge_test_index_v1,target=/usr/share/nginx/html/index.html \
--config source=test-edge_base_configv9,target=/etc/xxxx/config.yml \
test-private/test:1.1.3
nt1md6qp82hp2917v1s3358xt
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service nt1md6qp82hp2917v1s3358xt converged
It only takes 3-6 seconds to create.
Therefore, I suspect there's a difference in the internal processing mechanisms between docker stack / docker service update and docker service create. Why is it so slow?
The reason I care about this time is because services created using this mode cannot create multiple instances, nor can they use the tag order: start-first, which would cause excessively long service outages.
Expected behavior
I hope the container can be updated and created immediately.
Hope to optimize Docker stack
docker version
# manager
# docker version
Client:
Version: 28.5.1
API version: 1.51
Go version: go1.24.8
Git commit: e180ab8
Built: Wed Oct 8 12:16:30 2025
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 28.5.1
API version: 1.51 (minimum version 1.24)
Go version: go1.24.8
Git commit: f8215cc2
Built: Wed Oct 8 12:18:57 2025
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.7.28
GitCommit: b98a3aace656320842a23f4a392a33f46af97866
runc:
Version: 1.3.0
GitCommit: v1.3.0-0-g4ca628d
docker-init:
Version: 0.19.0
GitCommit: de40ad0
# node-de1
# docker version
Client:
Version: 28.5.1
API version: 1.51
Go version: go1.24.8
Git commit: e180ab8
Built: Wed Oct 8 12:16:30 2025
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 28.5.1
API version: 1.51 (minimum version 1.24)
Go version: go1.24.8
Git commit: f8215cc2
Built: Wed Oct 8 12:18:57 2025
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.25
GitCommit: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc:
Version: 1.3.0
GitCommit: v1.3.0-0-g4ca628d
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
# docker info
Client:
Version: 28.5.1
Context: default
Debug Mode: false
Server:
Containers: 12
Running: 11
Paused: 0
Stopped: 1
Images: 37
Server Version: 28.5.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: active
NodeID: hs2414gze7awxo16ldz6mruim
Is Manager: true
ClusterID: jqiah13jsiifckj3r0a37c26h
Managers: 4
Nodes: 15
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 1
Autolock Managers: false
Root Rotation In Progress: false
Node Address: manager-1-ip
Manager Addresses:
manager-4-ip:2377
manager-3-ip:2377
manager-2-ip:2377
manager-1-ip:2377
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b98a3aace656320842a23f4a392a33f46af97866
runc version: v1.3.0-0-g4ca628d
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.10.0-8-amd64
Operating System: Debian GNU/Linux 11 (bullseye)
OSType: linux
Architecture: x86_64
CPUs: 3
Total Memory: 2.425GiB
Name: seattle.zz
ID: a4632650-d7ac-45d2-a6d6-30e8dd2808c6
Docker Root Dir: /var/lib/docker
Debug Mode: false
Username: xxxxx
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false
Product License: Community EngineAdditional Info
- It should be noted that some servers in the cluster are VPC networks, and their public IP addresses are not directly bound to the network interface cards. However, the firewall allowed unimpeded access.
- The cluster is not an intranet, but rather servers distributed across different service providers.