[WORKFLOW SDK BUG] Activity concurrency is throttled too much

[olitomlinson]

runtime
1.17.rc2

SDK
dapr>=1.17.0rc2
dapr-ext-workflow>=1.17.0rc2

Expected Behavior

Activities are allowed to run concurrently

Actual Behavior

The SDK is throttling how many activities can be ran at any one point time

Steps to Reproduce the Problem

Create a workflow, fan out 100 instances of Activities, have each activity sleep for 60 seconds before completing.

Look at the logs and notice that there are at most 10-15 activities running at the same time. Despite all Activities being scheduled at the same time, Activities won't Activate untill previous activities have finished.

Look at the timestamps in the logs below

note : When comparing this exact same workflow authored in dotnet, dotnet allows all activities to run concurrently.

2026-01-27 12:49:47


2026-01-27 12:49:47,810 - hello_world_app.workflows.fan_out_workflow - INFO - Starting fan-out workflow with 100 iterations
2026-01-27 12:49:47


2026-01-27 12:49:47.812 durabletask-worker INFO: 0-312: Orchestrator yielded with 100 task(s) and 0 event(s) outstanding.
2026-01-27 12:49:47


2026-01-27 12:49:47,812 - hello_world_app.workflows.fan_out_workflow - INFO - Waiting for 100 activities to complete
2026-01-27 12:49:47


2026-01-27 12:49:47,812 - durabletask-worker - INFO - 0-312: Orchestrator yielded with 100 task(s) and 0 event(s) outstanding.
2026-01-27 12:49:47


2026-01-27 12:49:47,824 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #0 - scheduled=12:49:47 activated=12:49:47
2026-01-27 12:49:47


2026-01-27 12:49:47,826 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #1 - scheduled=12:49:47 activated=12:49:47
2026-01-27 12:49:47


2026-01-27 12:49:47,826 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #2 - scheduled=12:49:47 activated=12:49:47
2026-01-27 12:49:47


2026-01-27 12:49:47,828 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #3 - scheduled=12:49:47 activated=12:49:47
2026-01-27 12:49:47


2026-01-27 12:49:47,829 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #4 - scheduled=12:49:47 activated=12:49:47
2026-01-27 12:49:47


2026-01-27 12:49:47,833 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #5 - scheduled=12:49:47 activated=12:49:47
2026-01-27 12:49:47


2026-01-27 12:49:47,835 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #6 - scheduled=12:49:47 activated=12:49:47
2026-01-27 12:49:47


2026-01-27 12:49:47,846 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #7 - scheduled=12:49:47 activated=12:49:47
2026-01-27 12:49:47


2026-01-27 12:49:47,855 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #8 - scheduled=12:49:47 activated=12:49:47
2026-01-27 12:49:47


2026-01-27 12:49:47,865 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #9 - scheduled=12:49:47 activated=12:49:47
2026-01-27 12:49:47


2026-01-27 12:49:47,866 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #10 - scheduled=12:49:47 activated=12:49:47
2026-01-27 12:49:47


2026-01-27 12:49:47,869 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #11 - scheduled=12:49:47 activated=12:49:47
2026-01-27 12:49:47


2026-01-27 12:49:47,870 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #12 - scheduled=12:49:47 activated=12:49:47
2026-01-27 12:49:47


2026-01-27 12:49:47,872 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #13 - scheduled=12:49:47 activated=12:49:47
2026-01-27 12:49:47


INFO:     172.20.0.11:53174 - "POST /fanout-workflow HTTP/1.1" 200 OK
2026-01-27 12:50:47


2026-01-27 12:50:47,827 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #0 - scheduled=12:49:47 activated=12:49:47 finished=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,831 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #1 - scheduled=12:49:47 activated=12:49:47 finished=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,832 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #3 - scheduled=12:49:47 activated=12:49:47 finished=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,832 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #2 - scheduled=12:49:47 activated=12:49:47 finished=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,833 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #4 - scheduled=12:49:47 activated=12:49:47 finished=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,834 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #5 - scheduled=12:49:47 activated=12:49:47 finished=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,836 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #6 - scheduled=12:49:47 activated=12:49:47 finished=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,845 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #14 - scheduled=12:49:47 activated=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,845 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #15 - scheduled=12:49:47 activated=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,846 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #16 - scheduled=12:49:47 activated=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,847 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #17 - scheduled=12:49:47 activated=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,847 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #18 - scheduled=12:49:47 activated=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,848 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #19 - scheduled=12:49:47 activated=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,849 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #7 - scheduled=12:49:47 activated=12:49:47 finished=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,850 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #20 - scheduled=12:49:47 activated=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,852 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #21 - scheduled=12:49:47 activated=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,857 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #8 - scheduled=12:49:47 activated=12:49:47 finished=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,859 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #22 - scheduled=12:49:47 activated=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,867 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #9 - scheduled=12:49:47 activated=12:49:47 finished=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,868 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #10 - scheduled=12:49:47 activated=12:49:47 finished=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,870 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #23 - scheduled=12:49:47 activated=12:50:47
2026-01-27 12:50:47


2026-01-27 12:50:47,870 - hello_world_app.activities.super_slow_activity - INFO - 0-312 - super_slow_activity (1) #24 - scheduled=12:49:47 activated=12:50:47

Release Note

RELEASE NOTE:

[acroca]

The Python SDK is limiting concurrency intentionally: https://github.com/dapr/durabletask-python/blob/92af27ecaf07c9b7242883fb117fe82335c28756/durabletask/worker.py#L45

It seems other SDKs don't, they just execute whatever the runtime sends, and concurrency is controlled by the runtime.

We should do the same in python and remove all concurrency handling

[acroca]

I've been playing with this a bit here: https://github.com/acroca/dapr-durabletask-python/tree/unlimited-concurrency

It runs each activity in its own Thread... but that's not cheap in python.

If activities were async, it'd be cheaper to run multiple activities concurrently. I'm inclined to suggest to keep the ThreadPool for sync activities (with limited concurrency), and unlock 'unlimited' activities once we convert activities to async...