[DSIP-95][API] Complete the functionality of using dependencies in the complement data by det101 · Pull Request #18003 · apache/dolphinscheduler

det101 · 2026-02-27T08:17:24Z

Was this PR generated or assisted by AI?

close #17748

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

Copilot

Pull request overview

Implements downstream workflow triggering for complement/backfill runs in the API layer, adding support for “trigger dependent workflows” behavior and accompanying unit tests.

Changes:

Implemented doBackfillDependentWorkflow to fetch downstream workflow definitions and trigger backfill runs for them.
Added visited-code tracking intended to prevent self/cyclic triggering and duplicate downstream triggers.
Added BackfillWorkflowExecutorDelegateTest with basic scenarios for downstream triggering and filtering.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java	Adds dependent workflow backfill triggering logic and wiring for lineage + workflow definition lookup.
dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java	Adds unit tests for the new dependent backfill triggering logic (single-hop scenarios).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

.../org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

SbloodyS

You didn't fill in the content according to the PR template, please fix it.

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.../org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

…endent workflow - Problem 1: Replace single-level self-dep check with ThreadLocal visited set to detect and skip indirect circular dependencies (A→B→A), preventing StackOverflowError when allLevelDependent=true - Problem 2: Set startNodes=null for downstream workflows; upstream task node codes are not valid in a different workflow definition - Add tests for OFFLINE skip, not-found skip, and startNodes null assertion Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…tion ThreadLocal is unnecessary here since the call chain is synchronous and private. Passing visitedCodes as a parameter is simpler, clearer, and avoids ThreadLocal lifecycle management. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Align Java source and test formatting with project spotless rules to keep style checks consistent. Made-with: Cursor

Ensure dependent backfill triggering is evaluated per parallel date chunk, and batch-load downstream workflow definitions to avoid N+1 queries. Add regression coverage for parallel mode visited-code isolation. Made-with: Cursor

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java:119

In parallel mode, expectedParallelismNumber can be 0 (the validator only rejects values < 0). If it is 0, splitDateTime(listDate, expectedParallelismNumber) will divide by zero and throw ArithmeticException. Treat 0 the same as null (default to listDate.size()), or explicitly guard against <= 0 before calling splitDateTime.

        final BackfillWorkflowDTO.BackfillParamsDTO backfillParams = backfillWorkflowDTO.getBackfillParams();
        Integer expectedParallelismNumber = backfillParams.getExpectedParallelismNumber();

        List<ZonedDateTime> listDate = backfillParams.getBackfillDateList();
        if (expectedParallelismNumber != null) {
            expectedParallelismNumber = Math.min(listDate.size(), expectedParallelismNumber);
        } else {
            expectedParallelismNumber = listDate.size();
        }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

Copilot · 2026-03-25T08:56:54Z

.../org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java

+    public void testDoParallelBackfillWorkflow_ShouldIsolateVisitedCodesAcrossChunks() {
+        long upstreamCode = 500L;
+        WorkflowDefinition upstreamWorkflow =
+                WorkflowDefinition.builder().code(upstreamCode).releaseState(ReleaseState.ONLINE).build();
+        List<ZonedDateTime> dates = Arrays.asList(
+                ZonedDateTime.parse("2026-02-01T00:00:00Z"),
+                ZonedDateTime.parse("2026-02-02T00:00:00Z"),
+                ZonedDateTime.parse("2026-02-03T00:00:00Z"));
+        BackfillWorkflowDTO.BackfillParamsDTO params = BackfillWorkflowDTO.BackfillParamsDTO.builder()
+                .runMode(RunMode.RUN_MODE_PARALLEL)
+                .backfillDateList(dates)
+                .expectedParallelismNumber(2)
+                .backfillDependentMode(ComplementDependentMode.ALL_DEPENDENT)
+                .allLevelDependent(true)
+                .executionOrder(ExecutionOrder.ASC_ORDER)
+                .build();
+        BackfillWorkflowDTO dto = BackfillWorkflowDTO.builder()
+                .workflowDefinition(upstreamWorkflow)
+                .backfillParams(params)
+                .build();
+        Set<Long> baseVisitedCodes = new HashSet<>(Collections.singleton(upstreamCode));
+        List<Set<Long>> visitedSnapshotPerChunk = new java.util.ArrayList<>();
+
+        doAnswer(invocation -> {
+            Set<Long> chunkVisited = invocation.getArgument(2);
+            visitedSnapshotPerChunk.add(new HashSet<>(chunkVisited));
+            chunkVisited.add(9000L + visitedSnapshotPerChunk.size());
+            return null;
+        }).when(backfillWorkflowExecutorDelegate).doBackfillDependentWorkflowForTesting(any(), any(), any());
+
+        List<Integer> result = backfillWorkflowExecutorDelegate.executeWithVisitedCodes(dto, baseVisitedCodes);
+


This test calls executeWithVisitedCodes, which will run the real doBackfillWorkflow and attempt to use registryClient / Clients.withService(IWorkflowControlClient) to contact a master. Since neither is mocked/stubbed in this test, it will fail with NPE or a ServiceException before exercising the visited-codes isolation assertions. Consider refactoring to unit-test the chunk visited-code cloning without invoking the master trigger, or add a test seam/mocking for the backfill trigger step.

.../org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java

Made-with: Cursor

Delete brittle reflection-based tests for private doBackfillDependentWorkflow; keep higher-level parallel visited-code isolation. Made-with: Cursor

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

Use DependentWorkflowDefinition routing fields for dependent triggers, and force dependent runMode SERIAL when upstream is PARALLEL to traverse dependencies once per date chunk. Made-with: Cursor

Cover all-level recursion gating, cycle skipping, and offline/missing downstream skipping. Made-with: Cursor

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

.../org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java

…/api/executor/workflow/BackfillWorkflowExecutorDelegate.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Group dependent definitions by workflowDefinitionCode and aggregate distinct startNodes to avoid skipping nodes when visitedCodes blocks duplicate entries. Made-with: Cursor

Pass new HashSet instead of null for initial visitedCodes to simplify downstream handling. Made-with: Cursor

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…/api/executor/workflow/BackfillWorkflowExecutorDelegate.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…Exception

…rough executeWithVisitedCodes

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.../org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

…e visitedCodes across chunks

… for BackfillWorkflowExecutorDelegate and related tests

sonarqubecloud · 2026-03-27T02:31:03Z

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 60%)

See analysis details on SonarQube Cloud

det101 requested review from SbloodyS and caishunfeng as code owners February 27, 2026 08:17

det101 force-pushed the DSIP-95 branch from 91a1353 to a3fe99a Compare February 27, 2026 08:18

SbloodyS requested a review from Copilot March 20, 2026 03:50

Copilot started reviewing on behalf of SbloodyS March 20, 2026 03:51 View session

SbloodyS assigned det101 Mar 20, 2026

SbloodyS added backend DSIP labels Mar 20, 2026

SbloodyS added this to the 3.4.2 milestone Mar 20, 2026

github-actions bot added the test label Mar 20, 2026

Copilot AI reviewed Mar 20, 2026

View reviewed changes

SbloodyS reviewed Mar 20, 2026

View reviewed changes

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java Outdated Show resolved Hide resolved

SbloodyS closed this Mar 23, 2026

SbloodyS reopened this Mar 23, 2026

det101 requested review from SbloodyS and Copilot March 24, 2026 03:20

Copilot started reviewing on behalf of det101 March 24, 2026 03:21 View session

Copilot AI reviewed Mar 24, 2026

View reviewed changes

det101 force-pushed the DSIP-95 branch from 11c9f03 to b9ef592 Compare March 24, 2026 06:57

luxl and others added 6 commits March 25, 2026 16:51

close apache#17748

3fb7634

[Chore] Apply spotless formatting for backfill workflow executor files

ed24efe

Align Java source and test formatting with project spotless rules to keep style checks consistent. Made-with: Cursor

fix review

29905e8

det101 force-pushed the DSIP-95 branch from ca11eb6 to 1841eaf Compare March 25, 2026 08:51

det101 requested a review from Copilot March 25, 2026 08:51

Copilot started reviewing on behalf of det101 March 25, 2026 08:52 View session

Copilot AI reviewed Mar 25, 2026

View reviewed changes

fix(api): remove redundant dependent-trigger testing seam

cbe8a60

Made-with: Cursor

test(api): remove unit tests for private dependent backfill

47e3d17

Delete brittle reflection-based tests for private doBackfillDependentWorkflow; keep higher-level parallel visited-code isolation. Made-with: Cursor

det101 requested a review from Copilot March 26, 2026 01:40

Copilot started reviewing on behalf of det101 March 26, 2026 01:41 View session

Copilot AI reviewed Mar 26, 2026

View reviewed changes

luxl added 2 commits March 26, 2026 10:08

fix(api): align dependent routing and prevent double-splitting

72a2642

Use DependentWorkflowDefinition routing fields for dependent triggers, and force dependent runMode SERIAL when upstream is PARALLEL to traverse dependencies once per date chunk. Made-with: Cursor

test(api): add dependent backfill traversal branches

d2a8c96

Cover all-level recursion gating, cycle skipping, and offline/missing downstream skipping. Made-with: Cursor

det101 requested a review from Copilot March 26, 2026 02:18

Copilot started reviewing on behalf of det101 March 26, 2026 02:18 View session

Copilot AI reviewed Mar 26, 2026

View reviewed changes

det101 and others added 3 commits March 26, 2026 10:23

Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler…

c7ba4b9

…/api/executor/workflow/BackfillWorkflowExecutorDelegate.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

fix(api): aggregate dependent nodes per downstream workflow

b477d6b

Group dependent definitions by workflowDefinitionCode and aggregate distinct startNodes to avoid skipping nodes when visitedCodes blocks duplicate entries. Made-with: Cursor

chore(api): initialize visitedCodes at execute entry

6610bc2

Pass new HashSet instead of null for initial visitedCodes to simplify downstream handling. Made-with: Cursor

det101 requested a review from Copilot March 26, 2026 02:49

Copilot started reviewing on behalf of det101 March 26, 2026 02:50 View session

Copilot AI reviewed Mar 26, 2026

View reviewed changes

det101 and others added 5 commits March 26, 2026 10:59

Apply suggestion from @Copilot

eca546f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler…

f320007

…/api/executor/workflow/BackfillWorkflowExecutorDelegate.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler…

7c9acee

…/api/executor/workflow/BackfillWorkflowExecutorDelegate.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

fix: handle null/zero expectedParallelismNumber to prevent Arithmetic…

8b026ac

…Exception

fix: cap parallelism to listDate size and route dependent backfill th…

5579fe6

…rough executeWithVisitedCodes

det101 requested a review from Copilot March 26, 2026 06:52

Copilot started reviewing on behalf of det101 March 26, 2026 06:52 View session

Copilot AI reviewed Mar 26, 2026

View reviewed changes

luxl and others added 4 commits March 26, 2026 15:22

refactor: revert parallelism guard to original if/else style and shar…

af022b9

…e visitedCodes across chunks

docs: clarify force-SERIAL rationale and recursion depth assumption

42086c1

Refactor: translate all code comments to English and unify test logic…

2114213

… for BackfillWorkflowExecutorDelegate and related tests

Merge branch 'dev' into DSIP-95

8acc714

luxl added 2 commits March 27, 2026 16:49

fix spotless

ddbbdab

fix spotless

1363d8b

Conversation

det101 commented Feb 27, 2026

Was this PR generated or assisted by AI?

Purpose of the pull request

Brief change log

Verify this pull request

Pull Request Notice

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Mar 27, 2026

Quality Gate failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants