Skip to content

fix(node): Deduplicate sentry-trace and baggage headers on outgoing requests#19960

Open
Lms24 wants to merge 19 commits intodevelopfrom
lms/fix-double-baggage-propagation
Open

fix(node): Deduplicate sentry-trace and baggage headers on outgoing requests#19960
Lms24 wants to merge 19 commits intodevelopfrom
lms/fix-double-baggage-propagation

Conversation

@Lms24
Copy link
Copy Markdown
Member

@Lms24 Lms24 commented Mar 24, 2026

This PR fixes a bunch of closely related issues with our node fetch and http integrations for outgoing request header propagation.

Summary:

  • We now dedupe sentry-trace and baggage headers more aggressively, resolving multiple scenarios where duplicated sentry headers were attached to outgoing requests
  • We now always prefer the first sentry tracing headers pair set onto a request. This allows users to set custom sentry headers (for whatever reason) and ensures our instrumentation doesn't overwrite itself.
  • We no longer mix individual sentry- baggage entries when merging two headers where both contain sentry- entries. We only take one of the two and delete the other.

Sentry Propagator

(The propagator is used by both, fetch and node:http outgoing request instrumentation)

  • The propagator would add baggage (and optionally traceparent) headers to requests, even if a sentry-trace header was already set on the request
    • Consequence: Unpredictable dynamic sampling decisions on the sentry backend, potentially wrong SDK behaviour of downstream SDKs
    • Fix: Check for already set sentry-trace headers. Only apply our headers, if sentry-trace doesn't exist yet.

fetch instrumentation

  • mergeBaggageHeaders would mix up sentry- entries from both headers creating completely inconsistent baggage entries.
    • consequence: Unpredictable dynamic sampling decisions on the sentry backend, potentially wrong SDK behaviour of downstream SDKs
    • Fix: Remove all previous sentry- entries and add all new ones (merge non-sentry entries)
  • addTracePropagationHeadersToFetchRequest would correctly avoid setting a sentry-trace header if a previous one existed but happily still add new baggage and optionally traceparent headers.
    • consequence: Inconsistent data between sentry-trace and the other two values (e.g. distinct trace ids across the headers in one request).
    • Fix: Check for already set sentry-trace headers. Only apply our headers, if sentry-trace doesn't exist yet.
  • manually adding baggage headers with sentry- entries would cause multiple baggage headers to be present. This was due to insufficient deduplication in our own header adding logic. This surfaced because we didn't only have two competing headers in this case but three (user, OTel Undici, our instrumentation).
    • consequence: Incosistent/Unpredictable dynamic sampling decisions, parsing errors for downstream SDKs or unpredicatable (incorrect) trace continuation
    • Fix: Deduplicate multiple sentry-trace and baggage header if >1 are already set. Only add new headers, if no headers were set yet.
  • We'd incorrectly replace and merge string headers (undici on Node 18) whose keys contained baggage or sentry-trace (e.g. my-vendor-baggage) headers. Generally, the string replacement logic was brittle.
    • consequences: Fails on Node 18 to correctly dedupe headers, inconsistent headers, incosistent dynamic sampling decisions
    • Fix: Convert string headers to array headers and use the same, more tested and established logic

Http instrumentation

  • For node:http requests, we'd incorrectly set baggage (and traceparent) headers even if a sentry-trace header was already set.
    • Consequence: Unpredictable dynamic sampling decisions on the sentry backend, potentially wrong SDK behaviour of downstream SDKs
    • Fix: Check for already set sentry-trace headers. Only apply our headers, if sentry-trace doesn't exist yet.
  • For Node versions not supporting diagnostics channels we'd fall back to the propagator which had the same bug (see above)

Minor fixes

  • node-fetch: hardened the check for existing headers to no longer incorrectly return an existing header if we matched a header value instead of the key

Added tests for the various propagation mechanisms and integration tests asserting on traceid and spanid consistency across spans and all headers (as appropriate for various scenarios).

closes #19158

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 24, 2026

Semver Impact of This PR

🟢 Patch (bug fixes)

📋 Changelog Preview

This is how your changes will appear in the changelog.
Entries from this PR are highlighted with a left border (blockquote style).


New Features ✨

Deps

  • Bump babel-loader from 10.0.0 to 10.1.1 by dependabot in #19997
  • Bump handlebars from 4.7.7 to 4.7.9 by dependabot in #20008

Other

  • (browser) Replace element timing spans with metrics by logaretm in #19869
  • (bun) Add bunRuntimeMetricsIntegration by chargome in #19979
  • (core) Support embedding APIs in google-genai by nicohrubec in #19797
  • (node) Add nodeRuntimeMetricsIntegration by chargome in #19923
  • (nuxt) Support parametrized SSR routes in Nuxt 5 by s1gr1d in #19977

Bug Fixes 🐛

Node

  • Deduplicate sentry-trace and baggage headers on outgoing requests by Lms24 in #19960
  • Ensure startNewTrace propagates traceId in OTel environments by logaretm in #19963

Other

  • (e2e) Pin @opentelemetry/api to 1.9.0 in ts3.8 test app by logaretm in #19992
  • (opentelemetry) Convert seconds timestamps in span.end() to milliseconds by logaretm in #19958

Documentation 📚

  • (release) Update publishing-a-release.md by nicohrubec in #19982

Internal Changes 🔧

Core

  • Introduce instrumented method registry for AI integrations by nicohrubec in #19981
  • Consolidate getOperationName into one shared utility by nicohrubec in #19971

Deps

  • Bump amqplib from 0.10.7 to 0.10.9 by dependabot in #20000
  • Bump actions/upload-artifact from 6 to 7 by dependabot in #19569
  • Bump srvx from 0.11.12 to 0.11.13 by dependabot in #20001
  • Bump @apollo/server from 5.4.0 to 5.5.0 by dependabot in #20007

Deps Dev

  • Bump node-forge from 1.3.2 to 1.4.0 by dependabot in #20012
  • Bump yaml from 2.8.2 to 2.8.3 by dependabot in #19985

Other

  • (deno) Expand Deno E2E test coverage by chargome in #19957
  • (e2e) Add e2e tests for nodeRuntimeMetricsIntegration by chargome in #19989

🤖 This preview updates automatically when you update the PR.

@Lms24 Lms24 marked this pull request as draft March 24, 2026 18:03
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 24, 2026

size-limit report 📦

⚠️ Warning: Base artifact is not the latest one, because the latest workflow run is not done yet. This may lead to incorrect results. Try to re-run all tests to get up to date results.

Path Size % Change Change
@sentry/browser 25.69 kB - -
@sentry/browser - with treeshaking flags 24.17 kB - -
@sentry/browser (incl. Tracing) 42.17 kB - -
@sentry/browser (incl. Tracing, Profiling) 46.79 kB - -
@sentry/browser (incl. Tracing, Replay) 80.98 kB - -
@sentry/browser (incl. Tracing, Replay) - with treeshaking flags 70.6 kB - -
@sentry/browser (incl. Tracing, Replay with Canvas) 85.7 kB - -
@sentry/browser (incl. Tracing, Replay, Feedback) 97.97 kB - -
@sentry/browser (incl. Feedback) 42.48 kB - -
@sentry/browser (incl. sendFeedback) 30.35 kB - -
@sentry/browser (incl. FeedbackAsync) 35.4 kB - -
@sentry/browser (incl. Metrics) 26.96 kB - -
@sentry/browser (incl. Logs) 27.1 kB - -
@sentry/browser (incl. Metrics & Logs) 27.78 kB - -
@sentry/react 27.45 kB - -
@sentry/react (incl. Tracing) 44.52 kB - -
@sentry/vue 30.13 kB - -
@sentry/vue (incl. Tracing) 44.08 kB - -
@sentry/svelte 25.7 kB - -
CDN Bundle 28.39 kB - -
CDN Bundle (incl. Tracing) 43.2 kB - -
CDN Bundle (incl. Logs, Metrics) 29.76 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) 44.25 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) 68.56 kB - -
CDN Bundle (incl. Tracing, Replay) 80.08 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) 81.16 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) 85.62 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) 86.67 kB - -
CDN Bundle - uncompressed 82.93 kB - -
CDN Bundle (incl. Tracing) - uncompressed 128.07 kB - -
CDN Bundle (incl. Logs, Metrics) - uncompressed 87.07 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) - uncompressed 131.48 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) - uncompressed 210.06 kB - -
CDN Bundle (incl. Tracing, Replay) - uncompressed 244.95 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) - uncompressed 248.34 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) - uncompressed 257.86 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) - uncompressed 261.25 kB - -
@sentry/nextjs (client) 46.93 kB - -
@sentry/sveltekit (client) 42.67 kB - -
@sentry/node-core 56.7 kB +0.36% +198 B 🔺
@sentry/node 173.84 kB +0.15% +248 B 🔺
@sentry/node - without tracing 96.77 kB +0.25% +238 B 🔺
@sentry/aws-serverless 113.76 kB +0.21% +231 B 🔺

View base workflow run

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 24, 2026

node-overhead report 🧳

Note: This is a synthetic benchmark with a minimal express app and does not necessarily reflect the real-world performance impact in an application.

Scenario Requests/s % of Baseline Prev. Requests/s Change %
GET Baseline 8,934 - 9,185 -3%
GET With Sentry 1,659 19% 1,624 +2%
GET With Sentry (error only) 6,130 69% 6,039 +2%
POST Baseline 1,203 - 1,196 +1%
POST With Sentry 595 49% 581 +2%
POST With Sentry (error only) 1,052 87% 1,052 -
MYSQL Baseline 3,235 - 3,240 -0%
MYSQL With Sentry 465 14% 468 -1%
MYSQL With Sentry (error only) 2,605 81% 2,629 -1%

View base workflow run

@Lms24 Lms24 force-pushed the lms/fix-double-baggage-propagation branch from c4c5fbb to a11ca57 Compare March 24, 2026 18:14
@Lms24 Lms24 self-assigned this Mar 24, 2026
@Lms24 Lms24 force-pushed the lms/fix-double-baggage-propagation branch from 35051e1 to 8d53f05 Compare March 26, 2026 16:46
@Lms24 Lms24 marked this pull request as ready for review March 26, 2026 16:47
@Lms24 Lms24 force-pushed the lms/fix-double-baggage-propagation branch from fa73db1 to 27a400b Compare March 27, 2026 11:03
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the browser SDK already guarded against multiple sentry headers, I just strengthened the test assertions a bit.

'sentry-environment=myEnv',
'sentry-release=2.1.0',
expect.stringMatching(/sentry-sample_rand=\d+/),
'sentry-sample_rate=0.54',
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was a wrong assertion in the test

@Lms24 Lms24 requested review from a team, chargome and logaretm and removed request for a team March 27, 2026 13:18
@Lms24 Lms24 changed the title fix(node): Avoid duplicated sentry-trace and baggage headers on fetch requests fix(node): Deduplicate sentry-trace and baggage headers on fetch requests Mar 27, 2026
@Lms24 Lms24 changed the title fix(node): Deduplicate sentry-trace and baggage headers on fetch requests fix(node): Deduplicate sentry-trace and baggage headers on outgoing requests Mar 27, 2026
Copy link
Copy Markdown
Member

@chargome chargome left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks solid! The integration tests should harden for regressions here too.

if (key) {
headers.push(key, value);
}
} catch {}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

l: We drop malformed headers here, should we log it?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea! 4dd4ce8

Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

headers.push(key, value);
}
} catch {
debug.warn(`Failed to convert string request header to array header: ${header}`);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing DEBUG_BUILD guard on debug.warn call

Low Severity

The debug.warn() call is not guarded by DEBUG_BUILD &&, and DEBUG_BUILD is not imported in this file. Every other debug.log/warn/error call across packages/node-core/src/utils/ is consistently guarded with DEBUG_BUILD && (see outgoingHttpRequest.ts, errorhandling.ts). While debug.warn internally short-circuits via _maybeLog's own DEBUG_BUILD check, the external guard is what lets the calling package's bundler tree-shake the entire expression — including the template literal string allocation. Without it, the function call and string creation remain in the production bundle.

Fix in Cursor Fix in Web

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

baggage is sent twice when tracesSampleRate is set and Sentry.getTraceData() is added manually

2 participants