es_out: support Upstream Servers with configuration overriding

[mabrarov]

Implementation of Upstream feature for the Elasticsearch output plugin.

Refer to fluent/fluent-bit#7608 for the changes in Fluent Bit code.

Summary by CodeRabbit

• Documentation
  • Added Elasticsearch to the list of plugins that support Upstream.
  • Expanded Elasticsearch docs with full Upstream configuration support and examples for Classic mode and YAML, including sample upstream server layouts.
  • Added per-key override guidance and a dedicated overrides table for Upstream NODE usage.
  • Added AWS SigV4 authentication guidance and notes on how AWS/TLS settings interact with Upstream overrides.
  • Clarified write-operation behavior and field-name handling in Upstream contexts.
• Style
  • Added "AWS SigV4" to heading-capitalization exceptions.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[mabrarov]

This pull request is waiting for review of fluent/fluent-bit#7608

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[cnorris-cs]

I left suggestions, but I've added my technical writing-based approval based on that to keep this moving.

Primarily, I was looking to help the table be more consistent and readable, while implementing some style-based suggestions for consistency.

[mabrarov]

Hi @cnorris-cs,

I applied some (!) parts of your suggestions as well as introduced additional changes for consistency with your suggestions. Please take a look at this pull request one more time when you have time. Some of the changes are implemented in dedicated commits (for ease of revert in case of concerns) - please let me know if I need to squash all commits into a single one.

Thank you for your review.

[eschabell]

@esmerel can you look at the Vale check failures, I'm seeing this error before:

Raw Output:
{"message": "[FluentBit.Headings] 'Classic mode Configuration File with Upstream' should use sentence-style capitalization.", "location": {"path": "pipeline/outputs/elasticsearch.md", "range": {"start": {"line": 163, "column": 6}}}, "severity": "INFO"}
Error: reviewdog: Too many results (annotations) in diff.
You may miss some annotations due to GitHub limitation for annotation created by logging command.
Please check GitHub Actions log console to see all results.

[esmerel]

Making suggestions to correct vale warnings.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Added Elasticsearch to the list of Upstream-supported output plugins and expanded Elasticsearch output docs with per-key override rules, AWS SigV4/TLS interaction notes, and Classic/YAML Upstream configuration examples including upstream_servers usage.

Changes

Cohort / File(s)	Summary
Upstream Plugin Reference administration/configuring-fluent-bit/classic-mode/upstream-servers.md	Added Elasticsearch to the list of output plugins documented as supporting Upstream.
Elasticsearch Output Documentation pipeline/outputs/elasticsearch.md	Reworked parameter table to show per-key "Overrides allowed"; added detailed per-key overrides table, AWS SigV4 and TLS notes relevant to Upstream NODE overrides; added Classic and YAML Upstream/server and upstream_servers examples; adjusted field-name and write_operation guidance for Upstream usage.
Vale styles vale-styles/FluentBit/Headings.yml	Added "AWS SigV4" to heading-capitalization exception list.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• Vale: updating tests #2307 — also modifies vale-styles/FluentBit/Headings.yml to adjust heading exceptions.
• general cleanup to YAML configuration pages #2321 — touches YAML upstream_servers documentation and may overlap with added Upstream examples.
• Vale/general: various page updates to reduce messages #2167 — makes heading-exception edits similar to the "AWS SigV4" addition.

Suggested reviewers

• eschabell
• patrick-stephens

Poem

🐇 I hop through docs with floppy cheer,

Elasticsearch joins the Upstream sphere,
Keys, TLS, SigV4 tucked in rows,
Classic and YAML—off I goes! ✨

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding Upstream Servers support with configuration overriding to the Elasticsearch output plugin documentation.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

CodeRabbit can enforce grammar and style rules using `languagetool`.

Configure the reviews.tools.languagetool setting to enable/disable rules and categories. Refer to the LanguageTool Community to learn more.

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In `@pipeline/outputs/elasticsearch.md`:
- Around line 207-218: The YAML example uses an unquoted asterisk for the match
value ("match: *"), which YAML treats as an anchor reference; update the example
so the match key uses a quoted literal asterisk (e.g., change match: * to match:
"*" ) to ensure the YAML parses as intended and preserves the wildcard
semantics.
- Around line 226-249: The YAML has an invalid unquoted wildcard in the pipeline
output config: change the match field from an unquoted asterisk to a quoted
string (e.g., change match: * to match: "*" or match: "'*'") in the pipeline
outputs block so the YAML parser accepts the wildcard value; update the pipeline
outputs entry with the quoted wildcard for the match key.

📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d8aa628 and 9f9a5ee.

📒 Files selected for processing (2)

• administration/configuring-fluent-bit/classic-mode/upstream-servers.md
• pipeline/outputs/elasticsearch.md

🧰 Additional context used

🧠 Learnings (1)

📚 Learning: 2025-12-12T14:30:18.461Z

Learnt from: kalavt
Repo: fluent/fluent-bit-docs PR: 2294
File: pipeline/inputs/kafka.md:147-168
Timestamp: 2025-12-12T14:30:18.461Z
Learning: In Fluent Bit v4.0.4+, when using AWS MSK IAM authentication (rdkafka.sasl.mechanism: aws_msk_iam), the rdkafka.security.protocol is automatically set to SASL_SSL and the AWS region is auto-detected from the broker hostname for standard MSK endpoints. The aws_msk_iam_cluster_arn parameter was removed - users only need to set rdkafka.sasl.mechanism: aws_msk_iam (and optionally aws_region for custom DNS/PrivateLink).

Applied to files:

• pipeline/outputs/elasticsearch.md

🔇 Additional comments (7)

administration/configuring-fluent-bit/classic-mode/upstream-servers.md (1)

8-8: LGTM! Elasticsearch correctly added to Upstream-supported plugins.

The addition properly documents Elasticsearch as an Upstream-capable plugin, consistent with the extensive Upstream configuration documentation added in the elasticsearch.md file.

pipeline/outputs/elasticsearch.md (6)

13-20: LGTM! Clear documentation of Upstream override capability.

The introduction and table header effectively explain which configuration parameters can be overridden in Upstream NODE sections.

---

21-58: LGTM! Comprehensive override documentation for Elasticsearch parameters.

The "Allows overrides" column correctly identifies which parameters can be overridden in Upstream NODE sections, with helpful explanations for Host and Port requirements.

---

66-68: LGTM! Important clarification about AWS Sigv4 and Upstream interaction.

This section correctly distinguishes between TLS parameters for AWS authentication (not overridable in NODE) and TLS parameters for Elasticsearch connections (overridable in NODE), preventing potential configuration confusion.

---

163-163: Note: Vale style check flagged this heading.

Per the PR objectives, Vale reported that this heading should use sentence-style capitalization. However, "Upstream" appears to be used as a proper noun (feature name) throughout the documentation, so the current capitalization may be intentional. Please verify whether this should be "Classic mode configuration file with upstream" or if "Upstream" should remain capitalized as a feature name.

---

165-201: LGTM! Well-documented Classic mode Upstream configuration.

The Classic mode configuration examples and Upstream server definitions are syntactically correct and clearly demonstrate how to configure Elasticsearch with Upstream load balancing.

---

220-224: LGTM! Helpful explanation of YAML Upstream configuration options.

The explanation clearly documents that Upstream configuration can use either Classic mode format or YAML format, and can optionally be embedded in the main configuration file when using YAML.

_{✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.}

[eschabell]

@mabrarov I ran the two waiting checks and you've got Lint errors to fix before I can review.

[mabrarov]

Hi @eschabell,

@mabrarov I ran the two waiting checks and you've got Lint errors to fix before I can review.

Just curious is it a new linter? The only changes I did today is rebase on master branch.

[mabrarov]

Hi @eschabell,

I ran the two waiting checks and you've got Lint errors to fix before I can review.

I believe I fixed these errors. Could you please run same checks again?

Thank you.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pipeline/outputs/elasticsearch.md`:
- Line 40: In the table entries for the Logstash-related settings, fix two
grammar issues: in the description for `Logstash_Prefix` where the sentence
reads ", For example" change the "For" to lowercase ("for example"), and in the
other Logstash description that currently says "it append" change the verb to
third-person singular "it appends"; search for the strings `Logstash_Prefix` and
`Logstash_Format` to locate and update the two descriptions accordingly.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a09c65d7-1a25-46ef-a453-4c2039fc8dd2

📥 Commits

Reviewing files that changed from the base of the PR and between 2c92d49 and 96f70d0.

📒 Files selected for processing (2)

• pipeline/outputs/elasticsearch.md
• vale-styles/FluentBit/Headings.yml