[AWSX] feat(logs forwarder): update template and document custom source overriding (#1037)

Author: ge0Aja

aws/logs_monitoring/README.md

@@ -12,10 +12,10 @@ aliases:
 
 The Datadog Forwarder is an AWS Lambda function that ships logs from AWS to Datadog, specifically:
 
-- Forward CloudWatch and S3 logs.
-- Forward logs from SNS, and Kinesis events to Datadog.
-- Kinesis data stream events support CloudWatch logs only.
-- Forward metrics, traces, and logs from AWS Lambda functions to Datadog. Datadog recommends to use [Datadog Lambda Extension][1] to monitor Lambda functions.
+-   Forward CloudWatch and S3 logs.
+-   Forward logs from SNS, and Kinesis events to Datadog.
+-   Kinesis data stream events support CloudWatch logs only.
+-   Forward metrics, traces, and logs from AWS Lambda functions to Datadog. Datadog recommends to use [Datadog Lambda Extension][1] to monitor Lambda functions.
 
 For Serverless customers using the Forwarder to forward metrics, traces, and logs from AWS Lambda logs to Datadog, you should [migrate to the Datadog Lambda Extension][3] to collect telemetry directly from the Lambda execution environments. The Forwarder is still available for use in Serverless Monitoring, but will not be updated to support the latest features.
 
@@ -60,23 +60,23 @@ If you're using AWS Organizations, you can use CloudFormation StackSets to deplo
 1. Deploy the Forwarder template using [CloudFormation StackSets][151] in the AWS Console.
 2. In the **Permissions** section, optionally provide an IAM admin role ARN, or leave the default IAM execution role name (`AWSCloudFormationStackSetExecutionRole`).
 3. In the **Specify template** section:
-   1. Select **Amazon S3 URL**.
-   1. Enter the Forwarder template URL: `https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml`. 
-   1. Click **Next**.
+    1. Select **Amazon S3 URL**.
+    1. Enter the Forwarder template URL: `https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml`.
+    1. Click **Next**.
 4. In the **Specify stack set details** section:
-   1. Provide a stack set name.
-   1. Provide a valid Datadog API key using **one** of the `DdApiKey`, `DdApiKeySecretArn`, or `DdApiKeySsmParameterName` fields.
-   1. Select the appropriate Datadog site in the `DdSite` field. 
-   1. Optionally, configure other parameters.
-   1. Click **Next**.
+    1. Provide a stack set name.
+    1. Provide a valid Datadog API key using **one** of the `DdApiKey`, `DdApiKeySecretArn`, or `DdApiKeySsmParameterName` fields.
+    1. Select the appropriate Datadog site in the `DdSite` field.
+    1. Optionally, configure other parameters.
+    1. Click **Next**.
 5. In the **Configure stack set options** section:
-   1. Optionally, configure the provided options.
-   1. Check the box under **Capabilities**.
-   1. Click **Next**.
+    1. Optionally, configure the provided options.
+    1. Check the box under **Capabilities**.
+    1. Click **Next**.
 6. On the **Set deployment options** page:
-   1. Select **Deploy stacks in accounts** and enter your AWS account IDs. 
-   1. Under **Specify regions**, add target regions where you want to deploy the Forwarder.
-   1. Click **Next**.
+    1. Select **Deploy stacks in accounts** and enter your AWS account IDs.
+    1. Under **Specify regions**, add target regions where you want to deploy the Forwarder.
+    1. Click **Next**.
 7. Review the details, and click **Submit**.
 
 #### Multiple accounts
@@ -87,15 +87,15 @@ If you're using AWS Organizations, you can use CloudFormation StackSets to deplo
 2. Deploy from the organization management account using [CloudFormation StackSets][151].
 3. In the **Permissions** section, choose **Service-managed permissions**. This option allows CloudFormation StackSets to create the necessary IAM roles in target accounts automatically.
 4. In the **Specify template** section:
-   1. Select **Amazon S3 URL**.
-   1. Enter the Forwarder template URL: `https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml`. 
-   1. Click **Next**.
+    1. Select **Amazon S3 URL**.
+    1. Enter the Forwarder template URL: `https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml`.
+    1. Click **Next**.
 5. In the **Specify stack set details** section:
-   1. Provide a stack set name.
-   1. Provide a valid Datadog API key using **one** of the `DdApiKey`, `DdApiKeySecretArn`, or `DdApiKeySsmParameterName` fields.
-   1. Select the appropriate Datadog site in the `DdSite` field. 
-   1. Optionally, configure other parameters.
-   1. Click **Next**.
+    1. Provide a stack set name.
+    1. Provide a valid Datadog API key using **one** of the `DdApiKey`, `DdApiKeySecretArn`, or `DdApiKeySsmParameterName` fields.
+    1. Select the appropriate Datadog site in the `DdSite` field.
+    1. Optionally, configure other parameters.
+    1. Click **Next**.
 6. Choose **Deploy to organization** and specify target organizational units (OUs) or individual accounts, along with the regions where you want to deploy the Forwarder.
 
 **Note**: Regardless of whether you specify organizational units or individual accounts, you must [set up triggers][155] after the Forwarder is deployed.
@@ -184,10 +184,10 @@ If you encounter issues upgrading to the latest version, check the Troubleshooti
 
 Key breaking changes in v5.0.0:
 
-- **Log filtering behavior changed**: `IncludeAtMatch` and `ExcludeAtMatch` now match against the log message only, not the entire JSON structure
-- **TCP transport removed**: `DD_USE_TCP` parameter removed, all logs must use HTTP/HTTPS
-- **PrivateLink variable removed**: `DD_USE_PRIVATE_LINK` removed (PrivateLink is still supported via `DdUseVPC`)
-- **New tag enrichment**: Backend-based tag enrichment enabled by default via `DdEnrichS3Tags` and `DdEnrichCloudwatchTags`, reducing forwarder costs
+-   **Log filtering behavior changed**: `IncludeAtMatch` and `ExcludeAtMatch` now match against the log message only, not the entire JSON structure
+-   **TCP transport removed**: `DD_USE_TCP` parameter removed, all logs must use HTTP/HTTPS
+-   **PrivateLink variable removed**: `DD_USE_PRIVATE_LINK` removed (PrivateLink is still supported via `DdUseVPC`)
+-   **New tag enrichment**: Backend-based tag enrichment enabled by default via `DdEnrichS3Tags` and `DdEnrichCloudwatchTags`, reducing forwarder costs
 
 ### Upgrade an older version to 4.13.0+
 
@@ -361,13 +361,13 @@ If you must deploy the Forwarder to a VPC without direct public internet access,
 4. Ensure the `DdFetchLambdaTags`, `DdFetchStepFunctionsTags`, and `DdFetchS3Tags` options are disabled, because AWS VPC does not yet offer an endpoint for the Resource Groups Tagging API.
 5. If you are using HAProxy or NGINX:
 
-- Set `DdApiUrl` to `http://<proxy_host>:3834` or `https://<proxy_host>:3834`.
-- Set `DdTraceIntakeUrl` to `http://<proxy_host>:3835` or `https://<proxy_host>:3835`.
-- Set `DdUrl` to `<proxy_host>` and `DdPort` to `3837`.
+-   Set `DdApiUrl` to `http://<proxy_host>:3834` or `https://<proxy_host>:3834`.
+-   Set `DdTraceIntakeUrl` to `http://<proxy_host>:3835` or `https://<proxy_host>:3835`.
+-   Set `DdUrl` to `<proxy_host>` and `DdPort` to `3837`.
 
 Otherwise, if you are using Web Proxy:
 
-- Set `DdHttpProxyURL` to your proxy endpoint, for example: `http://<proxy_host>:<port>`, or, if your proxy has a username and password, `http://<username>:<password>@<proxy_host>:<port>`.
+-   Set `DdHttpProxyURL` to your proxy endpoint, for example: `http://<proxy_host>:<port>`, or, if your proxy has a username and password, `http://<username>:<password>@<proxy_host>:<port>`.
 
 7. Set `DdNoSsl` to `true` if connecting to the proxy using `http`.
 8. Set `DdSkipSslValidation` to `true` if connecting to the proxy using `https` with a self-signed certificate.
@@ -423,6 +423,9 @@ Datadog recommends using at least 10 reserved concurrency, but this defaults to
 `DdTags`
 : Add custom tags to forwarded logs, comma-delimited string, no trailing comma, such as `env:prod,stack:classic`.
 
+`DdSource`
+: Override the `ddsource` attribute for all logs forwarded by Lambda Forwarder. By default, the Forwarder automatically detects the source based on the log origin (for example: `lambda`, `s3`, `cloudwatch`, `rds`). When set, all logs will use the specified source value instead, and a `source_overridden:true` tag will be added to the logs.
+
 `DdMultilineLogRegexPattern`
 : Use the supplied regular expression to detect for a new log line for multiline logs from S3, such as `\d{2}\/\d{2}\/\d{4}` for multiline logs beginning with pattern "11/10/2014".
 
@@ -473,10 +476,10 @@ Filtering rules are applied to the log message. However, transformations applied
 
 Some examples of regular expressions that can be used for log filtering:
 
-- Include (or exclude) Lambda platform logs: `(START|END) RequestId:\s`. Datadog recommends keeping the `REPORT` logs, as they are used to populate the invocations list in the serverless function views.
-- Include CloudTrail error messages only: `errorMessage`.
-- Include only logs containing an HTTP 4XX or 5XX error code: `\b[4|5][0-9][0-9]\b`.
-- Include only CloudWatch logs where the `message` field contains a specific JSON key/value pair: `"awsRegion":"us-east-1"`.
+-   Include (or exclude) Lambda platform logs: `(START|END) RequestId:\s`. Datadog recommends keeping the `REPORT` logs, as they are used to populate the invocations list in the serverless function views.
+-   Include CloudTrail error messages only: `errorMessage`.
+-   Include only logs containing an HTTP 4XX or 5XX error code: `\b[4|5][0-9][0-9]\b`.
+-   Include only CloudWatch logs where the `message` field contains a specific JSON key/value pair: `"awsRegion":"us-east-1"`.
 
 To test different patterns against your logs, turn on [debug logs](#troubleshooting).
 
@@ -601,6 +604,9 @@ Datadog recommends using at least 10 reserved concurrency, but this defaults to
 `DD_TAGS`
 : Add custom tags to forwarded logs, comma-delimited string, no trailing comma, such as `env:prod,stack:classic`.
 
+`DD_SOURCE`
+: Override the `ddsource` attribute for all logs forwarded by Lambda Forwarder. By default, the Forwarder automatically detects the source based on the log origin (for example: `lambda`, `s3`, `cloudwatch`, `rds`). When set, all logs will use the specified source value instead, and a `source_overridden:true` tag will be added to the logs.
+
 `DD_MULTILINE_LOG_REGEX_PATTERN`
 : Use the supplied regular expression to detect for a new log line for multiline logs from S3, such as `\d{2}\/\d{2}\/\d{4}` for multiline logs beginning with pattern "11/10/2014".
 
@@ -651,10 +657,10 @@ Filtering rules are applied to the log message as read by the forwarder. Using a
 
 Some examples of regular expressions that can be used for log filtering:
 
-- Include (or exclude) Lambda platform logs: `(START|END) RequestId:\s`. Datadog recommends keeping the `REPORT` logs, as they are used to populate the invocations list in the serverless function views.
-- Include CloudTrail error messages only: `errorMessage`.
-- Include only logs containing an HTTP 4XX or 5XX error code: `\b[4|5][0-9][0-9]\b`.
-- Include only CloudWatch logs where the `message` field contains a specific JSON key/value pair: `"awsRegion":"us-east-1"`.
+-   Include (or exclude) Lambda platform logs: `(START|END) RequestId:\s`. Datadog recommends keeping the `REPORT` logs, as they are used to populate the invocations list in the serverless function views.
+-   Include CloudTrail error messages only: `errorMessage`.
+-   Include only logs containing an HTTP 4XX or 5XX error code: `\b[4|5][0-9][0-9]\b`.
+-   Include only CloudWatch logs where the `message` field contains a specific JSON key/value pair: `"awsRegion":"us-east-1"`.
 
 To test different patterns against your logs, turn on [debug logs](#troubleshooting).
 
@@ -756,12 +762,12 @@ To deploy the CloudFormation Stack with the default options, you need to have th
 
 The following capabilities are required when creating a CloudFormation stack:
 
-- CAPABILITY_AUTO_EXPAND, because the forwarder template uses macros such as the [AWS SAM macro][23].
-- CAPABILTY_IAM/NAMED_IAM, because the Forwarder creates IAM roles.
+-   CAPABILITY_AUTO_EXPAND, because the forwarder template uses macros such as the [AWS SAM macro][23].
+-   CAPABILTY_IAM/NAMED_IAM, because the Forwarder creates IAM roles.
 
 The CloudFormation Stack creates following IAM roles:
 
-- ForwarderRole: The execution role for the Forwarder Lambda function to read logs from S3, fetch your Datadog API key from Secrets Manager, and write its own logs.
+-   ForwarderRole: The execution role for the Forwarder Lambda function to read logs from S3, fetch your Datadog API key from Secrets Manager, and write its own logs.
 
 **IAM statements**
 
@@ -789,7 +795,7 @@ The CloudFormation Stack creates following IAM roles:
 ]
 ```
 
-- `ForwarderZipCopierRole`: The execution role for the ForwarderZipCopier Lambda function to download the Forwarder deployment ZIP file to a S3 bucket.
+-   `ForwarderZipCopierRole`: The execution role for the ForwarderZipCopier Lambda function to download the Forwarder deployment ZIP file to a S3 bucket.
 
 **IAM statements**:
 
@@ -826,7 +832,7 @@ The value of the `service` tag is determined based on multiple inputs. These inp
 
 Additional helpful documentation, links, and articles:
 
-- [Send AWS Services Logs With The Datadog Lambda Function][2]
+-   [Send AWS Services Logs With The Datadog Lambda Function][2]
 
 [1]: https://github.com/DataDog/datadog-lambda-extension
 [2]: https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function/

aws/logs_monitoring/template.yaml

@@ -75,6 +75,10 @@ Parameters:
     Type: String
     Default: ""
     Description: Add custom tags to forwarded logs, comma-delimited string, no trailing comma, e.g., env:prod,stack:classic
+  DdSource:
+    Type: String
+    Default: ""
+    Description: Override the source attribute for all logs forwarded by Lambda Forwarder. By default, the Forwarder automatically detects the source based on the log origin (e.g., lambda, s3, cloudwatch, rds). When set, all logs will use the specified source value instead, and a source_overridden:true tag will be added to the logs.
   DdEnrichS3Tags:
     Type: String
     Default: true
@@ -325,6 +329,8 @@ Conditions:
   SetS3SourceZip: !Equals [!Select [0, !Split [/, !Ref SourceZipUrl]], "s3:"]
   SetDdTags: !Not
     - !Equals [!Ref DdTags, ""]
+  SetDdSource: !Not
+    - !Equals [!Ref DdSource, ""]
   SetDdNoSsl: !Equals [!Ref DdNoSsl, true]
   SetDdUrl: !Not
     - !Equals [!Ref DdUrl, ""]
@@ -477,6 +483,10 @@ Resources:
             - SetDdTags
             - !Ref DdTags
             - !Ref AWS::NoValue
+          DD_SOURCE: !If
+            - SetDdSource
+            - !Ref DdSource
+            - !Ref AWS::NoValue
           DD_TAGS_CACHE_TTL_SECONDS: !Ref TagsCacheTTLSeconds
           DD_ENRICH_S3_TAGS: !Ref DdEnrichS3Tags
           DD_ENRICH_CLOUDWATCH_TAGS: !Ref DdEnrichCloudwatchTags