[Azure Monitor] Add default timegrain to Azure Storage Account metricset (#46786) (#46852)

Adds a new `default_timegrain` configuration option to allow users to customize the timegrain used in the Storage Account metricset. The default value remains PT5M, but users can now choose a different value.

Without this option, users can only collect metrics with a PT5M time grain. It is a sensible default, but some users want to collect metrics with a PT1M time grain.

To learn more, see elastic/integrations#15464.

(cherry picked from commit 8f145b9)

Co-authored-by: Maurizio Branca <[email protected]>

Author: mergify[bot]

CHANGELOG.next.asciidoc

@@ -260,6 +260,15 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
 - Add new metricset network for the vSphere module. {pull}40559[40559]
 - Add new metricset resourcepool for the vSphere module. {pull}40456[40456]
 - Add new metrics to vSphere Virtual Machine dataset (CPU usage percentage, disk average usage, disk read/write rate, number of disk reads/writes, memory usage percentage). {pull}44205[44205]
+- Added checks for the Resty response object in all Meraki module API calls to ensure proper handling of nil responses. {pull}44193[44193]
+- Add latency config option to Azure Monitor module. {pull}44366[44366]
+- Increase default polling period for MongoDB module from 10s to 60s {pull}44781[44781]
+- Upgrade github.com/microsoft/go-mssqldb from v1.7.2 to v1.8.2 {pull}44990[44990]
+- Add SSL support for sql module: drivers mysql, postgres, and mssql. {pull}44748[44748]
+- Add support for Kafka 4.0 in the Kafka module. {pull}44723[44723]
+- Add NTP response validation for system/ntp module. {pull}46184[46184]
+- Add vertexai_logs metricset to GCP for prompt response collection from VertexAI service. {pull}46383[46383]
+- Add default timegrain to Azure Storage Account metricset. {pull}46786[46786]
 
 *Metricbeat*
 

docs/reference/metricbeat/metricbeat-metricset-azure-storage.md

@@ -13,7 +13,6 @@ This is the storage metricset of the module azure.
 
 This metricset allows users to retrieve all metrics from specified storage accounts.
 
-
 ## Metricset-specific configuration notes [_metricset_specific_configuration_notes_11]
 
 `refresh_list_interval`
@@ -22,7 +21,6 @@ This metricset allows users to retrieve all metrics from specified storage accou
 `resources`
 :   This will contain all options for identifying resources and configuring the desired metrics
 
-
 ### Config options to identify resources [_config_options_to_identify_resources_11]
 
 `resource_id`
@@ -42,12 +40,61 @@ resources:
       service_type: ["queue", "file"]
 ```
 
-it will filter the metric values to be returned by specific metric namespaces. The supported metrics and namespaces can be found here [https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported#microsoftstoragestorageaccounts](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported#microsoftstoragestorageaccounts). The service type values allowed are `blob`, `table`, `queue`, `file` based on the namespaces  `Microsoft.Storage/storageAccounts/blobServices`,`Microsoft.Storage/storageAccounts/tableServices`,`Microsoft.Storage/storageAccounts/fileServices`,`Microsoft.Storage/storageAccounts/queueServices`. If no service_type is specified all values are applied.
+It will filter the metric values to be returned by specific metric namespaces. The supported metrics and namespaces can be found here [https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported#microsoftstoragestorageaccounts](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported#microsoftstoragestorageaccounts). The service type values allowed are `blob`, `table`, `queue`, `file` based on the namespaces  `Microsoft.Storage/storageAccounts/blobServices`,`Microsoft.Storage/storageAccounts/tableServices`,`Microsoft.Storage/storageAccounts/fileServices`,`Microsoft.Storage/storageAccounts/queueServices`. If no service_type is specified all values are applied.
 
 Also, if the `resources` option is not specified, then all the storage accounts from the entire subscription will be selected. The primary aggregation value will be retrieved for all the metrics contained in the namespaces. The aggregation options are `avg`, `sum`, `min`, `max`, `total`, `count`.
 
 A default non configurable timegrain of 5 min is set so users are advised to configure an interval of 300s or  a multiply of it.
 
+`default_timegrain`:
+:   (*string*) Sets the default time grain to use when collecting storage account metrics. Defaults to PT5M.
+
+To collect storage account metrics with a PT1M time grain, we recommend using one of the following configurations:
+
+```yaml
+# (1) With `period: 60s` and `default_timegrain: "PT1M"`, the metricset 
+# collects 1 data point every 60s.
+- module: azure
+  metricsets:
+  - storage
+  enabled: true
+  period: 60s
+  client_id: '${AZURE_CLIENT_ID:""}'
+  client_secret: '${AZURE_CLIENT_SECRET:""}'
+  tenant_id: '${AZURE_TENANT_ID:""}'
+  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
+  refresh_list_interval: 3600s # 1h
+  enable_batch_api: true
+  default_timegrain: "PT1M"
+```
+
+```yaml
+# (2) With `period: 300s` and `default_timegrain: "PT1M"`, the metricset
+# collects 5 data points every 300s (5 minutes) — one for each minute, 
+# but all data points arrive after 5 minutes
+- module: azure
+  metricsets:
+  - storage
+  enabled: true
+  period: 300s
+  client_id: '${AZURE_CLIENT_ID:""}'
+  client_secret: '${AZURE_CLIENT_SECRET:""}'
+  tenant_id: '${AZURE_TENANT_ID:""}'
+  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
+  refresh_list_interval: 3600s # 1h
+  enable_batch_api: true
+  default_timegrain: "PT1M"
+```
+
+These two configurations trade off scalability and freshness. Configuration (1) prioritizes freshness over scalability, while configuration (2) prioritizes scalability over freshness.
+
+Suggested changes:
+
+- `enable_batch_api: true`: Retrieves metric values for multiple Azure resources in one API call, supporting more storage accounts.
+- `refresh_list_interval: 3600s`: Looks for new storage accounts every 60 minutes instead of 10 minutes, helping to avoid or reduce gaps when monitoring many storage accounts.
+
+Note: By setting the collection `period: 1m`, the metricset only has 60s to collect all metric values instead of 300s, so it can handle fewer storage accounts. Keep in mind that the storage accounts metricset collects metrics for five different namespaces (storage account, blob, file, queue, and table).
+
 ## Fields [_fields]
 
 For a description of each field in the metricset, see the [exported fields](/reference/metricbeat/exported-fields-azure.md) section.

x-pack/metricbeat/module/azure/azure.go

@@ -73,7 +73,7 @@ var supportedMonitorMetricsets = []string{"monitor", "container_registry", "cont
 // NewMetricSet will instantiate a new azure metricset
 func NewMetricSet(base mb.BaseMetricSet) (*MetricSet, error) {
 	metricsetName := base.Name()
-	var config Config
+	config := createDefaultConfig()
 	err := base.Module().UnpackConfig(&config)
 	if err != nil {
 		return nil, fmt.Errorf("error unpack raw module config using UnpackConfig: %w", err)

x-pack/metricbeat/module/azure/client_utils.go

@@ -17,9 +17,6 @@ import (
 	"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/monitor/armmonitor"
 )
 
-// DefaultTimeGrain is set as default timegrain for the azure metrics
-const DefaultTimeGrain = "PT5M"
-
 var instanceIdRegex = regexp.MustCompile(`.*?(\d+)$`)
 
 // mapMetricValues should map the metric values

x-pack/metricbeat/module/azure/config.go

@@ -49,6 +49,22 @@ type Config struct {
 	BillingScopeAccountId  string `config:"billing_scope_account_id"` // retrieve usage details from billing account ID scope
 	// Use BatchApi for metric values collection
 	EnableBatchApi bool `config:"enable_batch_api"` // defaults to false
+	// DefaultTimeGrain sets the default time interval when the resource config
+	// doesn't specify one. If no time grain is configured, this value will be
+	// used whenever possible.
+	//
+	// When the metric definition doesn't support this time grain, we fall back
+	// to the smallest supported interval.
+	//
+	// Note: currently, this is only used for the storage metricset.
+	DefaultTimeGrain string `config:"default_timegrain"` // defaults to PT5M
+}
+
+// createDefaultConfig creates a default config for the metricset.
+func createDefaultConfig() Config {
+	return Config{
+		DefaultTimeGrain: "PT5M",
+	}
 }
 
 // ResourceConfig contains resource and metric list specific configuration.

x-pack/metricbeat/module/azure/storage/_meta/docs.md

@@ -2,7 +2,6 @@ This is the storage metricset of the module azure.
 
 This metricset allows users to retrieve all metrics from specified storage accounts.
 
-
 ## Metricset-specific configuration notes [_metricset_specific_configuration_notes_11]
 
 `refresh_list_interval`
@@ -11,7 +10,6 @@ This metricset allows users to retrieve all metrics from specified storage accou
 `resources`
 :   This will contain all options for identifying resources and configuring the desired metrics
 
-
 ### Config options to identify resources [_config_options_to_identify_resources_11]
 
 `resource_id`
@@ -31,8 +29,57 @@ resources:
       service_type: ["queue", "file"]
 ```
 
-it will filter the metric values to be returned by specific metric namespaces. The supported metrics and namespaces can be found here [https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported#microsoftstoragestorageaccounts](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported#microsoftstoragestorageaccounts). The service type values allowed are `blob`, `table`, `queue`, `file` based on the namespaces  `Microsoft.Storage/storageAccounts/blobServices`,`Microsoft.Storage/storageAccounts/tableServices`,`Microsoft.Storage/storageAccounts/fileServices`,`Microsoft.Storage/storageAccounts/queueServices`. If no service_type is specified all values are applied.
+It will filter the metric values to be returned by specific metric namespaces. The supported metrics and namespaces can be found here [https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported#microsoftstoragestorageaccounts](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported#microsoftstoragestorageaccounts). The service type values allowed are `blob`, `table`, `queue`, `file` based on the namespaces  `Microsoft.Storage/storageAccounts/blobServices`,`Microsoft.Storage/storageAccounts/tableServices`,`Microsoft.Storage/storageAccounts/fileServices`,`Microsoft.Storage/storageAccounts/queueServices`. If no service_type is specified all values are applied.
 
 Also, if the `resources` option is not specified, then all the storage accounts from the entire subscription will be selected. The primary aggregation value will be retrieved for all the metrics contained in the namespaces. The aggregation options are `avg`, `sum`, `min`, `max`, `total`, `count`.
 
 A default non configurable timegrain of 5 min is set so users are advised to configure an interval of 300s or  a multiply of it.
+
+`default_timegrain`:
+:   (*string*) Sets the default time grain to use when collecting storage account metrics. Defaults to PT5M.
+
+To collect storage account metrics with a PT1M time grain, we recommend using one of the following configurations:
+
+```yaml
+# (1) With `period: 60s` and `default_timegrain: "PT1M"`, the metricset 
+# collects 1 data point every 60s.
+- module: azure
+  metricsets:
+  - storage
+  enabled: true
+  period: 60s
+  client_id: '${AZURE_CLIENT_ID:""}'
+  client_secret: '${AZURE_CLIENT_SECRET:""}'
+  tenant_id: '${AZURE_TENANT_ID:""}'
+  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
+  refresh_list_interval: 3600s # 1h
+  enable_batch_api: true
+  default_timegrain: "PT1M"
+```
+
+```yaml
+# (2) With `period: 300s` and `default_timegrain: "PT1M"`, the metricset
+# collects 5 data points every 300s (5 minutes) — one for each minute, 
+# but all data points arrive after 5 minutes
+- module: azure
+  metricsets:
+  - storage
+  enabled: true
+  period: 300s
+  client_id: '${AZURE_CLIENT_ID:""}'
+  client_secret: '${AZURE_CLIENT_SECRET:""}'
+  tenant_id: '${AZURE_TENANT_ID:""}'
+  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
+  refresh_list_interval: 3600s # 1h
+  enable_batch_api: true
+  default_timegrain: "PT1M"
+```
+
+These two configurations trade off scalability and freshness. Configuration (1) prioritizes freshness over scalability, while configuration (2) prioritizes scalability over freshness.
+
+Suggested changes:
+
+- `enable_batch_api: true`: Retrieves metric values for multiple Azure resources in one API call, supporting more storage accounts.
+- `refresh_list_interval: 3600s`: Looks for new storage accounts every 60 minutes instead of 10 minutes, helping to avoid or reduce gaps when monitoring many storage accounts.
+
+Note: By setting the collection `period: 1m`, the metricset only has 60s to collect all metric values instead of 300s, so it can handle fewer storage accounts. Keep in mind that the storage accounts metricset collects metrics for five different namespaces (storage account, blob, file, queue, and table).

x-pack/metricbeat/module/azure/storage/client_helper.go

@@ -56,7 +56,7 @@ func mapMetrics(client *azure.Client, resources []*armresources.GenericResourceE
 			}
 
 			// some metrics do not support the default PT5M timegrain so they will have to be grouped in a different API call, else call will fail
-			groupedMetrics := groupOnTimeGrain(filteredMetricDefinitions)
+			groupedMetrics := groupOnTimeGrain(filteredMetricDefinitions, client.Config.DefaultTimeGrain)
 
 			for time, groupedMetricList := range groupedMetrics {
 				// metrics will have to be grouped by allowed dimensions
@@ -78,11 +78,11 @@ func mapMetrics(client *azure.Client, resources []*armresources.GenericResourceE
 }
 
 // groupOnTimeGrain - some metrics do not support the default timegrain value so the closest supported timegrain will be selected
-func groupOnTimeGrain(list []armmonitor.MetricDefinition) map[string][]armmonitor.MetricDefinition {
+func groupOnTimeGrain(list []armmonitor.MetricDefinition, defaultTimeGrain string) map[string][]armmonitor.MetricDefinition {
 	var groupedList = make(map[string][]armmonitor.MetricDefinition)
 
 	for _, metric := range list {
-		timegrain := retrieveSupportedMetricAvailability(metric.MetricAvailabilities)
+		timegrain := retrieveSupportedMetricAvailability(metric.MetricAvailabilities, defaultTimeGrain)
 		if _, ok := groupedList[timegrain]; !ok {
 			groupedList[timegrain] = make([]armmonitor.MetricDefinition, 0)
 		}
@@ -92,21 +92,17 @@ func groupOnTimeGrain(list []armmonitor.MetricDefinition) map[string][]armmonito
 }
 
 // retrieveSupportedMetricAvailability func will return the default timegrain if supported, else will return the next timegrain
-func retrieveSupportedMetricAvailability(availabilities []*armmonitor.MetricAvailability) string {
+func retrieveSupportedMetricAvailability(availabilities []*armmonitor.MetricAvailability, defaultTimeGrain string) string {
 	// common case in metrics supported by storage account - one availability
 	if len(availabilities) == 1 {
 		return *availabilities[0].TimeGrain
 	}
 	// check if the default timegrain is supported
 	for _, availability := range availabilities {
-		if *availability.TimeGrain == azure.DefaultTimeGrain {
-			return azure.DefaultTimeGrain
+		if *availability.TimeGrain == defaultTimeGrain {
+			return defaultTimeGrain
 		}
 	}
-	// select first timegrain, should be bigger than the min timegrain of 1M, timegrains are returned in asc order
-	if *availabilities[0].TimeGrain != "PT1M" {
-		return *availabilities[0].TimeGrain
-	}
 	return *availabilities[1].TimeGrain
 }
 

x-pack/metricbeat/module/azure/storage/client_helper_concurrent.go

@@ -65,7 +65,7 @@ func getStorageMappedResourceDefinitions(client *azure.BatchClient, resourceId s
 			filteredMetricDefinitions = append(filteredMetricDefinitions, *metricDefinition)
 		}
 		// some metrics do not support the default PT5M timegrain so they will have to be grouped in a different API call, else call will fail
-		groupedMetrics := groupOnTimeGrain(filteredMetricDefinitions)
+		groupedMetrics := groupOnTimeGrain(filteredMetricDefinitions, client.Config.DefaultTimeGrain)
 		for time, groupedMetricList := range groupedMetrics {
 			// metrics will have to be grouped by allowed dimensions
 			dimMetrics := groupMetricsByAllowedDimensions(groupedMetricList)

x-pack/metricbeat/module/azure/storage/client_helper_test.go

@@ -162,7 +162,7 @@ func TestFilterOnTimeGrain(t *testing.T) {
 		{MetricAvailabilities: availability2},
 		{MetricAvailabilities: availability3},
 	}
-	response := groupOnTimeGrain(list)
+	response := groupOnTimeGrain(list, "PT5M")
 	assert.Equal(t, len(response), 2)
 	result := [][]armmonitor.MetricDefinition{
 		{
@@ -180,11 +180,11 @@ func TestFilterOnTimeGrain(t *testing.T) {
 }
 
 func TestRetrieveSupportedMetricAvailability(t *testing.T) {
-	response := retrieveSupportedMetricAvailability(availability1)
+	response := retrieveSupportedMetricAvailability(availability1, "PT5M")
 	assert.Equal(t, response, time2)
-	response = retrieveSupportedMetricAvailability(availability2)
+	response = retrieveSupportedMetricAvailability(availability2, "PT5M")
 	assert.Equal(t, response, time3)
-	response = retrieveSupportedMetricAvailability(availability3)
+	response = retrieveSupportedMetricAvailability(availability3, "PT5M")
 	assert.Equal(t, response, time3)
 }