Seeing out of order samples scrapping from prometheus

[carlosm]

Hi, I'm come from prometheus issue, here: prometheus/prometheus#14973

I've tested in the last version v0.54.1 and also the prometheus 3.8.1, and I'm seeing Out of Order samples still in prometheus agent logs, not sure exactly which metrics is it but as the threadads there's plenty of people reporting the same issue. Is this known, is there a way to fix it in cAdvisor?

Thanks

[dims]

@carlosm is there a proposed patch/PR to fix things in cadvisor?

[carlosm]

Adding some example of errors, in this case is machine_* tags

time=2025-12-18T15:05:39.564Z level=DEBUG source=scrape.go:2045 msg="Out of order sample" component="scrape manager" scrape_pool=docker target=http://vbareprometheus.obs:9174/metrics series="machine_cpu_cores{boot_id=\"9e01ff89-c0a8-4cac-b419-4a69d3007ace\",machine_id=\"ab448c8ba268462baa863e52d51d7b48\",system_uuid=\"e94da1cc-2c57-11b2-a85c-efc134325f41\"}"
time=2025-12-18T15:05:39.564Z level=DEBUG source=scrape.go:2045 msg="Out of order sample" component="scrape manager" scrape_pool=docker target=http://vbareprometheus.obs:9174/metrics series="machine_cpu_physical_cores{boot_id=\"9e01ff89-c0a8-4cac-b419-4a69d3007ace\",machine_id=\"ab448c8ba268462baa863e52d51d7b48\",system_uuid=\"e94da1cc-2c57-11b2-a85c-efc134325f41\"}"
time=2025-12-18T15:05:39.564Z level=DEBUG source=scrape.go:2045 msg="Out of order sample" component="scrape manager" scrape_pool=docker target=http://vbareprometheus.obs:9174/metrics series="machine_cpu_sockets{boot_id=\"9e01ff89-c0a8-4cac-b419-4a69d3007ace\",machine_id=\"ab448c8ba268462baa863e52d51d7b48\",system_uuid=\"e94da1cc-2c57-11b2-a85c-efc134325f41\"}"
time=2025-12-18T15:05:39.564Z level=DEBUG source=scrape.go:2045 msg="Out of order sample" component="scrape manager" scrape_pool=docker target=http://vbareprometheus.obs:9174/metrics series="machine_memory_bytes{boot_id=\"9e01ff89-c0a8-4cac-b419-4a69d3007ace\",machine_id=\"ab448c8ba268462baa863e52d51d7b48\",system_uuid=\"e94da1cc-2c57-11b2-a85c-efc134325f41\"}"
time=2025-12-18T15:05:39.564Z level=DEBUG source=scrape.go:2045 msg="Out of order sample" component="scrape manager" scrape_pool=docker target=http://vbareprometheus.obs:9174/metrics series="machine_nvm_avg_power_budget_watts{boot_id=\"9e01ff89-c0a8-4cac-b419-4a69d3007ace\",machine_id=\"ab448c8ba268462baa863e52d51d7b48\",system_uuid=\"e94da1cc-2c57-11b2-a85c-efc134325f41\"}"
time=2025-12-18T15:05:39.564Z level=DEBUG source=scrape.go:2045 msg="Out of order sample" component="scrape manager" scrape_pool=docker target=http://vbareprometheus.obs:9174/metrics series="machine_nvm_capacity{boot_id=\"9e01ff89-c0a8-4cac-b419-4a69d3007ace\",machine_id=\"ab448c8ba268462baa863e52d51d7b48\",mode=\"app_direct_mode\",system_uuid=\"e94da1cc-2c57-11b2-a85c-efc134325f41\"}"
time=2025-12-18T15:05:39.564Z level=DEBUG source=scrape.go:2045 msg="Out of order sample" component="scrape manager" scrape_pool=docker target=http://vbareprometheus.obs:9174/metrics series="machine_nvm_capacity{boot_id=\"9e01ff89-c0a8-4cac-b419-4a69d3007ace\",machine_id=\"ab448c8ba268462baa863e52d51d7b48\",mode=\"memory_mode\",system_uuid=\"e94da1cc-2c57-11b2-a85c-efc134325f41\"}"
time=2025-12-18T15:05:39.564Z level=DEBUG source=scrape.go:2045 msg="Out of order sample" component="scrape manager" scrape_pool=docker target=http://vbareprometheus.obs:9174/metrics series="machine_swap_bytes{boot_id=\"9e01ff89-c0a8-4cac-b419-4a69d3007ace\",machine_id=\"ab448c8ba268462baa863e52d51d7b48\",system_uuid=\"e94da1cc-2c57-11b2-a85c-efc134325

[carlosm]

I discovered something using the simples config for scrapping, and the default for prometheus (agent) and cAdvisor, the issue is when running prometheus in the host (localhost:port) shown errors instead of running in docker (host.internal.docker:8080), not sure what's the reason for the different behaviour in that case.