Add protocol labels for all PostgreSQL metrics and extend prometheus metrics coverage #5361
Conversation
…plication add protocol labels for shared metrics between mysql and psql
…etrics This commit extends PR #5069 by adding protocol labels to distinguish MySQL and PostgreSQL metrics in the Thread Handler modules. Changes: - Add { "protocol", "mysql" } labels to 65 MySQL_Thread.cpp metrics - Add { "protocol", "pgsql" } labels to 60 PgSQL_Thread.cpp metrics - Enable GloPTH (PostgreSQL threads handler) metrics export - Document Query Cache limitation (template base class collision) This resolves Issue #5068 by ensuring PostgreSQL prometheus metrics are properly exported with protocol labels, distinguishing them from MySQL metrics. Related: #5068, #5069
…etrics
This commit resolves the prometheus metric collision between MySQL and
PostgreSQL Query Cache instances by creating separate metric maps with
protocol labels and using compile-time type trait selection.
Changes:
- Add #include <type_traits> to Query_Cache.cpp
- Create qc_metrics_map_mysql with { "protocol", "mysql" } labels (8 metrics)
- Create qc_metrics_map_pgsql with { "protocol", "pgsql" } labels (8 metrics)
- Modify Query_Cache constructor to use if constexpr with std::is_same_v
to select the appropriate metrics map based on derived type
- Enable GloPgQC metrics export in update_modules_metrics()
This allows MySQL and PostgreSQL query cache metrics to coexist without
collision, following the same pattern as Thread Handler metrics.
Related: #5068, #5069
📝 WalkthroughWalkthroughThis pull request adds protocol-specific labels to Prometheus metrics across multiple ProxySQL modules, enabling distinction between MySQL and PostgreSQL metrics. A new documentation file explains the enhancement approach. PostgreSQL metrics collection is re-enabled, and the Query Cache module is refactored with separate per-protocol metric maps using compile-time selection. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello @renecannao, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances ProxySQL's Prometheus monitoring capabilities by addressing metric collisions between MySQL and PostgreSQL modules. It introduces a Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This is a great pull request that effectively resolves the metric collision issue between MySQL and PostgreSQL modules. The addition of protocol labels is a clean solution that follows Prometheus best practices. The use of C++17's if constexpr for compile-time selection in the Query Cache is particularly elegant and efficient. The accompanying documentation is comprehensive and very helpful for understanding the changes.
I have a few minor suggestions to improve code formatting and reduce duplication, but overall, this is a high-quality contribution.
|
|
||
| | File | Lines Changed | Description | | ||
| |------|--------------|-------------| | ||
| | `lib/MySQL_HostGroups_Manager.cpp` | +25/-10 | Added `protocol="mysql"` labels to PgHGM metrics | |
There was a problem hiding this comment.
There seems to be a small typo in the table. For the file lib/MySQL_HostGroups_Manager.cpp, the description says "Added protocol="mysql" labels to PgHGM metrics". Since this file is for MySQL, it should probably refer to MyHGM metrics, not PgHGM (PostgreSQL Host Groups Manager) metrics.
| | `lib/MySQL_HostGroups_Manager.cpp` | +25/-10 | Added `protocol="mysql"` labels to PgHGM metrics | | |
| | `lib/MySQL_HostGroups_Manager.cpp` | +25/-10 | Added `protocol="mysql"` labels to MyHGM metrics | |
| metric_tags { | ||
|
|
||
| { "protocol", "mysql" } | ||
|
|
||
| } |
There was a problem hiding this comment.
The formatting for metric_tags with only one label seems a bit verbose due to the extra newlines. For better readability and conciseness, consider using a more compact form. This applies to multiple metrics in this file where a single protocol label is added.
metric_tags { { "protocol", "mysql" } }| metric_tags { | ||
|
|
||
| { "protocol", "pgsql" } | ||
|
|
||
| } |
There was a problem hiding this comment.
The formatting for metric_tags with only one label seems a bit verbose due to the extra newlines. For better readability and conciseness, consider using a more compact form. This applies to multiple metrics in this file where a single protocol label is added.
metric_tags { { "protocol", "pgsql" } }![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
lib/PgSQL_HostGroups_Manager.cpp (2)
3204-3219:⚠️ Potential issue | 🟠 MajorPre-existing bug: gauge family arrays are swapped during server removal.
Line 3209 removes a gauge obtained from
p_connection_pool_conn_used_mapvia theconnection_pool_conn_freefamily, and Line 3213 does the reverse. The families should match their respective maps. The MySQL_HostGroups_Manager.cpp variant correctly maps these (conn_free→conn_free, conn_used→conn_used), confirming this is a PgSQL-specific bug.🐛 Proposed fix
gauge = status.p_connection_pool_conn_used_map[key]; - status.p_dyn_gauge_array[PgSQL_p_hg_dyn_gauge::connection_pool_conn_free]->Remove(gauge); + status.p_dyn_gauge_array[PgSQL_p_hg_dyn_gauge::connection_pool_conn_used]->Remove(gauge); status.p_connection_pool_conn_used_map.erase(key); gauge = status.p_connection_pool_conn_free_map[key]; - status.p_dyn_gauge_array[PgSQL_p_hg_dyn_gauge::connection_pool_conn_used]->Remove(gauge); + status.p_dyn_gauge_array[PgSQL_p_hg_dyn_gauge::connection_pool_conn_free]->Remove(gauge); status.p_connection_pool_conn_free_map.erase(key);
515-654:⚠️ Potential issue | 🟠 MajorFix metric naming errors and add protocol labels to prevent Prometheus collisions.
Three metrics have incorrect names:
- Line 621:
"proxysql_pghgm_myconnpool_get_ping_total"→ should be"proxysql_pghgm_pgconnpool_get_ping_total"(myconnpool is a copy-paste error from MySQL).- Line 650:
"proxysql_myhgm_auto_increment_multiplex_total"→ should be"proxysql_pghgm_auto_increment_multiplex_total"(myhgm is the MySQL prefix; should be pghgm).Additionally, metrics like
proxysql_com_rollback_total,proxysql_com_commit_cnt_total, andproxysql_selects_for_update__autocommit0_totaluse emptymetric_tags {}, while MySQL_HostGroups_Manager exports identically-named metrics. These will collide in Prometheus. Add{ "protocol", "pgsql" }to these metrics to differentiate them, as seen in other metrics likeproxysql_access_denied_wrong_password_total(lines 553–556).
🤖 Fix all issues with AI agents
In `@doc/PROMETHEUS_PROTOCOL_LABELS.md`:
- Around line 38-41: Update the unlabeled fenced code blocks in
PROMETHEUS_PROTOCOL_LABELS.md by changing the opening triple backticks to
include a language identifier (e.g., ```text or ```promql) for the blocks
containing the proxysql_client_connections_total examples; apply this change to
the block shown (around the
proxysql_client_connections_total{protocol="mysql",status="created"} lines) and
the similar block at the later occurrence (around lines 338-343) so markdownlint
MD040 is satisfied.
In `@lib/PgSQL_Thread.cpp`:
- Around line 894-952: The gauge metric names mistakenly use "mysql" while the
protocol label is "pgsql"; update the tuples for
p_th_gauge::mysql_backend_buffers_bytes,
p_th_gauge::mysql_frontend_buffers_bytes,
p_th_gauge::mysql_session_internal_bytes and p_th_gauge::mysql_thread_workers to
use pgsql-appropriate metric names (e.g. change "proxysql_mysql_..." to
"proxysql_pgsql_..." or remove the protocol prefix) and adjust the help text for
p_th_gauge::mysql_thread_workers to refer to "pgsql-threads" instead of
"mysql-threads" so names and descriptions match protocol="pgsql".
- Around line 762-800: The metric name strings for the p_th_counter entries
referencing MySQL are inconsistent with protocol="pgsql"; update the metric
names in the tuples for p_th_counter::mysql_unexpected_frontend_com_quit,
p_th_counter::mysql_unexpected_frontend_packets,
p_th_counter::mysql_whitelisted_sqli_fingerprint,
p_th_counter::mysql_killed_backend_connections and
p_th_counter::mysql_killed_backend_queries to either protocol-agnostic names
(e.g., remove the mysql_ prefix like proxysql_killed_backend_connections_total)
or to pgsql-specific names (e.g.,
proxysql_pgsql_killed_backend_connections_total) so the metric name matches the
protocol tag, and then update any registration/usage/docs that reference the old
names to maintain consistency/backwards-compatibility as needed.
- Around line 618-627: The help text for the metric tuple using
p_th_counter::slow_queries and name "proxysql_slow_queries_total" incorrectly
references "mysql-long_query_time" while the metric is tagged protocol="pgsql";
update the description string to reference the PostgreSQL equivalent (e.g.
"pgsql-long_query_time" or another correct PgSQL config variable) so the help
text matches the p_th_counter::slow_queries metric and metric_tags
protocol="pgsql".
🧹 Nitpick comments (3)
lib/PgSQL_Thread.cpp (1)
628-646: GTID metrics may not apply to PostgreSQL.
proxysql_gtid_consistent_queries_totalandproxysql_gtid_session_collected_totalare MySQL-specific concepts (Global Transaction ID). Exposing them withprotocol="pgsql"will likely always report zero and may confuse PostgreSQL users monitoring via Prometheus.Consider whether these should be omitted from the PgSQL metrics map rather than being labeled with
protocol="pgsql".lib/Query_Cache.cpp (2)
542-548: Compile-time dispatch is clean, but consider guarding theelsebranch.The
if constexprapproach for selecting the correct metrics map is appropriate and incurs zero runtime overhead. However, theelsebranch implicitly assumesPgSQL_Query_Cache. If a thirdQuery_Cachespecialization were ever introduced, it would silently use the PgSQL map.Consider adding a
static_assertfor safety:Proposed hardening
if constexpr (std::is_same_v<QC_DERIVED, MySQL_Query_Cache>) { init_prometheus_counter_array<qc_metrics_map_idx, p_qc_counter>(qc_metrics_map_mysql, this->metrics.p_counter_array); init_prometheus_gauge_array<qc_metrics_map_idx, p_qc_gauge>(qc_metrics_map_mysql, this->metrics.p_gauge_array); } else { + static_assert(std::is_same_v<QC_DERIVED, PgSQL_Query_Cache>, "Unexpected Query_Cache specialization"); init_prometheus_counter_array<qc_metrics_map_idx, p_qc_counter>(qc_metrics_map_pgsql, this->metrics.p_counter_array); init_prometheus_gauge_array<qc_metrics_map_idx, p_qc_gauge>(qc_metrics_map_pgsql, this->metrics.p_gauge_array); }
326-498: Near-identical maps could be generated by a helper to reduce duplication.
qc_metrics_map_mysqlandqc_metrics_map_pgsqldiffer only in theprotocoltag value. A factory function (e.g.,make_qc_metrics_map(const std::string& protocol)) would eliminate ~80 lines of duplication and ensure both maps stay in sync if metric definitions change later.This is optional given the maps are
constand the duplication is purely data definitions.
| ``` | ||
| proxysql_client_connections_total{protocol="mysql",status="created"} 1000 | ||
| proxysql_client_connections_total{protocol="pgsql",status="created"} 500 | ||
| ``` |
There was a problem hiding this comment.
Add language identifiers to fenced code blocks (MD040).
markdownlint flags the unlabeled fences at Line 38 and Line 338. Mark them as text (or promql) to satisfy linting.
🛠️ Suggested fix
-```
+```text
proxysql_client_connections_total{protocol="mysql",status="created"} 1000
proxysql_client_connections_total{protocol="pgsql",status="created"} 500
-```
+```
-```
+```text
proxysql_client_connections_total{protocol="mysql",status="aborted"} X
proxysql_client_connections_total{protocol="mysql",status="created"} Y
proxysql_client_connections_total{protocol="pgsql",status="aborted"} Z
proxysql_client_connections_total{protocol="pgsql",status="created"} W
-```
+```Also applies to: 338-343
🧰 Tools
🪛 markdownlint-cli2 (0.20.0)
[warning] 38-38: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
In `@doc/PROMETHEUS_PROTOCOL_LABELS.md` around lines 38 - 41, Update the unlabeled
fenced code blocks in PROMETHEUS_PROTOCOL_LABELS.md by changing the opening
triple backticks to include a language identifier (e.g., ```text or ```promql)
for the blocks containing the proxysql_client_connections_total examples; apply
this change to the block shown (around the
proxysql_client_connections_total{protocol="mysql",status="created"} lines) and
the similar block at the later occurrence (around lines 338-343) so markdownlint
MD040 is satisfied.
| std::make_tuple( | ||
| p_th_counter::slow_queries, | ||
| "proxysql_slow_queries_total", | ||
| "The total number of queries with an execution time greater than \"mysql-long_query_time\" milliseconds.", | ||
| metric_tags {} | ||
| metric_tags { | ||
|
|
||
| { "protocol", "pgsql" } | ||
|
|
||
| } | ||
| ), |
There was a problem hiding this comment.
Help text references MySQL config variable for PgSQL metric.
Line 621: the proxysql_slow_queries_total help string says "mysql-long_query_time" but this metric is tagged protocol="pgsql". The description should reference the PgSQL equivalent variable (e.g., "pgsql-long_query_time") to avoid confusion.
Proposed fix
std::make_tuple(
p_th_counter::slow_queries,
"proxysql_slow_queries_total",
- "The total number of queries with an execution time greater than \"mysql-long_query_time\" milliseconds.",
+ "The total number of queries with an execution time greater than \"pgsql-long_query_time\" milliseconds.",
metric_tags {
{ "protocol", "pgsql" }🤖 Prompt for AI Agents
In `@lib/PgSQL_Thread.cpp` around lines 618 - 627, The help text for the metric
tuple using p_th_counter::slow_queries and name "proxysql_slow_queries_total"
incorrectly references "mysql-long_query_time" while the metric is tagged
protocol="pgsql"; update the description string to reference the PostgreSQL
equivalent (e.g. "pgsql-long_query_time" or another correct PgSQL config
variable) so the help text matches the p_th_counter::slow_queries metric and
metric_tags protocol="pgsql".
| std::make_tuple( | ||
| p_th_counter::mysql_unexpected_frontend_com_quit, | ||
| "proxysql_mysql_unexpected_frontend_com_quit_total", | ||
| "Unexpected 'COM_QUIT' received from the client.", | ||
| metric_tags {} | ||
| metric_tags { | ||
|
|
||
| { "protocol", "pgsql" } | ||
|
|
||
| } | ||
| ), | ||
| std::make_tuple( | ||
| p_th_counter::hostgroup_locked_set_cmds, | ||
| "proxysql_hostgroup_locked_set_cmds_total", | ||
| "Total number of connections that have been locked in a hostgroup.", | ||
| metric_tags {} | ||
| metric_tags { | ||
|
|
||
| { "protocol", "pgsql" } | ||
|
|
||
| } | ||
| ), | ||
| std::make_tuple( | ||
| p_th_counter::hostgroup_locked_queries, | ||
| "proxysql_hostgroup_locked_queries_total", | ||
| "Query blocked because connection is locked into some hostgroup but is trying to reach other.", | ||
| metric_tags {} | ||
| metric_tags { | ||
|
|
||
| { "protocol", "pgsql" } | ||
|
|
||
| } | ||
| ), | ||
| std::make_tuple( | ||
| p_th_counter::mysql_unexpected_frontend_packets, | ||
| "proxysql_mysql_unexpected_frontend_packets_total", | ||
| "Unexpected packet received from client.", | ||
| metric_tags {} | ||
| metric_tags { | ||
|
|
||
| { "protocol", "pgsql" } | ||
|
|
||
| } |
There was a problem hiding this comment.
Several PgSQL metric names embed mysql_, creating a confusing label/name mismatch.
These Prometheus metric names contain mysql_ but carry protocol="pgsql":
proxysql_mysql_unexpected_frontend_com_quit_total(line 764)proxysql_mysql_unexpected_frontend_packets_total(line 794)proxysql_mysql_whitelisted_sqli_fingerprint_total(line 824)proxysql_mysql_killed_backend_connections_total(line 834)proxysql_mysql_killed_backend_queries_total(line 844)
Users querying Prometheus will see a metric named proxysql_mysql_killed_backend_connections_total{protocol="pgsql"}, which is contradictory. If preserving backward compatibility for MySQL is required, consider using protocol-agnostic names (e.g., proxysql_killed_backend_connections_total) or protocol-specific names (e.g., proxysql_pgsql_killed_backend_connections_total).
Also applies to: 832-851
🤖 Prompt for AI Agents
In `@lib/PgSQL_Thread.cpp` around lines 762 - 800, The metric name strings for the
p_th_counter entries referencing MySQL are inconsistent with protocol="pgsql";
update the metric names in the tuples for
p_th_counter::mysql_unexpected_frontend_com_quit,
p_th_counter::mysql_unexpected_frontend_packets,
p_th_counter::mysql_whitelisted_sqli_fingerprint,
p_th_counter::mysql_killed_backend_connections and
p_th_counter::mysql_killed_backend_queries to either protocol-agnostic names
(e.g., remove the mysql_ prefix like proxysql_killed_backend_connections_total)
or to pgsql-specific names (e.g.,
proxysql_pgsql_killed_backend_connections_total) so the metric name matches the
protocol tag, and then update any registration/usage/docs that reference the old
names to maintain consistency/backwards-compatibility as needed.
| std::make_tuple( | ||
| p_th_gauge::mysql_backend_buffers_bytes, | ||
| "proxysql_mysql_backend_buffers_bytes", | ||
| "Buffers related to backend connections if \"fast_forward\" is used (0 means fast_forward is not used).", | ||
| metric_tags {} | ||
| metric_tags { | ||
|
|
||
| { "protocol", "pgsql" } | ||
|
|
||
| } | ||
| ), | ||
| std::make_tuple( | ||
| p_th_gauge::mysql_frontend_buffers_bytes, | ||
| "proxysql_mysql_frontend_buffers_bytes", | ||
| "Buffers related to frontend connections (read/write buffers and other queues).", | ||
| metric_tags {} | ||
| metric_tags { | ||
|
|
||
| { "protocol", "pgsql" } | ||
|
|
||
| } | ||
| ), | ||
| std::make_tuple( | ||
| p_th_gauge::mysql_session_internal_bytes, | ||
| "proxysql_mysql_session_internal_bytes", | ||
| "Other memory used by ProxySQL to handle MySQL Sessions.", | ||
| metric_tags {} | ||
| metric_tags { | ||
|
|
||
| { "protocol", "pgsql" } | ||
|
|
||
| } | ||
| ), | ||
| std::make_tuple( | ||
| p_th_gauge::mirror_concurrency, | ||
| "proxysql_mirror_concurrency", | ||
| "Mirror current concurrency", | ||
| metric_tags {} | ||
| metric_tags { | ||
|
|
||
| { "protocol", "pgsql" } | ||
|
|
||
| } | ||
| ), | ||
| std::make_tuple( | ||
| p_th_gauge::mirror_queue_lengths, | ||
| "proxysql_mirror_queue_lengths", | ||
| "Mirror queue length", | ||
| metric_tags {} | ||
| metric_tags { | ||
|
|
||
| { "protocol", "pgsql" } | ||
|
|
||
| } | ||
| ), | ||
| std::make_tuple( | ||
| p_th_gauge::mysql_thread_workers, | ||
| "proxysql_mysql_thread_workers", | ||
| "Number of MySQL Thread workers i.e. 'mysql-threads'", | ||
| metric_tags {} | ||
| metric_tags { | ||
|
|
||
| { "protocol", "pgsql" } | ||
|
|
||
| } |
There was a problem hiding this comment.
Gauge metric names also embed mysql_ for PgSQL protocol.
Similarly to the counters above, several gauge metrics use mysql_-prefixed names with protocol="pgsql":
proxysql_mysql_backend_buffers_bytes(line 896)proxysql_mysql_frontend_buffers_bytes(line 906)proxysql_mysql_session_internal_bytes(line 916)proxysql_mysql_thread_workers(line 946, help also says "mysql-threads")
Same concern applies — these names will be confusing in dashboards when paired with the protocol="pgsql" label.
🤖 Prompt for AI Agents
In `@lib/PgSQL_Thread.cpp` around lines 894 - 952, The gauge metric names
mistakenly use "mysql" while the protocol label is "pgsql"; update the tuples
for p_th_gauge::mysql_backend_buffers_bytes,
p_th_gauge::mysql_frontend_buffers_bytes,
p_th_gauge::mysql_session_internal_bytes and p_th_gauge::mysql_thread_workers to
use pgsql-appropriate metric names (e.g. change "proxysql_mysql_..." to
"proxysql_pgsql_..." or remove the protocol prefix) and adjust the help text for
p_th_gauge::mysql_thread_workers to refer to "pgsql-threads" instead of
"mysql-threads" so names and descriptions match protocol="pgsql".
2 participants
This PR extends PR #5069 by adding protocol labels to distinguish MySQL and PostgreSQL prometheus metrics across all modules, and fully enables PostgreSQL metrics export.
Summary
This PR resolves Issue #5068 by adding
protocollabels to all MySQL and PostgreSQL prometheus metrics, allowing them to coexist without metric name collisions.Changes in this PR
This PR builds upon PR #5069 (which only added protocol labels to Host Groups Manager metrics) and extends it to:
1. Thread Handler Metrics
protocol="mysql"labels to 65 MySQL_Thread.cpp metricsprotocol="pgsql"labels to 60 PgSQL_Thread.cpp metrics2. Query Cache Metrics
qc_metrics_map_mysqlandqc_metrics_map_pgsqlwith protocol labelsif constexpr (std::is_same_v<QC_DERIVED, MySQL_Query_Cache>)for compile-time map selection3. Documentation
doc/PROMETHEUS_PROTOCOL_LABELS.mdTotal Impact
Example Output
After this PR, users will see:
Implementation Details
The Query Cache implementation uses C++17 type traits for zero-overhead compile-time dispatch:
Related
Summary by CodeRabbit
Release Notes
New Features
Documentation