How to integrate GraphDB monitoring with the helm chart?

[jperville]

Hi Ontotext,

How to integrate prometheus monitoring using this chart?

I figured out a way using the extra* attributes of the chart (see snippet below),
but the documentation could be improved.

Generating the template of prometheus and prometheus-operator objects from the chart would be appreciated too.

Note : I am using https://github.com/prometheus/jmx_exporter#configuration to export the metrics that I need, but configuring prometheus serviceMonitor etc using the native GraphDB metrics would be as cumbersome.

Here is what I did using extra init containers, objects etc:

fullnameOverride: "graphdb"

image:
  tag: "10.8.11"

replicas: 1

license:
  existingSecret: "graphdb-license"
  licenseFilename: license

configuration:
  externalUrl: http://graphdb-backends.127.0.0.1.nip.io
  javaArguments: "-javaagent:/opt/graphdb/plugins/jmx_prometheus_javaagent.jar=0.0.0.0:9123:/opt/graphdb/jmx-exporter.yml"

# configuration of the custom prometheus exporter + ServiceMonitor follows
extraVolumes:
  - name: plugins
    emptyDir: {}
  - name: plugins-config
    configMap:
      name: graphdb-plugins # see extraObjects below
extraVolumeMounts:
  - name: plugins
    mountPath: /opt/graphdb/plugins
    readOnly: true
  - name: plugins-config
    mountPath: /opt/graphdb/jmx-exporter.yml
    subPath: jmx-exporter.yml
    readOnly: true
extraInitContainers:
  - name: plugins
    image: docker.io/library/alpine:latest
    command:
      - "/usr/bin/wget"
    args:
      - "-qO"
      - "/target/jmx_prometheus_javaagent.jar"
      - "https://github.com/prometheus/jmx_exporter/releases/download/$(JAVA_AGENT_PLUGIN_VERSION)/jmx_prometheus_javaagent-$(JAVA_AGENT_PLUGIN_VERSION).jar"
    env:
      - name: JAVA_AGENT_PLUGIN_VERSION
        value: 1.5.0
    volumeMounts:
      - name: plugins
        mountPath: /target
extraContainerPorts:
  - name: metrics
    containerPort: 9123
extraObjects:
  # prometheus-exporter configuration, mounted via extraVolumes/extraVolumeMounts
  - apiVersion: v1
    kind: ConfigMap
    metadata:
      name: graphdb-plugins
    data:
      jmx-exporter.yml: |
        # https://github.com/prometheus/jmx_exporter#configuration
        lowercaseOutputName: true
        lowercaseOutputLabelNames: false

        blacklistObjectNames:
          - "Tomcat*:*"
          - "jmx4perl*:*"
          - "jolokia*:*"

        rules:
          #- pattern: (.*)
          #  name: graphdb_$0

          # global worker metrics
          - pattern: 'com.ontotext.metrics<name=cache.(\w+)><>Count:'
            name: graphdb_global_cache_$1_count
            help: GraphDB global cache $1 count
            type: COUNTER

          # per-repository running query counter
          - pattern: 'com.ontotext<type=RepositoryMonitor, name="(\w+)"><>TrackRecordsNumber:'
            name: graphdb_running_queries_count
            labels:
              repository: "$1"
            help: GraphDB number of running queries per repository
            type: GAUGE

          # per-repository worker metrics
          - pattern: 'com.ontotext.metrics.(\w+)<name=epool.(\w+)><>Count:'
            name: graphdb_worker_entity_pool_$2_count
            labels:
              repository: "$1"
            help: GraphDB worker repository entity pool $2 count
            type: COUNTER
  # metrics service
  - apiVersion: v1
    kind: Service
    metadata:
      labels:
        name: graphdb-metrics
      name: graphdb-metrics
      namespace: backends
    spec:
      ports:
        - name: metrics
          port: 9123
          targetPort: metrics
      selector:
        name: graphdb
      type: ClusterIP
  # prometheus service monitor
  - apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      labels:
        name: graphdb
      name: graphdb
      namespace: backends
    spec:
      endpoints:
        - interval: 10s
          path: /metrics
          port: metrics
          relabelings:
            - action: replace
              regex: (.*)
              replacement: $1
              sourceLabels:
                - __meta_kubernetes_pod_name
              targetLabel: instance
          scrapeTimeout: 10s
      selector:
        matchLabels:
          name: graphdb-metrics