feat: Service discovery and exposition (#94)

* feat: Rename discovery service

* Set initial_cluster_manager_nodes only on cluster_manager nodes; Add opensearch-discovery scope only on cluster_manager nodes

* Fields discoveryServiceExposed and discoveryServiceListenerClass added to the CRD

* Rename opensearch-discovery service to opensearch-seed-nodes and remove the HTTP port

* wip

* Set seed nodes service scope on the internal TLS volume

* test: Use the discovery ConfigMap in all tests

* test(backup-restore): Add the option to disable TLS in S3

* Publish fully qualified domain names so that the SANs in the TLS certificates are matched

* test(opensearch-dashboards): Use the OpenSearch discovery ConfigMap for OpenSearch Dashboards

* Fix unit tests

* test: Use the images built by CI

* chore: Use constant for HTTP port name

* Fix regexes for attributed string types; Add unit tests for validation

* test: Add unit tests for Port

* Remove TODO after successful test

* test(smoke): Remove assertions about PVC statuses

* Add missing start and end tags to regular expressions

* Update changelog

* docs: Document the discovery ConfigMap

* docs: Reference Discovery ConfigMap in the ListenerClass usage guide

* Remove allow_k8s_contexts from Tiltfile

* test: Make unit tests less verbose

* test: Remove OpenSearch 3.4.0 for now

* feat: Add OPENSEARCH_HOSTS to discovery ConfigMap

* docs: Use the discovery ConfigMap in "First Steps"

* docs: Use the discovery ConfigMap in "OpenSearch Dashboards"

* docs: Fix clippy warnings

* docs: Update image with deployed Kubernetes resources

* docs: Update the description of the Kubernetes resources

* Run pre-commit

Author: siegfriedweber

CHANGELOG.md

@@ -14,6 +14,17 @@ All notable changes to this project will be documented in this file.
 - Enable the [restart-controller](https://docs.stackable.tech/home/nightly/commons-operator/restarter/), so that the Pods are automatically restarted on config changes ([#97]).
 - Configure OpenSearch to publish the fully-qualified domain names of the nodes instead of the IP
   addresses, so that TLS certificates can be verified ([#100]).
+- Add service discovery and exposition ([#94]):
+  - Service to set up the cluster renamed to `<cluster-name>-seed-nodes`.
+  - Discovery service named `<cluster-name>`, added.
+    The discovery service is used to populate the discovery ConfigMap.
+  - Discovery ConfigMap named `<cluster-name>`, added.
+    The ConfigMap contains the keys `OPENSEARCH_HOSTNAME`, `OPENSEARCH_PORT`, `OPENSEARCH_PROTOCOL`
+    and `OPENSEARCH_HOSTS`. Users should use this information to connect to the cluster.
+  - Configuration parameter `spec.nodes.roleConfig.discoveryServiceListenerClass` added to set the
+    ListenerClass for the discovery service.
+  - Configuration parameter `spec.nodes.roleGroups.<role-group-name>.config.discoveryServiceExposed`
+    added to expose a role-group via the discovery service.
 
 ### Changed
 
@@ -24,6 +35,7 @@ All notable changes to this project will be documented in this file.
 [#76]: https://github.com/stackabletech/opensearch-operator/pull/76
 [#91]: https://github.com/stackabletech/opensearch-operator/pull/91
 [#93]: https://github.com/stackabletech/opensearch-operator/pull/93
+[#94]: https://github.com/stackabletech/opensearch-operator/pull/94
 [#97]: https://github.com/stackabletech/opensearch-operator/pull/97
 [#100]: https://github.com/stackabletech/opensearch-operator/pull/100
 

deploy/helm/opensearch-operator/crds/crds.yaml

@@ -246,6 +246,10 @@ spec:
                               type: object
                               x-kubernetes-preserve-unknown-fields: true
                           type: object
+                        discoveryServiceExposed:
+                          description: Determines whether this role group is exposed in the discovery service.
+                          nullable: true
+                          type: boolean
                         gracefulShutdownTimeout:
                           description: |-
                             Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the
@@ -517,11 +521,19 @@ spec:
                       x-kubernetes-preserve-unknown-fields: true
                     roleConfig:
                       default:
+                        discoveryServiceListenerClass: cluster-internal
                         podDisruptionBudget:
                           enabled: true
                           maxUnavailable: null
                       description: This is a product-agnostic RoleConfig, which is sufficient for most of the products.
                       properties:
+                        discoveryServiceListenerClass:
+                          default: cluster-internal
+                          description: The [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) that is used for the discovery service.
+                          maxLength: 253
+                          minLength: 1
+                          pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
+                          type: string
                         podDisruptionBudget:
                           default:
                             enabled: true
@@ -600,6 +612,10 @@ spec:
                                     type: object
                                     x-kubernetes-preserve-unknown-fields: true
                                 type: object
+                              discoveryServiceExposed:
+                                description: Determines whether this role group is exposed in the discovery service.
+                                nullable: true
+                                type: boolean
                               gracefulShutdownTimeout:
                                 description: |-
                                   Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the

docs/modules/opensearch/examples/getting_started/getting_started.sh

@@ -75,25 +75,21 @@ kubectl rollout status --watch statefulset/simple-opensearch-nodes-default --tim
 # wait a bit for the port to open
 sleep 10
 
-echo "Starting port-forwarding of port 9200"
-# tag::opensearch-port-forwarding[]
-kubectl port-forward services/simple-opensearch 9200 > /dev/null 2>&1 &
-# end::opensearch-port-forwarding[]
-PORT_FORWARD_PID=$!
-# shellcheck disable=2064 # we want the PID evaluated now, not at the time the trap is
-trap "kill $PORT_FORWARD_PID" EXIT
-sleep 5
-
 echo "Using the REST API"
 # tag::rest-api[]
 export CREDENTIALS=admin:AJVFsGJBbpT6mChn
 
+OPENSEARCH_HOST=$(
+    kubectl get configmap simple-opensearch \
+        --output=jsonpath='{.data.OPENSEARCH_HOSTS}'
+)
+
 curl \
     --insecure \
     --user $CREDENTIALS \
     --request PUT \
     --json '{"name": "Stackable"}' \
-    https://localhost:9200/sample_index/_doc/1
+    "$OPENSEARCH_HOST/sample_index/_doc/1"
 
 # Output:
 # {"_index":"sample_index","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}
@@ -102,7 +98,7 @@ curl \
     --insecure \
     --user $CREDENTIALS \
     --request GET \
-    https://localhost:9200/sample_index/_doc/1
+    "$OPENSEARCH_HOST/sample_index/_doc/1"
 
 # Output:
 # {"_index":"sample_index","_id":"1","_version":1,"_seq_no":0,"_primary_term":1,"found":true,"_source":{"name": "Stackable"}}

docs/modules/opensearch/examples/getting_started/getting_started.sh.j2

@@ -75,25 +75,21 @@ kubectl rollout status --watch statefulset/simple-opensearch-nodes-default --tim
 # wait a bit for the port to open
 sleep 10
 
-echo "Starting port-forwarding of port 9200"
-# tag::opensearch-port-forwarding[]
-kubectl port-forward services/simple-opensearch 9200 > /dev/null 2>&1 &
-# end::opensearch-port-forwarding[]
-PORT_FORWARD_PID=$!
-# shellcheck disable=2064 # we want the PID evaluated now, not at the time the trap is
-trap "kill $PORT_FORWARD_PID" EXIT
-sleep 5
-
 echo "Using the REST API"
 # tag::rest-api[]
 export CREDENTIALS=admin:AJVFsGJBbpT6mChn
 
+OPENSEARCH_HOST=$(
+    kubectl get configmap simple-opensearch \
+        --output=jsonpath='{.data.OPENSEARCH_HOSTS}'
+)
+
 curl \
     --insecure \
     --user $CREDENTIALS \
     --request PUT \
     --json '{"name": "Stackable"}' \
-    https://localhost:9200/sample_index/_doc/1
+    "$OPENSEARCH_HOST/sample_index/_doc/1"
 
 # Output:
 # {"_index":"sample_index","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}
@@ -102,7 +98,7 @@ curl \
     --insecure \
     --user $CREDENTIALS \
     --request GET \
-    https://localhost:9200/sample_index/_doc/1
+    "$OPENSEARCH_HOST/sample_index/_doc/1"
 
 # Output:
 # {"_index":"sample_index","_id":"1","_version":1,"_seq_no":0,"_primary_term":1,"found":true,"_source":{"name": "Stackable"}}

docs/modules/opensearch/examples/getting_started/opensearch-dashboards-values.yaml

@@ -1,5 +1,4 @@
 ---
-opensearchHosts: https://simple-opensearch-nodes-default.default.svc.cluster.local:9200
 image:
   repository: oci.stackable.tech/sdp/opensearch-dashboards
   tag: 3.1.0-stackable0.0.0-dev
@@ -23,6 +22,11 @@ config:
       cookie:
         secure: true
 extraEnvs:
+  - name: OPENSEARCH_HOSTS
+    valueFrom:
+      configMapKeyRef:
+        name: simple-opensearch
+        key: OPENSEARCH_HOSTS
   - name: OPENSEARCH_PASSWORD
     valueFrom:
       secretKeyRef:

docs/modules/opensearch/examples/getting_started/opensearch-dashboards-values.yaml.j2

@@ -1,5 +1,4 @@
 ---
-opensearchHosts: https://simple-opensearch-nodes-default.default.svc.cluster.local:9200
 image:
   repository: oci.stackable.tech/sdp/opensearch-dashboards
   tag: 3.1.0-stackable{{ versions.opensearch }}
@@ -23,6 +22,11 @@ config:
       cookie:
         secure: true
 extraEnvs:
+  - name: OPENSEARCH_HOSTS
+    valueFrom:
+      configMapKeyRef:
+        name: simple-opensearch
+        key: OPENSEARCH_HOSTS
   - name: OPENSEARCH_PASSWORD
     valueFrom:
       secretKeyRef:

docs/modules/opensearch/examples/getting_started/opensearch.yaml

@@ -7,6 +7,8 @@ spec:
   image:
     productVersion: 3.1.0
   nodes:
+    roleConfig:
+      discoveryServiceListenerClass: external-stable
     roleGroups:
       default:
         replicas: 3

docs/modules/opensearch/images/opensearch_overview.drawio.svg

@@ -58,17 +58,6 @@ You can do so with this command:
 include::example$getting_started/getting_started.sh[tag=await-cluster]
 ----
 
-== Connecting to the HTTP endpoint
-
-Once the OpenSearch nodes are created, you can use the REST API of OpenSearch.
-
-To forward the HTTP port (`9200`) to localhost, run:
-
-[source,bash]
-----
-include::example$getting_started/getting_started.sh[tag=opensearch-port-forwarding]
-----
-
 == Using the REST API
 
 You can use the REST API as follows:

docs/modules/opensearch/pages/getting_started/first_steps.adoc

@@ -41,17 +41,18 @@ It helps you tune your cluster to your needs by configuring xref:usage-guide/sto
 
 === Kubernetes resources
 
-Based on the custom resources you define, the operator creates ConfigMaps, StatefulSets and Services.
+Based on the custom resources you define, the operator creates ConfigMaps, StatefulSets, Services and so on.
 
 image::opensearch_overview.drawio.svg[A diagram depicting the Kubernetes resources created by the operator]
 
 The diagram above depicts all the Kubernetes resources created by the operator, and how they relate to each other.
 
-For every xref:concepts:roles-and-role-groups.adoc#role-groups[role group] you define, the operator creates a StatefulSet with the amount of replicas defined in the role group.
-For every role group, a Service is created, as well as one for the whole cluster that references the cluster manager nodes.
+What should be highlighted, is the xref:reference/discovery.adoc[discovery ConfigMap] which is named the same as the OpenSearchCluster.
+It references the Service that should be used to connect to the cluster.
 
-Additionally, a ConfigMap is created for each role group.
-These ConfigMaps contain configuration files like `opensearch.yml`.
+For every xref:concepts:roles-and-role-groups.adoc#role-groups[role group] you define, the operator deploys OpenSearch as a StatefulSet with the amount of replicas defined in the role group.
+The pods of a StatefulSet use the configuration from the role group ConfigMap, i.e. they all use the same OpenSearch node roles, e.g. `cluster-manager` or `data`, and xref:opensearch:usage-guide/storage-resource-configuration.adoc[resource configurations].
+If you want dedicated `cluster-manager` and `data` nodes, you just define two role groups with according configurations as described in xref:opensearch:usage-guide/node-roles.adoc[].
 
 == Supported versions