DOC-5600 2101x

antora.yml

@@ -17,4 +17,16 @@ asciidoc:
     starlight-rabbitmq: 'Starlight for RabbitMQ'
     pulsar-reg: 'Apache Pulsar(TM)'
     pulsar: 'Apache Pulsar'
-    pulsar-short: 'Pulsar'
+    pulsar-short: 'Pulsar'
+
+    # Required for include::common partials that are shared with Astra Streaming
+    web-ui: 'Admin Console'
+    astra: 'Astra'
+    astra-db: 'Astra DB'
+    scb: 'Secure Connect Bundle (SCB)'
+    scb-short: 'SCB'
+    scb-brief: 'Secure Connect Bundle'
+    cass-reg: 'Apache Cassandra(R)'
+    cass: 'Apache Cassandra'
+    cass-short: 'Cassandra'
+    debezium-version: '1.7'

modules/ROOT/nav.adoc

@@ -22,7 +22,21 @@
 * xref:operations:auth.adoc[]
 * xref:operations:functions.adoc[]
 * xref:operations:functions-transform.adoc[]
-* xref:operations:io-connectors.adoc[]
-* xref:operations:io-elastic-sink.adoc[]
+* I/O connectors
+** xref:connectors:index.adoc[]
+** Sinks
+*** xref:connectors:sinks/astra-db.adoc[]
+*** xref:connectors:sinks/elastic-search.adoc[]
+*** xref:connectors:sinks/jdbc-clickhouse.adoc[]
+*** xref:connectors:sinks/jdbc-mariadb.adoc[]
+*** xref:connectors:sinks/jdbc-postgres.adoc[]
+*** xref:connectors:sinks/kafka.adoc[]
+*** xref:connectors:sinks/kinesis.adoc[]
+** Sources
+*** xref:connectors:sources/debezium-mongodb.adoc[]
+*** xref:connectors:sources/debezium-mysql.adoc[]
+*** xref:connectors:sources/debezium-postgres.adoc[]
+*** xref:connectors:sources/kafka.adoc[]
+*** xref:connectors:sources/kinesis.adoc[]
 * xref:operations:scale-cluster.adoc[]
 * xref:operations:troubleshooting.adoc[]

modules/ROOT/pages/faqs.adoc

@@ -80,7 +80,7 @@ https://github.com/datastax/pulsar-heartbeat[{pulsar-short} Heartbeat] monitors
 
 == What are the features provided by {company} {pulsar} Connector (`pulsar-sink`) that are not supported in `kafka-sink`?
 
-The https://pulsar.apache.org/docs/en/io-overview/[{pulsar-short} IO framework] provides many features that are not possible in Kafka, and has different compression formats and auth/security features. The features are handled by {pulsar-short}. For more, see xref:operations:io-connectors.adoc[Luna Streaming IO Connectors].
+The https://pulsar.apache.org/docs/en/io-overview/[{pulsar-short} IO framework] provides many features that are not possible in Kafka, and has different compression formats and auth/security features. The features are handled by {pulsar-short}. For more, see xref:connectors:index.adoc[Luna Streaming IO Connectors].
 
 The {company} {pulsar} Connector allows single-record acknowledgement and negative acknowledgements.
 

modules/ROOT/pages/index.adoc

@@ -17,7 +17,7 @@ In addition to the distribution of https://pulsar.apache.org/en/versions/[{pulsa
 
 * A xref:install-upgrade:quickstart-helm-installs.adoc[Helm chart] that deploys and manages {pulsar-short} on your current CNCF-conformant Kubernetes infrastructure
 
-* Cassandra, Elastic, Kinesis, Kafka, and JDBC xref:operations:io-connectors.adoc[connectors]
+* Cassandra, Elastic, Kinesis, Kafka, and JDBC xref:connectors:index.adoc[connectors]
 
 * xref:components:admin-console-vm.adoc[{pulsar-short} Admin Console] for simplified administration of your {pulsar-short} environment
 

modules/ROOT/partials/sinks/edit.adoc

@@ -0,0 +1,17 @@
+To update a connector, pass the new configuration definition to the connector.
+For example, if you created the connector with a configuration file, you can pass an updated configuration file.
+
+You can include the entire configuration or only the properties that you want to change.
+Additionally, some properties can be modified with specific arguments, such as `--parallelism`.
+
+To get the current configuration, see xref:connectors:index.adoc#get-sink-connector-configuration-data[Get sink connector configuration data].
+
+[source,shell,subs="+attributes"]
+----
+./bin/pulsar-admin sinks update \
+  --sink-type {connectorType} \
+  --name "$SINK_NAME" \
+  --inputs "persistent://$TENANT/$NAMESPACE/$TOPIC" \
+  --tenant "$TENANT" \
+  --parallelism 2
+----

modules/ROOT/partials/sinks/get-started.adoc

@@ -0,0 +1,32 @@
+. Optional: If you are using the `pulsar-admin` CLI, set the following commonly-used environment variables:
++
+[source,shell,subs="+quotes"]
+----
+export TENANT="**TENANT_NAME**"
+export TOPIC="**INPUT_TOPIC_NAME**"
+export NAMESPACE="**NAMESPACE_NAME**"
+export SINK_NAME="**SINK_CONNECTOR_UNIQUE_NAME**"
+----
++
+`**SINK_NAME**` is the name for your new sink connector.
+{company} recommends a memorable, human-readable name that summarizes the connector's purpose.
+For example: `{connectorType}-sink-prod-us-east-1`.
+
+. Create the connector using JSON-formatted connector configuration data.
+You can pass the configuration directly or with a configuration file.
++
+[source,shell,subs="+attributes"]
+----
+./bin/pulsar-admin sinks create \
+  --sink-type {connectorType} \
+  --name "$SINK_NAME" \
+  --inputs "persistent://$TENANT/$NAMESPACE/$TOPIC" \
+  --tenant "$TENANT" \
+  --sink-config-file configs.json
+----
++
+.Example configuration data structure
+[source,json]
+----
+include::common:streaming:example$connectors/sinks/{connectorType}/sample-data.json[]
+----

modules/ROOT/partials/sources/edit.adoc

@@ -0,0 +1,17 @@
+To update a connector, pass the new configuration definition to the connector.
+For example, if you created the connector with a configuration file, you can pass an updated configuration file.
+
+You can include the entire configuration or only the properties that you want to change.
+Additionally, some properties can be modified with specific arguments, such as `--parallelism`.
+
+To get the current configuration, see xref:connectors:index.adoc#get-source-connector-configuration-data[Get source connector configuration data].
+
+[source,shell,subs="+attributes"]
+----
+./bin/pulsar-admin sources update \
+  --source-type {connectorType} \
+  --name "$SOURCE_NAME" \
+  --destination-topic-name "persistent://$TENANT/$NAMESPACE/$TOPIC" \
+  --tenant "$TENANT" \
+  --parallelism 2
+----

modules/ROOT/partials/sources/get-started.adoc

@@ -0,0 +1,32 @@
+. Optional: If you are using the `pulsar-admin` CLI, set the following commonly-used environment variables:
++
+[source,shell,subs="+quotes"]
+----
+export TENANT="**TENANT_NAME**"
+export TOPIC="**OUTPUT_TOPIC_NAME**"
+export NAMESPACE="**NAMESPACE_NAME**"
+export SOURCE_NAME="**SOURCE_CONNECTOR_UNIQUE_NAME**"
+----
++
+`**SOURCE_NAME**` is the name for your new source connector.
+{company} recommends a memorable, human-readable name that summarizes the connector's purpose.
+For example: `{connectorType}-source-prod-us-east-1`.
+
+. Create the connector using JSON-formatted connector configuration data.
+You can pass the configuration directly or with a configuration file.
++
+[source,shell,subs="+attributes"]
+----
+./bin/pulsar-admin sources create \
+  --source-type {connectorType} \
+  --name "$SOURCE_NAME" \
+  --destination-topic-name "persistent://$TENANT/$NAMESPACE/$TOPIC" \
+  --tenant "$TENANT" \
+  --source-config-file configs.json
+----
++
+.Example configuration data structure
+[source,json]
+----
+include::common:streaming:example$connectors/sources/{connectorType}/sample-data.json[]
+----

modules/connectors/pages/index.adoc

@@ -0,0 +1,224 @@
+= {pulsar-short} I/O connectors in {product}
+:navtitle: {pulsar-short} I/O connectors reference
+:page-aliases: operations:io-connectors.adoc
+
+{product} offers fully-managed versions of https://pulsar.apache.org/docs/en/io-overview/[{pulsar-reg} sink and source connectors].
+
+[TIP]
+====
+There are three versions of the {company} Luna Streaming distribution.
+The `lunastreaming-all` version includes all connectors.
+====
+
+[#sink-connectors]
+== Sink connectors
+
+Sink connectors export messages from {pulsar-short} topics to external services.
+
+Supported sink connectors::
+The following sink connectors are included in {product}:
++
+* xref:connectors:sinks/astra-db.adoc[{astra-db} and {cass} sink]
+* xref:connectors:sinks/kafka.adoc[Apache Kafka sink]
+* xref:connectors:sinks/kinesis.adoc[AWS Kinesis sink]
+* xref:connectors:sinks/elastic-search.adoc[Elasticsearch sink]
+* xref:connectors:sinks/jdbc-clickhouse.adoc[JDBC ClickHouse sink]
+* xref:connectors:sinks/jdbc-mariadb.adoc[JDBC MariaDB sink]
+* xref:connectors:sinks/jdbc-postgres.adoc[JDBC PostgreSQL sink]
+
+[#source-connectors]
+== Source connectors
+
+Source connectors ingest messages from external services into {pulsar-short} topics.
+
+Supported source connectors::
+The following source connectors are included in {product}:
++
+* xref:connectors:sources/kafka.adoc[Apache Kafka source]
+* xref:connectors:sources/kinesis.adoc[AWS Kinesis source]
+* xref:connectors:sources/debezium-mongodb.adoc[Debezium MongoDB source]
+* xref:connectors:sources/debezium-mysql.adoc[Debezium MySQL source]
+* xref:connectors:sources/debezium-postgres.adoc[Debezium PostgreSQL source]
+
+[#create-and-manage-connectors]
+== Create and manage connectors
+
+You can use the {web-ui} and the `xref:components:admin-console-tutorial.adoc[pulsar-admin]` CLI to create, monitor, and manage sink and source connectors.
+
+Although you use the same base commands to create and update all {product} {pulsar-short} connectors, each connector has different configuration options.
+For example commands and configuration details, see the documentation for your preferred <<sink-connectors,sink connectors>> and <<source-connectors,source connectors>>.
+
+=== `pulsar-admin` CLI sink operations
+
+Get available sink connectors::
+Get a list of sink connectors that are available in your {product} {pulsar-short} tenant:
++
+[source,shell]
+----
+./bin/pulsar-admin sinks available-sinks
+----
+
+[#get-sink-connector-configuration-data]
+Get sink connector configuration data::
+Get the configuration for an existing sink connector:
++
+[source,shell]
+----
+# Get information about a connector
+./bin/pulsar-admin sinks get \
+  --namespace "$NAMESPACE" \
+  --name "$SINK_NAME" \
+  --tenant "$TENANT"
+----
+
+Start a sink connector::
++
+[source,shell]
+----
+# Start all instances of a connector
+./bin/pulsar-admin sinks start \
+  --namespace "$NAMESPACE" \
+  --name "$SINK_NAME" \
+  --tenant "$TENANT"
+
+# optionally add --instance-id to only start an individual instance
+----
+
+Stop a sink connector::
++
+[source,shell]
+----
+# Stop all instances of a connector
+./bin/pulsar-admin sinks stop \
+  --namespace "$NAMESPACE" \
+  --name "$SINK_NAME" \
+  --tenant "$TENANT"
+
+# optionally add --instance-id to only stop an individual instance
+----
+
+Restart a sink connector::
++
+[source,shell]
+----
+# Restart all instances of a connector
+./bin/pulsar-admin sinks restart \
+  --namespace "$NAMESPACE" \
+  --name "$SINK_NAME" \
+  --tenant "$TENANT"
+
+# optionally add --instance-id to only restart an individual instance
+----
+
+Get sink connector status::
++
+[source,shell]
+----
+# Check connector status
+./bin/pulsar-admin sinks status \
+  --instance-id "$SINK_INSTANCE_ID" \
+  --namespace "$NAMESPACE" \
+  --name "$SINK_NAME" \
+  --tenant "$TENANT"
+----
+
+Delete a sink connector::
++
+[source,shell]
+----
+# Delete all instances of a connector
+./bin/pulsar-admin sinks delete \
+  --namespace "$NAMESPACE" \
+  --name "$SINK_NAME" \
+  --tenant "$TENANT"
+----
+
+=== `pulsar-admin` CLI source operations
+
+Get available source connectors::
+Get a list of source connectors that are available in your {product} {pulsar-short} tenant:
++
+[source,shell]
+----
+./bin/pulsar-admin sources available-sources
+----
+
+[#get-source-connector-configuration-data]
+Get source connector configuration data::
+Get the configuration for an existing source connector:
++
+[source,shell]
+----
+# Get information about connector
+./bin/pulsar-admin sources get \
+  --namespace "$NAMESPACE" \
+  --name "$SOURCE_NAME" \
+  --tenant "$TENANT"
+----
+
+Start a source connector::
++
+[source,shell]
+----
+# Start all instances of a connector
+./bin/pulsar-admin sources start \
+  --namespace "$NAMESPACE" \
+  --name "$SOURCE_NAME" \
+  --tenant "$TENANT"
+
+# optionally add --instance-id to only start an individual instance
+----
+
+Stop a source connector::
++
+[source,shell]
+----
+# Stop all instances of a connector
+./bin/pulsar-admin sources stop \
+  --namespace "$NAMESPACE" \
+  --name "$SOURCE_NAME" \
+  --tenant "$TENANT"
+
+# optionally add --instance-id to only stop an individual instance
+----
+
+Restart a source connector::
++
+[source,shell]
+----
+# Restart all instances of a connector
+./bin/pulsar-admin sources restart \
+  --namespace "$NAMESPACE" \
+  --name "$SOURCE_NAME" \
+  --tenant "$TENANT"
+
+# optionally add --instance-id to only restart an individual instance
+----
+
+Get source connector status::
++
+[source,shell]
+----
+# Check connector status
+./bin/pulsar-admin sources status \
+  --instance-id "$SOURCE_INSTANCE_ID" \
+  --namespace "$NAMESPACE" \
+  --name "$SOURCE_NAME" \
+  --tenant "$TENANT"
+----
+
+Delete a source connector::
++
+[source,shell]
+----
+# Delete all instances of a connector
+./bin/pulsar-admin sources delete \
+  --namespace "$NAMESPACE" \
+  --name "$SOURCE_NAME" \
+  --tenant "$TENANT"
+----
+
+== Connector metrics
+
+{product} exposes Prometheus-formatted metrics for every connector.
+For more information, see xref:components:pulsar-monitor.adoc[].