PadoGrid | Catalogs | Manual | FAQ | Releases | Templates | Pods | Kubernetes | Docker | Apps | Quick Start

---

Jet CDC Tutorial

‼️ Deprecated. As of PadoGrid v1.0.0, Jet 3.x and Jet 4.x are no longer supported. This bundle runs only on PadoGrid v0.9.x. The cdc_tutorial bundle wraps the CDC tutorial available from the Hazelcast site https://jet-start.sh/docs/tutorials/cdc.

Installing Bundle

install_bundle -download bundle-jet-4-docker-cdc_tutorial

Use Case

This use case introduces the Debezium MySQL connector as a Jet job for ingesting database CDC records into a Jet cluster.

Required Software

• Docker

Tutorial Link

The following link provides detailed explanations to the data flow and source code.

https://jet-start.sh/docs/tutorials/cdc

Building App

cd_docker cdc_tutorial; cd bin_sh
./build_app

Starting MySQL Docker Container

If you have your own MySQL running, stop it first before starting the mysql container.

cd_docker cdc_tutorial; cd bin_sh

# Start mysql container
./start_mysql

Starting Jet Cluster

We can start a Jet cluster on the host machine or Docker containers.

Starting Jet Cluster on Host Machine

First, create a Jet cluster with the default settings. Any Jet cluster with the default settings will work.

create_cluster -cluster cdc_jet

Start the cluster.

switch_cluster cdc_jet
start_cluster

Starting Jet Cluster on Docker Containers

The create_docker command automatically selects a host IP address but it may not be accessible from the Docker containers if you have multiple addresses assigned to the host. You can specify the host IP address using the -host option.

If you are running Docker Desktop then you can specify the -host option with host.docker.internal, which resolves to the internal IP address used by the host. Otherwise, specify the IP address that the Docker containers have access to.

create_docker -product jet -cluster cdc_jet -host host.docker.internal

❗ The Jet's slf4j library might not be compatible with the version that the tutorial build packaged in the cdc-tutorial-1.0-SNAPSHOT.jar file. To avoid this problem, let's copy the jar file in the Jet cluster plugins directory.

cd_docker cdc_jet
cp ../cdc_tutorial/target/cdc_tutorial-1.0.0.jar padogrid/plugins/

Start the cluster.

cd_docker cdc_jet
docker-compose up

Submitting Debezium Connector Job

The default Jet cluster has been configured with the start port, 6701. Unless the cluster has been configured with the default port, 5701, we need to specify the port number when submitting a job.

The job jar we have built takes the host IP address as the optional argument. If you don't specify the argument, then it defaults to localhost (See the source code: src/main/java/org/example/JetJob.java). As mentioned before, the host IP must be accessible from the Docker containers.

cd_docker cdc_tutorial

# For host IP: localhost (if running on host machine)
jet --addresses localhost:6701 submit target/cdc_tutorial-1.0.0.jar

# For Docker Desktop: host.docker.internal
jet --addresses localhost:6701 submit target/cdc_tutorial-1.0.0.jar host.docker.internal

# For others:
jet --addresses localhost:6701 submit target/cdc_tutorial-1.0.0.jar <host_ip>

Upon submitting the connector, the customers table records will be populated in the customers map. You can view the customers map by running the read_cache script.

cd_docker cdc_tutorial; cd bin_sh
./read_cache

Output:

Currently there are following customers in the cache:
        Customer {id=1002, firstName=George, lastName=Bailey, [email protected]}
        Customer {id=1001, firstName=Sally, lastName=Thomas, [email protected]}
        Customer {id=1003, firstName=Edward, lastName=Walker, [email protected]}
        Customer {id=1004, firstName=Anne, lastName=Kretchmar, [email protected]}

Updating Database

cd_docker cdc_tutorial; cd bin_sh 
./start_mysql_cli

From the CLI, use the inventory database and view the customers table.

mysql> use inventory;
mysql> select * from customers;

Output:

+------+------------+-----------+-----------------------+
| id   | first_name | last_name | email                 |
+------+------------+-----------+-----------------------+
| 1001 | Sally      | Thomas    | [email protected] |
| 1002 | George     | Bailey    | [email protected]    |
| 1003 | Edward     | Walker    | [email protected]         |
| 1004 | Anne       | Kretchmar | [email protected]    |
+------+------------+-----------+-----------------------+

From the CLI, update the customers table.

mysql> UPDATE customers SET first_name='Anne Marie' WHERE id=1004;

Viewing Data Updates

Let's take a look at the customers map in the Jet cluster.

cd_docker cdc_tutorial; cd bin_sh
./read_cache

Output:

Currently there are following customers in the cache:
        Customer {id=1002, firstName=George, lastName=Bailey, [email protected]}
        Customer {id=1001, firstName=Sally, lastName=Thomas, [email protected]}
        Customer {id=1003, firstName=Edward, lastName=Walker, [email protected]}
        Customer {id=1004, firstName=Anne Marie, lastName=Kretchmar, [email protected]} 

Note that the firstName field is changed from Anne to Anne Marie for id=1004.

Tearing Down

# Stop cluster (if running on host machine)
stop_cluster

# Stop cluster (if running on Docker containers)
# Ctrl-C Docker Compose

# Stop and prune Docker containers
cd_docker cdc_tutorial; cd bin_sh
./cleanup

---

PadoGrid | Catalogs | Manual | FAQ | Releases | Templates | Pods | Kubernetes | Docker | Apps | Quick Start