Skip to content

Commit cf44c3b

Browse files
viktorKhanviktorKhan
committed
Merge branch 'main' of github.com:lucoenergia/conluz into main
2 parents a714e04 + 43aaa53 commit cf44c3b

File tree

94 files changed

+4001
-2270
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

94 files changed

+4001
-2270
lines changed

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,3 +38,7 @@ out/
3838

3939
### VS Code ###
4040
.vscode/
41+
42+
## InfluxDB
43+
/deploy/influxdb3/explorer/config/config.json
44+
/deploy/influxdb3/explorer/db/sqlite.db

‎README.md‎

Lines changed: 114 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -54,9 +54,9 @@ You don't need to do an extra effort of creating all the table manually, because
5454
> To have a PostgreSQL database up and running in a few seconds, you can use the docker compose file `deploy/docker-compose.yaml`. This file will do automatically all the configurations required transparently.
5555
> You just need to navigate to the `deploy` folder and execute the command `docker compose up -d`
5656
57-
2. **InfluxDB database**
57+
2. **InfluxDB 3 Core database**
5858

59-
Conluz uses InfluxDB as a time series database to store consumption, production and energy prices data gathered from different sources like datadis.es, Shelly meters or Huawei inverters.
59+
Conluz uses InfluxDB 3 Core as a time series database to store consumption, production and energy prices data gathered from different sources like datadis.es, Shelly meters or Huawei inverters.
6060

6161
#### Monitoring
6262

@@ -108,39 +108,75 @@ SPRING_DATASOURCE_URL="jdbc:postgresql://postgres_ip:5432/conluz_db"
108108
109109
If you are running Postgres locally on `localhost:5432` you don't need to provide this environment variable because that is the default configuration.
110110
111-
4. Configure InfluxDB database
111+
4. Configure InfluxDB 3 Core database
112112
113-
#### New InfluxDB installation
113+
#### New InfluxDB 3 Core installation
114114
115-
To have an InfluxDB database up and running in a few seconds, you can use the docker compose file `deploy/docker-compose.yaml`. This file will do automatically all the configurations required transparently.
116-
You just need to execute the command `docker compose up -d influxdb`
115+
To set up InfluxDB 3 Core, follow these steps:
117116
118-
#### Already existing InfluxDB installation
119-
If you already have an InfluxDB database up and running, you need to execute these commands:
117+
**Start the container:**
118+
```bash
119+
docker compose up -d influxdb3-core
120+
```
120121

121-
```sql
122-
CREATE DATABASE conluz_db
123-
CREATE USER luz WITH PASSWORD 'blank'
124-
GRANT ALL ON conluz_db TO luz
125-
CREATE RETENTION POLICY one_month ON conluz_db DURATION 30d REPLICATION 1
126-
CREATE RETENTION POLICY one_year ON conluz_db DURATION 365d REPLICATION 1
127-
CREATE RETENTION POLICY forever ON conluz_db DURATION INF REPLICATION 1 DEFAULT
122+
**Create an admin token:**
123+
```bash
124+
docker compose exec influxdb3-core influxdb3 create token --admin
125+
```
126+
127+
**Configure the token:**
128+
Open the `.env` file in the `deploy` folder and paste the token string for the `INFLUXDB_TOKEN` environment variable.
129+
130+
**Restart the container:**
131+
Down and up the container again to make the healthcheck read the previously created token:
132+
```bash
133+
docker compose down influxdb3-core && docker compose up -d influxdb3-core
128134
```
129-
These commands will:
130-
- Create a database called "conluz_db".
131-
- Create a user called "luz".
132-
- Grant privileges to the user "luz" over the database "conluz_db".
133-
- Create a set of policies required by the app.
134135

135-
By default, the InfluxDB settings are:
136+
**Verify the installation:**
137+
Verify that influxdb3-core is up and running by executing a query (replace with your actual token):
138+
```bash
139+
docker exec influxdb3-core influxdb3 show databases --token <your_token_here>
136140
```
137-
url=http://localhost:8086
138-
username=luz
139-
password=blank
140-
database=conluz_db
141+
142+
** Locate volume locally **
143+
If you want to know where exactly is influxdb3-core service storing the data through the volumne influxdb_data you can run this command to list all the volumes:
144+
```bash
145+
docker volume ls
141146
```
142147

143-
If you have your InfluxDB running with a different server, port, user credentials or database name, you would need to update the file `src/main/resources/application.properties` with your values.
148+
Locate the volume and copy the id of the volume. Then you can run this command to inspect the volume:
149+
```bash
150+
docker volume inspect {volume_id_here}
151+
```
152+
153+
You will get an output like this:
154+
```json
155+
[
156+
{
157+
"CreatedAt": "2025-10-11T16:45:51+02:00",
158+
"Driver": "local",
159+
"Labels": {
160+
"com.docker.compose.config-hash": "7c3d78d48af75c186f2c3dd11aa2637b1128e3b90aebad1780aa0811da74cb50",
161+
"com.docker.compose.project": "deploy",
162+
"com.docker.compose.version": "2.35.1",
163+
"com.docker.compose.volume": "influxdb_data"
164+
},
165+
"Mountpoint": "/var/lib/docker/volumes/deploy_influxdb_data/_data",
166+
"Name": "deploy_influxdb_data",
167+
"Options": null,
168+
"Scope": "local"
169+
}
170+
]
171+
```
172+
The `Mountpoint` field contains the path where the volume is mounted.
173+
174+
#### Already existing InfluxDB 3 Core installation
175+
176+
If you already have an InfluxDB 3 Core database up and running, you need to:
177+
- Create a database called `conluz_db`
178+
- Generate an admin token
179+
- Configure your application with the appropriate connection settings in `src/main/resources/application.properties`
144180

145181
5. Run the application
146182

@@ -154,6 +190,58 @@ If you have your InfluxDB running with a different server, port, user credential
154190

155191
The application will be accessible at https://localhost:8443.
156192

193+
**InfluxDB Explorer (UI)**
194+
195+
InfluxDB Explorer is a web-based user interface for querying and visualizing data stored in InfluxDB 3 Core. To run the Explorer and connect it to your influxdb3-core service:
196+
197+
> **Note:**
198+
>
199+
> You can find official documentation [here](https://docs.influxdata.com/influxdb3/explorer/install/?t=Docker+Compose).
200+
201+
**Pre-configure the connection:**
202+
203+
Create a `config.json` file in the `deploy/config` directory with the following content:
204+
205+
```json
206+
{
207+
"connections": [
208+
{
209+
"name": "conluz_db",
210+
"url": "http://influxdb3-core:8181",
211+
"token": "your_admin_token_here",
212+
"database": "conluz_db"
213+
}
214+
]
215+
}
216+
```
217+
218+
Replace `your_admin_token_here` with the admin token you created earlier (the same token from `INFLUXDB_TOKEN` in the `.env` file).
219+
220+
**Start InfluxDB Explorer:**
221+
```bash
222+
docker compose up -d influxdb3-explorer
223+
```
224+
225+
**Access the Explorer:**
226+
Open your browser and navigate to http://localhost:8888
227+
228+
The connection to influxdb3-core will be pre-configured and ready to use. You can now:
229+
- Run SQL queries against your time-series data
230+
- Browse databases and tables
231+
- Visualize data with charts and graphs
232+
- Explore the schema of your measurements
233+
234+
> **Note:**
235+
>
236+
> The Explorer container is configured to communicate with influxdb3-core through Docker's internal network using `http://influxdb3-core:8181`. If you need to access InfluxDB from outside the Docker network, use `http://localhost:8181` instead.
237+
238+
** Create a new database **
239+
240+
Once you have InfluxDB Explorer up and running, you can create a new database by following these steps:
241+
242+
- Go to the `Manage Databases` tab and click on the `Create New` button. Craete a databse with the same name as the one you configured in the `config.json` file.
243+
244+
![img.png](docs/db/timeseries/influxdb/images/create_database.png)
157245
158246
## Usage
159247

‎build.gradle‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ plugins {
55
}
66

77
group = 'org.lucoenergia'
8-
version = "1.0.43"
8+
version = "1.0.45"
99

1010
java {
1111
toolchain {
@@ -26,10 +26,15 @@ dependencies {
2626
implementation 'org.springframework.boot:spring-boot-starter-quartz'
2727
implementation 'org.springframework.boot:spring-boot-starter-actuator'
2828

29-
implementation 'org.liquibase:liquibase-core:4.28.0'
30-
implementation 'org.influxdb:influxdb-java:2.25'
29+
implementation 'org.liquibase:liquibase-core:4.33.0'
30+
implementation 'com.influxdb:influxdb3-java:1.4.0'
3131
runtimeOnly 'org.postgresql:postgresql'
3232

33+
implementation("com.squareup.okhttp3:okhttp:${okhttp3Version}")
34+
implementation("com.squareup.okhttp3:logging-interceptor:${okhttp3Version}")
35+
36+
implementation 'org.apache.commons:commons-lang3:3.18.0'
37+
3338
implementation 'org.springdoc:springdoc-openapi-starter-webmvc-ui:2.8.8'
3439

3540
implementation "io.jsonwebtoken:jjwt-api:${jjwtVersion}"
@@ -50,6 +55,11 @@ dependencies {
5055

5156
tasks.named('test') {
5257
useJUnitPlatform()
58+
// Add JVM arguments for Apache Arrow (required by InfluxDB 3 client)
59+
jvmArgs = [
60+
'--add-opens=java.base/java.nio=ALL-UNNAMED',
61+
'--add-opens=java.base/sun.nio.ch=ALL-UNNAMED'
62+
]
5363
}
5464

5565
jar {

‎deploy/INFLUXDB3_SETUP.md‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
# InfluxDB 3 Core Setup Guide
2+
3+
This guide explains how to set up and configure InfluxDB 3 Core for the Conluz project.
4+
5+
## Docker Configuration
6+
7+
The `docker-compose.yaml` includes an InfluxDB 3 Core service configured as follows:
8+
9+
```yaml
10+
influxdb3:
11+
image: influxdb:3-core
12+
container_name: influxdb3
13+
ports:
14+
- "8181:8181"
15+
environment:
16+
- INFLUXDB3_AUTH_TOKEN
17+
- INFLUXDB3_HTTP_BIND_ADDR=0.0.0.0:8181
18+
- LOG_FILTER=info
19+
volumes:
20+
- ${PATH_TO_INFLUXDB3_DATA}:/var/lib/influxdb3/data
21+
command: >
22+
influxdb3 serve
23+
--node-id conluz_node
24+
--object-store file
25+
--data-dir /var/lib/influxdb3/data
26+
```
27+
28+
## Environment Variables
29+
30+
Add these variables to your `.env` file in the `deploy/` directory:
31+
32+
```bash
33+
# InfluxDB 3 Core Configuration
34+
PATH_TO_INFLUXDB3_DATA=/path/to/influxdb3/data
35+
INFLUXDB3_AUTH_TOKEN=your-secret-token-here
36+
```
37+
38+
## Initial Setup
39+
40+
1. **Start the container:**
41+
```bash
42+
docker-compose up -d influxdb3
43+
```
44+
45+
2. **Create a database:**
46+
```bash
47+
docker exec influxdb3 influxdb3 create database conluz_db
48+
```
49+
50+
3. **Create an admin token (optional):**
51+
```bash
52+
docker exec influxdb3 influxdb3 create token --admin
53+
```
54+
55+
4. **Verify the setup:**
56+
```bash
57+
curl http://localhost:8181/api/v3/ping
58+
```
59+
60+
## Key Differences from InfluxDB 1.8
61+
62+
| Feature | InfluxDB 1.8 | InfluxDB 3 Core |
63+
|---------|-------------|-----------------|
64+
| **Port** | 8086 | 8181 |
65+
| **Query Language** | InfluxQL | SQL |
66+
| **Authentication** | Username/Password | Token-based |
67+
| **Data Organization** | Databases | Databases (no orgs/buckets like v2) |
68+
| **Retention Policies** | Multiple policies per DB | Per database |
69+
| **Time Aggregation** | `GROUP BY time(1d)` | `DATE_TRUNC('day', time)` |
70+
71+
## Application Configuration
72+
73+
Update your `application.properties`:
74+
75+
```properties
76+
# InfluxDB 3 configuration
77+
spring.influxdb3.url=http://localhost:8181
78+
spring.influxdb3.token=your-secret-token-here
79+
spring.influxdb3.org=your-org # Not used in Core but required by client
80+
spring.influxdb3.bucket=conluz_db
81+
```
82+
83+
## SQL Query Examples
84+
85+
InfluxDB 3 uses SQL instead of InfluxQL:
86+
87+
### Hourly Aggregation
88+
```sql
89+
SELECT
90+
DATE_TRUNC('hour', time) as time,
91+
SUM(consumption_kwh) AS consumption_kwh
92+
FROM datadis_consumption_kwh
93+
WHERE time >= '2024-01-01' AND time <= '2024-01-31'
94+
GROUP BY DATE_TRUNC('hour', time)
95+
```
96+
97+
### Monthly Aggregation (Main Benefit!)
98+
```sql
99+
SELECT
100+
DATE_TRUNC('month', time) as time,
101+
SUM(consumption_kwh) AS consumption_kwh
102+
FROM datadis_consumption_kwh
103+
WHERE time >= '2024-01-01' AND time <= '2024-12-31'
104+
GROUP BY DATE_TRUNC('month', time)
105+
```
106+
107+
### Yearly Aggregation
108+
```sql
109+
SELECT
110+
DATE_TRUNC('year', time) as time,
111+
SUM(consumption_kwh) AS consumption_kwh
112+
FROM datadis_consumption_kwh
113+
WHERE time >= '2020-01-01' AND time <= '2024-12-31'
114+
GROUP BY DATE_TRUNC('year', time)
115+
```
116+
117+
## Data Migration
118+
119+
To migrate data from InfluxDB 1.8 to InfluxDB 3:
120+
121+
1. **Export data from InfluxDB 1.8:**
122+
```bash
123+
cd deploy
124+
./migrate-influxdb-export.sh
125+
```
126+
127+
2. **Import data into InfluxDB 3:**
128+
```bash
129+
cd deploy
130+
export INFLUX_SETUP_TOKEN=your-token-here
131+
./migrate-influxdb-import.sh
132+
```
133+
134+
## Troubleshooting
135+
136+
### Container won't start
137+
- Check that port 8181 is not already in use
138+
- Verify the data directory path in `.env`
139+
- Check container logs: `docker logs influxdb3`
140+
141+
### Authentication errors
142+
- Ensure `INFLUXDB3_AUTH_TOKEN` is set in `.env`
143+
- Verify the token matches in application.properties
144+
- Token is required for all API requests
145+
146+
### Query errors
147+
- Remember: InfluxDB 3 uses SQL, not InfluxQL
148+
- Use `DATE_TRUNC()` instead of `GROUP BY time()`
149+
- Column names are case-sensitive in SQL
150+
151+
## Resources
152+
153+
- [InfluxDB 3 Core Documentation](https://docs.influxdata.com/influxdb3/core/)
154+
- [InfluxDB 3 SQL Reference](https://docs.influxdata.com/influxdb3/core/reference/sql/)
155+
- [Java Client Library](https://github.com/InfluxCommunity/influxdb3-java)

0 commit comments

Comments
 (0)