Skip to content

vives-devbit/summa-mqtt-sensors

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
mosquitto
mosquitto
 
 
.env.example
.env.example
 
 
.gitignore
.gitignore
 
 
README.md
README.md
 
 
compose.yaml
compose.yaml
 
 

Repository files navigation

Towards a Scalable and Automated Sensor Platform for Testing and Monitoring

In response to the need for efficient and scalable temperature and current monitoring, we would like to propose a custom-built solution. Instead of relying on commercial systems—which often have limited flexibility or are constrained by cloud-based restrictions—we suggest implementing a local setup based on ESP32 modules running Tasmota firmware.

Sensor data will be transmitted via MQTT to a backend composed of Docker containers. This backend includes InfluxDB for time-series data storage and Grafana for visualization. The setup allows for flexible, real-time data logging and monitoring within the local network, without depending on external platforms. Additionally, it can be extended with current sensors, cameras, or other components, making it a solid foundation for future developments such as automated firmware testing environments.

System Architecture

Overview of Components

  • ESP32 + Tasmota: Microcontroller modules flashed with Tasmota firmware, responsible for collecting sensor data (e.g., temperature, current) and publishing it over MQTT.
  • MQTT Broker: Acts as the central message hub, receiving sensor data from ESP32 devices and distributing it to subscribers (e.g., backend services).
  • InfluxDB: A time-series database that stores incoming sensor data for historical analysis and visualization.
  • Grafana: Visualization platform connected to InfluxDB, used to create dashboards and monitor sensor data in real time.
  • Node-RED: Optional low-code platform for wiring together data flows, enabling custom automation, alerting, or integration with other systems.

Network Diagram and Data Flow

flowchart LR
    ESP32[Tasmota ESP32 Devices]
    MQTT[MQTT Broker]
    NodeRED[Node-RED]
    InfluxDB[InfluxDB]
    Grafana[Grafana]

    ESP32 -- MQTT Publish --> MQTT
    MQTT -- Data Subscription --> NodeRED
    NodeRED -- Writes Data --> InfluxDB
    MQTT -- Data Subscription --> InfluxDB
    InfluxDB -- Data Source --> Grafana
Loading

Data Flow Description:

  1. Data Collection: ESP32 devices with Tasmota firmware read sensor values and publish them to the MQTT broker.
  2. Message Distribution: The MQTT broker receives sensor data and makes it available to subscribers (InfluxDB, Node-RED).
  3. Data Processing: Node-RED can process, filter, or transform incoming data and forward it to InfluxDB for storage.
  4. Storage: InfluxDB stores all time-series sensor data.
  5. Visualization: Grafana queries InfluxDB and presents the data in customizable dashboards for monitoring and analysis.

This architecture ensures modularity, scalability, and flexibility for future extensions.

Requirements

Hardware

  • ESP32 Development Boards: Microcontroller modules for sensor integration and wireless communication.
  • Temperature Sensors (e.g., DS18B20): Digital temperature sensors compatible with Tasmota.
  • Current Sensors (e.g., ACS712): For measuring electrical current.
  • Power Supply: Reliable 5V USB power adapters for ESP32 devices.
  • Wiring and Connectors: Jumper wires, breadboards, or terminal blocks for prototyping and connections.

Software

  • Tasmota Firmware: Open-source firmware for ESP32, flashed onto each device.
  • Docker: Containerization platform for backend services.
  • Docker Compose: Tool for defining and running multi-container Docker applications.
  • MQTT Broker: Such as Eclipse Mosquitto, for message handling.
  • InfluxDB: Time-series database for sensor data storage.
  • Grafana: Visualization and dashboard platform.
  • Node-RED (optional): For data processing and automation.
  • MQTT Client Tools: For testing and debugging (e.g., MQTT Explorer, mosquitto_sub).
  • Web Browser: For accessing Grafana dashboards and Node-RED editor.

Tasmota

For the sensor component, we recommend a system based on Tasmota. Tasmota is open-source firmware for ESP microcontrollers, supporting a wide range of common sensors and actuators. Communication via MQTT works out-of-the-box, and the flexible configuration allows you to build various applications without writing your own firmware.

Tasmota supports almost every ESP32, ESP32-S2, ESP32-S3, and ESP32-C3 microcontroller or development board. You are free to choose within this ecosystem. See the list of supported devices at: https://templates.blakadder.com/esp32.html

Tasmota also supports the sensors you have already ordered:

Web Installer

Tasmota offers a web installer, allowing you to flash most devices with firmware directly from your browser, without installing extra software. Visit https://tasmota.github.io/install/ to install the firmware. Note: your browser must support this (Edge, Chrome, and other Chromium-based browsers usually do; Firefox and derivatives do not). If you prefer traditional flashing methods, you can use tools like ESP Flash Tool or ESP.py/ESP Tool (CLI based).

Commands

When the Tasmota device is connected to your computer via USB, after flashing the firmware you can configure it using various 'Commands'. For example, you can set MQTT settings or specify which sensor is connected to which pin. Working via UART/Command is often not very intuitive for first-time users, so it is recommended to use the graphical WebUI.

WebUI

Through the WebUI, you can interactively view the device status and adjust its configuration.

MQTT

MQTT is one of the main ways Tasmota sends data. You can set up the broker and configure how data is published via command or the web UI.

Microcontroller Hardware

M5Stack offers ready-made microcontroller ecosystems with many expansion options. For this use case, the Atom and Core series are most suitable. The Atom series is very compact and affordable, while the Core series offers more functionality by default.

A good starting point is the M5Stack Atom development board. These are small, low-cost, and expandable. There are also various 'Base' boards that you can easily mount the Atom module on, adding extra functionality such as:

If the Atom boards are too limited, you can also choose the larger Core series. These offer more IO, often a (touch) display, battery, SD card slot, and DIN-rail mounting by default. A recommended option is the M5Stack CoreS3 Development Kit. Keep in mind that the extra features also increase complexity—but you are not required to use all features.

Where to Buy

Most components can be easily purchased from the Dutch webshop:
https://www.tinytronics.nl

Installation & Setup

Flashing and Configuring the ESP32 with Tasmota

1. Flashing Tasmota Firmware

  1. Download Tasmota:
    Visit the Tasmota releases page and download the appropriate .bin firmware for ESP32.
  2. Connect ESP32 to Computer:
    Use a USB cable to connect your ESP32 development board.
  3. Install Flashing Tool:
    Use Tasmotizer or esphome-flasher to flash the firmware.
  4. Flash the Firmware:
    • Open the flashing tool.
    • Select the downloaded .bin file.
    • Choose the correct serial port.
    • Click "Flash" and wait for completion.

2. Initial Configuration

After flashing, the ESP32 will start in Wi-Fi Access Point (AP) mode.

  1. Connect to Tasmota AP:
    On your computer or phone, connect to the Wi-Fi network named tasmota-xxxx.
  2. Open Configuration Portal:
    Navigate to 192.168.4.1 in your browser.

3. Setting Up Wi-Fi

  1. Enter Wi-Fi Credentials:
    In the portal, enter your local Wi-Fi SSID and password.
  2. Save and Reboot:
    The ESP32 will connect to your Wi-Fi network and display its new IP address.

4. Configuring MQTT

  1. Access Tasmota Web UI:
    Open the ESP32's IP address in your browser.
  2. Navigate to Configuration > Configure MQTT.
  3. Enter MQTT Broker Details:
    • Host: IP address or hostname of your MQTT broker.
    • Port: Usually 1883.
    • User/Password: (If required by your broker.)
    • Topic: Set a unique topic for each device (e.g., sensor/esp32_1).
  4. Save Settings:
    The device will restart and attempt to connect to the MQTT broker.

5. Verifying Operation

  • Check the Tasmota web UI for MQTT status.
  • Use an MQTT client (e.g., MQTT Explorer) to confirm messages are published to the correct topic.

You can now proceed to integrate the ESP32 with the rest of your monitoring stack.

Docker Setup

This section explains how to set up the backend services (MQTT broker, InfluxDB, Grafana, and Node-RED) using Docker Compose.

Example compose.yaml

The following compose.yaml file defines all required services. It uses environment variables for configuration, which you can override in a .env file or directly in the file.

services:
  influxdb:
    image: influxdb:1.11
    container_name: influxdb
    restart: unless-stopped
    ports:
      - "8086:8086"
    volumes:
      - ./influxdb:/var/lib/influxdb
    environment:
      INFLUXDB_DB: ${INFLUXDB_DB:-mydb}
      INFLUXDB_USER: ${INFLUXDB_USER:-myuser}
      INFLUXDB_USER_PASSWORD: ${INFLUXDB_USER_PASSWORD:-mypassword}
  
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    volumes:
      - grafana:/var/lib/grafana
    environment:
      GF_SECURITY_ADMIN_USER: ${GF_SECURITY_ADMIN_USER:-admin}
      GF_SECURITY_ADMIN_PASSWORD: ${GF_SECURITY_ADMIN_PASSWORD:-admin}
    depends_on:
      - influxdb

  nodered:
    image: nodered/node-red:latest
    container_name: nodered
    restart: unless-stopped
    ports:
      - "1880:1880"
    volumes:
      - ./nodered:/data
    environment:
      - NODE_RED_ENABLE_SAFE_MODE=true
      - NODE_RED_ENABLE_PROJECTS=true
    depends_on:
      - influxdb
      - grafana

  mqtt:
    image: eclipse-mosquitto:latest
    container_name: mqtt
    restart: unless-stopped
    ports:
      - "1883:1883"
      - "9001:9001"
    volumes:
      - ./mqtt/config:/mosquitto/config
      - ./mqtt/data:/mosquitto/data
      - ./mqtt/log:/mosquitto/log

volumes:
  influxdb:
  grafana:
  nodered:
  mqtt:

Note: You can customize usernames, passwords, and database names using environment variables or by editing the file directly.

Launching the Stack

  1. Install Docker and Docker Compose
    Make sure Docker and Docker Compose are installed on your system.

  2. Directory Structure
    Organize your project directory as follows:

    project-root/
  compose.yaml
  influxdb/
  grafana/
  nodered/
  mqtt/
    config/
    data/
    log/

  3. (Optional) Configure Mosquitto
    Place a mosquitto.conf file in mqtt/config/ if you need custom broker settings.

  4. Start the Services
    In your project directory, run:

    docker compose up -d

    This will pull images and start all containers in the background.

  5. Access the Services

  6. Stop the Stack
    To stop all services, run:

    docker compose down

This setup provides a modular and reproducible backend for your sensor platform.

MQTT Integration

Tasmota-enabled ESP32 devices use MQTT to publish sensor data to a central broker, enabling real-time communication with backend services.

How Tasmota Publishes Sensor Data via MQTT

After configuring MQTT settings in the Tasmota web UI, each ESP32 device connects to the broker and publishes sensor readings at regular intervals. Tasmota uses a topic structure that can be customized per device.

Topic Structure

By default, Tasmota uses the following topic format:

<tasmota_topic>/SENSOR
  • <tasmota_topic>: The unique topic name set in the device's MQTT configuration (e.g., tele/tasmota_D4F148).

Example:
tele/tasmota_D4F148/SENSOR

Payload Format

Sensor data is published as a JSON payload. The structure depends on the sensors attached and their configuration. For a device with a DS18B20 temperature sensor, the payload might look like:

{
    "Time": "2024-06-01T12:34:56",
    "DS18B20": {
        "Temperature": 23.5
    },
    "TempUnit": "C"
}

If you have additional sensors (e.g., current sensors), their readings will appear as extra fields in the JSON.

Subscribing to Data

Backend services (InfluxDB, Node-RED, etc.) subscribe to the relevant topics (e.g., tele/+/SENSOR) to receive updates from all devices.

Customization

  • You can change the topic prefix in Tasmota's MQTT settings for each device.
  • Use Node-RED or similar tools to parse, filter, and route incoming MQTT messages as needed.

Tip: Use an MQTT client (like MQTT Explorer) to inspect live messages and verify correct topic and payload formats.

Node-RED

Node-RED is an optional but powerful component in this sensor platform. It provides a visual, flow-based programming environment for wiring together devices, APIs, and online services. In this setup, Node-RED can be used to process, transform, and route MQTT sensor data, as well as trigger automations or alerts.

Key Features

  • Low-Code Data Processing: Easily parse and transform incoming MQTT messages using drag-and-drop nodes.
  • Automation: Set up rules to trigger notifications, control devices, or log events based on sensor data.
  • Integration: Connect with external APIs, databases, or other services for advanced workflows.
  • Visualization: Create simple dashboards for quick data visualization within Node-RED.

Example Use Cases

  • Filtering Data: Only forward temperature readings above a certain threshold to InfluxDB.
  • Alerting: Send an email or push notification if a sensor value exceeds safe limits.
  • Data Enrichment: Add timestamps, device metadata, or calculated fields before storing data.

Basic Flow Example

A typical Node-RED flow for this platform might look like:

  1. MQTT In Node: Subscribes to topics like sensor/+/SENSOR to receive all sensor data.
  2. JSON Node: Parses the incoming JSON payload.
  3. Function/Change Node: Processes or filters the data as needed.
  4. InfluxDB Out Node: Writes the processed data to InfluxDB for storage and visualization.

Getting Started

  1. Access Node-RED:
    Open http://localhost:1880 in your browser.

  2. Import Example Flow:
    You can import example flows from the Node-RED library or create your own.

  3. Install Additional Nodes:
    Use the "Manage palette" option to add nodes for InfluxDB, email, or other integrations as needed.

  4. Deploy Flows:
    After designing your flow, click "Deploy" to activate it.

Resources

Node-RED adds flexibility and extensibility to your sensor platform, making it easy to adapt to new requirements or integrate with other systems.

InfluxDB Configuration

This section describes how to configure InfluxDB for use with the project.
Note: InfluxDB v1 is used instead of v2 because it offers simpler integration with legacy MQTT sensor data pipelines and avoids the additional complexity of authentication and API changes introduced in v2.

This section explains how to set up InfluxDB for your sensor platform, including database creation, authentication, and optional retention policies.

1. Creating the Database

Once the InfluxDB container is running, you need to create a database for storing sensor data.

  1. Access the InfluxDB shell
    Run the following command to open the InfluxDB CLI inside the container: 
    docker exec -it influxdb influx
  2. Create a database
    In the InfluxDB shell, run: 
    CREATE DATABASE mydb
    Replace mydb with your preferred database name.

2. (Optional) Setting Up Authentication

By default, InfluxDB may run without authentication. To enable authentication:

  1. Set environment variables in compose.yaml: 
    environment:
  INFLUXDB_HTTP_AUTH_ENABLED: "true"
  INFLUXDB_ADMIN_USER: admin
  INFLUXDB_ADMIN_PASSWORD: strongpassword
  2. Restart the InfluxDB container 
    docker compose restart influxdb
  3. Create additional users (optional): In the InfluxDB shell: 
    CREATE USER myuser WITH PASSWORD 'mypassword'
GRANT ALL ON mydb TO myuser

3. (Optional) Configuring Retention Policies

Retention policies control how long data is kept in the database.

  • Create a retention policy (e.g., keep data for 30 days): 
    CREATE RETENTION POLICY "thirty_days" ON "mydb" DURATION 30d REPLICATION 1 DEFAULT
  • Verify retention policies: 
    SHOW RETENTION POLICIES ON mydb

4. Connecting Clients

  • Use the database name, username, and password (if enabled) in your Node-RED and Grafana configurations.
  • The default InfluxDB HTTP API endpoint is http://localhost:8086.

Tip: Always back up your InfluxDB data directory (./influxdb/) regularly.

For more details, see the InfluxDB documentation.

Grafana Visualization

Grafana provides powerful, customizable dashboards for visualizing sensor data stored in InfluxDB. This section explains how to connect Grafana to InfluxDB, create example dashboards, and manage dashboard imports/exports.

Connecting Grafana to InfluxDB

  1. Access Grafana:
    Open http://localhost:3000 in your browser. Log in with the default credentials (admin / password set in your compose.yaml).

  2. Add InfluxDB as a Data Source:

    • In the left sidebar, click Gear (⚙️) > Data Sources.
    • Click Add data source and select InfluxDB.
    • Set the following fields:
      • URL: http://influxdb:8086 (if using Docker Compose; otherwise, use http://localhost:8086)
      • Database: Name of your InfluxDB database (e.g., mydb)
      • User/Password: As configured in InfluxDB (if authentication is enabled)
    • Click Save & Test to verify the connection.

Example Dashboard

To visualize sensor data (e.g., temperature readings):

  1. Create a New Dashboard:

    • Click + > Dashboard in the sidebar.
    • Click Add new panel.

  2. Configure the Panel Query:

    • Select your InfluxDB data source.
    • Enter a query, for example: 
      SELECT mean("Temperature") FROM "DS18B20" WHERE $timeFilter GROUP BY time($__interval) fill(null)
      Adjust the measurement and field names to match your sensor data.

  3. Customize Visualization:

    • Choose a visualization type (e.g., Time series, Gauge).
    • Set panel titles, units (e.g., Celsius), and other options.

  4. Save the Dashboard:

    • Click Save dashboard, give it a name, and optionally add a description.

Importing and Exporting Dashboards

  • Export a Dashboard:

    • Open the dashboard, click the Dashboard settings (gear icon), then JSON Model.
    • Click Export to download the dashboard JSON.

  • Import a Dashboard:

    • In the sidebar, click + > Import.
    • Upload a dashboard JSON file or paste its contents.
    • Select the appropriate data source and click Import.

Tip: You can find ready-made dashboard templates for Tasmota, InfluxDB, and MQTT devices on Grafana's dashboard library.

With Grafana connected to InfluxDB, you can build real-time dashboards to monitor all your sensor data, set up alerts, and share insights with your team.

Usage & Validation

This section explains how to verify that your sensor platform is working correctly, how to check data flow through each component, and provides troubleshooting tips for common issues.

Verifying Data Flow

1. Check MQTT Data

  • Using MQTT Explorer or mosquitto_sub:
    • Connect to your MQTT broker (default: localhost:1883).
    • Subscribe to the relevant topic, e.g.: 
      mosquitto_sub -h localhost -t "sensor/+/SENSOR"
    • You should see JSON payloads from your ESP32 devices.

2. Confirm InfluxDB Data Ingestion

  • Via InfluxDB CLI:
    • Enter the InfluxDB shell: 
      docker exec -it influxdb influx
    • Query recent data: 
      USE mydb
SELECT * FROM DS18B20 ORDER BY time DESC LIMIT 5
    • Replace DS18B20 with your measurement name.

3. Validate Grafana Visualization

  • Open http://localhost:3000 in your browser.
  • Check your dashboard panels for live or recent sensor data.
  • If no data appears, verify your data source and query settings.

4. Node-RED Flow Testing (if used)

  • Open http://localhost:1880.
  • Use debug nodes to inspect incoming MQTT messages and processed outputs.
  • Ensure flows are deployed and connected to the correct topics and database.

Common Issues & Troubleshooting

Issue Possible Cause Solution
No MQTT messages received ESP32 not connected, wrong broker address, Wi-Fi issues Check ESP32 Wi-Fi and MQTT settings, verify broker is running
Data not appearing in InfluxDB Node-RED flow misconfigured, wrong database/measurement Check Node-RED logs, verify InfluxDB credentials and flow configuration
Grafana shows no data Data source misconfigured, wrong query, InfluxDB not running Check Grafana data source settings, ensure InfluxDB is up, review query syntax
ESP32 not connecting to Wi-Fi Incorrect SSID/password, weak signal Re-enter credentials, move device closer to router
MQTT broker fails to start Port conflict, config error Check Docker logs, ensure ports 1883/9001 are free, review mosquitto.conf

Tips:

  • Use container logs for debugging: 
    docker logs <container_name>
  • Restart containers after configuration changes.
  • Use MQTT clients to inspect live traffic and verify topic structure.

By following these steps, you can ensure your sensor platform is operating correctly and quickly resolve common issues.

Further Reading and References

To deepen your understanding of the technologies and architecture described in this document, the following resources provide comprehensive background, tutorials, and best practices:

General Concepts

Platform Components

Example Projects and Tutorials

Security and Best Practices

These resources will help you understand the rationale behind each component, how they interact, and how to extend or secure your own sensor platform.

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Contributors 2

  •  
  •