rcl_logging_syslog is alternative logging backend implementation that can be used for ROS 2 application via rcl_logging_interface.
rcl_logging_syslog uses SYSLOG(3) to send the log data to rsyslog a.k.a rocket-fast system for log processing 🚀.
The main objective is that Enabling ROS 2 logging system with Cloud-Native Log Management and Observability.
see overview slide deck for more information.
The logging data is critical especially for entire system observability and status, so that application can alert the administrator or even give the feedback to the system with adjusting the parameter. This importance rises once it comes to robotics and robot application, especially distributed system such as ROS 2 or edge computing because we must be able to specify what went wrong in the 1st place with these logging data.
rsyslog is available in default Ubuntu distribution managed by system service, performative, and many configuration supported including log data pipeline. So that user can choose the logging configuration depending on the application requirement and use case, sometimes file system sink, sometimes forwarding to remote rsyslogd, or even FluentBit.
FluentBit is a Fast Log Processor and Forwarder part of Graduated Fluentd Ecosystem and a CNCF sub-project.
See how it works 🔥
- rsyslog / FluebtBit
rcl_logging_syslog_to_fluent-bit.mp4
- rsyslog / Fluentd / Loki / Grafana
output.mp4
Supported ROS Distribution
| Distribution | Supported | Branch | Dynamic Loading |
|---|---|---|---|
| Rolling Ridley | ✅ | rolling (Development) |
✅ |
| Kilted Kaiju | ✅ | kilted |
❌ |
| Jazzy Jalisco | ✅ | jazzy |
❌ |
| Humble Hawksbill | ✅ | humble |
❌ |
Starting with Rolling Ridley, ROS 2 introduces rcl_logging_implementation, a package that enables runtime dynamic loading of logging backends, similar to how rmw_implementation works for middleware selection.
This abstraction layer allows users to switch between different logging implementations (such as rcl_logging_spdlog, rcl_logging_noop, or rcl_logging_syslog) without rebuilding RCL or application code.
See the ROS 2 Logging Documentation for more details.
The logging system supports two build configurations:
Dynamic Loading (Default, Rolling or later)
By default, rcl links against rcl_logging_implementation, which dynamically loads the logging backend at runtime.
This approach provides maximum flexibility, allowing the logging implementation to be changed via the RCL_LOGGING_IMPLEMENTATION environment variable without recompilation.
- The logging implementation is loaded as a shared library at runtime.
- No rebuild of
rclis required to switch between logging implementations. - Simply build
rcl_logging_syslogand set the environment variable to use it.
Static Linking (Kilted or older distributions)
For Kilted, Jazzy, and Humble distributions, the rcl_logging_implementation package is not available.
Users must rebuild rcl with the RCL_LOGGING_IMPLEMENTATION CMake/environment variable set at build time to statically link rcl_logging_syslog.
- The specified implementation is statically linked into
rclat build time. - Runtime switching is NOT available.
- Requires rebuilding
rclwhenever you want to change the logging backend.
- rsyslog installation
rcl_logging_syslog requires rsyslog package, which Ubuntu should have already in default.
But if you are using container, the situation is bit different from host system since there is no system services or rsyslogd is running by default.
In the case of container, we need to install rsyslog packages in the container root file system.
Note
We can enable the container with host system privileges but that is NOT recommended, especially for security.
### Install rsyslog package
apt install rsyslog- create
rosdirectory forrsyslog.
The following commands require root permission.
/var/log/ros is the root directory for rcl_logging_syslog, all logs are expected to be stored under this root directory.
mkdir -p /var/log/ros
chown syslog:adm /var/log/rosWarning
For production, user needs to configure appropriate group if necessary but owner should be syslog. Otherwise rsyslogd may meet the permission problem for the file system.
- rsyslog service check
If you are running the application on host system, check the rsyslog system service.
systemctl status rsyslogIf you are running the application on container environment, start the rsyslogd daemon process.
/usr/sbin/rsyslogd -n -iNONEOr you prefer to use rsyslogd in the host system from the application in the container, you need to bind the /dev/log to the container via -v /dev/log:/dev/log, so that the application running in the container can open the Unix Domain Socket for rsyslogd running in the host system.
In this setting, all rsyslogd configuration needs to be set in the host system.
- fluent-bit installation
if you want to use fluent-bit to forward the logging data from rsyslogd, please follow Fluent Bit Installation Manual to install it.
### Check fluent-bit package is installed
dpkg -l fluent-bit
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii fluent-bit 3.1.6 amd64 Fast data collector for LinuxNote
Update how to install the package here once it is released.
Please follow ROS 2 Official Development / Installation to build the rcl_logging_syslog package below.
There are two ways to build and use rcl_logging_syslog: dynamic loading or static linking.
See more details in the ROS 2 Logging Documentation and the rcl_logging_implementation section above.
Warning
Kilted or older distributions only support static linking and require rebuilding rcl.
See more details for ros2/rcl#1178.
-
Dynamic Loading (Rolling or later, Recommended)
With
rcl_logging_implementationavailable, you only need to buildrcl_logging_syslogand set theRCL_LOGGING_IMPLEMENTATIONenvironment variable at runtime. No rebuild ofrclis required.cd <YOUR_WORKSPACE>/src git clone https://github.com/fujitatomoya/rcl_logging_syslog.git cd <YOUR_WORKSPACE> colcon build --symlink-install --packages-select rcl_logging_syslog
Then, set the environment variable before running your application:
export RCL_LOGGING_IMPLEMENTATION=rcl_logging_syslog ros2 run <package> <executable>
-
Static Linking (Kilted or older)
For distributions that do not have
rcl_logging_implementation, you must rebuildrclwith theRCL_LOGGING_IMPLEMENTATIONenvironment variable set at build time.cd <YOUR_WORKSPACE>/src git clone https://github.com/fujitatomoya/rcl_logging_syslog.git export RCL_LOGGING_IMPLEMENTATION=rcl_logging_syslog colcon build --symlink-install --cmake-clean-cache --packages-select rcl_logging_syslog rcl
Following to the previous build procedure, we need to copy ros2-test.conf to /etc/rsyslog.d/, and restart the rsyslog.
sudo cp ./config/ros2-test.conf /etc/rsyslog.d/Then we can start the colcon test:
This test creates and uses /var/log/ros/rcl_logging_syslog directory in temporary for the test, and it checks the file existence and contents using rcl_logging_syslog.
colcon test --event-handlers console_direct+ --packages-select rcl_logging_syslogIf you are using dynamic loading (Rolling or later), set the logging implementation via the RCL_LOGGING_IMPLEMENTATION environment variable at runtime.
If you are using static linking (Kilted or older), the logging implementation is already linked into rcl at build time and this environment variable has no effect.
export RCL_LOGGING_IMPLEMENTATION=rcl_logging_syslog
# then run our application. e.g. "ros2 run <package> <executable>"See more details for rcl_logging_implementation and the ROS 2 Logging Documentation.
SYSLOG(3) is really simple that does not have much interfaces to control on application side, it just writes the log data on rsyslog Unix Domain Socket.
So we need to configure rsyslog how it manages the log message with /etc/rsyslog.conf, for example file system sink and forward the message to fluent-bit.
| environmental variable | default | Note |
|---|---|---|
RCL_LOGGING_SYSLOG_FACILITY |
LOG_LOCAL1 |
syslog facility: either of LOG_CRON, LOG_DAEMON, LOG_SYSLOG, LOG_USER, and LOG_LOCAL0-7. (/etc/rsyslog.conf must be configured accordingly.) |
Examples:
export RCL_LOGGING_SYSLOG_FACILITY="LOG_LOCAL2"See more details for https://www.rsyslog.com/doc/index.html.
Copy ros2-logging.conf to /etc/rsyslog.d/, and replace <FLUENTBIT_IP> with your own IP address where fluent-bit listens on.
sudo cp ./config/ros2-logging.conf /etc/rsyslog.d/Check the configuration file to see if there is no error,
rsyslogd -N1
rsyslogd: version 8.2312.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.then, restart the rsyslog system service, or simply restart rsyslog daemon when using container.
### via systemd (not container environment)
sudo systemctl restart rsyslog
systemctl status rsyslog
### container environment
/usr/sbin/rsyslogd -n -iNONEWith above configuration, rsyslogd also sends log data to TCP port 5140 that fluent-bit listens with the following configuration.
Replace <FLUENTBIT_IP> with your own IP address in fluent-bit.conf.
### Create working directory
mkdir fluentbit_ws; cd fluentbit_ws
### Copy the original files to the local directory
cp -rf /etc/fluent-bit/fluent-bit.conf fluent-bit.conf
cp -rf /etc/fluent-bit/parsers.conf parsers.conf
### Modify the configuration as below
cat fluent-bit.conf
[SERVICE]
Flush 1
Parsers_File parsers.conf
[INPUT]
Name syslog
Parser syslog-rfc3164
Listen <FLUENTBIT_IP>
Port 5140
Mode tcp
[OUTPUT]
Name stdout
Match *
# Start fluent-bit
/opt/fluent-bit/bin/fluent-bit --config=./fluent-bit.confLogging messages from ROS 2 application will be sent to rsyslogd and forwarded to fluent-bit if that is configured.
The following is an example output for file system and fluent-bit.
Start ROS 2 demo application:
ros2 run demo_nodes_cpp talker
[INFO] [1724453270.445092186] [talker]: Publishing: 'Hello World: 1'
[INFO] [1724453271.444981999] [talker]: Publishing: 'Hello World: 2'
[INFO] [1724453272.445056310] [talker]: Publishing: 'Hello World: 3'
...Logging files under file system:
With example configuration and template above, file name is <PROGRAMNAME>_<PID>_<YEAR>-<MONTH>-<DATE>.log.
And log data is prefixed with hostname, facility name, log level and program name as well.
root@tomoyafujita:/var/log# find ros
ros
ros/tomoyafujita
ros/tomoyafujita/talker_31348_2024-08-28.log
root@tomoyafujita:/var/log# cat ros/tomoyafujita/talker_31348_2024-08-28.log
Aug 28 11:30:18 tomoyafujita local1.info: talker[31348]: [INFO] [1724869818.620827927] [talker]: Publishing: 'Hello World: 1'
Aug 28 11:30:19 tomoyafujita local1.info: talker[31348]: [INFO] [1724869819.620758249] [talker]: Publishing: 'Hello World: 2'
Aug 28 11:30:20 tomoyafujita local1.info: talker[31348]: [INFO] [1724869820.620777277] [talker]: Publishing: 'Hello World: 3'FluentBit output:
/opt/fluent-bit/bin/fluent-bit --config=./fluent-bit.conf
Fluent Bit v3.1.6
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
______ _ _ ______ _ _ _____ __
| ___| | | | | ___ (_) | |____ |/ |
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __ / /`| |
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / \ \ | |
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /.___/ /_| |_
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ \____(_)___/
[2024/08/23 16:02:00] [ info] [fluent bit] version=3.1.6, commit=, pid=1686882
[2024/08/23 16:02:00] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/08/23 16:02:00] [ info] [cmetrics] version=0.9.4
[2024/08/23 16:02:00] [ info] [ctraces ] version=0.5.5
[2024/08/23 16:02:00] [ info] [input:syslog:syslog.0] initializing
[2024/08/23 16:02:00] [ info] [input:syslog:syslog.0] storage_strategy='memory' (memory only)
[2024/08/23 16:02:00] [ info] [in_syslog] TCP server binding 43.135.146.89:5140
[2024/08/23 16:02:00] [ info] [sp] stream processor started
[2024/08/23 16:02:00] [ info] [output:stdout:stdout.0] worker #0 started
...
[0] syslog.0: [[1724844661.000000000, {}], {"pri"=>"142", "time"=>"Aug 28 11:31:01", "host"=>"tomoyafujita", "ident"=>"talker", "pid"=>"31348", "message"=>"[INFO] [1724869861.620769614] [talker]: Publishing: 'Hello World: 44'"}]
[0] syslog.0: [[1724844662.000000000, {}], {"pri"=>"142", "time"=>"Aug 28 11:31:02", "host"=>"tomoyafujita", "ident"=>"talker", "pid"=>"31348", "message"=>"[INFO] [1724869862.620773450] [talker]: Publishing: 'Hello World: 45'"}]
[0] syslog.0: [[1724844663.000000000, {}], {"pri"=>"142", "time"=>"Aug 28 11:31:03", "host"=>"tomoyafujita", "ident"=>"talker", "pid"=>"31348", "message"=>"[INFO] [1724869863.620773763] [talker]: Publishing: 'Hello World: 46'"}]
...