(UE5 / Humble) De-/serialization problem when trying to publish sensor_msgs/Image or CompressedImage

[agaertner]

stacktrace:

[rosbridge_tcp-1] [INFO] [1673531368.773922849] [rosbridge_tcp]: [Client 0] Subscribed to /unreal/observer_image
[rosbridge_tcp-1] [ERROR] [1673531373.971018822] [rosbridge_tcp]: [Client 0] [id: publish:/unreal/observer_image:288] publish: Message type sensor_msgs/Image does not have a field header.seq
[rosbridge_tcp-1] [ERROR] [1673531373.980958077] [rosbridge_tcp]: [Client 0] [id: publish:/unreal/observer_image:290] publish: Message type sensor_msgs/Image does not have a field header.seq
[rosbridge_tcp-1] [ERROR] [1673531374.040446020] [rosbridge_tcp]: [Client 0] Exception in deserialization of BSON
[rosbridge_tcp-1] exc in handle/recv_bson: (<class 'SyntaxError'>, SyntaxError('invalid syntax', ('<string>', 0, 0, '', 0, 0)), <traceback object at 0x7f9a6b1cc680>)
[rosbridge_tcp-1] Traceback (most recent call last):
[rosbridge_tcp-1]   File "/home/agaertner/ros2_humble/ros2_ws/install/rosbridge_server/local/lib/python3.10/dist-packages/rosbridge_server/tcp_handler.py", line 109, in handle
[rosbridge_tcp-1]     if self.recv_bson() is None:
[rosbridge_tcp-1]   File "/home/agaertner/ros2_humble/ros2_ws/install/rosbridge_server/local/lib/python3.10/dist-packages/rosbridge_server/tcp_handler.py", line 85, in recv_bson
[rosbridge_tcp-1]     data = self.recvall(msglen - BSON_LENGTH_IN_BYTES)
[rosbridge_tcp-1]   File "/home/agaertner/ros2_humble/ros2_ws/install/rosbridge_server/local/lib/python3.10/dist-packages/rosbridge_server/tcp_handler.py", line 74, in recvall
[rosbridge_tcp-1]     return eval(bytes(data,'utf-8'))
[rosbridge_tcp-1]   File "<string>", line 0
[rosbridge_tcp-1]     
[rosbridge_tcp-1] SyntaxError: invalid syntax
[rosbridge_tcp-1] exc in handle/recv_bson: (<class 'SyntaxError'>, SyntaxError('invalid syntax', ('<string>', 0, 0, '', 0, 0)), <traceback object at 0x7f9a6b1afb00>)
[rosbridge_tcp-1] exc in handle/recv_bson: (<class 'SyntaxError'>, SyntaxError('invalid syntax', ('<string>', 0, 0, '', 0, 0)), <traceback object at 0x7f9a6b1afb00>)
[rosbridge_tcp-1] Traceback (most recent call last):
[rosbridge_tcp-1]   File "/home/agaertner/ros2_humble/ros2_ws/install/rosbridge_server/local/lib/python3.10/dist-packages/rosbridge_server/tcp_handler.py", line 109, in handle
[rosbridge_tcp-1]     if self.recv_bson() is None:
[rosbridge_tcp-1]   File "/home/agaertner/ros2_humble/ros2_ws/install/rosbridge_server/local/lib/python3.10/dist-packages/rosbridge_server/tcp_handler.py", line 85, in recv_bson
[rosbridge_tcp-1]     data = self.recvall(msglen - BSON_LENGTH_IN_BYTES)
[rosbridge_tcp-1]   File "/home/agaertner/ros2_humble/ros2_ws/install/rosbridge_server/local/lib/python3.10/dist-packages/rosbridge_server/tcp_handler.py", line 74, in recvall
[rosbridge_tcp-1]     return eval(bytes(data,'utf-8'))
[rosbridge_tcp-1]   File "<string>", line 0
[rosbridge_tcp-1]     
[rosbridge_tcp-1] SyntaxError: invalid syntax
[rosbridge_tcp-1] Traceback (most recent call last):
[rosbridge_tcp-1]   File "/home/agaertner/ros2_humble/ros2_ws/install/rosbridge_server/local/lib/python3.10/dist-packages/rosbridge_server/tcp_handler.py", line 109, in handle
[rosbridge_tcp-1]     if self.recv_bson() is None:
[rosbridge_tcp-1]   File "/home/agaertner/ros2_humble/ros2_ws/install/rosbridge_server/local/lib/python3.10/dist-packages/rosbridge_server/tcp_handler.py", line 85, in recv_bson
[rosbridge_tcp-1]     data = self.recvall(msglen - BSON_LENGTH_IN_BYTES)
[rosbridge_tcp-1]   File "/home/agaertner/ros2_humble/ros2_ws/install/rosbridge_server/local/lib/python3.10/dist-packages/rosbridge_server/tcp_handler.py", line 74, in recvall
[rosbridge_tcp-1]     return eval(bytes(data,'utf-8'))
[rosbridge_tcp-1]   File "<string>", line 0
[rosbridge_tcp-1]     
[rosbridge_tcp-1] SyntaxError: invalid syntax
[rosbridge_tcp-1] Traceback (most recent call last):
[rosbridge_tcp-1]   File "/home/agaertner/ros2_humble/ros2_ws/install/rosbridge_server/local/lib/python3.10/dist-packages/rosbridge_server/tcp_handler.py", line 109, in handle
[rosbridge_tcp-1]     if self.recv_bson() is None:
[rosbridge_tcp-1]   File "/home/agaertner/ros2_humble/ros2_ws/install/rosbridge_server/local/lib/python3.10/dist-packages/rosbridge_server/tcp_handler.py", line 85, in recv_bson
[rosbridge_tcp-1]     data = self.recvall(msglen - BSON_LENGTH_IN_BYTES)
[rosbridge_tcp-1]   File "/home/agaertner/ros2_humble/ros2_ws/install/rosbridge_server/local/lib/python3.10/dist-packages/rosbridge_server/tcp_handler.py", line 74, in recvall
[rosbridge_tcp-1]     return eval(bytes(data,'utf-8'))
[rosbridge_tcp-1]   File "<string>", line 0
[rosbridge_tcp-1]     
[rosbridge_tcp-1] SyntaxError: invalid syntax
[rosbridge_tcp-1] exc in handle/recv_bson: (<class 'SyntaxError'>, SyntaxError('invalid syntax', ('<string>', 0, 0, '', 0, 0)), <traceback object at 0x7f9a6b1afb00>)

I am using naxsm fork of rosbridge_suite and have tried the fork of tsender both error with a missing header.seq field.

My message creation functions for Compressed or NonCompressed images are as follows:

bool UTopic::PublishCompressedImageMessage(const TArray<uint8>& imageData, const ECompressedImageFormat& format, const FString& frameId) {
	
	FString fmt;
	switch (format) 
	{
		case ECompressedImageFormat::Jpeg: 
		{
			fmt = "jpeg"; 
			break;
		} 
		case ECompressedImageFormat::Png: 
		{
			fmt = "png"; 
			break;
		}
		default: return false;
	}

	if (!_State.Advertised)
	{
		if (!Advertise())
		{
			return false;
		}
	}

	TSharedPtr<ROSMessages::sensor_msgs::CompressedImage> msg = MakeShareable(new ROSMessages::sensor_msgs::CompressedImage);

	TSharedPtr<ROSMessages::std_msgs::Header> header = MakeShareable(new ROSMessages::std_msgs::Header(1, FROSTime::Now(), frameId, true));
	msg->header = *header;
	msg->data_size = imageData.Num();
	msg->data = imageData.GetData();
	msg->format = fmt;
	return _Implementation->Publish(msg);
}

bool UTopic::PublishImageMessage(const int32& height, const int32& width, const FString& encoding, const bool& isBigEndian, const int32& rowLength, const TArray<uint8>& imageData, const FString& frameId) {
	if (!_State.Advertised)
	{
		if (!Advertise())
		{
			return false;
		}
	}

	TSharedPtr<ROSMessages::sensor_msgs::Image> msg = MakeShareable(new ROSMessages::sensor_msgs::Image);

	TSharedPtr<ROSMessages::std_msgs::Header> header = MakeShareable(new ROSMessages::std_msgs::Header(1, FROSTime::Now(), frameId, true));
	msg->header = *header;
	msg->height = height;
	msg->width = width;
	msg->encoding = encoding;
	msg->is_bigendian = isBigEndian;

	msg->step = imageData.Num();
	msg->data = imageData.GetData();

	return _Implementation->Publish(msg);
}
``

[tsender]

If you wish to use ROS2 with this plugin, you will need to use my foxy branch of rosbridge_suite and my feature/specify_ros_version branch of ROSIntegration. Note, that in my updated branch for ROSIntegration, you will need to indicate the ROS version in your game instance details window (the same area where you specify the rosbridge IP, etc.)

Also, the title of your issue says you are using UE5 (although your post does not mention it). If you are trying to use UE5 with this plugin, then I cannot help you on that front.

[agaertner]

If you wish to use ROS2 with this plugin, you will need to use my foxy branch of rosbridge_suite and my feature/specify_ros_version branch of ROSIntegration. Note, that in my updated branch for ROSIntegration, you will need to indicate the ROS version in your game instance details window (the same area where you specify the rosbridge IP, etc.)

Also, the title of your issue says you are using UE5 (although your post does not mention it). If you are trying to use UE5 with this plugin, then I cannot help you on that front.

That's a great! However, I am using humble not foxy. I will try and use a humble rosbridge_suite version and use your feature/specify_ros_version branch of the plugin.

[tsender]

I should have clarified. You will still need to use my foxy branch of rosbridge_suite. All the code should run fine on galactic and humble without problem. I do not know of anyone else that managed to work through all the bugs needed to bring back the TCP handler.