Openthread RCP using USB to control it

[Mavioux]

Hello everyone,

I am having an esp32 Openthread application that I use in RCP mode. I am flashing the ot_rcp esp example to my esp32c6 and then I am controlling it through a ubuntu host machine. This works perfectly when I am connecting it to the UART port of my esp32c6. But unfortunately I have a PCB that only exposes the USB port of my esp32c6. When I switch to that port the host machine does not receive any response from the esp32 in RCP mode. Does openthread in esp32 only work through native UART and SPI, or can it also work through USB?

If it cannot work through USB natively is there any way I can modify the openthread stack to support that?

Or should I redirect USB traffic to a UART controller? (any hint or direction on how to do that would be appreciated)

[chshu]

@Mavioux The USB port RCP support is under progress, will update here once it's available.

[gytxxsy]

@Mavioux
We will officially support USB-based ESP ot_rcp in the future. This is a patch for temporary support of USB-based ot_rcp:
0001-feat-openthread-support-rcp-over-usb.zip

In addition to applying this patch, you must ensure that your branch includes the following commits:

• feat(usb_serial_jtag): Update vfs read to be POSIX compliant
• feat(usb_cdc): Update vfs read() to comply with POSIX standards
• feat(usb_serial_tag_vfs): Add test for read exit conditions
  
• fix(driver): remove unecessary if conditions in the read function
  

These commits fixed USB Serial JTAG issues. Without this commit, the USB-based ot_rcp may not function properly.

Usage example:

1. Based on esp-idf commit 68b583f3, copy the patch to the esp-idf/ directory and apply it using the following command:

git apply 0001-feat-openthread-support-rcp-over-usb.patch

2. Navigate to examples/openthread/ot_rcp, set the target, and build:

idf.py set-target esp32c6  
idf.py build

3. Connect your PC to the device via USB and flash the firmware:

idf.py -p /dev/ttyACM0 flash monitor

After that, you can communicate with the ot_rcp device over USB (for example, use it as a sniffer).

[Mavioux]

Thank you very much for the patch!

I applied it and at first it seemed that it worked! The host managed to control the RCP through the USB port (It seemed like it worked as expected based on the behaviour when connected through the UART port). But then I disconnected the cable and reconnected it to the host. Then the host could not receive a response. If I plug the cable in and out multiple times and restart the host application that controls the RCP node, then the communication is restored at some point.

Is this because this is a patch and not finished? Have you experiences anything similar? Shall I expect that this would be fixed in the final release? Or should I look at the host application to fix this issue?

Any help is appreciated!

[MaikVermeulen]

I just tried it with the CONFIG_OPENTHREAD_RCP_USB_SERIAL_JTAG option introduced here, configured in the ot_rcp example, flashed on an ESP32C6 DevKitM-1. I got an RCP communication timeout, so ot-br-posix exited. I tried it a couple more times, no luck. When unplugging and replugging the ESP32, it started working. However that's not reliable enough for our customer facing use case. Also when replugging it a couple more times, it doesn't work every time.

Is this something you also see?

I tried getting OpenThread/Spinel debug logs over UART but didn't succeed yet. Will try to spend a bit more time on that, and troubleshooting, today or tomorrow.

[chshu]

@Mavioux @MaikVermeulen RCP over USB has been supported since this commit: 51bcec7.

I tested the plug-and-unplug scenario, and the only case where the connection fails to recover is when the port number changes, for example, /dev/ttyACM0 becomes /dev/ttyACM1.

Could you please double-check the following:

• Ensure that no other tool (e.g., IDF monitor or any serial terminal) is accessing the port at the same time.
• After unplugging and plugging the device back in, confirm that the port number remains the same as configured in the otbr-agent settings.

If the issue persists after verifying these points, please share the ot-br-posix commit ID and the otbr-agent logs so we can further investigate.

[MaikVermeulen]

Thanks @chshu , I'm using that commit and the new CONFIG_OPENTHREAD_RCP_USB_SERIAL_JTAG option.
And indeed no other tools are competing for the port, and the port number stays the same.

It usually takes me around 15 plug-and-unplugs to get the issue again, using ot_rcp with the above option.

For troubleshooting, I reverted to my custom OpenThread application on our Linux host.
Furthermore, on the RCP I enabled debug ESP & OpenThread verbose logging and added some debug statements to the otPlatUartSend() and otPlatUartReceive() functions. However, that seemed to decrease how often the issue reproduced!

Some more observations: it seemed that even if the issue happened, the RCP was reporting to send bytes with otPlatUartSend(), which of course seems good 🙂 :

I (732) phy: libbtbb version: ec2ecba, Mar  3 2025, 16:01:27
D (732) intr_alloc: Connected src 12 to int 12 (cpu 0)
E (732) OPENTHREAD: RCP USB rval: -1
E (732) OPENTHREAD: RCP USB otPlatUartSend rval: 8
E (742) OPENTHREAD: RCP USB rval: -1
I (592) main_task: Returned from app_main()
E (4952) OPENTHREAD: RCP USB rval: 1
E (4982) OPENTHREAD: RCP USB rval: 7
E (4982) OPENTHREAD: RCP USB otPlatUartSend rval: 8
E (4982) OPENTHREAD: RCP USB rval: -1
E (6982) OPENTHREAD: RCP USB rval: 7
E (6982) OPENTHREAD: RCP USB otPlatUartSend rval: 9
E (6982) OPENTHREAD: RCP USB rval: -1
E (8982) OPENTHREAD: RCP USB rval: 7
E (8982) OPENTHREAD: RCP USB otPlatUartSend rval: 9
E (8982) OPENTHREAD: RCP USB rval: -1
E (10992) OPENTHREAD: RCP USB rval: 7
E (10992) OPENTHREAD: RCP USB otPlatUartSend rval: 9
E (10992) OPENTHREAD: RCP USB rval: -1

However, when the issue happened and it was not recovering on our Linux device, I stopped the OpenThread host application and used the attached Python script to check if the RCP was indeed responding or not. It was responding, but with weird output:

# python3 test_rcp.py
[INFO] Bytes waiting in RX buffer before sending: 0
[INFO] Sending Spinel RESET command...
[INFO] Read 141 bytes: 45202832373231363729204f50454e5448524541443a2052435020555342207276616c3a20370a7e80060070ee747e45202832373231363729204f50454e5448524541443a2052435020555342206f74506c61745561727453656e64207276616c3a20380a45202832373231363729204f50454e5448524541443a2052435020555342207276616c3a202d310a
[OK] Final response (hex): 45202832373231363729204f50454e5448524541443a2052435020555342207276616c3a20370a7e80060070ee747e45202832373231363729204f50454e5448524541443a2052435020555342206f74506c61745561727453656e64207276616c3a20380a45202832373231363729204f50454e5448524541443a2052435020555342207276616c3a202d310a

Decoding all those bytes leads to:

E (272167) OPENTHREAD: RCP USB rval: 7
~��pît~E (272167) OPENTHREAD: RCP USB otPlatUartSend rval: 8
E (272167) OPENTHREAD: RCP USB rval: -1

which matches with my added logs. Could it be that debug logs are interfering with the USB RCP traffic?
I have attached the used sdkconfig below

rcp_sdkconfig.txt
test_rcp.zip

[chshu]

@MaikVermeulen If log output is enabled on the same USB port used for Spinel communication, it may cause issues.
By default, logging is disabled in the ot_rcp example, as shown here:
https://github.com/espressif/esp-idf/blob/master/examples/openthread/ot_rcp/sdkconfig.defaults#L47

Could you please try using the default configuration as-is, and only enable the CONFIG_OPENTHREAD_RCP_USB_SERIAL_JTAG option to switch the Spinel transport from UART to USB?

[MaikVermeulen]

@chshu the log output was enabled to troubleshoot, not the other way around, so it's not the cause of the issue we're having.

When you tried reproducing, how many times did plug/unplug? Sometimes it takes 20 attempts for me to reproduce, other times it's way quicker. In any case, even it would be 1 in a 100, we are not confident to roll out to our fleet with those numbers, as it would still incur a big operational support load.

[MaikVermeulen]

Did some more tests, and collected a USB pcap which can be found attached. The pcap snippet is first of some successful replugs (so replug -> working Spinel comms), followed by a replug after which Spinel is not working anymore. It does not recover, also not when I unbind & rebind my CDC-ACM driver for example.
constant_replugging_working_into_nonworking.zip

Furthermore, I found something interesting:
When I take our ESP32C6 DevKit-M1, with the default ot_rcp example but of course configured for RCP over USB, I did this:
I tested with keeping the devkit powered over UART and constantly replugging the USB, it just kept working after way too many replugs.

So it seems to stay healthy if the chip/stack doesn't restart, so perhaps a race issue during power on of the ESP firmware

I believe this indicates that the issue is from the ESP's side, not our host USB stack or our application.

I would like to highlight that this is blocking us from going to production with our actual custom ESP32C6 based board

[MaikVermeulen]

spinel_driver.cpp:87 is a SuccessOrDie(), the application is expected to exit after this log and then restart to recover. Could you check why your application does not exit at this point?

Our service also exists, and restarts, and crashes with the same log again, forever. I just didn't include those logs. The application is not the issue, it's just using the same stack as otbr-agent is also using

Perhaps having a device wirelessly communicating to the RCP is relevant to reproducing the issue.

This is a strange finding, but based on the current behavior, it seems more like a host recovery flow issue than an RCP issue.

I disagree that it seems like a host recovery issue, because then the constant re-plugging of USB while keeping the DevKit powered over UART should also reproduce the issue. Furthermore, @Skyler84 's findings also seem to point to the RCP.

Can we escalate this issue within ESP? I don't feel like you are taking this seriously enough, while it may cause us to choose another chip for our products.

[chshu]

Regardless of whether the issue is on the host or the RCP side, we are fully committed to reproducing and fixing it. What doesn’t make sense to me is that it occurs (or more likely to occur) only when another node is present in the network. The replug behavior should have nothing to do with the presence of other nodes in the network, right?

Could you help verify whether the issue occurs only with RCP over USB and not with RCP over UART? RCP over USB is a newly supported feature on ESP platform, whereas RCP over UART is more mature.

Please also contact our support team via https://www.espressif.com/en/contact-us/technical-inquiries and share your project schedule. We will take it as a priority.

[MaikVermeulen]

Thank you! Have you also had a chance to review the USB pcap I shared earlier?

What doesn’t make sense to me is that it occurs (or more likely to occur) only when another node is present in the network. The replug behavior should have nothing to do with the presence of other nodes in the network, right?

I have not tried it often enough to be 100% sure, and did not collect hard data on it, so it might not be relevant. Though, having a node causing traffic to the RCP could impact it's performance/timing, so in that case it could impact replug behavior?

Could you help verify whether the issue occurs only with RCP over USB and not with RCP over UART? RCP over USB is a newly supported feature on ESP platform, whereas RCP over UART is more mature.

I have flashed the classic ot_rcp with UART to our C6 DevKit M1, and replugged it ~70 times. It kept working without issues

[gytxxsy]

@Mavioux
We have currently identified this as a USB Serial JTAG-related issue, and we are actively working on a solution internally. Here is a workaround: you can temporarily resolve the issue by manually flushing the TX FIFO after sending data. You can use this patch:
0001-patch-for-ot_rcp-over-USB.zip

I have tested it: before applying the workaround, I was able to reproduce the issue within ten hardware resets in my local environment. After applying the workaround, I performed approximately fifty hardware resets, and the issue did not occur. If you have any questions, feel free to ask. Thanks.

[MaikVermeulen]

@gytxxsy indeed it seems stable now, thank you for the investigation and fix! 🚀