LWIP bridge w/ USB-NCM (IDFGH-14930)

[mickeyl]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

I tried adding USB-NCM to the LWIP bridge basing on code that has been developed by @kostaond as per #5697.

I've used @chegewara's usb-netif component here and slightly tweaked it (don't set IP and don't run DHCPS, since this will be happening on the bridge interface).

My example is @ https://github.com/mickeyl/esp-usb-wifi-eth-bridge -- but unfortunately it doesn't work.

The USB network interface appears, but despite adding it to the bridge interface (via esp_netif_br_glue_add_port, the USB interface is not coming up, i.e. there's no IP. It should share 192.168.4.1 with the WiFI and the W5500, shouldn't it? Do I need to setup the MAC differently?

What am I missing?

[chegewara]

I think you should study those examples
https://github.com/espressif/esp-idf/tree/master/examples/network/sta2eth
https://github.com/espressif/esp-idf/tree/master/examples/network/eth2ap

My component is based on the first example.

[mickeyl]

I don't think so. As far as I can see these examples are doing the forwarding "on their own". The bridge example does this in a higher level way basing on LWIP. As it already works fine with WiFi and ETH, I think there can only be a small thing missing or wrong.

[kostaond]

Hi @mickeyl, I'm terribly sorry but I currently don't have any time to have a look at it in details...

Check the followig:

• does USB sets its netif to UP? E.g. does it call esp_netif_action_connected anywhere? Port must be up to be forwarded.
• At least one of the interfaces either Ethernet or WiFi needs to be UP to set bridge netif UP. This is currently a limitation of esp_netif_br_glue.c. It natively only monitors for ETH and WiFi actions to start the bridge but once started just by one interface, there is no problem.

IP address e.g. 192.168.4.1 is IP address of bridge netif. Physical interfaces like ETH, WiFi doesn't have it's IP address. They are there to pass just Ethernet frames.

[mickeyl]

I call esp_netif_action_connected right after starting the interface.

When adding log output to netif_recv_callback and l2_free I see that these functions are called regularly:

D (136689) usb-netif: netif_recv_callback: 333
D (136689) usb-netif: l2_free
D (136689) usb-netif: netif_recv_callback: 110
D (136689) usb-netif: l2_free
D (136959) usb-netif: netif_recv_callback: 110
D (136959) usb-netif: l2_free
D (137309) usb-netif: netif_recv_callback: 86
D (137309) usb-netif: l2_free
D (138339) usb-netif: netif_recv_callback: 110
D (138339) usb-netif: l2_free
D (138359) usb-netif: netif_recv_callback: 62
D (138359) usb-netif: l2_free

The stack seems to receive the packets my computer connected via USB_NCM is sending (333 is the length of the DHCP DISCOVER), but there don't seem to be any answers.

Therefore the interface does not seem to get an IP from the DHCPS whereas both the WiFi and the W5500 interfaces are getting one.

How can I debug this further?

[kostaond]

• Could you please enable debug logs for the lwIP bridge? Component config->LWIP->Enable LWIP debug->Enable bridge *
• Make sure your USB_NCM is set to promiscuous mode.
• Capture a traffic in computer connected to USB_NCM.

[mickeyl]

bool p = true;
esp_eth_ioctl(usb_netif, ETH_CMD_S_PROMISCUOUS, &p);

crashes. Perhaps I need a special way to do this on USB_NCM?

XXX setting USB netif to promiscuous mode
D (2234) esp_netif_br_glue: bridge netif 0x3fcbe1d0 is started
D Guru Meditation Error: Core  0 panic'ed (InstrFetchProhibited). Exception was unhandled.

Core  0 register dump:
PC      : 0x812d6282  PS      : 0x00060130  A0      : 0x8200e26f  A1      : 0x3fcad810
A2      : 0x3fcbdfdc  A3      : 0x00000008  A4      : 0x3fcad90e  A5      : 0x00000000
A6      : 0x3fcbe06c  A7      : 0x3fcbdfdc  A8      : 0x82023cfa  A9      : 0x3fcad7f0
A10     : 0x42020fc4  A11     : 0x00000001  A12     : 0x40387714  A13     : 0x00000000
--- 0x42020fc4: ethernetif_init at /home/mickey/Dokumente/late/misc/esp-idf/components/esp_netif/lwip/netif/ethernetif.c:158
0x40387714: free at /home/mickey/Dokumente/late/misc/esp-idf/components/newlib/heap.c:38

A14     : 0x00000000  A15     : 0x3fcaa848  SAR     : 0x0000001a  EXCCAUSE: 0x00000014
EXCVADDR: 0x812d6280  LBEG    : 0x400556d5  LEND    : 0x400556e5  LCOUNT  : 0xfffffff5
--- 0x400556d5: strlen in ROM
0x400556e5: strlen in ROM



Backtrace: 0x412d627f:0x3fcad810 0x4200e26c:0x3fcad850 0x4209e50f:0x3fcad980 0x4037f4d9:0x3fcad9b0
--- 0x4200e26c: app_main at /home/mickey/Dokumente/late/misc/eth-usb-bridge/main/bridge_example_main.c:220
0x4209e50f: main_task at /home/mickey/Dokumente/late/misc/esp-idf/components/freertos/app_startup.c:208
0x4037f4d9: vPortTaskWrapper at /home/mickey/Dokumente/late/misc/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

[kostaond]

@mickeyl you can't use esp_eth_ioctl with netif handle, this must be used with Ethernet handle. Check if the current driver of USB_NCM support esp_eth_ioctl. Configuring the promiscuous mode is MAC specific feature.

[chegewara]

Im not networking expert, actually i am pretty lame, but i spent few hours today trying to figure out if i can do it.
My conclusion is that USB netif is missing a glue code, which we have for wifi sta/ap and eth.

The idea i have is to make, but im not sure if that solution is satisfying (and not sure if/how it may works):

• usb-netif to eth
• bridge eth with wifi sta/ap

[mickeyl]

Since I'm not sure whether the problem is on the sending or the receiving side, I figured I try to trace the
way of the DHCP DISCOVER packet through the stack.

On the USB side, Wireshark reports that it properly gets sent to the USB interface.
It looks like it gets picked up ok by TinyUSB, since our glue code [usb_stack.c:35] gets called:
static esp_err_t netif_recv_callback(void *buffer, uint16_t len, void *ctx) w/ len being 333.

usb_stack.c now calls esp_netif_receive(usb_netif_p, buf_copy, len, NULL) where in
[esp_net_lwip.c:1335] it gets forwarded via esp_netif->lwip_input_fn(esp_netif->netif_handle, buffer, len, eb).

We continue in [ethernetif.c:119] esp_netif_recv_ret_t ethernetif_input(void *h, void *buffer, size_t len, void *l2_buff) where in line 139 we call
the esp_netif's input function to put it up the TCP/IP stack: netif->input(p, netif) which points to tcpip_input [tcpip.c:288].

In there we check whether NETIF_FLAG_ETHARP or NETIF_FLAG_ETHERNET is set to determine whether we have to pass the Ethernet frame or the IP frame. Since at least one of those flags is set on our interface, the code calls
tcpip_inpkt(p, inp, ethernet_input) [tcpip.c:245] which posts it to the tcpip_thread via sys_mbox_trypost(&tcpip_mbox, msg).

Fast forward to the context of the tcpip_thread in tcpip_thread_handle_msg [tcipip.c:156], where we pick up our message from the queue, switch over its type (since it could be an input packet, an API call, or what not) and call ethernet_input [ethernet.c:81], since the USB/esp_netif stack has marked this as the interface's input function.

In this function we correctly detect that the packet as a link-layer broadcast and then switch over the type. It is being recognized as IP packet and forwarded via ip4_input(p, netif) in [ethernet.c:186].

Starting w/ [ip4.c:487], we do a lot of target and sanity checks before finally sending the packet forward via udp_input(p, inp) in [ip4.c:780].

In [udp.c:194], we iterate through all UDP PCBs, but nothing is registered, hence none matches. If I see things correctly, then udp_input_local_match is returning 0 in our case, due to

/* check if PCB is bound to specific netif */
  if ((pcb->netif_idx != NETIF_NO_INDEX) &&
      (pcb->netif_idx != netif_get_index(ip_data.current_input_netif))) {
    return 0;
  }

With UDP debugging on, I can see the following in the case of the wired interface:

D (4092) lwip: udp_input: received datagram of length 298
D (4092) lwip: UDP header:
D (4112) lwip: +-------------------------------+
D (4112) lwip: |        68     |        67     | (src port, dest port)
D (4122) lwip: +-------------------------------+
D (4122) lwip: |       298     |     0xc4a1    | (len, chksum)
D (4132) lwip: +-------------------------------+

D (4132) lwip: udp (
D (4132) lwip: 255.255.255.255
D (4142) lwip: , 67) <-- (
D (4142) lwip: 0.0.0.0
D (4142) lwip: , 68)

D (4142) lwip: pcb (
D (4152) lwip: 192.168.4.1
D (4152) lwip: , 67) <-- (
D (4152) lwip: 0.0.0.0
D (4152) lwip: , 0)

D (4152) lwip: udp_input: calculating checksum
D (4172) lwip: udp_send
D (4172) lwip: udp_send: added header in given pbuf 0x3fcad2f4
D (4182) lwip: udp_send: sending datagram of length 556
D (4182) lwip: udp_send: UDP packet length 556
D (4182) lwip: udp_send: UDP checksum 0x3919
D (4192) lwip: udp_send: ip_output_if (,,,,0x11,)

When the packet from USB comes in, it looks like that:

D (626242) lwip: udp_input: received datagram of length 299
D (626242) lwip: UDP header:
D (626262) lwip: +-------------------------------+
D (626262) lwip: |        68     |        67     | (src port, dest port)
D (626272) lwip: +-------------------------------+
D (626272) lwip: |       299     |     0x423c    | (len, chksum)
D (626272) lwip: +-------------------------------+

D (626282) lwip: udp (
D (626282) lwip: 255.255.255.255
D (626282) lwip: , 67) <-- (
D (626292) lwip: 0.0.0.0
D (626292) lwip: , 68)

D (626292) lwip: pcb (
D (626292) lwip: 192.168.4.1
D (626302) lwip: , 67) <-- (
D (626302) lwip: 0.0.0.0
D (626302) lwip: , 0)

D (626302) usb-netif: l2_free
D (626942) lwip: dhcp_coarse_tmr()

[mickeyl]

In LWIP's bridgeif.c, it says:

* - When adding a port netif, its input function is changed to call into the bridge.

Could it be that this doesn't happen for the USB interface and that's the reason why we see udp_input_local_match failing?

In fact, I don't see bridgeif_add_port being called for the USB interface, although we're calling start and connected on it. Is it possibly a race condition?

[mickeyl]

@chegewara

Im not networking expert, actually i am pretty lame, but i spent few hours today trying to figure out if i can do it. My conclusion is that USB netif is missing a glue code, which we have for wifi sta/ap and eth.

But which part?

The idea i have is to make, but im not sure if that solution is satisfying (and not sure if/how it may works):

* usb-netif to eth

* bridge eth with wifi sta/ap

Sounds like a plan B, but it would be a mixture of doing it ourselves and having LWIP bridge ETH and AP.
After all, the bridge code seems to be written with multiple ethernet interfaces in mind and from the viewpoint of LWIP, USB-NCM is ethernet. I'm pretty sure it's just a tiny thing we're overlooking to make it work with the builtin bridge.