AF_XDP issues on kernel 6.14 with Intel I225-LM — packet not passed to kernel stack and redirect_map not working

[YiHsiuChiu]

Hi, I’m currently testing AF_XDP on a system running Linux kernel 6.14 with an Intel I225-LM NIC, and I’ve encountered two issues:

1. When I bind an AF_XDP socket to an RX queue, packets on that queue are not passed to the kernel stack as expected.

   • My XDP program only uses redirect_map for specific types of packets, and other packets should be passed normally to the kernel stack.
   • However, once the AF_XDP socket is bound, it seems that all packets on that queue are blocked from reaching the kernel.

2. I tried binding the AF_XDP socket to a different RX queue (e.g., queue ID 3) and used redirect_map to redirect packets from another queue (e.g., queue ID 0) to the AF_XDP socket’s queue ID (3).

   • However, the redirected packets do not arrive at the AF_XDP socket.

I’d like to confirm whether these two setups are expected to work or if there are known limitations or configuration requirements I might have missed.

Any guidance or hints on what might be misconfigured would be greatly appreciated.

Thank you!

[tohojo]

Are you using AF_XDP in zero-copy mode for this?

Also, ping @magnus-karlsson

[YiHsiuChiu]

@tohojo Thank you so much for your quick response!

I now understand the difference between copy mode and zero-copy mode, and after switching to copy mode, it seems that issue 1 has been resolved.

Just to confirm — is it expected behavior that return XDP_PASS does not work properly in zero-copy mode?

[tohojo]

No, that should still work (just be slower). If it doesn't, that may be a driver bug (which I'm hoping @magnus-karlsson can say more about)

[magnus-karlsson]

If you receive a packet on queue id X, you can only direct it to a socket bound to queue id X. So your case 2 will not work. Case 1 should work. Have you tried with anoth NIC in case it is a driver bug?

[YiHsiuChiu]

Thanks for the explanation! Since I only have one machine available, it might take me some time to test this further.

Also, I have a question regarding the note in advanced03-AF_XDP’s README, which says:

Depending on your use-case, it can still make sense to use the “copy-mode” on a “zero-copy” capable driver. If for some reason, not all traffic on a RX-queue is for the AF_XDP socket, and the XDP program multiplexes between XDP_REDIRECT and XDP_PASS, then “copy-mode” can be relevant. As in “zero-copy” mode doing XDP_PASS has a fairly high cost, which involves allocating memory and copying over the frame.

Does this mean that, in practice, it’s generally better to avoid mixing XDP_REDIRECT and XDP_PASS in the same setup? (And just to confirm, the XDP_REDIRECT here refers to redirect_map, right?)

[magnus-karlsson]

You can mix them, no problem. But if you have more XDP_PASS than XDP_REDIRECT (to a map), you are better off with copy-mode as it will give higher performance in that case. XDP_PASS is cheaper in copy-mode while XDP_REDIRECT (to an xskmap) is cheaper in zc-mode.

[YiHsiuChiu]

Thanks for the clarification!

I tried several versions of the igc driver (from kernel 6.14, 6.11, and 6.8), but I’m still seeing the issue where packets with XDP_PASS are not successfully transmitted — although I can capture them with Wireshark, so they seem to leave the NIC?

Do you think this is more likely to be a software/kernel configuration issue, or could it be related to a hardware limitation or behavior specific to the I225-LM NIC?

Thank you!

[magnus-karlsson]

Now I am a little bit confused. You say that you perform an XDP_PASS on a packet and that it is not transmitted but it seems to leave the NIC. But XDP_PASS can only be performed on packets that are received. And in your first post you mentioned Rx. Are you saying that received packets are not being passed to the kernel stack but you can capture them with Wireshark as they come in?

[YiHsiuChiu]

Sorry for the confusion — you’re right. What I meant is that packets with XDP_PASS are not being delivered to the kernel stack, not that they fail to transmit.

[magnus-karlsson]

Thanks for the clarification. If you can see the packets with Wireshark, they have been delivered to the stack since it is after XDP. Your packets are being dropped by the stack for some reason. "perf record -e skb:kfree_skb" can give you some hint.

[YiHsiuChiu]

This is the result from running perf record -e skb:kfree_skb. It seems that the packets are indeed received by the kernel. Do you have any thoughts or suggestions on what might be happening in this case?

[YiHsiuChiu]

I switched to an X540 NIC (with the ixgbe driver), and the issue no longer occurs. It seems the problem might be related to the I225-LM hardware or the igc driver.

[magnus-karlsson]

OK, good to know. Likely a driver issue then.