"Problem" with endless dmesg/console logs: "icmpv6_send: no reply to icmp error" / Open Design Questions at forwarding level

[maverick3005]

Since feeled 1 month i see on the EA6x00 arm devices a huge amount of messages from this type:

root@mgmt1:/tmp/home/root# dmesg -c

icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error

On the other EA-Router, which serve the home office with just PCs and some Media-Devices, there are just 4-5 such Messages/day.

The First Router serves for Solar/IoT/ESP32-devices, which are using ip4 and ip6 and connects to the manufacturer clouds, and
it looks like, they all have his own methods to check if destination is alive or maybe check out different locations and try to measure the
distance by latency tests for selecting the nearest cloud, something like ?

I had no chance actualy to figure out the real reason, why i have thousands (!) of this lines in the router log-buffer each day since a few weeks.

The only differences between booth routers is, that the Office-Device has CTF enabled, and the IoT-serving Router has CTF disabled, because
of problems with bridge2bridge+wireless/L2+L3 not working when CTF is enabled.

Note: The Router in front/or when directly connected to ISP by dhcp6 has NO NAT or any type of filtering, so it is completly open
for incoming packets on ipv6 - which is a need here for some devices to be full reachable.

Question: Do we filter with ip6tables "hard coded" still some importent icmp6 messages, whe should not ?
The above messages comes only from LAN/WAN-forwarded traffic, not the local router itself.

This is the actual forwarding table i have running:

Chain FORWARD (policy DROP)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             rt type:0
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere             state INVALID
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
ACCEPT     ipv6-nonxt --  anywhere             anywhere             length 40
ACCEPT     ipv6-icmp --  anywhere             anywhere             ipv6-icmp destination-unreachable
ACCEPT     ipv6-icmp --  anywhere             anywhere             ipv6-icmp packet-too-big
ACCEPT     ipv6-icmp --  anywhere             anywhere             ipv6-icmp time-exceeded
ACCEPT     ipv6-icmp --  anywhere             anywhere             ipv6-icmp parameter-problem
ACCEPT     ipv6-icmp --  anywhere             anywhere             ipv6-icmp echo-request limit: avg 5/sec burst 5
ACCEPT     ipv6-icmp --  anywhere             anywhere             ipv6-icmp echo-reply
ACCEPT     esp  --  anywhere             anywhere
ACCEPT     udp  --  anywhere             anywhere             udp dpt:500
wanin      all  --  anywhere             anywhere
wanout     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

So nothing special - all on default. So why the Router itself fill up the log with this type of message (5 new messages at nearly every 10 seconds...) ?

Should we change this hardcoded "forced" setting anyway:

ACCEPT ipv6-icmp -- anywhere anywhere ipv6-icmp echo-request limit: avg 5/sec burst 5 <--- !

Such things should be handled from the client in case of forwarded traffic and not interfer by the router. For the router itself this rule make sence (INPUT-Chain).

This rules are another question:

DROP       all  --  anywhere             anywhere             state INVALID
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED

It looks for me like firewalling ip6 to the lan-clients - with no further (web based) tuning option to change it.

• except use "Basic IP6-Forwarding", where it can be defined which outside destination can connect to which port/IP6 at the inside...
  Here is the next Problem: This works only with local-fe80:: addresses reliable - if the ip6 prefix changed by ISP, so the internal ip6 native IPs will completly change. With typical windows/android and active "privacy extensions" it is anyway not working reliable with external ip6...

From my knowledge ip6 has "no NAT" - and is open to the devices - The devices itself need a local firewalling by handling
incoming traffic - windows, linux, as example, have this by default implemented at client side through the local firewall.

The hardcoded settings makes sense, when you have a lot of "broken" or "insecure" devices in the LAN, which can easy be hacked, because of
a not really working implemented firewall. So the result is: Always the sessions needs to be opened from inside to outside and must be
tracked from the router by conntrack. (With the usual timeout-issues and extra tuning eventually....)

AVM uses for ip6 another aproach: For specific internal devices there is an option to forward all incoming traffic without filtering,
wheras the basic setting is equaly as tomato: only inside -> outside. In case of "double-NAT" or having a Tomato behind a AVM-Device,
this Tomato gets all ip6 traffic directly handed over - just with a static mac-address-rule/fe80::-peer - so Tomato can handle the traffic byself.

(for that we have the Basic-IP6-Forwarding with above internal limitations to use only local-link fe80:: addresses...)

It is a design question generally - but when it comes to CGN ip4 + native ip6, this is the only way to reach inside (servers) from outside.

@MarsTomato

Any ideas, how we can make it a little "better" ? Can you check the actual hardcoded forward chain, too - have we forget to include some
more icmp6 message types ? Removing of the Burst-limit for icmp6-echo-request only in forward-mode ?

[maverick3005]

@pedro0311

it looks like i found an typo here for this entry:

ACCEPT ipv6-icmp -- anywhere anywhere ipv6-icmp echo-request limit: avg 5/sec burst 5

WAN Ping is ENABLED, so nvram variable is set to: block_wan=0
So the icmp blocking rule is NOT used in ipv4, but still active in ipv6

further checking the rc/firewall.c file i found this snippet:

  /* ICMPv6 rules */
                for (i = 0; i < sizeof(allowed_icmpv6)/sizeof(int); ++i) {
                        if ((allowed_icmpv6[i] == 128) && !nvram_get_int("block_wan")) /* ratelimit ipv6 ping */
                                ip6t_write("-A FORWARD -p ipv6-icmp --icmpv6-type %i -m limit --limit 5/s -j %s\n", allowed_icmpv6[i], chain_in_accept);
                        else
                                ip6t_write("-A FORWARD -p ipv6-icmp --icmpv6-type %i -j %s\n", allowed_icmpv6[i], chain_in_accept);
                }

!nvram_get_int("block_wan") is negated and will SET a rate limit, so the "else"-statement is not used.

...or it is a misunderstanding of the rule and it is only used on type 128 AND when block_wan=1,
so all types defined in the array "allowed_icmpv6" are rate limited - which is not wished to limit in FORWARD mode a huge
amount of icmp messages. As i stated, rate limiting is ok for the router itself, the forwarding ratelimit should the LAN/WLAN-devices handle byself ;-) ? - or we should add a web based button, so the user can choose to "protect" LAN devices from more icmp6 "ddos" ?

Please recheck my finding.

I will make some experiments by disable all forward ratelimiting on v6 for now. I expect the icmp error comes from the many IoT-Devices with her continous low icmp6-traffic to outside - so when you have a lot of them and all work at the sime time, rate limit can have some sideeffects... not an expert here, to be fair.

[maverick3005]

Found one thing more: The INPUT rule for the device has the same issue. But my "invert"-logic is wrong.

There are two settings in the firewall-menu: WAN-Ping ON/OFF and "Limit communications to x"
These variables are only correct for the ipv4 section.

When set the limit to "3" communications, the ipv6 section has still always "5" setted for INPUT and FORWARD, so
the function is completly ignored and the ratelimiting is ALWAYS on on ipv6 (INPUT&FORWARD) - no matter what the wan setting is.

• this is an error.

Long story short: The WAN-Ping & ratelimit functions needs to be checked and synced for booth: ipv4 and ipv6 ip(6)tables.

For more local testing i removed the icmp-ratelimit for INPUT and FORWARD rules by delete the if/else lines for now.

[maverick3005]

Just for info, for local testing i use at the moment this "fix":

--- release/src-rt-6.x.4708/router/rc/firewall.c
+++ release/src-rt-6.x.4708/router/rc/firewall.c
@@ -1304,9 +1304,6 @@

                /* ICMPv6 rules */
                for (i = 0; i < sizeof(allowed_icmpv6)/sizeof(int); ++i) {
-                       if ((allowed_icmpv6[i] == 128) && !nvram_get_int("block_wan")) /* ratelimit ipv6 ping */
-                               ip6t_write("-A FORWARD -p ipv6-icmp --icmpv6-type %i -m limit --limit 5/s -j %s\n", allowed_icmpv6[i], chain_in_accept);
-                       else
                                ip6t_write("-A FORWARD -p ipv6-icmp --icmpv6-type %i -j %s\n", allowed_icmpv6[i], chain_in_accept);
                }

@@ -1556,9 +1553,6 @@

        /* ICMPv6 rules */
        for (n = 0; n < sizeof(allowed_icmpv6)/sizeof(int); n++) {
-               if ((allowed_icmpv6[n] == 128) && !nvram_get_int("block_wan")) /* ratelimit ipv6 ping */
-                       ip6t_write("-A INPUT -p ipv6-icmp --icmpv6-type %i -m limit --limit 5/s -j %s\n", allowed_icmpv6[n], chain_in_accept);
-               else
                        ip6t_write("-A INPUT -p ipv6-icmp --icmpv6-type %i -j %s\n", allowed_icmpv6[n], chain_in_accept);
        }
        for (n = 0; n < sizeof(allowed_local_icmpv6)/sizeof(int); n++) {

but i still try to figure out the icmp6_send error "issue". After googling i found some answers, which are possible for the used kernel 2.6:

One answer is:

"The "icmpv6_send: no reply to icmp error" indicates your system tried to send an IPv6 ICMP error message (like destination unreachable) but never received the expected confirmation or response, often pointing to firewall blocks, misconfigured routing (especially with tunnels), network issues, or bugs in kernel/software handling IPv6, common with IPv6 transition technologies like SixXS or Xen virtualization"

and some more info:

_Common Causes & Troubleshooting:

• Firewall Rules (iptables/nftables): A firewall might be dropping the reply or blocking the ICMPv6 packets entirely. Check rules for DROP or REJECT on ICMPv6 types.
• Router/Neighbor Issues: Network devices between your server and the destination might be faulty or misconfigured._

and found more info:

/*
         *      Never answer to a ICMP packet.
         */

        if (is_ineligible(skb)) {
                if (net_ratelimit())
                        printk(KERN_DEBUG "icmpv6_send: no reply to icmp error\n");
                return;
        }


The function is_ineligible() itself carries the comment:

 We do not reply, if:
       - it was icmp error message.
       - it is truncated, so that it is known, that protocol is ICMPV6
         (i.e. in the middle of some exthdr)

My conclusion is, that we should local patch evtl. the kernel and outcomment this debug message or set an higher debug level, which is not the default for the local console - so it does no longer flood the ssh console (and serial console).

With a simple "ip" check i try further to search in the own network for some events or problems:

root@mgmt1:/tmp/home/root# dmesg -c
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error
icmpv6_send: no reply to icmp error

root@mgmt1:/tmp/home/root# ip -6 -s monitor
fe80::cad7:19ff:fe45:66d2 dev br0  router ref 8 used 0/68/0 probes 3 FAILED
fe80::ce8d:a2ff:fe46:f5cc dev br0  ref 2 used 1/435/0 probes 6 FAILED
fe80::cad7:19ff:fe45:66d2 dev br0  router ref 8 used 0/71/0 probes 6 FAILED
fe80::4af8:b3ff:fe9a:4475 dev br1 lladdr 48:f8:b3:9a:44:75 router ref 1 used 0/60/0 probes 0 STALE
fe80::cad7:19ff:fe45:66d2 dev br0 lladdr c8:d7:19:45:66:d2 router ref 9 used 0/60/0 probes 5 DELAY
fe80::ce8d:a2ff:fe46:f5cc dev br0  ref 2 used 1/439/0 probes 6 FAILED
fe80::ce8d:a2ff:fe46:f5cc dev br0  ref 2 used 0/443/0 probes 6 FAILED
fe80::cad7:19ff:fe45:66d2 dev br0  router ref 11 used 0/68/0 probes 3 FAILED
fe80::ce8d:a2ff:fe46:f5cc dev br0  ref 2 used 0/447/0 probes 6 FAILED
fe80::cad7:19ff:fe45:66d2 dev br0  router ref 2 used 0/71/0 probes 6 FAILED
fe80::cad7:19ff:fe45:66d2 dev br0 lladdr c8:d7:19:45:66:d2 router ref 3 used 0/60/0 probes 4 DELAY
fe80::ce8d:a2ff:fe46:f5cc dev br0  ref 2 used 0/451/0 probes 6 FAILED
fe80::ce8d:a2ff:fe46:f5cc dev br0  ref 2 used 0/455/0 probes 6 FAILED
fe80::cad7:19ff:fe45:66d2 dev br0  router ref 5 used 0/68/0 probes 3 FAILED
fe80::ce8d:a2ff:fe46:f5cc dev br0  ref 2 used 0/459/0 probes 6 FAILED
fe80::cad7:19ff:fe45:66d2 dev br0 lladdr c8:d7:19:45:66:d2 router ref 6 used 0/60/0 probes 6 DELAY
fe80::ce8d:a2ff:fe46:f5cc dev br0  ref 2 used 0/463/0 probes 6 FAILED
fe80::ce8d:a2ff:fe46:f5cc dev br0  ref 2 used 0/467/0 probes 6 FAILED
fe80::cad7:19ff:fe45:66d2 dev br0  router ref 8 used 0/68/0 probes 3 FAILED
fe80::ce8d:a2ff:fe46:f5cc dev br0  ref 2 used 0/471/0 probes 6 FAILED
fe80::cad7:19ff:fe45:66d2 dev br0  router ref 8 used 0/71/0 probes 6 FAILED
fe80::cad7:19ff:fe45:66d2 dev br0 lladdr c8:d7:19:45:66:d2 router ref 9 used 0/60/0 probes 4 DELAY
fe80::ce8d:a2ff:fe46:f5cc dev br0  ref 2 used 0/475/0 probes 6 FAILED

i can verify that when there are probe-errors shown in the ip-monitor, a new icmpv6 error is logged.
So i looked closer for a reason, why i have so much probes "FAILED".

Checked my Shelly Smartmeters + Sensors which uses ESP32 devices, and they use "privacy extensions" and continous
check ipv4 and ipv6 network health & gateways! - When the rooter is rebooted, the old temp ip6-data is no longer valid, this is normal and should be stopped after a while - but this is not the case here. After 24h i have thousands of this debug messages still running... all few seconds 5-10 more new debug messages. Compared this with the office-router where i got just 3-5 messages a day, and yes: windows clients & privacy extensions needs way longer to take a new temporary address. So i asume the
old addresses are no longer valid and this way, there is no more a response - and this is debug-logged on default level.

Not an expert here, I´am always open for better explanations!

Just one more note to the icmp-rate limiting for ip6 in general: ip6 makes huge usage of all sorts of icmp types compared with ip4.
So i suggest, when we rate limit v6-icmp, we should consider decent settings which cover traffic of a bigger internal network with a lot of devices and computers.

EDIT: After more checking the API/docs from Shelly Devices i figured out, that ALL devices with the latest firmware have NO real ipv6 routing support! - they communicate only via ip4/DNS to the outside at the moment. The ipv6 local-link and the external ip6 address is got bei NDP self discovery and ONLY used localy via mDNS advertisement - so ip6 can be used for now only "inside". The side effect is now the enormous logspam. A Quickfix for all this LAN connected devices is now to set an ACL in the L2 switch to block The Ehernet Frametype "86DD" for all Ports with Shelly LAN-devices - which will not work for the WIFI connected devices. Disabling ip6 for the entire bridge is not an option, because the Solar-Equipment in the same local network uses real ipv6 and should not be blocked. It´s a shame that ip6 CAN´t be disabled in the device-web interface now. It´s always enabled, so i need to check with manufacturer for a cleaner solution in the next Firmwares :-(

Only other option directly on Tomato is compiling a huge list of all collected devices and create a user firewall script which blocks +100 devices mac by mac and block the ethernet-frame type here. Basicly useless for me now.

[MarsTomato]

Source was RMerl/asuswrt-merlin.ng@efeeab6
moved to FreshTomato with
cee1193
update with
3d17125

Just for info, for local testing i use at the moment this "fix":

do you touch the limit? have you checked? AsusWRT does have 1/s <--> we have at least 5/s

with option "WAN interfaces respond to Ping and Traceroute" checked nvram block_wan will be 0 and ONLY type 128 (echo) will be limited; otherwise no limit (+ALL other types no limit)

[maverick3005]

Yes, i noted it with type = 128 after i removed the forced (unswitchable hardcoded limit).

One "Problem" is still, that it´s not in sync with the WAN-Ping-Option and "Ratelimit" which is working only for ipv4.

OK... maybe not very importent, the real boring issues is the kernel 2.6 debug level in "standard mode", with thousands of
logged icmp6 errors. Default Level for console print is "emergency" - this is a known bug in the old 2.6 kernel, where we can //remark the entire debug message - or rise by kernel-patch the logging level higher, so we can ignore it "as given" - but we no longer flooding terminal, serial port or remote-syslog with them.

Anyway the issues comes from the many Shellys, which all makes NDP, get an external IP6, but never use it for active outside communication - so it´s a massive client problem. Anyway we should mute this error. Users whith just a small Houseinstall and 3 such devices can create an extra Bridge and disable ip6 for the entire bridge - so work only outgoing (with CGNAT....).
When ipv6 in router interface is disabled, NDP no longer works and the Shellys only generate the local-link addresses
for direct communications inside the same bridge/vlan.

[MarsTomato]

One "Problem" is still, that it´s not in sync with the WAN-Ping-Option and "Ratelimit" which is working only for ipv4.

Yes, there is room for improvement and IPv6 (hardcoded limit) - maybe i have some time next week

icmpv6_send: no reply to icmp error

I can see that message too, sometimes - just checked, no entry(s) currently. Also have some shelly stuff/clients in my network. So far i just ignored it ... maybe we can mute it. have to look at it

[maverick3005]

muting would be nice, because in final stage more then +130 Shelly/IoT-devices must be handled here combined with LAN&WLAN, and that becomes really huge with this type of power logging ;-)

[maverick3005]

@MarsTomato

Is this ratelimting ONLY from WAN -> LAN ? - What is with the response from internal requests to get from outside fast enough
all answsers "undelayed"...

I asked because, all these many IoI devices makes icmp-measurements to a.) check continous the default-gw for v4 and v6 and b.) when cloud services are used, they trace a pre compiled list, maybe by using dns, for the nearest cloud, and this is more or less an action which accours at the same time. I like to make sure, when many internal devices with this 2 two checks, that there is no rate limiting for v4 and v6 active, or maybe with a much higher rate. When some of these tests are negative, the devices exclude
with "not reachable" some clouds for a while.... and try again minutes later.

With regulary 2-3 devices and the standard Household Situation with some TVs, Android Devices and a few Computers, this is never a problem. In my case i have to manage 20+ Appments centralized - with 3+ ESP32 in each Household, mixed up with Solakon/FoxESS-Solar & Growatt Equipment, a huge amount of different Shelly-Equipment, Shelly Smoke Detectors, Shellys Relasis and further more are a lot of more temperature/WET-Detect Sensors, all wlan based comes to the bigger Area near End of 2026.