OVN `ipvX.address.external` partially breaks inbound TCP/UDP traffic

[0xk1f0]

Is there an existing issue for this?

• There is no existing issue for this bug

Is this happening on an up to date version of Incus?

• This is happening on a supported version of Incus

Incus system details

config:
  cluster.https_address: 10.196.163.11
  core.https_address: 10.196.163.11
  network.ovn.ca_cert: |-
    -----BEGIN CERTIFICATE-----
    REDACTED
    -----END CERTIFICATE-----
  network.ovn.client_cert: |-
    -----BEGIN CERTIFICATE-----
    REDACTED
    -----END CERTIFICATE-----
  network.ovn.client_key: |-
    -----BEGIN RSA PRIVATE KEY-----
    REDACTED
    -----END RSA PRIVATE KEY-----
  network.ovn.northbound_connection: ssl:[10.196.163.11]:6641,ssl:[10.196.163.12]:6641,ssl:[10.196.163.13]:6641
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- network_sriov
- console
- restrict_dev_incus
- migration_pre_copy
- infiniband
- dev_incus_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- dev_incus_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- backup_compression
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- images_all_projects
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
- network_txqueuelen
- cluster_member_state
- instances_placement_scriptlet
- storage_pool_source_wipe
- zfs_block_mode
- instance_generation_id
- disk_io_cache
- amd_sev
- storage_pool_loop_resize
- migration_vm_live
- ovn_nic_nesting
- oidc
- network_ovn_l3only
- ovn_nic_acceleration_vdpa
- cluster_healing
- instances_state_total
- auth_user
- security_csm
- instances_rebuild
- numa_cpu_placement
- custom_volume_iso
- network_allocations
- zfs_delegate
- storage_api_remote_volume_snapshot_copy
- operations_get_query_all_projects
- metadata_configuration
- syslog_socket
- event_lifecycle_name_and_project
- instances_nic_limits_priority
- disk_initial_volume_configuration
- operation_wait
- image_restriction_privileged
- cluster_internal_custom_volume_copy
- disk_io_bus
- storage_cephfs_create_missing
- instance_move_config
- ovn_ssl_config
- certificate_description
- disk_io_bus_virtio_blk
- loki_config_instance
- instance_create_start
- clustering_evacuation_stop_options
- boot_host_shutdown_action
- agent_config_drive
- network_state_ovn_lr
- image_template_permissions
- storage_bucket_backup
- storage_lvm_cluster
- shared_custom_block_volumes
- auth_tls_jwt
- oidc_claim
- device_usb_serial
- numa_cpu_balanced
- image_restriction_nesting
- network_integrations
- instance_memory_swap_bytes
- network_bridge_external_create
- network_zones_all_projects
- storage_zfs_vdev
- container_migration_stateful
- profiles_all_projects
- instances_scriptlet_get_instances
- instances_scriptlet_get_cluster_members
- instances_scriptlet_get_project
- network_acl_stateless
- instance_state_started_at
- networks_all_projects
- network_acls_all_projects
- storage_buckets_all_projects
- resources_load
- instance_access
- project_access
- projects_force_delete
- resources_cpu_flags
- disk_io_bus_cache_filesystem
- instance_oci
- clustering_groups_config
- instances_lxcfs_per_instance
- clustering_groups_vm_cpu_definition
- disk_volume_subpath
- projects_limits_disk_pool
- network_ovn_isolated
- qemu_raw_qmp
- network_load_balancer_health_check
- oidc_scopes
- network_integrations_peer_name
- qemu_scriptlet
- instance_auto_restart
- storage_lvm_metadatasize
- ovn_nic_promiscuous
- ovn_nic_ip_address_none
- instances_state_os_info
- network_load_balancer_state
- instance_nic_macvlan_mode
- storage_lvm_cluster_create
- network_ovn_external_interfaces
- instances_scriptlet_get_instances_count
- cluster_rebalance
- custom_volume_refresh_exclude_older_snapshots
- storage_initial_owner
- storage_live_migration
- instance_console_screenshot
- image_import_alias
- authorization_scriptlet
- console_force
- network_ovn_state_addresses
- network_bridge_acl_devices
- instance_debug_memory
- init_preseed_storage_volumes
- init_preseed_profile_project
- instance_nic_routed_host_address
- instance_smbios11
- api_filtering_extended
- acme_dns01
- security_iommu
- network_ipv4_dhcp_routes
- network_state_ovn_ls
- network_dns_nameservers
- acme_http01_port
- network_ovn_ipv4_dhcp_expiry
- instance_state_cpu_time
- network_io_bus
- disk_io_bus_usb
- storage_driver_linstor
- instance_oci_entrypoint
- network_address_set
- server_logging
- network_forward_snat
- memory_hotplug
- instance_nic_routed_host_tables
- instance_publish_split
- init_preseed_certificates
- custom_volume_sftp
- network_ovn_external_nic_address
- network_physical_gateway_hwaddr
- backup_s3_upload
- snapshot_manual_expiry
- resources_cpu_address_sizes
- disk_attached
- limits_memory_hotplug
- disk_wwn
- server_logging_webhook
- storage_driver_truenas
- container_disk_tmpfs
- instance_limits_oom
- backup_override_config
- network_ovn_tunnels
- init_preseed_cluster_groups
- usb_attached
- backup_iso
- instance_systemd_credentials
- cluster_group_usedby
- bpf_token_delegation
- file_storage_volume
- network_hwaddr_pattern
- storage_volume_full
- storage_bucket_full
- device_pci_firmware
- resources_serial
- ovn_nic_limits
- storage_lvmcluster_qcow2
- oidc_allowed_subnets
- file_delete_force
- nic_sriov_select_ext
- network_zones_dns_contact
- nic_attached_connected
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
auth_user_name: root
auth_user_method: unix
environment:
  addresses:
  - 10.196.163.11:8443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    REDACTED
    -----END CERTIFICATE-----
  certificate_fingerprint: REDACTED
  driver: qemu | lxc
  driver_version: 10.2.0 | 6.0.5
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    uevent_injection: "true"
    unpriv_binfmt: "true"
    unpriv_fscaps: "true"
  kernel_version: 6.12.69+deb13-amd64
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Debian GNU/Linux
  os_version: "13"
  project: default
  server: incus
  server_clustered: true
  server_event_mode: full-mesh
  server_name: 2282152-n001
  server_pid: 1784
  server_version: "6.21"
  storage: dir | ceph | lvm
  storage_version: 1 | 18.2.7 | 2.03.31(2) (2025-02-27) / 1.02.205 (2025-02-27) /
    4.48.0
  storage_supported_drivers:
  - name: btrfs
    version: "6.14"
    remote: false
  - name: ceph
    version: 18.2.7
    remote: true
  - name: cephfs
    version: 18.2.7
    remote: true
  - name: cephobject
    version: 18.2.7
    remote: true
  - name: lvm
    version: 2.03.31(2) (2025-02-27) / 1.02.205 (2025-02-27) / 4.48.0
    remote: false
  - name: dir
    version: "1"
    remote: false
  - name: truenas
    version: 0.7.3
    remote: true

Instance details

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Debian trixie amd64 (20260209_06:20)
  image.os: Debian
  image.release: trixie
  image.serial: "20260209_06:20"
  image.type: disk-kvm.img
  image.variant: default
  limits.cpu: "1"
  limits.memory: 1024MiB
  migration.stateful: "True"
  security.guestapi: "False"
  security.secureboot: "False"
  volatile.base_image: 7b9a082ad2aaea28d92b4113c650b5370755bf20c1aa61bdc7a7bf233e59ea4a
  volatile.cloud-init.instance-id: f25201fa-6645-4be3-910e-1c775b04b756
  volatile.eth0.host_name: tap69136d75
  volatile.eth0.hwaddr: 10:66:6a:4e:a9:34
  volatile.eth0.last_state.ip_addresses: 10.136.245.2,fd1e:6977:36c9::2
  volatile.last_state.power: STOPPED
  volatile.uuid: cbb4cec2-c345-4703-af60-b3aea40b6086
  volatile.uuid.generation: cbb4cec2-c345-4703-af60-b3aea40b6086
  volatile.vm.definition: pc-q35-10.1
  volatile.vm.rtc_adjustment: "-3"
  volatile.vm.rtc_offset: "0"
  volatile.vsock_id: "3286343076"
devices:
  eth0:
    ipv4.address: 10.136.245.2
    ipv4.address.external: <REDACTED_PUBLIC_IP>
    ipv6.address: fd1e:6977:36c9::2
    limits.egress: 100Mbit
    limits.ingress: 100Mbit
    network: 019c1def-6eb8-7ff3-ab73-23bd2124389b
    type: nic
  root:
    path: /
    pool: ceph-default
    size: 10000MB
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

Instance log

No response

Current behavior

Currently, the feature ipvX.address.external allows the user to bind an address from UPLINK network to a corresponding instance, which will set a corresponding snat rule under the hood in OVN. This should set the ip address seen by outter peers to the new address.
To allow inbound connections via this IP, a corresponding network forward must be present, so inbound traffic can find its way back to the correct instance.

---

When doing this as described in the documentation, inbound connectivity works when connections are being initiated from outter peers, f.E. a user visiting a website hosted on the instance via the public IP. But when trying to establish connections over TCP or UDP, f.E. DNS requests, timeouts occur, since packets are not able to find their way back to the instance initiating them. Pinging a host works thugh, so this seems to be unrelated to ICMP-based traffic.

Expected behavior

The expected behaviour would be working inbound traffic, when connections are established from the instance itself, as long as a corresponding forward is present (which is a requirement currently). Since the issue seems to be limited to TCP/UDP traffic, this could be some kind of NAT-Table mismatch, in the case that the newly added snat rule is being created with a different NAT table, which could cause the instance to confuse incoming connections, causing packets to be dropped.

---

For my specific use-case, what I want to achieve is essentially a "fully bound" public IP to an instance. What I further experimented is just setting a dnat_and_snat rule for this use-case in OVN manually.

ovn-nbctl lr-nat-add <ROUTER_ID> dnat_and_snat <example_public_ip> <example_private_ip>

This does exactly what I want and does not have the requirement of needing to manually set a forward for the corresponding address. I do however think that this functionality should be a seperate feature, since it kind of defeats the naming and purpose of address.external, which as the naming suggests should only modify the outbound address. It could be realized via a new config option such as, ipv4.address.bind or likewise. I would be happy to contribute someting like that.

Steps to reproduce

1. Create a network forward with listen_address as <example_public_ip> and config.target_address as <example_private_ip>
2. Add ipv4.address.external with <example_public_ip> on the instance with the corresponding <example_private_ip>
3. Try to ping a public ip address, it should work as expected, f.E. ping 1.1.1.1
4. Try to do a dns resolution, f.E. nslookup google.com, it will timeout and fail

[0xk1f0]

CC: @DarkressX

[DarkressX]

Another finding is, that if the forward to the private ip is an actual port-forward, the instance will be able to establish connections with the assigned external ip.
If the forward however is a 1:1 forward (via "config": {"target_address": "x.x.x.x" }), then connections cannot be established.

[0xk1f0]

Chiming in again here to ask if a 1:1 NAT feature using dnat_and_snat as proposed above would be of interest and accepted into upstream? If yes, should I create a seperate feature issue for this functionality or file a PR directly, as I would start implementing it on the basis of the ipvX.address.external implementation.

[stgraber]

Did you try setting snat=true on the port definition of your forward?
You'll most likely need to setup two full port ranges, one for TCP and one for UDP but that would let you get the dnat_and_snat.

[stgraber]

Btw, just worth noting that for a fully bound IP address, our recommendation is to have the IP itself routed to the instance directly, avoiding all the nat/snat flows entirely. So effectively setting ipv4.routes.external=IP/32 in the instance's NIC definition, then adding the IP address to the instance's network configuration.

The only downside of this approach is that it requires configuration inside of the instance, but when possible, it's definitely the way to go.

[0xk1f0]

Thanks for the suggestions!

Did you try setting snat=true on the port definition of your forward?

As per documentation:
"The snat property is currently only supported on managed bridge networks and with the nftables firewall driver."
So for OVN it doesn't seem like we can utilize this feature.

You'll most likely need to setup two full port ranges, one for TCP and one for UDP but that would let you get the dnat_and_snat.

In my testing, specifying full port ranges, which creates two LB's with a lot of port definitions, put a lot of stress on OVN, which with a couple of instances in place will not really scale that well.

Btw, just worth noting that for a fully bound IP address, our recommendation is to have the IP itself routed to the instance directly, avoiding all the nat/snat flows entirely. So effectively setting ipv4.routes.external=IP/32 in the instance's NIC definition, then adding the IP address to the instance's network configuration.

I agree that this is certainly an option for use-cases when configuration of the IP is possible inside the instance. However for situations where that is not the case, we don't really have an option right now that (in my opinion) solves this in a way a dnat_and_snat rule would.

That's just my two cents on this.

[stgraber]

So for OVN it doesn't seem like we can utilize this feature.

Oops, I thought I had implemented it for OVN, but actually looking back at my notes, there was an issue (missing feature) on the OVN case which got in the way at the time.

[stgraber]

What version of OVS and OVN are you using in your environment?
I haven't yet attempted to reproduce the issue here, but figured I'd start from the basics as OVN definitely has seen some activity in that area recently.

[0xk1f0]

What version of OVS and OVN are you using in your environment?

OVN:

~# ovn-appctl -t ovn-controller version
ovn-controller 25.09.0
Open vSwitch Library 3.6.0

OVS:

~# ovs-vsctl --version
ovs-vsctl (Open vSwitch) 3.6.0
DB Schema 8.8.0