Skip to content



The Egress IP feature enables a cluster administrator to ensure that the traffic from one or more pods in one or more namespaces has a consistent source IP address for services outside the cluster network.
East-West traffic (including pod -> node IP) is excluded from Egress IP.

For more info, consider looking at the following links: - Egress IP CRD - Assigning an egress IP address - Managing Egress IP in OpenShift 4 with OVN-Kubernetes


An example of EgressIP might look like this:

kind: EgressIP
  name: egressip-prod
      - key: environment
        operator: NotIn
          - development
      app: web
It specifies to use or egressIP for pods that are labeled with app: web that run in a namespace without environment: development label. Both selectors use the generic kubernetes label selectors.

Traffic flows

If the Egress IP(s) are hosted on the OVN primary network then the implementation is redirecting the POD traffic to an egress node where it is SNATed and sent out.

Using the example EgressIP and a matching pod with IP, the following logical router policies are configured in ovn_cluster_router:

Routing Policies
  1004 inport == "rtos-ovn-control-plane" && ip4.dst == /* ovn-control-plane */                            reroute
  1004 inport == "rtos-ovn-worker" && ip4.dst == /* ovn-worker */                                          reroute
  1004 inport == "rtos-ovn-worker2" && ip4.dst == /* ovn-worker2 */                                        reroute

   102 (ip4.src == $a12749576804119081385 || ip4.src == $a16335301576733828072) && ip4.dst == $a11079093880111560446  allow    pkt_mark=1008
   102 ip4.src == && ip4.dst ==                                                           allow
   102 ip4.src == && ip4.dst ==                                                           allow

   100 ip4.src ==                                                                                          reroute,
- Rules with 1004 priority are responsible for redirecting pod -> local host IP traffic.
- Rules with 102 priority are added by OVN-Kubernetes when EgressIP feature is enabled, they ensure that east-west traffic is not using egress IPs. - The rule with 100 priority is added for the pod matching egressip-prod EgressIP, and it redirects the traffic to one of the egress nodes (ECMP is used to balance the traffic between next hops).

Once the redirected traffic reaches one of the egress nodes it gets SNATed in the gateway router:

ovn-nbctl lr-nat-list GR_ovn-worker
ovn-nbctl lr-nat-list  GR_ovn-worker2

Lets now imagine the Egress IP(s) mentioned previously, are not hosted by the OVN primary network and is hosted by a secondary host network which is assigned to a standard linux interface, a redirect to the egress-able node management port IP address:

Routing Policies
  1004 inport == "rtos-ovn-control-plane" && ip4.dst == /* ovn-control-plane */                            reroute
  1004 inport == "rtos-ovn-worker" && ip4.dst == /* ovn-worker */                                          reroute
  1004 inport == "rtos-ovn-worker2" && ip4.dst == /* ovn-worker2 */                                        reroute

   102 (ip4.src == $a12749576804119081385 || ip4.src == $a16335301576733828072) && ip4.dst == $a11079093880111560446  allow    pkt_mark=1008
   102 ip4.src == && ip4.dst ==                                                           allow
   102 ip4.src == && ip4.dst ==                                                           allow

   100 ip4.src ==                                                                                          reroute,

IPTables will have the following chain in NAT table and also rules within that chain to source NAT to the correct IP address:

sh-5.2# iptables-save 
# Generated by iptables-save v1.8.7 on Tue Jul 25 13:09:39 2023
:PREROUTING ACCEPT [14087:9430205]
:INPUT ACCEPT [13923:9397241]
:OUTPUT ACCEPT [11270:1030982]
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A OVN-KUBE-EGRESS-IP-MULTI-NIC -s -o dummy -j SNAT --to-source
-A OVN-KUBE-EGRESS-SVC -m mark --mark 0x3f0 -m comment --comment DoNotSNAT -j RETURN
-A OVN-KUBE-SNAT-MGMTPORT -o ovn-k8s-mp0 -m comment --comment "OVN SNAT to Management Port" -j SNAT --to-source

IPRoute2 rules will look like the following - note rule with priority 6000 and also the table 1111:

sh-5.2# ip rule
0:  from all lookup local
30: from all fwmark 0x1745ec lookup 7
6000:   from lookup 1111
32766:  from all lookup main
32767:  from all lookup default

And the default route in the correct table 1111:

sh-5.2# ip route show table 1111
default dev dummy

No NAT is required on the OVN primary network gateway router. OVN-Kubernetes (ovnkube-node) takes care of adding a rule to the rule table with src IP of the pod and routed towards a new routing table specifically created to route the traffic out the correct interface. IPTables rules are also altered and an entry is created within the chain OVN-KUBE-EGRESS-IP-Multi-NIC for each selected pod to allow SNAT to occur when egress-ing a particular interface. The routing table number 1111 is generated from the interface name. Routes within the main routing table who's output interface share the same interface used for Egress IP are also cloned into the VRF 1111.

Pod to node IP traffic

When a cluster networked pod matched by an egress IP tries to connect to a non-local node IP it hits the following logical router policy in ovn_cluster_router:

# $<all_eip_pod_ips> - address-set of all pod IPs matched by any EgressIP
# $<all_esvc_pod_ips> - address-set of all pod IPs matched by any EgressService
# $<all_node_ips> - address-set of all node IPs in the cluster
102 (ip4.src == $<all_eip_pod_ips> || ip4.src == $<all_esvc_pod_ips>) && ip4.dst == $<all_node_ips>  allow    pkt_mark=1008
In addition to simply allowing the pod -> node IP traffic so that EgressIP reroute policies are not matched upon, it is also marked with the 1008 mark.
If a pod is hosted on an egressNode the traffic will first get SNATed to the egress IP, and then it will hit following flow on breth0 that will SNAT the traffic to local node IP:
# output truncated, 0x3f0 == 1008
priority=105,pkt_mark=0x3f0,ip,in_port=2 actions=ct(commit,zone=64000,nat(src=<NodeIP>),exec(load:0x1->NXM_NX_CT_MARK[])),output:1
This is required to make pod -> node IP traffic behave the same regardless of where the pod is hosted.
Implementation details:

For local gateway mode, in which an Egress IP is assigned to a non-primary interface, an IP rule is added to send packets to the main routing table at a priority higher than that of EgressIP IP rules, which are set to priority 6000:

5999:   from all fwmark 0x3f0 lookup main
Note: 0x3f0 is 1008 in hexadecimal. Lower IP rule priority number indicates higher precedence versus higher IP rule priority number.

This ensures all traffic to node IPs will not be selected by EgressIP IP rules. However, reply traffic will not have the mark 1008 and would be dropped by reverse path filtering, therefore we add an IPTable rule to the mangle table to save and restore the 1008 mark:

sh-5.2# iptables -t mangle -L  PREROUTING
target     prot opt source               destination
CONNMARK   all  --  anywhere             anywhere             mark match 0x3f0 CONNMARK save
CONNMARK   all  --  anywhere             anywhere             mark match 0x0 CONNMARK restore

Dealing with non SNATed traffic

Egress IP is often configured on a node different from the one hosting the affected pods.
Due to the fact that ovn-controllers on different nodes apply the changes independently, there is a chance that some pod traffic will reach the egress node before it configures the SNAT rules. The following flows are added on breth0 to address this scenario:

# Commit connections from local pods so they are not affected by the drop rule below, this is required for ICNIv2
priority=109,ip,in_port=2,nw_src=<nodeSubnet> actions=ct(commit,zone=64000,exec(set_field:0x1->ct_mark)),output:1

# Drop non SNATed egress traffic coming from non-local pods
priority=104,ip,in_port=2,nw_src=<clusterSubnet> actions=drop

# Commit connections coming from IPs not in cluster network
priority=100,ip,in_port=2 actions=ct(commit,zone=64000,exec(set_field:0x1->ct_mark)),output:1

Special considerations for Egress IPs hosted by standard linux interfaces

If you wish to assign an Egress IP to a standard linux interface (non OVS type), then the following is required: * Link is up * IP address must have scope universe / global * Links and their addresses must not be removed during runtime after an egress IP is assigned to it. If you wish to remove the link, first remove the Egress IP and then remove the address / link. * IP forwarding must be enabled for the link

Egress Nodes

In order to select which node(s) may be used as egress, the following label must be added to the node resource:

kubectl label nodes <node_name>""

Egress IP reachability

Once a node has been labeled with, the EgressIP operator in the leader ovnkube-master pod will periodically check if that node is usable. EgressIPs assigned to a node that is no longer reachable will get revalidated and moved to another useable node.

Egress nodes normally have multiple IP addresses. For sake of Egress IP reachability, the management (aka internal SDN) addresses of the node are the ones used. In deployments of ovn-kubernetes this is known to be the ovn-k8s-mp0 interface of a node.

Even though the periodic checking of egress nodes is hard coded to trigger every 5 seconds, there are attributes that the user can set:

  • egressIPTotalTimeout
  • gRPC vs. DISCARD port


This attribute specifies the maximum amount of time, in seconds, that the egressIP operator will wait until it declares the node unreachable. The default value is 1 second.

This value can be set in the following ways: - ovnkube binary flag: --egressip-reachability-total-timeout=<TIMEOUT> - inside config specified by --config-file flag:


Note: Using value 0 will skip reachability. Use this to assume that egress nodes are available.

gRPC vs. DISCARD port

Up until recently, the only method available for determining if an egress node was reachable relied on the TCP port unreachable icmp response from the probed node. The TCP port 9 (aka DISCARD) is the port used for that.

Later implementation of ovn-kubernetes is capable of leveraging secure gRPC sessions in order to probe nodes. That requires the ovnkube node pods to be listening on a pre-specified TCP port, in addition to configuring the ovnkube master pod(s).

This value can be set in the following ways: - ovnkube binary flag: --egressip-node-healthcheck-port=<TCP_PORT> - inside config specified by --config-file flag:


Note: If not specifying a value, or using 0 as the egressip-node-healthcheck-port will make Egress IP reachability probe the egress nodes using the DISCARD port method. Unlike egressip-reachability-total-timeout, it is important that both node and master pods of ovnkube get configured with the same value!

Additional details on the implementation of the gRPC probing: