NSX Advanced load balancer (AVI) is a 100% software-defined multi-cloud application services platform with Software Load Balancers, Intelligent WAF (iWAF), and Container Ingress.
All modern applications use load balancing nowadays. Load balancing is the process of distributing network traffic across multiple servers.
The goal of this blog
The goal of this blog is to configure L3 Direct Server Return (DSR) on Avi. I will be using IP-in-IP encapsulation. Avi can also do L2 DSR. See Direct Server Return on NSX Advanced Load Balancer for more information.
DSR is often applicable to audio and video applications as these applications are very sensitive to latency.
In my example, I am using TCP. You can also use UDP if you want.
I am not going over the Avi controller, SE, and cloud setup.
What I am using
For my setup, I am using the following.
AVI 22.1.4, using vCenter cloud
Ubuntu 22.04.1 LTS
Nsx is not needed for DSR to work. I am just using NSX in my lab.
If you do use NSX, URPF mode needs to be on ‘none’.
URPF stands for Unicast Reverse Path Forwarding.
When URPF is enabled, the T1 only forwards packets if they are received on the same interface that would be used to forward the traffic to the source of the packet. If the route to the source address of the packet is through a different interface than the one it is received on, the packet is dropped.
This is what my network setup looks like. For this demonstration, both segments are connected to the same T1.
How does it work
In the diagram, the VIP (192.168.35.241) on the ServiceEngines, is also configured on the loopback interface of the servers. This is needed for the server to be able to use that address as the source address.
Step 1: The client connects to http://192.168.35.241, which is the VIP address.
Step 2: The ServiceEngines encapsulate the packet (source:192.168.11.2, dest:192.168.35.241) in IP-in-IP packet (source:192.168.33.7 dest:192.168.33.10)
Looking at the packet capture
The packet comes from 192.168.33.7 (SE1) sent to 192.168.33.10 (web01) with IP-in-IP from 192.168.11.2 (client) to 192.168.35.241 (web01 Loopback)
08:32:28.557485 IP 192.168.33.7 > 192.168.33.10: IP 192.168.11.2.50668 > 192.168.35.241.80: Flags [P.], seq 1:392, ack 1, win 8212, length 391: HTTP: GET / HTTP/1.1
Step 3: The server sent back the packet directly to the client (source:192.168.35.241 dest:192.168.11.2)
Looking at the packet capture
The packet returns from 192.168.35.241 (web01 Loopback) directly to 192.168.11.2 (client)
08:32:28.557931 IP 192.168.35.241.80 > 192.168.11.2.50668: Flags [P.], seq 1:655, ack 392, win 501, length 654: HTTP: HTTP/1.1 200 OK
How do we configure DSR on Avi and the Servers?
Configure the servers
First, we need to take care of some sysctl settings. Keep in mind that interface names might be different in your case. I have also added IPV6. Keep in mind that L3 DSR ipv6 is not (yet) supported.
Add the following lines to /etc/sysctl.conf net.ipv4.ip_forward=1 net.ipv4.conf.all.rp_filter=0 For IPV6 add net.ipv6.conf.all.forwarding=1
Edit your Netplan configuration. Configuration names and interfaces might be different in your case.
vi /etc/netplan/01-netcfg.yaml network: ethernets: ens160: dhcp4: false addresses: - 192.168.33.10/24 - fc00::10/64 routes: - to: default via: 192.168.33.1 - to: ::/0 via: fc00::1 nameservers: search: [lab.fqdn.nl] addresses: [192.168.11.2] lo: dhcp4: false addresses: - 192.168.35.241/32 - fc00:3::24a/128 tunnels: tunl0: mode: ipip local: 192.168.33.10 remote: 0.0.0.0 addresses: - 192.168.35.241/32 ip6tnl0: mode: ip6ip6 local: fc00::10 remote: ::0 addresses: - fc00:3::24a/128 version: 2
It’s time to configure Avi.
1. Create a network profile
Goto Templates->Profiles->TCP/UDP and press Create
I am making a TCP profile. Select TCP Fast Path under Type.
Select Enable DSR under Direct Server Return. Type L3, encapsulation ipinip
2. Create (or edit) a virtual service
Goto Applications->Virtual Services and create or edit a virtual service.
Under profiles, select Application profile System-L4-Application and the created Network Profile.
Under pool, create or select a pool.
3. Create a pool
As a server IP address, use the IP address of the server itself. Not the VIP.
You can also configure a health monitor if that is needed.
Goto Applications->Dashboard and check the status of your virtual service.
DSR can be very useful in some use cases. I.e audio and video. It can also be used with http (as I did in my example) or DNS.
If you have any questions or suggestions, please leave a comment.