Network Connectivity
There are very few scenarios where NetScaler requires L2 connectivity to any backend. Mostly involving DSR setups (Direct Server Return) where we rarely do it from practical reasons. There are such scenarios like Load Balancing large amount of traffic as a backup and we wish to save bandwidth (maybe hitting a licence limit), but that's the exception, not the rule. Most of the time the backend needs the SRC IP, but it's not foreseen by the vendor to be placed behind a reverse proxy (which is simply ridiculous in 2023), because of this you cannot insert a header and to be done with it.
DSR Setup explained here.
The majority of the setups where the NetScaler has direct access to a network where backend servers live are not DSR setups. They're out like this because of negligence or because the temp setup went into prod. What's the big deal you would ask? Maybe that's a dedicated network for specific service Backends with nothing else in it. The short answer - it leaves a huge security loophole in your network access and we're going to explain why in this article.
I would not leave a NetScaler appliance without an IDS\IPS Firewall on the front door and I would always recommend backend connectivity to traverse through an internal firewall instance.
Floating IPs
It all has to do with the concept of floating IPs. Floating IP are generally the Virtual IPs that you'll configure on your Load Balancer and for a long time I thought we all call them like this because they are active only on the Primary Node (althoug same is valid for SNIP addresses, my own understanding was lacking logic). I could not have been further from the truth.
Any other network device requires several basic things when you configure an IP.
- IP
- Network (subnet)
- Interface (vlan)
When you configure these 3 things your device (Server, router) is placed in that specific network. A device can have many IPs from same Subnet, but can also have different IPs from different subnets. Different IPs will have different functions on a host (e.g. internal console, public services, iSCSI). When we assign these properties we're able to identify that IP in specific network segment. You can access these services only if you your packet (connection) arrives on the correct interface.
OS Interface and vlan are interchangeable practically as vlan is actually sub-interface with a tag.
Below image shows and example of what we discussed:
NetScaler's floating IPs does not listen on a specific interface. Virtual IPs listen on all interfaces. That's why if you have 3 interfaces and you add a Virtual IP it does not require interface assignment to work.
In our host-service representation we have practically the following concept:
So far so good. Let's examine a complete communication flow:
- The SSH Service request will never reach the iSCSI Interface as the firewall/router will not forward the packet.
- Even if the first packet is delivered (incorrectly) the return traffic depends on Host Routing table. If exits the same iSCSI Interface (VLAN 60) the firewall will block it as IP Spoofing. If exits the SSH interface (VLAN 40) you have asymmetric routing with some chance of success.
This "feature" does not have any practical implication, because it does not line up with how the rest of your network works. If anything creates a peculiar security issue when placed in L2 context!
L2 Context
Let's examine the communication flow when the client and the server live in the same Subnet. The client will no longer be in VLAN 60, but placed in VLAN 70, same as the iSCSI interface.
This time there will be no firewall to enforce proper forwarding of the first packet, because it will happen based on MAC address with incorrect destination IP (in VLAN 40 - 10.10.40.5 - SSH Service). The return traffic will take the Directly accessible interface in VLAN 70. Voila the communication is possible!
NetScaler Context
So far our example was with a host having just two services. Let's palce our NetScaler into context.
- Your organization is hosting mutiple services on different IPs pointed to different Groups of Backends.
- Some backends are in L2 with the NetScaler, some has to traverse via Firewall as they're in different subnets.
- Access to Virtual IPs and between backends (east-west) traffic is controlled via Firewall and allow wherever necessary.
If a service hosted in same L2 network as the NetScaler gets exploited by a threat actor, that service will gain access to all Virtual IPs hosted on the NetScaler. Practically gaining access to all services hosted on the NetScaler, using it as a Next Hop and bypassing your firewall.
The Floating IP behaviour behaviour is called routing. Disabling the the L3 Mode on the NetScaler does not affect this!
Exactly for this reason, network security guys do not like machines having IPs in more than one Subnet. If configured incorrectly devices could start routing, effectively bypassing firewall restrictions.
To illustrate the malicious communication:
- VIP #1 forwards traffic to Server C
- VIP #2 forwards traffic to Server Group A
- Server C is exploited
- Server C sends packet towards VIP #2 with Next Hop Netscaler's SNIP address
- NetScaler allows the access because of the Floating IP concept
Preventive options
- VIP's return traffic towards the compromised system relies on the DIRECT L2 Route in your routing table. If you have configured PBRs for the VIP ranges the NetScaler will not honor the DIRECT route, hence traversing via the Firewall.
- Configure Extended ACLs
Both options not ideal. What we need is the ability to disable this floating IP behaviour.
Thank you for reading!