Note

LVS Variants: LVS was originally based on the masquerading code of Linux-2.0. The director had the VIP, to let the router know where to send packets for the virtual server. There was no port for the service listening on the director's VIP, so ip_vs() forced acceptance of the packet by hooking into LOCAL_IN. The design of ip_vs() of VIP hook in LOCAL_IN are historical rather than a technical restriction. If other clues can be given to the router, to route packets, then the VIP is not needed on the director. Current thinking is that ip_vs() should be moved to the FORWARD chain to allow the director to function more like a router. This (and the following) section(s) show attempts at alternate designs for LVS, e.g. to move ip_vs() out of LOCAL_IN.

	Note
	This is NOT LocalNode. It's experimental code from Horms to allow running realservers on the director. This allows you to test LVS on one box.

Viktors Rotanovs 1 Sep 2006

Is it possible to do port redirection using iptables _after_ director on localnode? I've changed NF_IP_LOCAL_IN to NF_IP_PRE_ROUTING at ip_vs_in_ops in ip_vs_core.c, and now it bypasses NAT, but I'm not a kernel hacker and I don't know which priority should be set and if it's possible to solve the problem that way.

Siim Poder windo (at) p6drad-teel (dot) net 01 Sep 2006

If the LVS grabs a packet, you cant do any NAT on it any more. The packet is as good as lost for those purposes (currently it seems so, at least).

However, LVS does it's own NAT, is there a reason why you have to first let LVS do its own nat and then have iptables nat again? couldn't you just have the right LVS real servers (with right ports) in the first place (using fwmarks, if that source address is important).

Joe

Although LVS has always hooked into LOCAL_IN, it could hook in anywhere and perhaps it would be good to write this into the ip_vs code. Both Horms and Ludo have fiddled around here with no apparent ill-effect.

Horms 2 Sep 2006

Here is my take on this problem Local Nodes, msg00102 Local Nodes, msg00113. Here's the local_nodes.patch. If people tested it and gave feedback we could merge it into the kernel :)

Dave Whitla 20 Jun 2005

I am trying to load balance to two "real" servers which are actually listening on virtual IPs on the load-balancing host. Why would I want to do this? To build a test environment for a web application which usually runs on an IPVS cluster. The purpose of the test environment is to test for databasecache contention issues before we deploy to our production cluster. The catch is I must make the test environment (lvs director + 2 x application server instances) run on one physical host (each developer's development machine).

The man page for ipvsadm makes specific mention of forwarding to realservers which are in fact running on local interfaces stating that the load balancing "forward" mechanism specified for a virtual service is completely ignored if the kernel detects that a real server's IP is actually a local interface. The "Local Node" page describes a configuration in which I could load balance between a "real" server running on the director's virtual service IP and a real server running on another host. This does not solve my problem however as I must bind each instance of my application to a different IP address on the same physical box.

You may be thinking "Why not run the two instances on different ports on the same IP (the virtual service IP)?". Sadly the application is not a simple web-site, and source code and deployment container dependencies on certain port numbers exist. eg RMI-IIOP listeners.

Does anyone know of some config or kernel hack, patch or whatever which might make my ipvs present forwarded packets to the virtual interfaces as though they had appeared on the wire so that my forward directives are not ignored and the packets are not simply presented to the TCP stack for the virtual service IP? I guess this is like NAT to local destination addresses (as opposed to NAT of locally originated connections which is supported in the kernel).

Horms

this is a pretty interesting problem that crops up all the time. I have often wondered how hard it would be to make nat work locally (not that LVS-Tun and LVS-DR don't/can't support portmaping anyway). The patch linked above is for 2.6.12 that allows nat to work locally by:

Please note that this completely breaks NAT for non-Local hosts. It could be cleaned up and made so it doesn't do that. But I thought I'd throw it onto the list before putting any more time into it.

Horms 21 Jun 2005

The patch is my second attempt. This should automatically switch local real-servers to Local unless the requested forwarding method is Masq and the real port differs from the virtual port. That is, if you want to do portmaping on a local service it will use Masq, otherwise it will use Local. It seems to work, but there are probably a few gotchas in there and I haven't tested a whole lot.