13. LVS: Non Linux Realservers

This list of realservers is from Ratz (Joe: from about 2003). About the only thing he hasn't tried yet is Plan 9. Remember to set netmask=/32 for the VIP on LVS-DR and LVS-Tun (for LVS-NAT you can setup with any netmask you like). If you are running non-Linux unix realservers, you can usually handle the arp problem by configuring the device carrying the VIP with the -arp switch.

Ratz's code to setup non-linux realservers is now in the configure script (http://www.austintek.com/LVS/LVS-HOWTO/mini-HOWTO/LVS-mini-HOWTO.html#configure_script) This part of the script has not been well tested (you might find that it doesn't setup your non-linux unix box properly yet, please contact me - Joe).

Note
(In the 3yrs the configure script has been out, I've not heard of anyone using this part of the code, so there seems no point in maintaining it.)

Here's the original info from Ratz for realservers with non-Linux OS's. On some Unixes you have to plumb the interface before assigning an IP. The plumb instruction is not included here.

#uname      : FreeBSD
#uname -r   : 3.2-RELEASE
#<command>  : ifconfig lo0 alias <VIP> netmask 0xffffffff -arp up
#ifconfig -a: lo0: flags=80c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST>mtu 16837
#                  inet 127.0.0.1 netmask 0xff000000
#                  inet <VIP> netmask 0xffffffff

#uname      : IRIX
#uname -r   : 6.5
#<command>  : ifconfig lo0 alias <VIP> netmask 0xffffffff -arp up
#ifconfig -a: lo0: flags=18c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST,CKSUM>
#                  inet 127.0.0.1 netmask 0xff000000
#                  inet <VIP> netmask 0xffffffff

#uname      : SunOS
#uname -r   : 5.7
#<command>  : ifconfig lo0:1 <VIP> netmask 255.255.255.255 up
#ifconfig -a: lo0:  flags=849<UP,LOOPBACK,RUNNING,MULTICAST>mtu 8232
#                   inet 127.0.0.1 netmask ff000000
#             lo0:1 flags=849<UP,LOOPBACK,RUNNING,MULTICAST>mtu 8232
#                   inet <VIP> netmask ffffffff

#uname      : HP-UX
#uname -r   : B.11.00
#<command>  : ifconfig lan1:1 10.10.10.10 netmask 0xffffff00 -arp up
#ifconfig -a: lan0:   flags=842<BROADCAST,RUNNING,MULTICAST>
#                     inet <some IP> netmask ffffff00
#             lan0:1: flags=8c2<BROADCAST,RUNNING,NOARP,MULTICAST>
#                     inet <VIP> netmask ffffff00
#

Ratz 16 Apr 2001

in most cases (when using the NOARP option) you need alias support. Some Unices have no support for aliased interfaces or only limited, such as QNX, Aegis or Amoeba for example. Others have interface flag inheritance problems like HP-UX where it is impossible to give an aliased interface a different flag vector as for the underlying physical interface (as happens with Linux 2.2 and 2.4 - Joe). So for HP/UX you need a special setup because with the standard depicted setup for DR it will NOT work. I've done most Unices as Realserver and was negatively astonished by all the different implementation variations of the different Unix flavours. This maybe resulted from unclear statements from the RFC's.

Gregory Boehnlein

I'm going to be working with a bunch of Solaris 9 boxes in the near future, and I would like to add them to my existing LVS cluster. Does anyone have information on what/how Solaris 9 can be used as the Real Servers in an LVS-DR cluster? On linux, I implement the hidden-arp patch. How is this accomplished on Solaris boxen?

Roberto Nibali ratz (at) drugphish (dot) ch 11 Aug 2003

Solaris doesn't have this issue ;).

Chris Kennedy ckennedy (at) iland (dot) net

The thing I have found out is that on Solaris 2.6, and probably other versions of Solaris, you have to to some magic to get the loopback alias setup. You must run the following commands one at a time:

ifconfig lo0:1 <VIP>
ifconfig lo0:1 <VIP> <VIP>
ifconfig lo0:1 netmask 255.255.255.255
ifconfig lo0:1 up

Which works well and is actually a pointopoint link like ppp which must be the way Solaris defines aliases to the lo interface. It will not let you do this all at once, just each step at a time or you have to start over from scratch on the interface.

Ramon Kagan rkagan (at) YorkU (dot) ca 05 Jun 2002

Just in case anybody is interested. You can do the following on lo0:1 or for paranoid people like me hme1.

ifconfig <intfc> plumb
ifconfig <intfc> <VIP>
ifconfig <intfc> <VIP> <VIP>
ifconfig <intfc> netmask 255.255.255.255
ifconfig <intfc> up

This is from the FAQ but I'm adding that this doesn't have to be on lo0.

For LVS-NAT, anything with a tcpip stack can be a realserver, even a printer.

For LVS-DR, you either must hide the realserver from arp requests, or have it not answer arp requests. HPUX and Linux (kernel > 2.0) are the only Unices that don't honour the -noarp flag.

For LVS-Tun, you need a realserver that decapsulates IPIP packets. Windows used to do this but doesn't anymore.

13.1. Loopback interface on Windows/Microsoft/NT/W2K

Windows is not handled by the configure script (http://www.austintek.com/LVS/LVS-HOWTO/mini-HOWTO/LVS-mini-HOWTO.html#configure_script).

According to Horms you don't need anything special to handle the ARP problem; presumably Windows honours the noarp flag.

Instructions for setting up windows realservers is one of the more common questions on the mailing list. This must be a difficult part of the HOWTO to find ;-\. There seem to be many ways of doing it. Here are some of the answers. Setting the metric to 254 seems to be critical.

Wensong's original recipe for setting up the lo device on a NT realserver.

If you don't have MS Lookback Adapter Driver installed on your NT boxes, enter Network Control Panel, click the Adapter section, click to add a new adapter, select the MS Loopback Adapter. Your NT cdrom is needed here. Then add your VIP (Virtual IP) address on the MS Loopback Adapter, do not enter a gateway address on the Loopback Adapter. Since the netmask 255.255.255.255 is considered invalid in M$ NT, you just accept the default netmask, then enter MS-DOS prompt, remove the extra routing entry.


c:route delete <VIP's network> <VIP's netmask>

This will make the packets destined for this network will go through the other network interface, not this MS Loopback interface. As I remember, setting its netmask to 255.0.0.0 also works.

Jerome Richard jrichard (at) virtual-net (dot) fr

On Windows NT Server, you just have to install a network adapter called "MS Loopback" (Provided on the Windows NT CDROM in new network section) and then you setup the VIP on this interface.

o1004g o1004g (at) nbuster (dot) com

  1. Click Start, point to Settings, click Control Panel, and then double-click Add/Remove Hardware.
  2. Click Add/Troubleshoot a device, and then click Next.
  3. Click Add a new device, and then click Next.
  4. Click No, I want to select the hardware from a list, and then click Next.
  5. Click Network adapters, and then click Next.
  6. In the Manufacturers box, click Microsoft.
  7. In the Network Adapter box, click Microsoft Loopback Adapter, and then click Next.
  8. Click Finish.

robert.gehr (at) web2cad (dot) de; 24 Oct 2001

The MS-Loopback adapter is a virtual device under Windows that does not answer any arp requests. It should be on a Server Edition CD of WinNT/2000. Install and assign it the appropriate IP Address. Because MS would not let you assign a "x.x.x.x/32" netmask to the MS-Loopback adapter, you will end up having two routes pointing into the same net. Lets say your RIP is 10.10.10.10 and your MS-Loopback VIP is 10.10.10.11. You will have two routes in your routing table both pointing to the 10.0.0.0 net. You delete the route that is bound to the VIP on the MS-Loopback adapter with a command like

 
route del 10.0.0.0 mask 255.0.0.0 10.10.10.11
 

Johan Ronkainen jr (at) mpoli (dot) fi

True with Windows NT4. However with Win2000 you can just configure high metric value for loopback interface. I tried this about year ago with metric value 254 without problems. I think you could change netmask to /32 with regedit. Haven't tried that with WinNT4/2000 though. I've used that trick with Win98SE and it worked.

Brent Knotts brent (dot) knotts (at) mylocalgov (dot) com 28 Jan 2003

Concerning Windows 2000 realservers:

It is possible to change the subnet mask to 255.255.255.255 in the registry, and it works fine for my DR setup. This is easier than deleting the extra route to the local interface after each boot.

In Windows 2000, the interfaces are found in:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\Interfaces

Find the interface with the appropriate IP address and change the subnet mask. Rebooting is not necessary, but you should bring the interface down and up.

Foreign Languages for MS

Sebastien Sebastien.Bonnet (at) experian (dot) fg 12 Apr 2002

The "MS Lookback Adapter" is called "carte de bouclage"

Malcolm Turnbull

I've set up a simple LVS DR loadbalancer with 4 IIS web servers (win2K) behind it. I've setup a local loopback adapter on each 2K server and set the priority to 254. The loadbalancer works fine... But seems to confuse Windows Networking, SMB Network Browsing no longer seems to work. (Required by RoboCopy).

Martijn Klingens mklingens (at) ism (dot) nl 10 Sep 2002. Subject: Re: LVS DR setup with NT2K servers

We're using an almost similar setup, just that we're currently using a simple xcopy and want to migrate to DFS. That doesn't change the basics though. Things that I encountered:

  • Windows always adds a route to the subnet on the loopback adapter. If that subnet is also available on the normal LAN your routing table will get confused. In our case we migrated sites with existing IPs to LVS so we had to pick an entire class C subnet as mask. Solution: manually delete the route to the /24 on the loopback interface.
  • Disabling the file and print services on the loopback interface usually also helps.
  • Check the default gateways and the DNS on the various interfaces. If they are not identical you may find very strange behaviour (although there are cases where it's useful to have them differ, this is not that common).

Hope this helps. If not, please specify a bit more about your setup. If you have problems on the realservers, are they able to resolve another realserver using dns and/or ping another realserver?

Oh, and you mentioned network browsing, which is not entirely the same as opening a share on a computer with a well-known name. Do you really need the browser? We usually have the computer browser service stopped anyway so an intruder isn't instantly able to see all other w2k hosts on the net. A bit moot, since DNS provides the same data, but it doesn't really hurt either.

lstep (at) banquise (dot) org 11 Aug 2004

I'm trying to understand why you set the metric of the localhost interface to 254 (according to the doc found on LVS with HA for Win2k Terminal Servers (http://wiki.linuxquestions.org/wiki/LVS_with_HA_for_Win2k_Terminal_Servers) when configuring a W2K realserver's loopback in a Direct Routing mode? If I don't do this what problem(s) will I get?

Malcolm Turnbull malcolm (at) loadbalancer (dot) org

NT can be a pain in the backside when its routing table gets confused :-) In my experience setting the metric to 254 solves 95% of issues.

Ratz

I'm sorry but this doesn't really convince me. Where can I read about this? What does NT really do when you set such a high metric?

Using the registry to set the mask to 255.255.255.255 does the other 5% Seems to happen more often in NT style domains or servers with multiple NICs. Any NT realserver with a problem is easy to spot as it won't work in DR full stop, and the routing table will be incorrect.

Malcolm Turnbull malcolm (at) loadbalancer (dot) org 2005/04/15

We have a small .net based tool to setup the loopback adpater for DR mode on windows 2000/2003 web servers. The idea was to install the loopback and setup as many VIPs as required and then set the netmask to 255.255.255.255. But from testing we've found that windows will sometimes stop responding to the load balancers RIP (for health checks), whereas if you use a netmask of 255.0.0.0 Windows ignores this route because it looks for the smallest subnet in the routing table first i.e. you 255.255.255.0 (or whaterver your using for your RIP.)

The small utility requires the .net framework 1.1. When you start iti, click on the red warning 'no adapter found' to install the loopback Then add as many VIPs as you want and click save. Slightly pointless for anyone who knows what they are doing, but someone may find it usefull.

You can download binary and source (http://www.loadbalancer.org/download/nt/). NB. The one marked BETA will set the netmask to 255.255.255.255 (if thats what you want).

Graeme Fowler graeme (at) graemef (dot) net 06/11/2005

Adding a WIN NT machine to LVS setup: If you go to "add new hardware" and "select from list", you should find it listed under Network Adapters - Microsoft.

Install the "Microsoft Loopback Adapter" network device, and assign it the relevant address(es). IIRC that it doesn't ARP at all, and I've used it in production for a number of SLB setups - not behind LVS though, these were Cisco IOS or ExtremeWare setups, either using NAT or DR.

Stuart walmsley wrote: 2007-05-16

The fact that all the real servers are Windows based and will not take a 255.255.255.255 mask on the loopback stops this from working.

Graeme Fowler graeme (at) graemef (dot) net16 May 2007

No it doesn't - set the realserver loopback interface address as per normal, then edit the registry to change them from 255.255.255.0 to 255.255.255.255. Unfortunately I forget exactly where the key is buried, but I suspect it will be somewhere within:

HKLM\SYSTEM\CurrentControlSet\Tcpip\Parameters\Interfaces\{SID_VALUE}

An alternative is to set the netmask to 255.0.0.0, since a more specific route for the local net will already exist and will be favoured over it; this can however cause unexpected problems for inter-real-server communication.

Graeme Fowler graeme (at) graemef (dot) net 16 May 2007

Note
when connecting from Windows LVS-NAT realservers to the VIP on the director using http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-NAT.html#clients_on_LVS-NAT_realserver_contacting_services_on_VIP the realservers need a /32 netmask.

set the realserver loopback interface address as per normal, then edit the registry to change them from 255.255.255.0 to 255.255.255.255. Unfortunately I forget exactly where the key is buried, but I suspect it will be somewhere within:

HKLM\SYSTEM\CurrentControlSet\Tcpip\Parameters\Interfaces\{SID_VALUE}

An alternative is to set the netmask to 255.0.0.0, since a more specific route for the local net will already exist and will be favoured over it; this can however cause unexpected problems for inter-real-server communication.

13.2. Windows Server 2008

Malcolm has figured this out Direct Server Return on Windows 2008 using loopback adapter (http://www.loadbalancer.org/blog/direct-server-return-on-windows-2008-using-loopback-adpter/)

Malcolm found you can't handle it by adding a 2nd NIC (as you could in the early days of Linux) as windows disables the NIC if there's no link.

13.3. Mac OS X (and Solaris)

Malcolm Turnbull

Has anyone used Mac OS X for a real server in LVS/DR ?

Jerome RICHARD jrichard (at) virtual-net (dot) fr 20 Nov 2002

If you wish to configure multiple IP on Mac OS X, you should use the following command :

 
sudo ifconfig lo0 alias 10.0.0.80 netmask 255.255.0.0
 

There is more information in the AppleCare Documents

Roberto Nibali ratz (at) drugphish (dot) ch 20 Nov 2002

Yes, I gave a presentation about a year ago and I remember that one of the attendees had a new shiny G4 with Mac OS X. It worked flawlessly.

Is it OK to use the loopback interface ?

You can but you don't have to. OS X is BSDish, so you need to use BSDish syntax:

 
ifconfig lo0 alias VIP netmask 255.255.255.255 -arp up
 

Malcolm Turnbull Malcolm.Turnbull (at) crocus (dot) co (dot) uk 22 Nov 2002

Got it working:

  • Solaris:
     
    ifconfig lo0:1 plumb
    ifconfig lo0:1 <vip-addr> netmask 255.255.255.255 up
     
  • Mac OS X:
     
    ifconfig lo0 alias VIP netmask 255.255.255.255 -arp up
     

13.4. Windows servers in Active Directory Domain

Note
Joe: I'm not a windows guy, so I don't understand this. I've just spliced it out of the ml as best as I can. Please send corrections.

David Dyer-Bennet dd-b (at) dd-b (dot) net 10 Oct 2008

We're running into a problem with windows boxes being on a private LAN inside the LVS; they can't join the domain (apparently Active Directory has to be able to initiate connections to the system), and now that's starting to interfere with their deployment of what they call "tcp" protocol since it authenticates service users (obviously they're not talking about the real tcp proptocol; Microsoft must be working *really* hard to obfucate things in this area!).

I'm not a Windows guy, but according to our Windows IT team, a computer can't be part of a windows domain unless the domain controller can initiate a connection to it. So these hidden servers can't be in our corporate domain. It's not an issue with additional services, it's the base domain membership.

Malcolm Turnbull malcolm (at) loadbalancer (dot) org 10 Oct 2008

Normally you wouldn't want load balanced servers to be in an active directory domain... but if it is a requirement then either use: Direct Routing (but make sure DNS is set to manual in both active directory and on the real servers) otherwise active directory stupidly registers the loopback adpater address :-0.

Or you can try single network NAT and make sure that:

Note
Joe: for more details see One Network LVS-NAT

On the load balancer: In order for one arm NAT to work correctly you must modify the firewall script on the load balancers to disable ICMP redirects:

# For single NIC NAT you will also need to disable re-directs
# director is gw for realservers so turn OFF icmp redirects (1 on, 0 off)
echo "0" >/proc/sys/net/ipv4/conf/all/send_redirects
echo "0" >/proc/sys/net/ipv4/conf/default/send_redirects
echo "0" >/proc/sys/net/ipv4/conf/eth0/send_redirects
echo "0" >/proc/sys/net/ipv4/conf/eth1/send_redirects
echo "0" >/proc/sys/net/ipv4/conf/eth2/send_redirects

Make sure that these lines are active by removing the # at the start of each echo command.

Then configure the routing on the windows real servers:

Route configuration for Windows Server with one arm NAT mode When a client on the same subnet as the real server tries to access the virtual server on the load balancer the request will fail. The real server will try to use the local network to get back to the client rather than going through the load balancer and getting the correct network translation for the connection. To rectify this issue we need to add a route to the the load balancer that takes priority over Windows default routing rules. This is a simple case of adding a permanent route: route add -p 192.168.1.0 mask 255.255.255.0 metric 1 NB. Replace 192.168.1.0 with your local subnet address. The default route to the local network has a metric of 10, so this new route overrides all local traffic and forces it to go through the load balancer as required. Any local traffic (same subnet) is handled by this route and any external traffic is handled by the default route (which also points at the load balancer).

I'm not sure what happens if the active directory is on a routed network, but I think it will still work. Please let me know.

Graeme 10 Oct 2008

Hrm... it depends on the management tools you're using as to whether other domain member servers need to reach the realservers you're talking about. I certainly haven't ever come across a situation where the domain controllers initiate connections to member servers without being asked to (like someone running a computer management application to control a service on the realservers).

If this were me, I'd put a domain controller into the "private" LAN which has firewall holes to the main AD domain controllers. That way firewall restrictions should force the local systems use the local DC (or DCs, for better resilience) which can then do all the fancy AD replication back to the other DCs.

Not ideal, but it *might* work.

David Dyer-Bennet dd-b (at) dd-b (dot) net

The benefits I saw initially for load balancing windows boxes in an active directory domain are:

  • The people maintaining it can use their normal windows logons, meaning I don't have to maintain a parallel set of accounts, and the Windows software people don't have to remember yet more passwords.
  • Access to these systems can be controlled through the normal windows active directory mechanisms.

In addition to that, apparently when users are connecting from in-house applications on Windows boxes, it's easy for the windows people to extend authentication through the Web Service connect to supply authentication for the database and information services being accessed by the request. Or so they say. I hadn't thought that would be an issue, and it may not be in the end still.

My desktop system is part of the corporate domain. So are the desktops of the people doing Windows development. Why would making a server part of the domain be any more dangerous than that? And that's standard anywhere that does Windows development.

Graeme

You're personally fairly unlikely to run code as a system account, especially when developing - you're more likely to run it as yourself. Of course, many developers and sysadmins make themselves admins on their own machines (makes installing software just *so* much more convenient than doing "runas") so the security arguments in those cases are slightly damaged anyway :)

Allowing arbitrary code (think of the mass of .NET examples out there) to be executed under the IIS framework is a dangerous game, especially (as is often the case) when it's being executed by a user with elevated privileges (like the Network Service user which IIRC is the default user for IIS code execution).

This is, of course, a massive Catch-22 for hosting operations, and is the reason why app pools came along in IIS6 which allowed almost complete segregation of execution environments which themselves ran as non-privileged users. Much tidier than it used to be.

In your environment you might not be exposing the web servers to that nasty Intertubes thingmy, which makes security all the easier to manage.