LVS-HOWTO <author>Joseph Mack (C) 1999-2001, released under GPL <tt/jmack@wm7d.net/ <date>v1.5, May 2001 <abstract>Install, testing and running of a Linux Virtual Server with 2.2.x and 2.4.x kernels</abstract> <toc> <sect>Introduction <sect1>ChangeLog LVS-HOWTO <itemize> <item>v0.1 12 Jun 99, sent to Wensong for checking <item>v0.2 18 Jul 99, for kernel 2.0.36 <item>v0.3 6 Nov 99, for kernel 2.2.12, LVS v0.9.2, ipvsadm v1.5 (17Oct99 release) <item>v0.4 7 Nov 99, for kernel 2.2.13, LVS v0.9.3, ipvsadm v1.5 (7Nov99 release) <item>v0.5 14 Nov 99, for kernel 2.2.13, LVS v0.9.4, ipvsadm v1.5 (11Nov99 release) <item>v0.6 17 Nov 99, updates off the mailing list <item>v0.7 1 Dec 99, fixes to "arp problem" <item>v1.0 Mar 2001, Transparent proxy for VS-DR (Horms' method) and for VS-Tun (from Julian) corrections to VS-NAT instructions for VS-DR for non-Linux real-servers (from ratz). Change to sgml format. Lots of updates <item>v1.1 Mar 2001, transparent proxy for 2.4.x <item>v1.2 Apr 2001, fwmark <item>v1.3 May 2001, kernels 0.2.12-2.4.4, 1.0.7-2.2.19. e-commerce comments from Ratz, ftp is not secure, netscape persistence, dynamic web pages. <item>v1.4 May 2001, kernels 0.8.0-2.4.4, 1.0.7-2.2.19 <item>v1.5 May 2001, dead links fixed (thanks to Dave Byrne <tt/dave@glynjones.com/) </itemize> posted to the LVS site,<url url="http://www.linuxvirtualserver.org/" name="LVS website http://www.linuxvirtualserver.org"> <sect1>Purpose of this HOWTO To enable you to understand how a Linux Virtual Server (LVS) works. Another document, the LVS-mini-HOWTO, tells you how to setup and install an LVS without understanding how the LVS works. The mini-HOWTO was extracted from this HOWTO. Eventually the redundant material will be dropped from this HOWTO. The material here covers directors and real-servers with 2.2 kernels. For 2.4 kernels, the coverage is less extensive. In general ipvsadm commands and services have not changed between kernels. Otherwise there will be separate information for 2.2 and 2.4 kernels in each section. If there's nothing on 2.4 kernels, it's probably because I haven't written it yet. <sect1>Nomenclature/Abbreviations Preferred names: <itemize> <item> director: the node that runs the ipvs code. Clients connect to the director. The director forwards packets to the servers. <item> real-servers: the servers that handle the requests from the clients. <item> LVS director + real-servers. The virtual server which appears as one machine to the clients. </itemize> The code that patches the linux kernel is called ipvs (or IPVS). <sect2>synonyms <itemize> <item> director: load balancer, dispatcher, redirector. <item> real-server: servers, real servers. <item> LVS: the whole cluster, the (linux) virtual server (LVS) </itemize> <sect2>other The real-servers sometimes are front-ends to other (back-end) servers. (The client does not connect to the back-end servers and they are not in the ipvsadm table.) e.g. <itemize> <item> a real-server may run a web application. The web application in turn connects to a database on another (back-end) server. <item> a real-server which is a webcache (e.g. a squid). The squid connects to (back-end) webserver(s). </itemize> <sect2>Nomenclature is not uniform Anarchy still rules with nomenclature on the mailing list. People call both the director and the real-servers "the server" (in the ipvsadm man page the director is called the "LVS server/host" - this is a historical accident which will be fixed), the real-server is called the "real server" and the whole system the "virtual server". Because of the ambiguity involved, I would rather avoid the term "the server" with LVS. Most people use "real server" for what I call "real-server". I use "real-server" as I despair of finding a reference to a real server in a webpage using the search keys "real" and "server". Until a standardised nomenclature is adopted, you should be prepared for anything on the mailing list. <sect2>IPs The following abbreviations are used in the HOWTO (and in the mailing list). <verb> client IP = CIP virtual IP = VIP (the IP on the director that the client connects to) director IP = DIP real-server IP = RIP (and RIP1, RIP2...) director GW = DGW real-serer GW = RGW </verb> <sect1>What is an LVS? A Linux Virtual Server (LVS) is a piece of software which makes a cluster of servers appear to be one server to an outside client. This apparent single server is called here a "virtual server". The individual servers (real-servers) are under the control of a director (or load balancer), which runs a patched Linux kernel. The client is connected to one particular real-server for the duration of its connection to the LVS. This connection is chosen and managed by the director. The real-servers serve services (eg ftp, http, dns, telnet, nntp, smtp) such as are found in /etc/services or inetd.conf. The LVS presents one IP on the director (the virtual IP, VIP) to clients. An LVS is useful <itemize> <item> for higher throughput. The cost of increasing throughput by adding real-servers in an LVS increases linearly, whereas the cost of increased throughput by buying a larger single machine increases faster than linearly <item> for redundancy. Individual machines can be switched out of the LVS, upgraded and brought back on line without interuption of service to the clients. Machines can move to a new site and brought on line one at a time while machines are removed from the old site, without interruption of service to the clients. <item> for adaptability. If the throughput is expected to change gradually (as a business builds up), or quickly (for an event), the number of servers can be increased (and then decreased) transparently to the clients. </itemize> The Client/Server relationship is preserved in an LVS since <itemize> <item> IPs of all servers is mapped to one IP (the VIP). Client sees only one IP address <item> servers at different IP addresses believe they are contacted directly by the client. </itemize> Here is the basic setup for an LVS. <tscreen><code> ________ | | | client | (local or on internet) |________| | (router) | -- | L Virtual IP i ____|_____ n | | (director can have 1 or 2 NICs) u | director | x |__________| | V | i | r ---------------------------------- t | | | u | | | a | | | l _____________ _____________ _____________ | | | | | | S | real-server | | real-server | | real-server | e |_____________| |_____________| |_____________| r v e r --- </code> </tscreen> In the computer bestiary, the director is a layer 4 (L4) switch. The director makes decisions at the IP layer and just sees a stream of packets going between the client and the real-servers. In particular an L4 switch makes decisions based on the IP information in the headers of the packets. Here's a description of an L4 switch from <url url="http://www.supersparrow.org/ss_paper/" name="Super Sparrow Global Load Balancer documentation"> " Layer 4 Switching: Determining the path of packets based on information available at layer 4 of the OSI 7 layer protocol stack. In the context of the Internet, this implies that the IP address and port are available as is the underlying protocol, TCP/IP or UCP/IP. This is used to effect load balancing by keeping an affinity for a client to a particular server for the duration of a connection. " The director does not inspect the content of the packets and cannot make decisions based on the content of the packets (e.g. if the packet contains a <ref id="cookie" name="cookie">, the director doesn't know about it and doesn't care). The director doesn't know anything about the application generating the packets or what the application is doing. Because the director does not inspect the content of the packets (layer 7, L7) it is not capable of session management or providing service based on packet content. L7 capability would be a useful feature for LVS and perhaps this will be developed in the future. The director is basically a router, with routing tables set up for the LVS function. These tables allow the director to forward packets to real-servers for services that are being LVS'ed. If http (port 80) is a service that is being LVS'ed then the director will forward those packets. The director does not have a socket listener on VIP:80 (i.e. netstat won't see a listener). John Cronin <tt/jsc3@havoc.gtf.org/ (19 Oct 2000) calls these types of servers (i.e. lots of little boxes appearing to be one machine) "RAILS" (Redundant Arrays of Inexpensive Linux|Little|Lightweight|L* Servers). Lorn Kay <tt/lorn_kay@hotmail.com/ calls them RAICs (C=computer), pronounced "rake". The director uses 3 different methods of <ref id="forwarding" name="forwarding"> <itemize> <item> VS-NAT based on network address translation (NAT) <item> VS-DR (direct routing) where the MAC addresses on the packet are changed and the packet forwarded to the real-server <item> VS-Tun (tunnelling) where the packet is IPIP encapsulated and forwarded to the real-server. </itemize> Some modification of the real-server's ifconfig and routing tables will be needed for VS-DR and VS-Tun forwarding. For VS-NAT the real-servers only need a functioning tcpip stack (ie the real-server can be a networked printer). LVS works with all services tested so far (single and 2 port services) except that VS-DR and VS-Tun cannot work with services that initiate connects from the real-servers (so far; identd and rsh). The real-servers can be indentical, presenting the same service (eg http, ftp) working off file systems which are kept in sync for content. This type of LVS increases the number of clients able to be served. Or the real-servers can be different, presenting a range of services from machines with different services or operating systems, enabling the virtual server to present a total set of services not available on any one server. The real-servers can be local/remote, running Linux (any kernel) or other OS's. Some methods for setting up an LVS have fast packet handling (eg VS-DR which is good for http and ftp) while others are easier to setup (eg transparent proxy) but have slower packet throughput. In the latter case, if the service is CPU or I/O bound, the slower packet throughput may not be a problem. For any one service (eg httpd at port 80) all the real-servers must present identical content since the client could be connected to any one of them and over many connections/reconnections, will cycle through the real-servers. Thus if the LVS is providing access to a farm of web, database, file or mail servers, all real-servers must have identical files/content. You cannot split up a database amongst the real-servers and access pieces of it with LVS. An LVS requires a Linux director (Intel and <ref id="Alpha" name="Alpha"> versions known to work, other versions eg Sparc expected to work, but haven't been tested). There are differences in the coding for LVS for the 2.0.x, 2.2.x and 2.4.x kernels. Development of LVS on 2.0.36 kernels has stopped (May 99). Code for 2.2.x kernels is production level and this HOWTO is up to date for 2.2.18 kernels. Code for 2.4.x kernels is relatively new and this HOWTO may not contain all the information neccessary to run 2.4.x kernel ipvs (check on the mailing list). The 2.0.x and 2.2.x code is based on the masquerading code. Even if you don't explicitely use ipchains (eg with VS-DR or VS-Tun), you will see masquerading entries with `ipchains -M -L` (or `netstat -M`). Because 2.2.x kernels are not SMP for kernel code, LVS can only use one processor at a time. Code for 2.4.x kernels was rewritten to be compatible with the netfilter code (i.e. its entries will show up in netfilter tables). It is production level for VS-DR and VS-Tun. Because of incompatibilities with VS-NAT for 2.4.x LVS is development (Jan 2001) with problems are expected to be resolved shortly. Directors running 2.4.x kernel LVS's will run in SMP mode in the kernel allowing a higher throughput of packets than with a 2.2.x kernel running on the same (SMP) hardware. If you are setting up an LVS for the first time, then set up the director with the latest 2.2.x kernel or 2.4.x kernel. You can have almost any OS on the real-servers. LVS works on ethernet. There are some limitations on using <ref id="ATM" name="ATM">. LVS is continually being developed and usually only the more recent kernel and kernel patches are supported. Usually development is incremental, but with the 2.2.14 kernels the entries in the /proc file system changed and all subsequent 2.2.x versions were incompatible with previous versions. For more documentation, look at the LVS web site (eg a talk I gave on how LVS works on <url url="http://www.linuxvirtualserver.org/Joseph.Mack" name="2.0.36 kernel directors">) <sect1>Minimal knowledge required<label id="minimal_knowledge"> (from Ratz <tt/ratz@tac.ch/) To be able to setup/maintain an LVS it would be helpfull if you <itemize <item>know how to patch and compile a kernel <item>the basics of shell-scripting <item>have intermediate knowledge of TCP/IP <item>have read the man-page, the online-documentation and LVS-HOWTO (this document) (and the LVS-mini-HOWTO) <item>know basic system administration tools (e.g. ipchains, syslog) </itemize> <sect1>LVS Failure The LVS itself does not provide high availability. Other software (eg mon, ldirectord, or the Linux HA code) is used in conjunction with LVS to provide high availability (i.e. to switch out a failed real-server/service or a failed director). Another package <url url="http://keepalived.sourceforge.net/" name="keepalived"> is designed to work with LVS watching the health of services, but no-one has reported using it. Julian has written <url url="http://www.linuxvirtualserver.org/~julian/" name="Netparse">, which is suffering the same fate. There are two types of failures with an LVS. <itemize> <item>director failure This is handled by having a redundant director available. Director failover is handled by the <url url="http://ultramonkey.sourceforge.net" name="Ultra Monkey Project">. <item>real-server failure, or failure of a service on a real-server This is relatively simple to handle (compared to director failover). An agent running on the director monitors the services on the real-servers. If a service goes down, that service is removed from the ipvsadm table. When the service comes back up, the service is added back to the ipvsadm table. There is no separate handling of real-server failure (e.g. it catches on fire, a concern of Mattieu Marc <tt/marc.mathieu@metcelo.com/) - the agent on the director will just remove all that real-server's services from the ipvsadm table. In the <url url="http://ultramonkey.sourceforge.net" name="Ultra Monkey Project">, service failure is monitored by ldirectord. The <ref id="configure" name="configure script"> monitors services with mond. Setting up mon is covered in <ref id="failover" name="Failover protection"> </itemize> In both cases (failure of director, or failure of a service), the client's session with the real-server will be lost (as would happen in the case of a single server). With failover however, the client will be presented with a new connection when they reconnect. <sect1>Thanks Contributions to this HOWTO came from the mailing list and are attibuted to the poster (with e-mail address). Postings may have been edited to fit them into the flow of the HOWTO. The LVS logo (Tux with 3 lighter shaded penguins behind him representing a director and 3 real-servers) is by Mike Douglas spike@bayside.net <url url="http://www.linuxvirtualserver.org" name="LVS homepage"> in Germany hosted by Lars <tt/lmb@suse.de/ <sect1>Mailing list, subscribing, unsubscribing and searchable archives <url url="http://www.linuxvirtualserver.org/mailing.html" name="mailing list details"> Thanks to Hank Leininger for the mailing list archive which is searchable not only by subject and author, but by strings in the body. Hank's resource has been of great help in assembling this HOWTO. <sect1>getting technical help <itemize> <item> mailing list <item> LVS-mini-HOWTO. This will setup a simple 3 node (client, director, real-server) LVS without you needing to understand how an LVS works. Presumably you will have read the mini-HOWTO before this HOWTO. <item> updates/problems - post to the mailing list and cc: mailto:jmack@wm7d.net </itemize> <sect1>Posting problems/questions to the mailing list<label id="problem_report"> There's always new ideas and questions being posted on the mailing list. We don't expect this to stop. If you have a problem with your LVS not working, before you come up on the mailing list, please - <itemize> <item> Read the LVS-mini-HOWTO <item> Set up a simple LVS (3 nodes: client, director, real-server) with VS-DR or VS-NAT forwarding, with the service telnet using the instructions in the LVS-mini-HOWTO. You should be able to do this starting from a freshly downloaded kernel and ipvs-patch pair. <item> LVS is distribution neutral. It only needs the kernel (from ftp.kernel.org) and the ipvs patch (from www.linuxvirtualserver.org). If you can get an LVS to work with these files but not with the kernel on your distro, then you have a problem with your distro. If your problem is distribution specific (eg Redhat ships with LVS patches applied, with a version which will be old by the time you get to talk to us; SuSE is planning on doing the same thing) then we may know what the problem is, but it would be better if you contacted your distro - they need the feedback, not us. <item>If you are using one of the packages that can be used with LVS (eg heartbeat from the Linux HA project http://www.henge.com/~alanr/ha, or piranha from Redhat), again we may know what the problem is, but they need the feedback that you can't get it to work, not us. Many of us are on each others' mailing lists and we try to help when we can but the best people to handle the problem are the developers for each package. <item> Read the LVS-HOWTO <item> Consult the <url url="http://marc.theaimsgroup.com/?l=linux-kernel&ero;m=91754108723334&ero;w=2" name="LVS archives"> <item>Please use our jargon as best you can. The machine names will be client, director, real-server1, real-server2... IPs are CIP, VIP, RIP, DIP. If you do this, we won't have to translate "susanne" and "annie" to their functional names as we scan your posting. <item>When you come up on the mailing list we need to know your kernel (eg 2.2.14) and the patch that was applied to it (eg 0.9.11), whether you are using VS-DR, VS-NAT or VS-Tun. Tell us what you did, what you expected and what you got and why that's a problem. <item>We didn't know everything when we first started and we don't expect you to be any better at LVS than we were at first, but we'll feel much better about helping you if you've done the above first. We can't help you if you just tell us that you've read the documents and your LVS doesn't work. </itemize> If you don't understand your problem well, Here's a suggested submission format from Roberto Nibali <tt/ratz@tac.ch/ <enum> <item> System information, such as kernel, tools and their versions. Example: <verb> hog:~ # uname -a Linux hog 2.2.18 #2 Sun Dec 24 15:27:49 CET 2000 i686 unknown hog:~ # ipvsadm -L -n | head -1 IP Virtual Server version 1.0.2 (size=4096) hog:~ # ipvsadm -h | head -1 ipvsadm v1.13 2000/12/17 (compiled with popt and IPVS v1.0.2) hog:~ # </verb> <item>Short description and maybe sketch of what you intended to setup. Example (LVS-DR): <verb> o Using LVS-DR, gatewaying method. o Load balancing port 80 (http) non-persistent. o Network Setup: ________ | | | client | |________| | CIP | | | (router) | | | | GEP (packetfilter, firewall) | GIP | | __________ | DIP | | +------+ director | | VIP |__________| | | | +-----------------+----------------+ | | | | | | RIP1, VIP RIP2, VIP RIP3, VIP ____________ ____________ ____________ | | | | | | |real-server1| |real-server2| |real-server3| |____________| |____________| |____________| CIP = 212.23.34.83 GEP = 81.23.10.2 (external gateway, eth0) GIP = 192.168.1.1 (internal gateway, eth1, masq or NAT) DIP = 192.168.1.2 (eth0) VIP1 = 192.168.1.100 (eth0:4 on director, lo:1 on realserver) RIP1 = 192.168.1.11 (belonging to VIP1) RIP2 = 192.168.1.12 (belonging to VIP1) RIP3 = 192.168.1.13 (belonging to VIP1) DGW = 192.168.1.1 (GIP for all realserver) o ipvsadm -L -n hog:~ # ipvsadm -L -n IP Virtual Server version 1.0.2 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.1.10:80 wlc -> 192.168.1.13:80 Route 0 0 0 -> 192.168.1.12:80 Route 0 0 0 -> 192.168.1.11:80 Route 0 0 0 hog:~ # The output from ifconfig from all machines (abbreviated, just need the IP, netmask etc), and the output from netstat -rn. </verb> <item>What doesn't work. Show some output like tcpdump, ipchains, ipvsadm and kernlog. Maybe we even ask you for a more detailed configuration like routing table, OS-version or interface setup on some machines used in your setup. Tell us what you expected. Example: <verb> o ipchains -L -M -n (2.2.x) or cat /proc/net/ip_conntrack (2.4.x) o echo 9 > /proc/sys/net/ipv4/vs/debug_level && tail -f /var/log/kernlog o tcpdump -n -e -i eth0 tcp port 80 o route -n o netstat -an o ifconfig -a </verb> </enum> <sect1>ToDo List <sect2>for the HOWTO Combining HA and LVS (e.g. Ultramonkey). I realise that information in here isn't all that easy to locate yet (there's no index and you'll have to search with your editor) and that the ordering of sections could be improved. I'll work on it as I have time. <sect2>for LVS Does anyone want to write a <ref id="MIB" name="MIB"> for LVS? <sect1>Other load balancing solutions from lvs@spiderhosting.com <url url="http://dmoz.org/Computers/Software/Internet/Site_Management/Load_Balancing" name="a list of load balancers"> <sect1>Software/Information useful/related to LVS <url url="http://ultramonkey.sourceforge.net/" name="Ultra Monkey"> is LVS and HA combined. from lvs@spiderhosting.com <url url="http://supersparrow.sourceforge.net/" name="Super Sparrow Global Load Balancing"> using BGP routing information. From ratz, there's a write up on load imbalance with persistence and sticky bits at our friends at <url url="http://www.microsoft.com/TechNet/win2000/nlbovw.asp?a=printable" name="M$">. From ratz, Zero copy patches to the kernel to speed up network throughput, <url url="ftp://ftp.kernel.org/pub/linux/kernel/people/davem" name="Dave Miller's patches">, <url url="http://www.surriel.com/patches" name="Rik van Riel's vm-patches"> and <url url="http://www.linux-mm.org/" name="more of Rick van Riel's patches">. The Zero copy patches may not work with LVS and may not work with netfilter either (from Kate<tt/john@antefacto.com/). From Michael Brown <tt/michael_e_brown@dell.com/, the <url url="http://www.redhat.com/products/software/webservers/tux" name="TUX kernel level webserver">. From Lars <tt/lmb@suse.de/ <url url="http://www.backhand.org/" name="mod_backhand">, a method of balancing apache httpd servers that looks like ICP for web caches. A lightweight and simple webbased cluster monitoring tool designed for beowulfs <url url="http://www.phy.duke.edu/brahma/" name="procstatd">, the latest version was 1.3.4 (you'll have to look around on this page). From Putchong Uthayopas <tt/pu@ku.ac.th/ a heavyweight (lots of bells and whistles) cluster monitoring tool <url url="http://smile.cpe.ku.ac.th/software/" name="KCAP"> <sect>Getting Files <url url="http://www.linuxvirtualserver.org" name="LVS homepage"> Go to the "software" link. Get the latest patch tarball for the director (available in 2.2.x and 2.4.x flavors). This will contain the kernel patch and source code for various programs and utilities. <sect1>Director Code <label id="Alpha"> You have to patch the standard kernel with the matching ipvs patch. The director operates in one (or several) forwarding methods; VS-NAT (network address translation); VS-DR (direct routing) and VS-Tun (tunneling). The mode is chosen individually for each real-server/service by ipvsadm. The director code is well tested on Intel. As well several people are running on Alpha hardware (Flavio Pescuma <tt/edtfopa@malmen.ericsson.se/ on a mailserver, and Philip Copeland <tt/copeland@redhat.com/ who is using it alongside an Intel director to test director failover). You have to edit the code to run it on Alpha: from Philip (this has been fixed in recent code). <tscreen> <code> >remember use socklen_t NOT size_t for network calls (See >page 27 of R.Stevens UNIX Net. prog. vol1 ed 2) and everything >suddenly falls into place. Certainly socklen_t is the correct >type to use for portability between 32/64 machines, you'll >need to update any other code to use socklen_t thats 64 bit >eg ia64 sparc64 etc </code> </tscreen> <sect1>RealServer Code The real-servers must be configured appropriately for the LVS forwarding method. You must <itemize> <item>handle the <ref id="arp_problem" name="arp problem"> (VS-DR/VS-Tun; Linux 2.2.x/2.4.x kernels) <item>set the default gw </itemize> Here's this list of OS's that have been tested with the forwarding methods. (we expect all OS's can be made to work one way or another.) <itemize> <item> Real-servers by Mode - <itemize> <item> VS-NAT - any OS <item> VS-DR <itemize> (from Ratz <tt/ratz@tac.ch/) <item>most OS's (all expected to succumb eventually) <item>Linux 2.0.36, 2.2.x <item>Solaris 2.5.1, 2.6, 2.7; <item>FreeBSD 3.1, 3.2, 3.3 <item>NT (although Webserver would crash): 4.0 no SP <item>IRIX 5.3-6.5 <item>HPUX 11 </itemize> <item> VS-Tun - only linux </itemize> <item> real-servers by OS - <itemize> <item>Other OS <itemize> <item> VS-NAT - no modifications required to real-server <item> VS-DR - works for most OS's so far, although some trickery has been required to find the commands neccessary to configure the VIP on the real-servers for some OS's. If running Linux-2.2.x, or Linux-2.4.x, you have to handle the <ref id="arp_problem" name="arp problem"> <item> VS-Tun - only linux (real-server must be able to decapsulate IPIP packets). If running Linux-2.2.x, or Linux-2.4.x, you have to handle the <ref id="arp_problem" name="arp problem"> </itemize> <item>Linux 2.0.36: no modifications or patches required. Works with all LVS modes VS-NAT, VS-Tun and VS-DR. <item>Linux 2.2.x,2.4.x: <itemize> <item> VS-NAT - no modification to real-server. <item> VS-DR and VS-Tun - (drum roll - this is the difficult bit, otherwise known as the <ref id="arp_problem" name="arp problem"> - see also the section on the <ref id="Wisdom" name="Wisdom of the mailing list">). The <ref id="arp_problem" name="arp theory"> can be skipped on a first read. You'll need to understand this to set up a working LVS. Here's the summary: <code> IF ( (you are using VS-DR or VS-Tun on the director) AND (you are running a Linux 2.2.x, 2.4.x kernel on a real-server) AND ( the VIP on the real-server is on an ethernet device eg lo:0, tunl0:0 i.e. is not being accepted by transparent proxy ) AND ( the real-servers can answer arp requests from the client/router (the real-servers are on the same piece of wire|network as the director) ) ) THEN { YOU MUST HANDLE THE ARP PROBLEM } FI </code> In general you'll have to handle the <ref id="arp_problem" name="arp problem">. </itemize> </itemize> </itemize> The most common way to handle the arp problem is to hide the VIP from arp requests <itemize> <item> <ref id="2.2_arp" name="2.2.x: use the prepatched 2.2.x kernel in the standard distribution for the real-server"> <item> <ref id="2.4_arp" name="2.4.x: patch the standard 2.4.x kernel for the real-server">. </itemize> Although not used very much (I don't know why), another simple method is to put an extra NIC into the real-servers for the VIP and not connect it to the network. The NIC doesn't handle any packets, it's just a way of putting the VIP onto the real-server. The NIC can be an old 10Mbps ISA card. The cost of some 100Mbps PCI tulip cards now is less than the salary you'd pay for the time to recompile a 2.4.x kernel with the hidden patch. All methods of handling the arp problem work, have about the same performance (throughput, latency) and are about equally easy/difficult to setup. The method of hiding the devices from arp requests is the closest to the standard NOARP unix behaviour and is the method most commonly used on Linux real-servers. <sect1>Configure Script<label id="configure"> You can set up the LVS by hand with ipvsadm. This is somewhat tedious and error prone. While do-able for a single LVS configuration, this is not the way to go for setting up lots of different LVS configurations. Some <ref id="configure" name="configure"> tools are available. The configure script was designed to set up LVS's quickly so I could do testing. The current version (0.8) does a wide range of checks, hopefully catching the usual errors. This only sets up a single director and cannot handle director failover. For director failover try <url url="http://ultramonkey.sourceforge.net/" name="Ultra Monkey">. The configure script uses mon to handle failure of services on real-servers. <sect>The arp Problem<label id="arp_problem"> <sect1>The problem If you follow the instructions and setup the examples in the LVS-mini-HOWTO, then you don't need to know about the arp problem. If you're going to setup grander LVS's, then you'll need to understand the arp problem. Although this section comes early in the HOWTO, it has lots of pitfalls. You shouldn't be reading this unless you've at least setup a working VS-NAT and VS-DR LVS using the canned instructions in the mini-HOWTO. The LVS allows several machines to function as one machine. For VS-DR and VS-Tun some trickery was needed to split the various handshakes etc involved in establishing and maintaining a tcpip connection so that some parts of it came from one machine and other parts from another machine. Most of these problems are handled, and some problems only occur for certain services (eg <ref id="authd" name="identd">) and we've learned to live with them. The worst problem, which ironically happens with real-servers running Linux 2.2.x and 2.4.x kernels, is the "arp problem" (it's just as well we have the source code). With VS-DR and VS-Tun, all the machines (director, real-servers) in the LVS have an extra IP, the VIP. Here's a VS-DR in a test setup where all machines and IPs are on the same network. <tscreen> <code> ________ | | | client | |________| | | (router) | | | __________ | DIP | | |------| director | | VIP |__________| | | | ------------------------------------ | | | | | | RIP1, VIP RIP2, VIP RIP3, VIP ______________ ______________ ______________ | | | | | | | real-server1 | | real-server2 | | real-server3 | |______________| |______________| |______________| </code> </tscreen> When the client requests a connection to the VIP, it must connect to the VIP on the director and not to the VIP on the real-servers. The director box acts as an IP router, accepting packets destined for the VIP and then sending them on to a real-server (where the real work is done and a reply is generated). When the client (or router) puts out the arp request "who has VIP, tell client", the client/router must receive the MAC address of the director for the LVS to work. After receiving the arp reply, the client will send the connect request to the director. (The director will then forward the connect request packet to the appropriate real-server and update its internal tables to keep track of connections). If instead, the client gets the MAC address of one of the real-servers, then the packets will be sent directly to that real-server, bypassing the LVS action of the director. If the client's packets are consistently sent to the same real-server, then the client will have a normal session connected to that real-server. You can't count on this happening, the MAC address the client gets might change in the middle of a session and a new real-server will start getting packets for connections it knows nothing about (the real-server will send tcp resets). If nothing is done to direct arp requests for the VIP specifically to the director, then in some setups, one particular real-server's MAC address will be in the client/router's arp table for the VIP and the client will only see one real-server. (In my setup, the machine with the fastest CPU is in the client's arp table, suggesting that it's the first machine to reply that gets in. Horms and Steven WIlliams have written that they think it's the last machine to reply whose entry in in the client's arp table.) In other setups where the real-servers are identical, the client will connect to different real-servers each time the arp cache times out (see comment by Steven WIlliams elsewhere). There the client's connection will hang as the new real-server will be presented with packets from an established connection that it knows nothing about. If the director always gets its MAC address in the router arp table, then the LVS will work without any changes to the real-servers (as happened in my case with a director with the fastest CPU in the LVS), although this may not be a reliable solution for production. Getting the MAC address of the director (instead of the real-servers) to the client when the client/router does an arp request is the key to solving the "arp problem". The arp problem is handled in 2.0.x kernels as serveral devices which don't reply to arp requests (eg dummy0, tunl0, lo:0) were available for the the VIP. For other OS's, the NOARP flag for ifconfig would stop the VIP on the real-servers from replying to arp requests. However with 2.2.x (and now 2.4.x) kernels, the devices which didn't reply to arp requests in 2.0.x, now reply to arp requests. There is a "-arp" (NOARP) option for ifconfig which (according to the man pages) turns off replies to arp requests for that device, and an "arp" option which turns them back on again. Linux does not always honour this flag (you couldn't turn on replies to arp requests for the dummy0 devices in 2.0.36 kernels and you can't turn it off for tunl0 in 2.2.x kernels. eth0 behaves properly in 2.0.36 but in 2.2.x kernels it arps even when you tell it not to arp). This behaviour of not honouring the NOARP flag in the Linux 2.2.x kernels is not regarded as a "problem" by those writing the Linux TCPIP code and is not going to be "fixed". Julian 22 May 2001 <quote> The flag is used to allow arp requests for the specified device. Although "lo" doesn't reply to arp requests, the requests for the VIP go through eth*, and so the NOARP flag is of no help to us. We can't drop the flag for eth. </quote> Another wrinkle is that in 2.0.36 kernels, aliased devices (eg eth0:1) could be setup independantly of the options on the primary (eth0) device. Thus eth0:1 behaved as if it were on a separate NIC and it's arp'ing behaviour could be set independantly of the primary interface. The settings of an aliased device belonged to the IP. With the 2.2.x kernels, the aliased devices are now just alternate names for each other: you change an option (eg -arp) or up/down of one alias (or primary) the other aliases follow. With 2.2.x kernels, the settings of the aliased device belong to the primary device (there is only one device with several IPs). When LVS was running on 2.0.36 machines, the VIP was usually configured as an alias (eg lo:0, tunl0) on the main ethernet device (eth0), allowing the nodes in an LVS to have only one NIC. With 2.2.x kernels care is needed when only one NIC is used on the real-server (the usual case). On a real-server with eth0 carrying the RIP, and the real-server having only one NIC, eth0 must reply to arp requests (to receive packets), then eth0:1 carrying the VIP will reply to arp requests too, even if you ifconfig it with -noarp. Thus if a real-server is running a 2.2.x kernel and has the VIP on an ip_alias, then the VIP on the real-server will reply to arp requests received from the router. <sect1>The cure(s) Several cures have been produced in an attempt to solve the arp problem. They involve either <itemize> <item> stopping the real-servers from replying to arp requests for the VIP. <item> hiding the VIP on the real-servers so that they don't see the arp requests. <item> priming the client/router in front of the director with the correct MAC address for the VIP. <item> allowing the real-server to accept a packet with dst=VIP even though the real-server does not have a device with this IP. <item> stopping arp requests for the VIP getting to the real-servers. </itemize> Pick one - Note: Some of these cures involve applying a patch to the kernel on the Linux 2.2.x or 2.4.x real-server. The patch (e.g. the "hidden" patch), which you apply to the real-server, is different to the patch which you apply to the director (the "ipvs" patch). <sect2>2.2.x kernels<label id="2.2_arp"> The "hidden" patches for kernel >=2.2.14 are now in the standard linux distribution (ie you can use the "hidden" feature with a standard kernel and don't have to patch the kernel on the real-server anymore). The arp patches allow you to hide a device from arp requests, returning to the no_arp behaviour of the 2.0.x kernels. To hide devices from arp calls<label id="hidden">, on the real-servers do <verb> #to activate the hidden feature echo 1 > /proc/sys/net/ipv4/conf/all/hidden #to make lo:0 -arp, put lo here echo 1 > /proc/sys/net/ipv4/conf/<interface_name>/hidden </verb> To test that the network device (here lo:0) is hidden from arp requests - <itemize> <item>before you hide the lo:0, ping the VIP from another machine, then run arp -a and see that the MAC address for the VIP matches that for eth0 on the real-server <item>Clear the entry for the VIP with "arp -d VIP", and show that the arp entry is gone for the VIP (with arp -a) <item>ping the VIP and look for the reappearance of the arp entry for the VIP. <item>Then hide the lo interface and ping the VIP again from the outside machine. The VIP will most likely reply to the ping since the entry for the VIP is still in the arp table of the outside machine. <item>Clear the arp entry (arp -d VIP) and ping the VIP again - this time you'll get no reply. </itemize> There is a possible race condition in hiding the VIP - On Thu, 15 Feb 2001, Kyle Sparger wrote: <quote> I've found an interesting, but not totally unexpected race condition under DR in 2.2.x that I've managed to create when installing VIP's on a machine in DR mode. Basically, the cause is this: </quote> <verb> ifconfig dummy0 10.0.1.15 echo 1 > /proc/sys/net/ipv4/conf/dummy0/hidden </verb> <quote> You'll notice that there's going to be a small gap between the two which allows an ARP request to come in, and for the server to reply. And yes, it is big enough to be bitten by -- I've been bitten twice by it so far :) </quote> Julian On boot: <verb> echo 1 > /proc/sys/net/ipv4/conf/all/hidden # For each hidden interface: modprobe dummy0 ifconfig dummy0 0.0.0.0 up echo 1 > /proc/sys/net/ipv4/conf/dummy0/hidden # Now set any other IP address </verb> Kyle's suggestion <verb> echo 1 > /proc/sys/net/ipv4/conf/default/hidden ifconfig dummy0 10.0.1.15 echo 0 > /proc/sys/net/ipv4/conf/default/hidden </verb> <quote> The echo 0 command is incase I want to configure other interfaces later that I _do_ want responding to ARP requests. Technically, it's not necessary, I just find it useful in my particular setup. </quote> For older kernels, you apply the arp patches to the kernel code of the 2.2.x real-servers. These patches are separate from the ipvs patch applied to the kernel on the director. For kernels <2.2.12, Julian's patch is on the lvs website. http://www.linuxvirtualserver.org/arp_invisible-2213-2.diff The patch by Stephen WillIams is at http://www.linuxvirtualserver.org/sdw_fullarpfix.patch This patch is against a 2.2.5 kernel but can be applied to later kernels (tested to 2.2.13). The file appears to have DOS carriage control. Depending what you get on your disk, you may have to convert the file to unix carriage control (with `tr -d '\015'`) (the unix line extension of '\' doesn't work in combination with DOS carriage control). The whitespace may not match your file so do <verb> $ cd /usr/src/linux $ patch -p1 -l < sdw_fullarpfix.patch </verb> If you are running one of these old kernels, you could upgrade to your kernel. <sect2>2.4.x kernels<label id="2.4_arp"> Julian's hidden patch to the standard 2.2.x kernel is not being included in the 2.4.x kernels. For early 2.4.x kernels (eg x=0), the patch is available at http://www.linuxvirtualserver.org/hidden-2.3.41-1.diff. (This patches a part of the kernel that isn't being actively fiddled with, so hopefully the patch will work against later 2.4.x kernels too.) The 2.4.x "hidden" patch in now being actively maintained and is included in ipvs-x.x.x/contrib/patches/hidden-x.x.x.diff Assuming you are patching 2.4.2 with the ipvs-0.2.5 files <verb> cd /usr/src/linux patch -p1 <../ipvs-0.2.5/contrib/patches/hidden-2.4.2-1.diff </verb> Then build the kernel (can use same options as for the 2.4 director kernel build). You activate the hidden feature as for 2.2 (see <ref id="hidden" name="hidden">). As to why the hidden patch is in the 2.2 kernels but not the 2.4 kernels see the <url url="http://marc.theaimsgroup.com/?l=linux-kernel&m=98032243112274&w=2" name="the mailing list archives"> or for <url url="http://marc.theaimsgroup.com/?t=98019795800013&w=2&r=1" name="the thread"> <sect2>Put an extra NIC on the real-server to carry the VIP (on eth1) Possible cards would be a discarded ISA card (WD80x3), or a cheap 100Mbit PCI card (eg Netgear FA310TX, $16 in USA in Nov 99) There is no traffic going through this NIC and it doesn't matter that it's an old slow card. The extra card is only required so that the real-server can have the VIP on the machine. With 2.2.x kernels you can't stop this device (eth1) from replying to arp requests, but if you don't connect the cable to it or don't put a route to it in the real-server's routing table, then the client won't be able to send it an arp request. To set this up with the configure script, enter eth1 as the device for the VIP on the real-server. <sect2> Put the real-servers on a different network to the VIP, and setup routing tables so that the client cannot route to this network (Lars' method) This method requires 2 NICS on the director and for the director to be a firewall (see VS-DR, VS-Tun for details). <sect2>On the client(router), set the routing to the VIP to go only to the director You can hardwire the MAC address of the director as the MAC address of the VIP. You can do this with <verb> #arp -s lvs.mack.net 00:80:C8:CA:A7:E4 or arp -f /etc/ethers. </verb> Here is my /etc/ethers file (on the client) <verb> lvs.mack.net 00:80:C8:CA:A7:E4 </verb> This requires no extra NICs or patching of real-servers. However in a production environment, redundant directors with heartbeat/failover may be required and some method (eg running send-arp) will be needed to change the static arp entry as the failover occurs. If multiple NICs are involved, it is possible that the above instruction will result in a route through the wrong NIC. In this case bring up the NIC of interest first and then run the above command. Alternately if the router has serveral NICs, use one for the director and another for the real-servers. Route the VIP to the director. <sect2>Use transparent proxy allow the incoming packet to be accepted locally - Horms' method. see VS-DR and VS-Tun for details. The configure script will set this up for you. <sect1>The ARP problem, the first inklings History: ARP behaviour changed with 2.2.x kernels. Here's the original posting by Wensong <verb> Date: Wed, 24 Mar 1999 From: Wensong Zhang <tt/wensong@iinchina.net/ Subject: The problem of Linux 2.2.3 tunnel device </verb> Today I upgraded the kernel to 2.2.3 with tunneling support on one of a real server, and found a problem that the Linux 2.2.3 tunnel device answers ARP requests. Even if I used the NOARP options as follows: ifconfig tunl0 172.26.20.110 -arp netmask 255.255.255.255 broadcast 172.26.20.110 It still answers the ARP requests. This will greatly affect the virtual server via tunneling work properly. In fact, the tunnel device shouldn't answer the ARP requests from the ethernet. I think it is a bug of linux/net/ipv4/ipip.c, which is now a clone of ip_gre.c not the original tunneling code. If you are interested, you can test yourself on kernel 2.2.3, choose a free IP address of your ethernet and configure it on the tunl0 device, then telnet to that IP address from other host, I guess you can. Finally, have a look at the ipip.c, maybe you can debug it. :-) -- A reply to Wensong about the change in arp characteristics in 2.2 kernels, from Kuznet (2.2 tcpip author) <verb> From: kuznet@ms2.inr.ac.ru To: Wensong Zhang <tt/wensong@iinchina.net/ Cc: netdev@nuclecu.unam.mx Subject: Re: A little patch for linux/net/ipv4/arp.c for 2.2.5 Hello! > But, what is the IFF_NOARP flag of the tunnel device for? IFF_NOARP means that ARP is not used by THIS device. On normal IPIP tunnels it does not make much of sense, but may be used f.e. to turn on/off endpoint reachability detection. I do not see any reasons to disable answering ARP in such curcumstances. Isolation of VPNs on adjucent segments is impossible at routing/arp level, it is just not well-defined behaviour. If the isolation is made with firewall policy rules, then it is clear that arp policy must be handled at this level too. > In kernel 2.0.x, the tunnel device doesn't answer ARP requests. Yes. > Yeah, we can have link-local addresses that doesn't answer ARP requests in > kernel 2.2.x. For example, we can configure all the hosts in a network > with the following command: > ifconfig lo:0 192.168.0.10 up > There will no collision. The lookback alias interfaces don't answer ARP > requests. Are you sure? I am not. Please, test. BTW you risk adding non-loopback addresses on loopback device. They have the HIGHEST preference to be used as router identifier. so that VPN addresses cannot be added to loopback at all. > No, it doesn't fail. I tested it with kernel 2.0.36, it worked. It does not work under 2.2. To be honest, I am about to stop to understand you. You talk about 2.2, but all your tests are made for 2.0. 8) Alexey </verb> -- <sect1>A posting to the mailinglist by Peter Kese <tt/peter.kese@ijs.si/ explaining the "arp problem" (saved for posterity by Ted Pavlic, minor editing by Joe) Before we start, let's assume we have following network configuration for an LVS running VS-DR. <verb> client 10.10.10.10 gw 192.168.1.1 director 192.168.1.10 IP for admin (director IP) 192.168.1.110 VIP (responds to arp requests) real server 192.168.1.11 IP to which each service is listening (real-server IP) 192.168.1.110 VIP (DOES NOT respond to arp requests) </verb> The virtualserver is the combination of the director and the real-server running LVS. Or goal is: <enum> <item> Virtual server should respond to arp requests for both the VIP and the director IP. <item> The real-server should respond to arp requests for the real-server IP but NOT the VIP. <item> Gateway sends packets for the VIP to the director IP load balancer no matter what. </enum> Problem 1: Interface aliases Real-server and director need to have an interface with the VIP in order to respond to packets for virtual server. A real interface is not needed, an IP alias will do just fine and this interface alias could be either eth0:0 or lo:0. On the 2.0 kernels, the ARP responding ability of an interface alias (eg eth0:0) could either be enabled or disabled independantly of the main (eth0) interface. If you wanted eth0:0 not to respond to ARP requests, you could simply say: ifconfig eth0:0 192.168.1.2 -arp up Thus in the 2.0 kernels it is possible, on a real-server, to have the real-server IP (on eth0) respond to arp requests and for the VIP (on eth0:0) to not respond. In the 2.2 kernels this doesn't work any more. Whether the an interface alias responds to ARP requests or not, depends only on the way the real interface is configured. So if eth0 responds to ARP requests (which it normally will), eth0:0 carrying the VIP will also respond to ARP requests no matter what. This means an ethernet alias (eth0:0) is not permitted on real servers, because real servers should not respond ARP requests. On the other hand, loopback aliases never respond ARP requests, which means that the loopback alias (lo:0) must not be used on the director for the VIP. Problem 2: Loopback aliases I haven't done much checking on loopback interface problem, but it seems that if an alias is used on a loopback interface (as is required for VS-DR) on a real server running kernel 2.2.x, the whole ARP gets screwed. It appears that loopback interfaces get special ARP treatment in the kernel, so I suggest avoiding the loopback aliases as whole. The question now is: What kind of an interface can I use on real servers? As I already noted, eth0:0 alias can not be used, because such aliases respond to ARP requests. lo:0 aliases can not be used, because they make ARP problems too. In case of tunneling VS configuration, the answer is trivial: tunl0. But to be honest, tunl0 interface can also be used for direct routing. (from Joe, the dummy device is OK too) With direct routing, the only thing we need an interface for is to let kernel know we posses an additional IP address. This means, we can set up any kind of an interface, as long as it doesn't respond ARP requests. Instead of tunl0, you could also set up a ppp0, slip0, eth1 or whatever. I suggest setting up a tunl0: ifconfig tunl0 192.168.1.2 -arp up Problem 3: Real server ARP requests. Suppose we have set up a virtual server as described at the beginning. All computers are running, but no requests have been made. Then the client sends a request to the VIP. When the packet arrives to gateway, the gateway makes an ARP query for the VIP and the director responds. Gateway remembers the director's MAC address and sends the packet to the director. Director receives the packet, looks up its ipvsadm/LVS tables and chooses the real server and forwards the packet to the real server by direct routing or tunneling method. Real server receives the packet and generates a response packet with destination=client, source=VIP. (until now everything works correctly) When real server wants to send the response packet to the gateway, it finds out, that it does not know the gateway's MAC address. It sends an ARP request to the local network and asks for the gateway MAC address. This should look like: ARP, who has 192.168.1.1 (gw), tell 192.168.1.11 (real-server IP) But in reality, real server asks something like: ARP, who has 192.168.1.1 (gw), tell 192.168.1.110 (VIP), because it takes the source address from the packet it wants to send. Here the problems come in. Gateway receives the packet and responds to it, which is correct. But at the same time, gatweay does a little optimization. It finds out, that the real-server's MAC address is not listed in its ARP tables and adds the entry into the table, just in case it might need that address in the near future. The ARP request contained the VIP address and the real-server's MAC address, so from now on, the gateway will send all packets destined for the VIP to the real server instead (due to MAC address). This means all packets that follow will avoid the virtual server as whole and get responded by the real-server. If the real server's ARP request would be: ARP, who has 192.168.1.1 (gw), tell 192.168.1.11 (real-server IP) all this would not have happened. Therefore I have patched the 2.2 VS kernel in such a way, that it composes ARP requests based on the address of the interface selected by the routing tables instead of the address taken from the packet itself. In order for virtual server to work correctly, the real servers should have patched kernels as well, or at least copy the patched /usr/src/linux/net/ipv4/arp.c file to the real servers before compiling the kernels. Conclusion Those were my experience with ARP problems, and the 2.2 kernel virtual server. I think it would be wise to add this letter to the web site and notify the network developers about our findings at some point in time. Here are some golden rules I stick to, when I do virtual server configuration: <verb> Rule 1: Do not use lo:0 alias on the director. Use eth0:0 alias instead. Rule 2: Avoid using lo:0 alias, not even on real-servers. Use tunl0 or some other simulated interface on real servers instead. (Joe: use dummy0) Rule 3: Apply the VS patch to kernels on real servers. </verb> <sect1>random mailings on the arp problem (from Stephen Williams <tt/sdw@lig.net/, Stephen wrote one of the patches that stop devices in 2.2.x kernels from replying to arp requests) symptoms of real-servers arp'ing: If you don't use the patch you'll find that the 'active' box will bounce from machine to machine as each one sends an ARP reply that is heard last. Additionally you will get TCP Reset's as connections that were on one box suddenly start going to others. Very nasty and unusable. (Lars) I have thought about how the ARP problem can occur at all with direct routing, because I never noticed it. Then it occured to me that your virtual IP comes from the same subnet as the real IP of the LVS and also all the real servers share this media. To avoid the "ARP problem" in this case without adding a kernel patch or anything else, you can just add a direct route for the VIP using the real IP of the LVS as a gateway address on the router in front of the LVS. ("ip route VIP 255.255.255.255 real_ip" on a Cisco, or "route add -host VIP gw RIP" on Linux) Since I just used 2 ethernet cards and had the LVS act as gateway/firewall anyway, I never noticed the ARP problem. (We have 2 LVS in a standby configuration to eliminate the SPOF) and a reply from Wensong (just to show this subject isn't obvious) For the clients who reach the virtual server through the router, there is no problem if a static route for VIP is added. However, fot the clients who are in the network of virtual server, the "ARP problem" will arise. There is fight in ARP response, and the clients don't know send the packets to the load balancer or the real server. In my point of view, the VIP address is shared by the load balancer and real servers in VS-Tun or VS-DR, only the load balancer does ARP response for VIP to accept request packets, and the real servers has the VIP but don't, so that they can process packets destined for VIP. <sect1>Is the arp behaviour of 2.2.x kernel a bug? (Julian Anastasov replying to correct an error in a previous version of the HOWTO where I state that the dummy0 device in 2.2.x kernels does not arp. Julian wrote one of the real-server patches which fix the "arp problem"). <verb> > In fact, the documentation is incorrect. There is no difference, > all devices are reported in the ARP replies: lo, tunl and dummy. So, only > the ARP patch can solve the problem. This can be tested using this > configuration with any device (before the patch applied): > > Host A: > eth:x 192.168.0.1 > > Host B: > eth:x 192.168.0.2 > lo, dummy, tunl: 192.168.0.3 > > > On host A try: ping 192.168.0.3 > > Host B replies for 192.168.0.3 through 192.168.0.2 device > > So, the ARP problem means: "All local interfaces are reported" > until the ARP patch is used. In fact, all ARP patches which use IFF_NOARP > to hide the interface are incorrect. I don't expect them in the kernel. </verb> (Stephen WIlliams, who wrote another of the patches to fix the arp problem). <verb> >> Of course the ARP code in the kernel needs to be fixed so my filter code isn't >> needed. Still, I'm confused by this statement. The IFF_NOARP flag determines >> whether a device arp replies or not. What's wrong with honoring that? >> >> If you mean that arp replies should never be sent on another interface, that is >> what I currently believe to be correct. > (Julian) > My understanding is that 2.2.x ARP code is not buggy and > there is no need to be "fixed". I must say that your patch is > working for the LVS folks but not for all linux users. > > IFF_NOARP means "Don't talk ARP on this device", > from the 'man ifconfig': > > [-]arp Enable or disable the use of the ARP protocol on > this interface. > > So, where is the bug ? The ARP code never talks through > lo, dummy and tunl devices when they are set NOARP. It uses > eth (ARP) device. > > If You hide all NOARP interfaces from the ARP protocol > this is a bug. One example: > > +--------+ppp0 +------+ > | Host A |------------ppp link----------|ROUTER|------ The World > +--------+A.B.C.1 (www.domain.com) +------+ > |eth0 > |A.B.C.2 > | > |A.B.C.3 > +--------+ > | Host B | > +--------+ > > Is it possible after your patch Host B to access www.domain.com ? > How ? Host A doesn't send replies for A.B.C.1 through eth0 after > your patch. OK, may be this is not fatal. Tell it to all kernel > users. You hide all their NOARP interfaces. May be there are other > examples where this is a problem too. Or may be there is something > wrong in this configuration? > > I want to say that this patch hurts all users if present > in the kernel. On Nov 6 I posted one patch proposal to the > linux-kernel list which adds the ability to hide interfaces > from the ARP queries and replies. But the difference is that > only specified interfaces are not replied, not all NOARP > interfaces. Its arp_invisible sysctl can be used by LVS > folks to hide lo, tunl or dummy interfaces but this feature > doesn't hurt all kernel users. I think, this patch is more > acceptable and can be included in the 2.2 kernel, may be after > some tunning. And I'm still expecting comments from the net > folks and from all LVS users. </verb> -- <sect1>How to tell if an interface is replying to arp requests on the machine with that IP (usually the VIP) $ ping VIP look in /proc/net/arp for MAC address on a machine on a network (eg 192.168.1.0/24) to see which addresses are replying to arp requests $ ping 192.168.1.255 then before the arp tables expire (15secs - 2mins depending on the OS) $ arp -a <sect1>Arp caching defeats Heartbeat switchover From: Claudio Di-Martino <tt/claudio@claudio.csita.unige.it/ I've set up a VS using direct routing composed of two linux-2.2.9 boxes with the 0.4 patch applied. The load balancer acts as a local node too. I configured mon to monitor the state of the services and update the redirect table accordingly. I also configured heartbeat so that when the load balancer fails the second machine takes over the virtual ip, sets up the redirect table and starts mon. When the load balancer restarts, the backup reconfigures itself as a real server, drops the interface alias that carries the virtual ip, stops mon, clears the redirect table. Although the configuration of the two machines is set up correctly it fails to restore the load balancer due to arp caching problems. It seems that the local gateway keeps routing requests for the virtual ip to the load balancer backup. Sending gratuitous arp packets from the load balancer doesn't have effect since the interface of the backup is still alive and responding. Has anyone encountered a similar problem and is there a hack or a proper solution to take back control of the virtual ip? From: "Antony Lee" <tt/AntonyL@hwl.com.hk/ I am new to LVS and I have a problem in setting up two LVSes for failover issue. The problem is related to the ARP caching of the primary LVS' MAC address in the real servers and the router connected to the Internet. The problem leads all the Internet connections stalled until all ARP caching in Web Servers and router to be expired. Can anyone help to solve the problem by making some changes in the Linux LVS ? ( It is because I am not able to change the router ARP cache time. The router is not owned by the Web hosting company not by me.) In each LVS, there are two network card installed. The eth0 is connected to a router which is connected to the Internet. The eth1 is connected to a private network which is the same segment as the two NT IIS4. <verb> The eth0 of the primary LVS is assigned an IP address 202.53.128.56 The eth0 of the backup LVS is assigned an IP address 202.53.128.57 The eth1 of the primary LVS is assigned an IP address 192.128.1.9 The eth1 of the primary LVS is assigned an IP address 192.128.1.10 In addition, both primary and backup LVS have enabled the IPV4 FORWARD and IPV4 DEFRAG. In the file /etc/rc.d/rc.local the following command was also added: ipchains -A -j MASQ 192.168.1.0/24 -d 0.0.0.0/0 </verb> I use the piranha to configure the LVS so that the two LVS have a common IP address 202.53.128.58 in the eth0 as eth0:1. And have a IP address 192.128.1.1 in the eth1 as eth1:1 The pulse daemon is also automatically be run when the two LVSes were booted. In my configuration, the Internet clients can still access to our Web server with one of the NT was disconnected from the LVS. The backup LVS --CAN AUTOMATICALLY-- take up the role of the primary LVS when the primary LVS is shut down or disconnected from the backup LVS. However, I found that all the NT Web Servers cannot reach the backup LVS through the common IP address 192.128.1.1, and all the Internet clients stalled to connect to our web servers. Later, I found that the problem may due to the ARP caching in the Web Servers and router. I tried to limit the ARP cache time to 5 seconds in the NT servers and half of the problem has solved ,i.e. the NT Web servers can reach the backup LVS through the common IP address 192.128.1.1 when the primary LVS was down. However, it is still cannot be connected through the Internet clients when the LVS failover occur. (Wensong) I just tried two LVS boxes with piranha 0.3.15. When the primary LVS stops or fails, the backup will take over and send out 5 Gratuitous Arp packets for the VIP and the NAT router IP respectively, which should clean the ARP caching in both the web servers and the external router. After the LVS failover occurs, the established connections from the clients will be lost in the current version, and the clients need to re-connection the LVS. <verb> .. 5 ARP packets for each IP address, and 10 for both the VIP and the NAT router IP. I saw the log file as follows: Mar 3 11:12:14 PDL-Linux2 pulse[4910]: running command "/sbin/ifconfig" "eth0:5" "192.168.10.1" "up" Mar 3 11:12:14 PDL-Linux2 pulse[4908]: running command "/usr/sbin/send_arp" "-i" "eth0" "192.168.10.1" "00105A839CBE" "172.26.20.255" "ffffffffffff" Mar 3 11:12:14 PDL-Linux2 pulse[4913]: running command "/sbin/ifconfig" "eth0:1" "172.26.20.118" "up" Mar 3 11:12:14 PDL-Linux2 kernel: send_arp uses obsolete (PF_INET,SOCK_PACKET) Mar 3 11:12:14 PDL-Linux2 pulse[4909]: running command "/usr/sbin/send_arp" "-i" "eth0" "172.26.20.118" "00105A839CBE" "172.26.20.255" "ffffffffffff" Mar 3 11:12:17 PDL-Linux2 nanny[4911]: making 192.168.10.2:80 available </verb> I don't know if the target addresses of the 2 send_arp commands are set correctly. I am not sure if it is different when broadcast or source IP is used as target address, or any target address is OK. (Horms) Are there just 5 ARPs or 5 to start this and then more gratuitous ARPs at regular intervals. If the gratuitous ARPs only occur at fail-over then once the ARP caches on hosts expire there is a chance that a failed host - whose kernel is still functional - could reply to an ARP request. From: <tt/wanger@redhat.com/ When we put this together, I talked to Alan Cox about this. His opinion was that send 5 ARPs out at 2 seconds apart. If there is something out there listening and cares, then it will pick it up. THe way piranha works, as long as the kernel is alive, the backup (or failed node) will not maintain any interfaces that are Piranha managed. In other words, it removes any of those IPs/interfaces from its routing table upon failure recovery. <sect1>More on the arp problem<label id="arp"> ARP requests/replies are thought of as coming from a device and people make statements like "the dummy device in 2.0.x kernels does not reply to arp requests while the same device in 2.2.x kernels does reply". It is the kernel that handles arp requests according to a set of rules and not the device. The code for the dummy device is the same in 2.0.x and 2.2.x kernels and is not responsible for the change in arp behaviour. The RPC for ARP is at ftp://ftp.isi.edu/in-notes/std/std37.txt. (also see rfc826 and rfc1122). The model system used there is 2 machines on a single ethernet. It doesn't shed any light on the implementation of ARP on multi-interface systems like LVS. <sect1>Properties of devices for the VIP In a previous version of the HOWTO I stated that the dummy0 device did not arp in 2.2.x kernels and therefore could be used as the device for the VIP on an unpatched 2.2.13 real-server. Julian Anastasov replied that they did arp (see below for his posting and the ensuing discussions). I hadn't actually tested whether the dummy0 device arp'ed but had concluded that it wasn't arp'ing because I had a working LVS using the dummy0 interface for the VIP on unpatched 2.2.x real-servers and because as everyone knows ;-) an LVS needs to have a non-arp'ing device on the VIP of the real-servers. I had a VS-DR LVS which worked with dummy0, lo:0 and tunl0 as the VIP device and which on further testing, I found also worked with eth0:1 or eth1 as the VIP device on 2.2.13 real-servers. Whatever the arp'ing status of dummy0, lo:0 or tunl0, clearly eth1 replies to arp requests, so despite the conventional wisdom, it is possible to build an LVS with arp'ing VIP's on the real-servers. On investigating why this LVS worked, I found that the MAC address for the VIP in the client's arp cache (# arp -a) was always the director. I assume this was because the director is 3-4x the speed of the other machines in the LVS and it replies to arp requests first for the VIP (another posting from Stephen WIlliams says that the address which replies last is stored in the arp cache - we'll figure out what's really going on here eventually). On another LVS where the real-servers were all identical hardware with 2.2.13 unpatched kernels, one particular real-server always was the machine in the client's arp cache for the VIP (to check, delete entry for VIP with arp -d, then ping again, then look in arp cache). I found that I could get a working LVS using almost anything to hold the VIP on the real-servers, including eth0:1 and eth1 (another NIC in the real-server). These devices carrying the VIP were pingable from the client and I could get the corresponding MAC addresses in the arp table of the client if the director was not setup with a VIP. When I setup a working LVS this way, I found each time that the MAC address for the VIP in the client's arp cache was the director's MAC address. For some reason, that I don't know, whenever the client does an arp request for the VIP, it gets the director's MAC address. Possible reasons for the MAC address of the director always being associated with the VIP in my LVS - 1. I configure the director first (I can't imagine the client asking for the MAC address of the VIP until it makes a request - this doesn't happen till after I've configured the real-servers). 2. The director is 3 times faster (CPU speed) than the next machine in the LVS and it always replies to arp request first. 3. I was lucky. Since you can make a working VS-DR LVS with the real-server VIP on an arp'ing eth0:1 device I decided that the relevent piece of information about arp'ing was - an LVS will work if the client always gets the MAC address of the director when it asks for the MAC address of the VIP This is easy - you tell the client (or the router) the MAC address of the VIP with arp -s or arp -f . here's my /etc/ethers lvs.mack.net 00:A0:CC:55:7D:47 After installing the MAC address of the DIP (director) as the MAC address of the VIP (lvs) in the arp table ($arp -f /etc/ethers) I get <verb> client:/usr/src/temp/lvs# arp -a real-server1.mack.net (192.168.1.1) at 00:90:27:66:CE:EB [ether] on eth0 lvs.mack.net (192.168.1.110) at 00:A0:CC:55:7D:47 [ether] PERM on eth0 director.mack.net (192.168.1.10) at 00:A0:CC:55:7D:47 [ether] on eth0 </verb> notice the "PERM" in the VIP entry on the client. removing the permanent entry <tscreen><verb> client:/usr/src/temp/lvs# arp -d lvs.mack.net client:/usr/src/temp/lvs# arp -a real-server1.mack.net (192.168.1.1) at 00:90:27:66:CE:EB [ether] on eth0 lvs.mack.net (192.168.1.110) at <incomplete> on eth0 director.mack.net (192.168.1.10) at 00:A0:CC:55:7D:47 [ether] on eth0 </verb></tscreen> If I edited /etc/ethers changing the MAC address of lvs to anything else, the LVS did not work anymore. So the arp information is coming from /etc/ethers rather than some uncontrolled variable I'm not aware of. I had thought that in an LVS with the VIP on real-servers on an arping device that the VIP would hop from one machine to another (see the postings in the MISC section). Since naturally occuring LVS's with arping VIP's on real-servers existed and worked well (mine), I set up an LVS by making a permanent entry for the VIP of the director in the arp cache of the client (router). This can be done by <verb> $ arp -f /etc/ethers or $ arp -s 192.168.1.110 MAC_ADDRESS </verb> There are 2 results of this 1. the real-servers can have the VIP on an an arp'ing device (eg eth0:1, eth1) - you don't need lo or dummy0, tunl0 for real-servers with 2.0.36 and 2.2.x kernels. 2. If two (or more) directors are setup in failover mode, the mechanism by for changing the VIP from one to another is broken by making a permanent entry for VIP on the director in the arp cache of the router. This is not a problem for a test setup to demonstrate an LVS but may be a problem in a high availability environment (a solution may be found n the meantime too). The normal method for changing drectors (eg with heartbeat) includes a gratuitous arp. To force a gratuitous arp (Julian) You can use Yuri Volobuev's send_arp.c from the 'fake' package or Alexey Kuznetsov's arping from its iputils package: <verb> fake - http://vergenet.net/linux/fake/ iputils - ftp://ftp.inr.ac.ru/ip-routing/iputils-ss991024.tar.gz (iputils is also used for IPAT, IP address takeover)) </verb> Here's some tests I did <verb> LVS equipment: 2.2.13 client, and 0.9.4/2.2.13 director. 2 real-servers a) 2.0.36 kernel, libc5, gcc-2.7.2.3, net-tools 1.42. b) 2.2.13 kernel, glibc, gcc-2.95, net-tools 1.52 </verb> Experiment 1: Result - arp'ing is independant of [-]arp Summary: the -arp/+arp option for ifconfig had no effect on any devices back to 2.0.36 kernels with net-tools 1.42. If it normally arps then -arp had no effect, if it normally doesn't arp, than "arp" doesn't turn it on (data below). Method: IP=192.168.1.1/24 with VIP=192.168.1.110/32. The VIP was on dummy0. The test was to see if the VIP was pingable from another (external) machine on the 192.168.1.0/24 network or pingable from the machine itself (ie internally from the console). (I assume I had a route add -host for the VIP although I didn't record this). The test was done with ifconfig using arp or -arp (the output of ifconfig -a didn't change) <verb> -----2.0.36------- -----2.2.13------ ping from internal external internal external VIP device dummy ARP + - + + NOARP + - + + down - - - - (control) </verb> Experiment2: Can the VIP be on a separate NIC? Summary: yes, as long as the NIC doesn't have a cable plugged into it. Method: same as above except VIP on eth1 (another NIC). <verb> -----2.0.36------- ping from internal external VIP device eth1 has cable connected to 192.168.1.0 network eth1 ARP + + NOARP + + eth1 cable to network removed eth1 ARP + - NOARP + - works as real-server in LVS - yes </verb> One of the reasons an no-arp interface is used on the real-server is that it is not visible to the rest of the network. Does the LVS work if the eth1 VIP on the real-server is not visible to the rest of the network? Conclusion: for 2.0.36 dummy0 doesn't arp, and eth1 does arp. the arp/-arp option to ifconfig has no effect on arp behaviour. LVS works with both dummy0 and eth1, I assume since VIP need only be resolved as local on the real-server and does not need to be visible to the network. Experiment 3: What devices and netmasks are neccessary for a working LVS? Using the /etc/ethers approach for setting the MAC address of the VIP I then set up an LVS with pair of real-servers serving telnet. All IPs are 192.168.1.x, all machines have a route to 192.168.1.0 via eth0. There is no default route. <verb> 1. 2.0.36, libc5, gcc 2.7.2.3, net-tools 1.42 2. 2.2.13, glibc-2.1.2, gcc-2.95, net-tools 1.52 </verb> with the following devices holding the VIP, tunl0, eth0:1, lo:0, dummy0, eth1. In each case there was no route entry for the VIP device and there was no cable connected to eth1 when it was used for the VIP. The table below shows whether the LVS worked. The VIP is installed with ifconfig $DEVICE 192.168.1.110 netmask $NETMASK broadcast $BROADCAST <verb> with $NETMASK="255.255.255.255" $BROADCAST="192.168.1.110" or $NETMASK="255.255.255.0" $BROADCAST="192.168.1.255" </verb> the result belong to 1 of 3 groups <verb> + works fine - doesn't work (at $ prompt on client get "unable to connect to remote host. Protocol not available" then client returns to regular unix $ prompt) hang - client hangs, real-server cannot access network anymore, have to run rc.inet1 from console prompt on real-server to start network again. </verb> netmask of VIP=255.255.255.255 (normal LVS setup) <verb> LVS type -----VS-Tun------ ----VS-DR------ kernel 2.0.36 2.2.13 2.0.36 2.2.13 VIP on tunl0 + + + + eth0:1 + - + + lo:0 + - + + dummy0 + - + + eth1 + - + + </verb> netmask of VIP=255.255.255.0 (not normally used for LVS) <verb> VIP on tunl0 + + + + eth0:1 + - + + lo:0 + hangs + hangs dummy0 + - + + eth1 + - + + </verb> It would seem that any device and any netmask can be used for the VIP on a 2.0.36 real-server for both VS-Tun and VS-DR. For 2.2.13 real-server, VS-Tun, VIP on a tunl0 device only, any netmask (ie you need tunl0 on VS-Tun with 2.2.x kernels) <verb> VS-DR, lo:0 device netmask /32 only all other devices any netmask </verb> For VS-DR then on solaris/DEC/HP/NT... LVS can probably use a regular eth0 device rather than an lo:0 device (more work for Ratz to do :-). Does anyone know why the lo:0 device has to be /32 for VS-DR on kernel 2.2.13 while the other devices can be /24? Jean-Francois Nadeau <tt/jna@microflex.ca/ 6 Dec 99 In kernel 2.2.1x with a virtual interface on lo:0 and netmask of 255.255.255.0 that the interface no longer arps. Does anyone know why only the tunl0 device works for VS-Tun on 2.2.x kernels? Experiment 4: Effect of route entry for VIP and connection to VIP The VIP normally has an entry in the routing table eg route add -host 192.168.1.110 $DEVICE I found in Experiment 2 that a route entry was not neccessary for the LVS to work when the real-server had the VIP on eth0:1. Since I had always used a route entry for the VIP I wanted to find out when it was needed. The same LVS was used as for Experiment 3. The variables were <verb> 1) a route entry/no route entry for VIP/32 2) for eth1 whether the NIC was connected to the network by a cable. kernel ------2.0.36------- -------2.2.13------- VIP eth1 eth1_nc eth0:1 eth1 eth1_nc eth0:1 no route LVS + + + + + + ping internal - - - + + + ping external + - + + + + route LVS + + + + + + ping internal + + + + + + ping external + - + + + + </verb> Conclusion 1: LVS works when for both cases of route/no_route for the VIP for eth0:1 and eth1 (ie you don't need a route entry for the VIP on the real-servers). Conclusion 2: having a network cable/no network cable does not affect whether the LVS works. Conclusion 3: for 2.0.36 kernels you can choose to have the VIP pingable from the outside world but not pingable by the local host by having it on eth1 with a cable connection (this seems wierd and I can't think of any use for it just yet) or the reverse - pingable from the localhost but not by the external world by not have a cable connection. (Note: using a hosts routable IP as the target - the IP on eth0 say - you can make a host unpingable from the console if you down the lo. The host is still pingable from elsewhere on the net.) <sect1>Topologies for VS-DR and VS-Tun LVS's<label id="topology"> <sect2>Traditional The conventional VS-DR/VS-Tun topology which allows maximum scalability has each real-server with its own default gateway (to a router). (In a routerless test setup, the client would be the default gateway for the real-servers. In a disk- or compute-bound situation, only one router may be needed. The changes in topology/routing are made by changing the IP of the default gw for the real-servers) Some method of handling the arp problem is needed here. The packets sent to the real-servers from the director, generate replies which go directly to the client. Failure messages (eg if a real-servers is not available) do not get returned to the director, who cannot tell if a real-server has failed (see discussion of <ref id="agent" name="monitoring agents">). <verb> -------------clients----------------------- | | | | (router) (router)(router)(router) | | | | _________ | | | | | | | VIP | | | | director |--- DIP | | | |__________| | | | | | | | | | | | | --------------------------------- | | | | | | | | | | | | | | | RIP1 RIP2 RIP3 | | | VIP VIP VIP | | | _____________ _____________ _____________ | | | | | | | | | | | | | real-server | | real-server | | real-server | | | | |_____________| |_____________| |_____________| | | | | | | | | | | | ---------- | | | ----------------------------------- | ---------------------------------------------------------- </verb> <sect2>Director sees replies (from Julian Anastasov) This discussion led to Julian's <ref id="martian" name="martian modification">. If the default gw for each real-server is changed to the DIP (see the Martian modification section) then 1. The director has to handle the reply packets as well as in the incoming packets, doubling the network load. 2. The director sees all the reply packets. Connection failure can be detected (in principle). <verb> clients | router | __________ | | | | VIP | director |--- DIP |__________| | | | ------------------------------------ | | | | | | RIP1 RIP2 RIP3 VIP VIP VIP _____________ _____________ _____________ | | | | | | | real-server | | real-server | | real-server | |_____________| |_____________| |_____________| </verb> <verb> >From: Horms <tt/horms@vergenet.net/ > >Hi, I have been setting up a test network to benchmark IPVS, >the topology is as follows. > > node-1 node-6 node-7 > (client) (client) (client) > | | | client-net > ---------+---------+----------+------ 192.168.2.0/24 > | > node-3 (router) > | server-net > ------+--------+----------+--- 192.168.1.0/24 > | | | > node-2 node-4 node-5 > (IPVS) (server) (server) > > >The question that I have is that the network I would really like >to be testing is; > > node-1 node-6 node-7 > (client) (client) (client) > | | | client-net > ---------+---------+----------+------ 192.168.2.0/24 > | > node-2 (IPVS) > | server-net > ---------+-----+----+--------- 192.168.1.0/24 > | | > node-4 node-5 > (server) (server) > </verb> .. > other than using NAT, which has > performance problems, is this possible? I tried this topology > with direct routing and packets from the clients were multiplexed > to the servers fine, but return packets from the servers to the > client were not routed by the IPVS box. (Lars) Yes. The LVS box silently drops the return packets, since they have a src ip which is also bound as a local interface on the LVS. This is meant to be a simple anti-spoofing protection. (Note from Joe - the return packet from the real-server has src=VIP, dest=CIP. If this packet is routed via the director, which also has the VIP, the director will be receiving a packet from another machine with the the src being an one of its own IPs and the director will drop the packet). You can enable logging these packets via echo 1 >/proc/sys/net/ipv4/conf/all/log_martians The only way around this with current Linux kernels is to disable the check in the kernel source or to use a separate box as the outward gateway. (Which is how DR is meant to be used for full performance) <verb> > This is not a problem as such as it probably makes a lot of sense > on not to use an IPVS box as your gateway router, </verb> Actually it makes a lot of sense to do just that IMHO. Less points of failure, less hard- & software to duplicate in a failover configuration. from: Lars Marowsky-Bree <tt/lmb@teuto.net/ To: Ray Bellis <tt/rpb@community.net.uk/ <verb> > It needs to be made more explicit in the documentation that LVS-DR will > *only* work if you have a different return path. ... or if you have a suitably patched kernel. > We spent several man days trying to get this to work before figuring out why > the packets were being dropped, at which point we had no alternative but to > use LVS-NAT instead. </verb> I agree. We still assume too much knowledge on the network admin side. <verb> > FYI, we have our LVS system working now, with LVS redundancy achieved by > running OSPF routing (gated) on the LVS-NAT servers and having the VIP > within the same IP subnet as the RIPs so that IGP routing policies > automatically determine which LVS router the packets arrive on. </verb> Yes, thats one option. Even better than heartbeat and IPAT, if all your systems support running a routing protocol. (IPAT = IP address takeover, part of heartbeat) (In essence, heartbeat & IPAT is nothing but reinventing a subset of the functionality of a hardened routing protocol like OSPF/RIPv2/EIGRP) <sect2>On other schemes for director/real-servers to exchange roles<label id="promote"> Julian Anastasov <tt/uli@linux.tu-varna.acad.bg/ has pointed out on the mailing list that the prototype LVS can be redrawn as <verb> | | | client | |________| | | (router) | | ------------------------------------ | | | | | | DIP, VIP RIP1, VIP RIP2, VIP ____________ ______________ ______________ | | | | | | | director | | real-server1 | | real-server2 | |____________| |______________| |______________| </verb> and that any real-server is in a position to replace a failed director. No-one has bothered to write the code for this. It seems it's easier do have extra boxes in the director role (ready for failover) and others in real-server role. It's easier to wheel in another box for a spare director than to configure real-servers to do two jobs reliably. To: Wensong Zhang <tt/wensong@iinchina.net/ <verb> > The director and the backup are in a shared > network for incoming traffic, the backup sniff packets and change its > connection state the same as the director (because the director is just on > half client-to-server connection in LVS/TUN and LVS/DR), then drop > packets. > It needs some investigation and probably lots of additional code too. ;-) </verb> I don't even think so - the main trick is getting the kernel to sniff the packets, which is probably quite easy with a little messing around. Not sending the packets out again (which would confuse the real-servers) is easy with a ipchains output rule which silently drops them. This doesn't work with a switch though, you need a shared network like a hub. However, I have been talking with rusty about this. The problem is more general - HA shared-state firewalls are asked for all the time, so we want to do a generic thing for everything which builds upon Netfilter's state machine. This would not only cover LVS, but also masquerading and packet filtering in general. We intend to discuss this in greater detail at the Ottawa Linux Symposium latest. <verb> > You can see,the connections depend on the initalize status and realsevers > realtime status. So another method is that when Director is down, backup-sever > setup the ipvs with the connections,but it seems too late. How do you think > about this? </verb> TCP/IP should be able to cope with a few seconds delay and lost packets. You want to heartbeat once per second and take over after 3-4s though - this usually means takeover is complete in <10s, which TCP/IP should swallow. <sect2>Geographically distributed LVS From: Michael Sparks <tt/zathras@epsilon3.mcc.ac.uk/ <verb> > I'm curious about the physical architecture of a cluster of servers > where "the real-servers have their own route to the client." (Like in > LVS-DR and LVS-Tun) How have people achived this in real life? Does > each real server actually have it's own dedicated router and Internet > connection? Do you set up groups of real servers where each group > shares one line? </verb> It could do or it can share things. We've got 3 LVS based clusters, based around VS-Tun. The reason for this is because one of the clusters is at a different location (about 200 miles from where I'm sitting) , and this allows us to configure all the real-servers in the same way thus: <verb> tunl0:1 - IP of LVS balanced cluster1 tunl0:2 - IP of LVS balanced cluster2 tunl0:3 - IP of LVS balanced cluster3 (remote) </verb> The only machines that ends up getting configured differently then are just the directors. So whilst machines are nominally in one of the three clusters, if (say) the remote cluster is overloaded, it can take advantage of the extra machines in the other two clusters, which then reply directly back to the client - and vice versa. In that situation a client in (say) Edinburgh, could request an object via the director at Manchester, and if the machines are overloaded there, have the request forwarded to London, which then requests the object via a network completely separate from the director's and returns teh object to the client. That UK Nat cache likely to be introducing another node at another location in the country at some point in the near future which will be very useful. (The key advantage is that at each location we gain X more Mbit/s of bandwidth to utilise making service better for users.) <sect1>A discussion about the arp problem (Joe and Julian) <verb> >(Julian Anastasov <tt/uli@linux.tu-varna.acad.bg/) >There is no difference between devices in 2.2.x, all devices >are reported in the ARP replies: lo, tunl and dummy. >This can be tested using this configuration with any device: > >Host A: > eth:x 192.168.0.1 > >Host B: > eth:x 192.168.0.2 > lo, dummy, tunl: 192.168.0.3 > > >On host A try: ping 192.168.0.3 > > Host B replies for 192.168.0.3 through 192.168.0.2 device > >The ARP problem means: "All local interfaces are reported" >until the ARP patch is used. In fact, all ARP patches which use IFF_NOARP >to hide the interface are incorrect. I don't expect them in the kernel. </verb> ARP problem, some rules: ARP responses <itemize> <item> all local IP addresses are replied: lo, eth, tunl*, dummy* but with some exceptions (see the next rules) <item> 127.0.0.0/8(LOOPBACK) and 224.0.0.0/4(MULTICAST) are not replied <item> there is one exception for the "lo" interface: it is possible the kernel to ignore the ARP request if the source IP is from the same net as the net used to configure "lo" alias. The specified network is treated as local. </itemize> For example: real-server# ifconfig lo:0 192.168.1.1 netmask 255.255.255.0 broadcast 192.168.1.255 up "real" treats all packets with source addr from 192.168.1.0/24 which come from the other devices (eth0) as invalid, i.e. source address validation works in this case and the ARP request are not replied. The kernel thinks: "The incoming packet arrived with saddr=local_IP1 and daddr=local_IP2(VIP), so it is invalid". By this way the host from the LAN can't talk to the real server if its lo alias is configured with netmask != 255.255.255.255 <verb> ifconfig dummy0 192.168.1.1 netmask 255.255.255.255 </verb> registers only 192.168.1.1 as local ip but: <verb> ifconfig lo:0 192.168.1.1 netmask 255.255.255.0 </verb> all 256 IPs are local. All IFF_LOOPBACK devices treat all IPs as local according to the used netmask. > (I assume IFF_LOOPBACK devices are lo, lo:0..n?) Yes, currently only lo is marked as loopback. It is used to mark whole subnets as local. > lo:0 is not marked as loopback? lo:0 is just attached IP address to the same device "lo". You can try "ifconfig lo:0 192.168.0.1 netmask 255.255.255.255" and display the interfaces using "ifconfig". There is LOOPBACK flag for lo:0 which is inherited from the device "lo". In Linux 2.2 all aliases inherit the device flags. Only the IFF_UP flag is used to add/delete the aliases. <verb> > Assume VS-DR with VIP, RIPs all on the same /24 network on eth0 devices, > real-servers all have lo:0 with VIP/24 and have the standard 2.2.x kernel > (no patches to hide interfaces). Router says "who has VIP", the arp > request arrives at the real-servers via eth0. Device lo:0 finds arp request > which arrived on eth0 from router is on the same subnet as lo:0 and does > not reply to the arp request. </verb> Before checking if to answer the ARP the routing tables are checked, i.e. the source validation of the packet is performed. If 192.168.0.2 asks "who-has 192.168.1.1 tell 192.168.1.2" the real servers assumes that this is invalid packet, i.e. from one local IP to another local IP (from me to me => drop). <verb> > I notice that with the 2.2.x kernel, that lo:0 has to have > netmask=255.255.255.255 to work, whereas with the 2.0.x kernels (where > lo:0 doesn't reply to arp requests), that lo:0 can have the VIP on a > 255.255.255.0 netmask and still work. </verb> The rule is to use netmask 255.255.255.255 and to hide lo. The ARP works in different way in 2.2. It looks the "local" table to validate the source of the ARP request and after that it lookups the same table to check if daddr of the ARP request is local ip. ARP requests: - all local addresses can be used by the kernel to announce them as the source for the ARP request. <verb> > is it OK to say > > the kernel can (does?) use all local addresses as the source > of ARP requests </verb> It can and does. The real server thinks that it can use any local ip address as saddr in the ARP request and the answer will be returned back if this ip is uniq in the LAN. <verb> > (do you mean "the real-server will receive a reply if the s_addr is > unique in the LAN"?) </verb> The real server will receive answer if it uses RIP as saddr in the ARP request because the VIP(HIP) is hidden or when using transparent proxy because it is not local (the VIP). Real server must know how to ask (using uniq IP) or the trafic for the asked IP (ROUTER) will be blocked. But the hidden addresses are not used because they are not uniq (2.2.14) and the answer will be returned to the Director. > (do you mean "the non-hidden VIP on the director"?) Yes, when the real server ask "who-has ROUTER tell VIP" the ARP reply is received in the Director and the transmission in the real servers is stopped. The ROUTER sends everything destined to VIP to the Director. This is true for all clients on the LAN too if they are not in this cluster (if they don't handle packets for VIP). <verb> > (I would have thought that the main device on each NIC, eg eth0, eth1 > would have been used as the source address). </verb> No, it is extracted from the outgoing datagram and if saddr is local ip it is used. But if this is not local ip, i.e. when using transparent proxy or the address is marked as hidden the main device ip is used. <verb> >(how is arping part of transparent proxy?) </verb> It is not. When VIP is not local IP address in the real server this IP is not used from the ARP code. It is not in the "local" table. But TCP, UDP and ICMP use it via transparent proxy support. They are extracted from the outgoing packet. <verb> > what is "They"? the source addresses? When you say "extracted", do you > mean "removed from packet" or "looked at/detected" </verb> The saddr from the data packet is used to build the ARP request. <verb> > We tell the kernel > that these addresses are not uniq by setting > <interface>/hidden=1 (starting with kernel 2.2.14). > By this way the kernel select the devices primary IP > as the source of the ARP request. > (the kernel can use any local address as s_addr but the > code for hiding IPs from arp requests prevents the > kernel from using hidden addresses as > s_addr in an arp request?) </verb> Yes, the code to hide the addresses is already part of the source address autoselection (saddr in the ARP request in our case). We never autoselect hidden addresses, i.e. if the source address is not specified from the higher level. The code to hide interface: <verb> - ignores ARP replies for hidden local addresses - doesn't select hidden local addresses as source of the ARP request - doesn't autoselect hidden local addresses for the IP level > > > > We expect it is uniq in the LAN. > > (do you mean - > we expect you've set up your network properly and that > you don't have the same RIP on 2 real-servers? :-) > ) </verb> The LVS administrator must ensure that the RIPs are uniq, only the VIP is shared. > We expect it is uniq in the LAN. We tell the kernel that these addresses are not uniq by setting <verb><interface></verb>/hidden=1 (2.2.14). By this way the kernel select the devices primary IP as the source of the ARP request. We expect it is uniq in the LAN. So, the recommendation for using the "lo" interface in the real servers is: - use netmask 255.255.255.255 when configuring lo alias. By this way source validation doesn't drop the incoming packets to this IP. LVS users usually define the net route through the eth interface, so we can talk to other hosts from this network, for example to send the packets to the client through the default gateway. It is not needed to configure the alias with mask != 255.255.255.255 So, the interfaces which can be used in the real servers to listen for VIP are: <verb> - lo aliases with netmask 255.255.255.255 - tunl* - dummy* </verb> All these devices must be marked as hidden to solve the ARP problem when using Linux 2.2. In the Director: there is no problem to configure the VIP even on lo alias or dummy interface. If the interface is not marked as hidden this VIP is visible for all hosts on the LAN. <sect1>ATM/ethernet and router problems<label id="ATM"> LVS has only been tested on ethernet. One person had an ATM setup which didn't work with VS-DR as the ATM router expects packets from the VIP to have the same MAC address (in VS-DR packets coming from the VIP could have the MAC address of any of the real-servers). Apparently this is not easily fixable in the ATM world. It should be possible to use one of Julian's <ref id="martian" name="martian modifications"> to make VS-DR work on ATM, but the person with the ATM setup disappeared off the mailing list without us convincing him of the joy in having the first ATM LVS. Other people have found similar problems with ethernet - From: Kyle Sparger <tt/ksparger@dialtoneinternet.net/ I don't know if someone has gone over this, but here's a consideration I've come across when setting up LVS in DR mode: When the real servers reply, cisco routers (ours do, at least) will pick up on the fact that it's replying from a different MAC address, and will start arping soon thereafter. This is sub-optimal, as it causes a constant flood of arp requests on the network. Our solution has been to hardcode the MAC address into the router, but this can cause other issues, for example during failover. That can be worked around, as you can set the MAC address on most cards, but that in itself may cause other issues. Has anyone else experienced this? Has anyone else come up with a better solution than hardcoding it into the router? <sect>Collect Hardware <sect1>minimum setup You will need a minimum of 3 machines (you can do it with 2, but this doesn't demonstrate how to scale up an LVS farm, read <ref id="localnode" name="localnode"> to see how this is done). You need <itemize> <item> Director: running Linux patched kernel 2.2.x or 2.4.x (VS-NAT is still in development for 2.4.x, in Feb 2001, check on the mailing list for latest info). <item> Client: any machine, any OS, with a client for the service (eg netscape, fetch/xterm/...) <item> Real-server(s) <itemize> <item>VS-NAT - any machine, any OS, running some service of interest (eg httpd, ftpd, telnetd, smtp, nntp, dns, daytime ...) <item>VS-Tun - real-server needs to run an OS that can tunnel (only Linux so far). <item>VS-DR - OS known to work are listed in <ref id="VS-DR" name="VS-DR"> </itemize> </itemize> <sect1>Gotchas<label id="gotcha"> You need an outside client. The LVS functions as one machine. You must access the LVS from a client that is NOT a member of the LVS. You cannot access an LVS controlled service (eg http, telnet) from any of the machines in the LVS; access from the director will hang, access from a real-server will connect to the service locally, bypassing the LVS. Minimum 3 machines: client, director, real-server(s) <sect1>Test with telnet (or netcat) All testing of your LVS should be done with telnet. Telnet has <itemize> <item>a simple client (available on all OSs) <item>a simple protocol (one port) <item>exchanges are all ascii (can be watched with tcpdump) <item>non-persistent connection (you can test round robin scheduling) <item>telnetd usually listens to all IP's on the server (i.e. to 0.0.0.0) at least under inetd <item>most importantly, when you get a connection, you'll see an entry in the ActConn column of the output of ipvsadm. </itemize> For security reasons, you'll be turning off telnet later, but whenever you're testing your LVS or your new service, always look to see if telnet works if you're having trouble. If telnet is not being LVS'ed then you should go back and fix that first. Another useful client is netcat. See <ref id="netcat" name="examples using netcat as a client with ftp">. <sect>Choose LVS Forwarding Type<label id="forwarding"> <sect1>Comparison of VS-NAT, VS-DR and VS-Tun The instructions in the following sections show how to set up LVS in VS-NAT, VS-Tun and VS-DR modes for the service telnet. If you just want to demonstrate to yourself that you can setup an LVS, then VS-NAT has the advantage that any OS can be used on the real-servers, and that no modifications are needed for the kernel on the real-server(s). If you have a linux machine with a 2.0.x kernel, then it can be used as a real-server for an LVS operating in any mode without any modifications. Because VS-NAT was the first mode of LVS developed, it was the type first used by people setting an LVS. For production work VS-DR scales to much higher throughput and is the most common setup for production. However for a simple test, VS-NAT only requires patching 1 machine (the director) and an unmodified machine of any OS for a real-servers. After a simple test, unless you need the features of VS-NAT (ability to use real-servers that provide services not found on Linux machines, port remapping, real-servers with primitive tcpip stacks - e.g./printers, services that initiate connect requests such as identd), then it would be best to move to VS-DR. Here are the constraints for choosing the various flavors of LVS: VS-NAT (network address translation), VS-Tun (tunnelling) and VS-DR (direct routing). <verb> VS-NAT VS-Tun VS-DR server OS any must tunnel most server mods none tunl must not arp lo must not arp port remapping yes no no server network private on internet local (remote or local) - server number low high high client connnects to VIP VIP VIP server default gw director own router own router </verb> <sect1>Expected LVS performance<label id="performance"> If you are just setting up an LVS to see if you can set one up, then you don't care what your performance is. When you want to put one on-line for other people to use, you'll want to know the expected performance. On the assumption that you have tuned/tweeked your farm of real-servers and you know that they are capable of delivering data to clients at a total rate of bits/sec or packets/sec, you need to design a director capable of routing this number of requests and replies for the clients. Before you can do this, some background information on networking hardware is required. At least for Linux (the OS I've measured, see <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance data for single real-server LVS">), a network rated at 100Mbps is not 100Mbps all the time. It's only 100Mbps when continuously carrying packets of mtu size (1500bytes). A packet with 1 bit of data takes as long to transmit as a full mtu sized packet. If your packets are <ack>s, or 1 character packets from your telnet editing session or requests for http pages and images, you'll barely reach 1Mbps on the same network. On the <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance page">, you'll notice that you can get higher hit rates on a website as the size of the hit targets (in bytes) gets smaller. Hit rate is not neccessarily a good indicator of network throughput. Tcpip can't use the full 100Mbps of 100Mbps network hardware, as most packets are paired (data, ack; request, ack). A link carrying full mtu data packets and their corresponding <ack>s, will presumably be only carrying 50Mbps. A better measure of network capacity is the packet throughput. An estimate of the packet throughput comes from the network capacity (100Mbps)/mtu size(1500bytes) = 8333 packets/sec. (Thanks to Pat O'Rourke for the correct number here. In previous versions of this document, I've stated "800" here. This was because my calculator didn't accept the last "0" when I entered 10^9). Thinking of a network as 100Mbps rather than ca.8000packets/sec is a triumph of marketing. When offered the choice, everyone will buy network hardware rated at 100Mbps even though this capacity can't be used with your protocols, over another network which will run continuously at 8000packets/sec for all protocols. Only for applications like ftp will near full network capacity be reached (then you'll be only running at 50% of the rated capacity as half the packets are <ack>s). A netpipe test shows that some packets must be "small". Julian's <ref id="show_traffic" name="show_traffic script"> shows that for small packets (<128bytes), the throughput is constant at 1200packets/sec. As packets get bigger (upto mtu size), the packet throughput decreases to 700packets/sec, and then increasing to 2600packets/sec for large packets. (My real-servers are 75MHz pentiums and can't saturate the 100Mbps network.) The constant througput in packets/sec is a first order approximation of of tcpip network throughput and is the best information we have to predict director performance. In the case where a client is in an exchange of small packets (<mtu size) with a real-server in a VS-DR LVS, each of the links (client-director, director-realserver, realserver-client) would be saturated with packets, although the bps rate would be low. This is the typical case for non-persistent http when 7 packets are required for the setup and termination of the connection, 2 packets are required for data passing (eg the request GET /index.html and the reply) and an <ack> for each of these. Thus only 1 out of 11 packets is likely to be near mtu size, and throughput will be 10% of the rated bps throughput even though the network is saturated. The first thing to determine then is the rate at which the real-servers are generating/receiving packets. If the real-servers are network limited, i.e. the real-servers are returning data in memory cache (eg a disk-less squid) and have 100Mpbs connections, then each real-server will saturate a 100Mbps link. If the service on the real-server requires disk or CPU access, then each real-server will be using proportionately less of the network. If the real-server is generating images on demand (and hence is compute bound) then it may be using very little of the network and the director can be handling packets for another real-server. The forwarding method affects packet throughput. With VS-NAT all packets go through the director in both directions. As well the VS-NAT director has to rewrite incoming and reply packets for each real-server. This is a compute intensive process. In a VS-DR or VS-Tun LVS, the incoming packets are just forwarded (requiring little intervention by the director's CPU) and replies from the real-servers return to the client directly by a separate path (via the real-server's default gw) and aren't seen by the director. In a network limited LVS, for the same hardware, because there are separate paths for incoming and returning packets with VS-DR and VS-Tun, the maximum (packet) throughput is twice that of VS-NAT. Because of the rewriting of packets in VS-NAT, the load average on a VS-NAT director will be higher than for a VS-DR or VS-Tun director managing twice the number of packets. In a network bound situation, a single real-server will saturate a director of similar hardware. This is a relatively unusual case for the LVS's deployed so far. However it's the situation where replies are from data in the memory cache on the real-servers (eg squids). With a VS-DR LVS, the real-servers have their own connection to the internet, the rate limiting step is the NIC on the director which accepts packets (mostly <ack>s) from the clients. The incoming network is saturated for packets but is only carrying low bps traffic, while the real-servers are sending full mtu sized packets out their default gw (presumably the full 100Mbps). The information needed to design your director then is simply the number of packets/sec your real-server farm is delivering. The director doesn't know what's in the packets (being an L4 switch) and doesn't care how big they are (1 byte of payload or full mtu size). If the real-servers are network limited, then the director will need the same CPU and network capacity as the total of your real-servers. If the real-servers are not network limited, then the director will need correspondingly less capacity. If you have 7 network limited real-servers with 100Mbps NICs, then they'll be generating an average of 7x8000 = 50k packets/sec. Assuming the packets arrive randomly the standard deviation for 1 seconds worth of packets is +/- sqrt(50000)=200 (ie it's small compared to the rate of arrival of packets). You should be able to connect these real-servers to a 1Gbps NIC via a switch, without saturating your outward link. If you are connected to the outside world by a slow connection (eg T1 line), then no matter how many 8000packet/sec real-servers you have, you are only going to get 1.5Mbps throughput (or half that, since half the packets are <ack>s). Note: The carrying capacity of 100Mbps network of 8000packets/sec may only apply to tcpip exchanges. My 100Mbps network will carry 10,000 SYN packets/sec when tested with Julian's testlvs program. Wayne <tt/wayne@compute-aid.com/ 03 Apr 2001 <quote> The performance page calculate the ack as 50% or so the total packets. I think that might not accurate. Since in the twist-pair and full duplex mode, ack and request are travelling on two different pairs. Even in the half duplex mode, the packets for two directions are transmit over two pairs, one for send, one for receive, only the card and driver can handle them in full duplex or half duplex mode. So the packets would be 8000 packets/sec all the times for the full duplex cards. </quote> Unfortunately we only can approximately predict the performance of an LVS director. Still the best estimates come from comparing with a similar machine. The <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance page"> shows that a 133MHz pentium director can handle 50Mbps throughput. With VS-NAT the load average on the director is unusably high, but with VS-DR, the director has a low load average. Statements on the website indicate that a 300MHz pentium VS-DR director running a 2.2.x kernel can handle the traffic generated by a 100Mbps link to the clients. Other statements indicate that single CPU high end (800MHz) directors cannot handle 1Gbps networks. Presumably multiple directors or SMP directors will be needed for Gbps networks. From: Jeffrey A Schoolcraft <tt/dream@dr3amscap3.com/ 7 Feb 2001 <quote> I'm curious if there are any known DR LVS bottlenecks? My company had the opportunity to put LVS to the test the day following the superbowl when we delivered 12TB of data in 1 day, and peaked at about 750Mbps. In doing this we had a couple of problems with LVS (I think they were with LVS). I was using the latest lvs for 2.2.18, and ldiretord to keep the machines in and out of LVS. The LVS servers were running redhat with an EEPro100. I had two clusters, web and video. The web cluster was a couple of 1U's with an acenic gig card, running 2.4.0, thttpd, with a somewhat performance tuned system (parts of the C10K). At peak our LVS got slammed with 40K active connections (so said ipvsadmin). When we reached this number, or sometime before, LVS became in-accessible. I could however pull content directly from a server, just not through the LVS. LVS was running on a single proc p3, and load never went much above 3% the entire time, I could execute tasks on the LVS but http requests weren't getting passed along. A similar thing occurred with our video LVS. While our real servers aren't quite capable of handling the C10K, we did about 1500 a piece and maxed out at about 150Mbps per machine. I think this is primarily modem users fault. I think we would have pushed more bandwidth to a smaller number of high bandwidth users (of course). I know this volume of traffic choked LVS. What I'm wondering is, if there is anything I could do to prevent this. Until we got hit with too many connections (mostly modems I imagine) LVS performed superbly. I wonder if we could have better performance with a gig card, or some other algorithm (I started with wlc, but quickly changed to wrr because all the rr calculations should be done initially and never need to be done again unless we change weights, I thought this would save us). Another problem I had was with ldirectord and the test (negotiate, connect). It seemed like I needed some type of test to put the servers in initially, then too many connections happened so I wanted no test (off), but the servers would still drop out from ldirectord. That's a snowball type problem for my amount of traffic, one server gets bumped because it's got too many connections, and then the other servers get over-loaded, they'll get dropped to, then I'll have an LVS directing to localhost. So, if anyone has pushed DR LVS to the limits and has ideas to share on how to maximize it's potential for given hardware, please let me know. </quote> <sect1>Initial setup steps Here's some rules of thumb until you know enough to make informed decisions. <sect2>Choose forwarding type choose in this order <itemize> <item>VS-DR default, has high throughput, can be setup on most OS's. <item>VS-NAT, lower throughput, higher latency. Real-server only needs tcpip stack (ie any OS, even a network printer). Works for multiple instances of services (ie multiple copies of demon running on several different ports). <item>VS-Tun, Linux only real-servers. Same throughput as VS-DR. Needed if real-server on different network to director (eg in another location). </itemize> <sect2>Choose number of networks The real-servers are normally run on a private network (eg 192.168.1.0/24). They are not contacted by clients directly. Sometimes the real-servers are machines (on the local network) also being used for other things, in which case leave them on this network. The director is contacted by client(s) on the VIP. If the VIP must be publically available, then it will be on a different network to the real-servers, in which case you will have a two network LVS. The VIP then will be in the network that connects the director to the router (or test client). If the client(s) are on the same network as the real-servers, then you'll only have a one network LVS. <sect2>Choose number of NICs on director If the director is in a 2 network LVS, then having 2 NICs on the director (one for each network) will increase throughput (as long as something else doesn't become rate limiting, eg the CPU speed or the PCI bus). You can have 1 NIC on the director with a 2 network LVS. This is easy to do for VS-NAT (for which there is an example conf file). Doing the same this for VS-DR or VS-Tun requires more though and is left as an exercise for the reader. The configure script will handle 2 NICs on the director. To increase throughput further, the director could have a NIC for each real-server. The configure script doesn't handle this (yet - let me know if you'd like it). <sect2>Pick a configure script If you're using the <ref id="configure" name="configure script"> to setup your LVS, pick the appropriate lvs*.conf.* file and edit to suit you. <sect>Install - General You can setup a working LVS from the LVS-mini-HOWTO. Presumably it will be more upto date, clearer and simpler than this section. Abreviations/conventions for setup/testing/configuring <verb> client: client's IP =CIP gateway: gateway/router's IP =DGW (router will be the client in most test setups) director: director's IP =DIP on director eth0 virtual IP =VIP on director eth0:x (eg eth0:1) real-server: real-server IP =RIP on real-server eth0 virtual IP =VIP on real-server eth0:x/lo:0/tunl0/dummy0 gateway =SGW </verb> <sect1>Director <sect2>2.2.x kernels (with suggestions from John Cronin <tt/jsc3@havoc.gtf.org/) Get a fresh kernel from ftp.kernel.org and the matching ipvs patch from the software page on www.linuxvirtualserver.org. Apply the kernel patch 2.2.x using the instructions in the tarball. You'll do something like director:/usr/src/linux# patch -p1 <../ipvs-0.9.8-2.2.14/ipvs-0.9.8-2.2.14.patch (Note: kernels from RedHat are not the standard kernels off ftp.kernel.org. They are pre-patched with ipvs. If you are going to use RedHat files, follow RedHat's instructions, not ours. If you follow our instructions and try to patch a RH kernel you'll get <verb> > Hunk #1 succeeded at 121 with fuzz 2 (offset 18 lines). > Hunk #2 FAILED at 153. > Hunk #3 FAILED at 163. > Hunk #4 FAILED at 177. > 3 out of 4 hunks FAILED -- saving rejects to include/linux/ip_masq.h.rej > > patching file `include/net/ip_masq.h' > Reversed (or previously applied) patch detected! Assume -R? [n] </verb> The pre-applied patch on the RedHat kernel will likely be older than the one on the LVS website.) Compile the 2.2.x kernel and reboot. The actual kernel compile instructions will vary with kernel patch number. Here's what I used for ipvs-0.9.9 on kernel 2.2.15pre9 in the Networking options. The relevant options are marked. Some of the options are not explicitely required for LVS to work, but you'll need them anyhow - e.g. ip aliasing if you need to constuct a director with only one NIC, or tunneling if you are going to run VS-Tun. Until you know what you're doing activate all of the options with an '*' at the start of the line. You need to turn on "Prompt for developmental code... " or whatever under the "Code Maturity" section. In the "Networking" section, you have to turn on IP:masquerading before you get the ipvs option. Do all the other kernel stuff - make modules, install the modules, copy the new kernel into / or /boot, edit lilo.conf and run lilo. Make sure you leave the old kernel config in lilo too, so you can recover if all does not go well. When loading the new kernel, make sure the ip_vs* modules get loaded. The README in the kernel source tree has all the necessary info in there. <tscreen><verb> [*] Kernel/User netlink socket [*] Routing messages < > Netlink device emulation * [*] Network firewalls [*] Socket Filtering <*> Unix domain sockets * [*] TCP/IP networking [ ] IP: multicasting * [*] IP: advanced router * [*] IP: policy routing [ ] IP: equal cost multipath [ ] IP: use TOS value as routing key [ ] IP: verbose route monitoring [ ] IP: large routing tables [ ] IP: kernel level autoconfiguration * [*] IP: firewalling [ ] IP: firewall packet netlink device * [*] IP: use FWMARK value as routing key (NEW) * [*] IP: transparent proxy support * [*] IP: masquerading --- Protocol-specific masquerading support will be built as modules. * [*] IP: ICMP masquerading --- Protocol-specific masquerading support will be built as modules. * [*] IP: masquerading special modules support * <M> IP: ipautofw masq support (EXPERIMENTAL) * <M> IP: ipportfw masq support (EXPERIMENTAL) * <M> IP: ip fwmark masq-forwarding support (EXPERIMENTAL) * [*] IP: masquerading virtual server support (EXPERIMENTAL) * (12) IP masquerading VS table size (the Nth power of 2) * <M> IPVS: round-robin scheduling * <M> IPVS: weighted round-robin scheduling * <M> IPVS: least-connection scheduling * <M> IPVS: weighted least-connection scheduling * [*] IP: optimize as router not host * <M> IP: tunneling <M> IP: GRE tunnels over IP [*] IP: broadcast GRE over IP [ ] IP: multicast routing [*] IP: PIM-SM version 1 support [*] IP: PIM-SM version 2 support * [*] IP: aliasing support [ ] IP: ARP daemon support (EXPERIMENTAL) * [*] IP: TCP syncookie support (not enabled per default) --- (it is safe to leave these untouched) < > IP: Reverse ARP [*] IP: Allow large windows (not recommended if <16Mb of memory) < > The IPv6 protocol (EXPERIMENTAL) </verb></tscreen> The default LVS <ref id="Hash_Table" name="hash table"> size (2^12 entries) originally meant 2^12 simultanous connections. If you are editing the .config by hand look for CONFIG_IP_MASQUERADE_VS_TAB_BITS. Each entry (for a connection to a client) takes 128 bytes, 2^12 entries requires 512kbytes. If you have 128M spare memory you can have 10^6 entries if you set the table size to 2^20. (Note: not all connections are active - some are waiting to timeout). Early versions of ipvs would crash your machine if you alloted too much memory to this table. This problem has been fixed in 0.9.9. (Note "top" reports memory allocated, not memory you are using. No matter how much memory you have, Linux will eventually allocate all of it as you continue to run the machine and load programs.) As of ipvs-0.9.9 the hash table is different. <quote> From: Julian Anastasov <tt/uli@linux.tu-varna.acad.bg/ With CONFIG_IP_MASQUERADE_VS_TAB_BITS we specify not the max number of the entries (connections in your case) but the number of the rows in a hash table. This table has columns which are unlimited. You can set your table to 256 rows and to have 1,800,000 connections in 7000 columns average. But the lookup is slower. The lookup function chooses one row using hash function and starts to search all these 7000 entries for match. So, by increasing the number of rows we want to speedup the lookup. There is _no_ connection limit. It depends on the free memory. Try to tune the number of rows in this way that the columns will not exceed 16 (average), for example. It is not fatal if the columns are more (average) but if your CPU is fast enough this is not a problem. All entries are included in a table with (1 << IP_VS_TAB_BITS) rows and unlimited number of columns. 2^16 rows is enough. Currently, LVS 0.9.7 can eat all your memory for entries (using any number of rows). The memory checks are planned in the next LVS versions (are in 0.9.9?). </quote> <verb> > How long are the connection entries held for ? (Column 8 of > /proc/net/ip_masquerade ?) </verb> <quote> (Julian Anastasov <tt/uli@linux.tu-varna.acad.bg/) The default timeout value for TCP session is 15 minutes, TCP session after receiving FIN is 2 miniutes, and UDP session 5 minutes. You can use "ipchains -M -S tcp tcpfin udp" to set your own time values. </quote> <verb> > If we assume a clunky set of web servers being > balanced that take 3s to serve an object, then if the connection entries > are dropped immediately then we can balance about 20 million web requests > per minute with 128M RAM. If however the connection entries are kept for a > longer time period this puts a limit on the balancer. </verb> <quote> Yeah, it is true. </quote> <verb> > Eg (assuming column 8 is the thing I'm after!) </verb> <quote> Actually, the column 8 is the delta value in sequence numbers. The timeout value is in column 10. </quote> <verb> > [zathras@consus /]$ head -n 1000 /proc/net/ip_masquerade |sed > -e "s/ */ /g"|cut -d" " -f8|sort -nr|tail -n500|head -n1 > 8398 > > ie Held for about 2.3 hours, which would limit a 128Mb machine to balance > about 10.4 million requests per day. (Which is definitely on the low side > knowing our throughput...) </verb> <sect2>2.4.x kernels The patches to the early versions of the 2.4.x kernels were configured and installed separately to the "make menuconfig" for the kernel. This required moving files into the /lib/modules directories and loading the modules by hand. To avoid this, you should start with 0.2.7-2.4.2 (there were two versions, the hand install and the version which is configured by "make configure"; get the 2nd one). Here's the networking config <verb> <*> Packet socket [ ] Packet socket: mmapped IO [*] Kernel/User netlink socket [*] Routing messages <*> Netlink device emulation [*] Network packet filtering (replaces ipchains) [*] Network packet filtering debugging [*] Socket Filtering <*> Unix domain sockets [*] TCP/IP networking [ ] IP: multicasting [*] IP: advanced router [*] IP: policy routing [*] IP: use netfilter MARK value as routing key [*] IP: fast network address translation [*] IP: equal cost multipath [*] IP: use TOS value as routing key [*] IP: verbose route monitoring [*] IP: large routing tables [*] IP: kernel level autoconfiguration [ ] IP: BOOTP support [ ] IP: RARP support <M> IP: tunneling < > IP: GRE tunnels over IP [ ] IP: multicast routing [ ] IP: ARP daemon support (EXPERIMENTAL) [ ] IP: TCP Explicit Congestion Notification support [ ] IP: TCP syncookie support (disabled per default) IP: Netfilter Configuration ---> IP: Virtual Server Configuration ---> < > The IPv6 protocol (EXPERIMENTAL) < > Kernel httpd acceleration (EXPERIMENTAL) [ ] Asynchronous Transfer Mode (ATM) (EXPERIMENTAL) --- </verb> Here's my config for the IP: Virtual Server configuration (turn it all on) <verb> <M> virtual server support (EXPERIMENTAL) [*] IP virtual server debugging (NEW) (12) IPVS connection table size (the Nth power of 2) (NEW) --- IPVS scheduler <M> round-robin scheduling (NEW) <M> weighted round-robin scheduling (NEW) <M> least-connection scheduling scheduling (NEW) <M> weighted least-connection scheduling (NEW) <M> locality-based least-connection scheduling (NEW) <M> locality-based least-connection with replication scheduling (NEW) <M> destination hashing scheduling (NEW) <M> source hashing scheduling (NEW) --- IPVS application helper <M> FTP protocol helper (NEW) </verb> Here is my config for the netfilter section <verb> <M> Connection tracking (required for masq/NAT) <M> FTP protocol support <M> Userspace queueing via NETLINK (EXPERIMENTAL) <M> IP tables support (required for filtering/masq/NAT) <M> limit match support <M> MAC address match support <M> netfilter MARK match support <M> Multiple port match support <M> TOS match support <M> Connection state match support <M> Unclean match support (EXPERIMENTAL) <M> Owner match support (EXPERIMENTAL) <M> Packet filtering <M> REJECT target support <M> MIRROR target support (EXPERIMENTAL) <M> Full NAT <M> MASQUERADE target support <M> REDIRECT target support <M> Packet mangling <M> TOS target support <M> MARK target support <M> LOG target support < > ipchains (2.2-style) support < > ipfwadm (2.0-style) support </verb> Note I have removed the ipchains option here. This was <M> in the last version of the HOWTO. However this raised problems as some people didn't understand the <ref id="ipchains_compatability" name="ipchains compatability problems">. <sect2>The LVS code does not swap Joe Feb 2001: With sufficient number of connections, a director could start to swap out its tables (is this true?) In this case, throughput could slow to a crawl. I presume the kernel would have to retrieve parts of the table to find the real-server associated with incoming packets. I would think in this case it would be better to drop connect requests than to accept them. In earlier verions of LVS, you could set the amount of memory for the tables (in bytes). Now you allocate a number of hashes, whose size could (in the worst case) grow without limit. Julian <verb> > IMO, this is not true. LVS uses GFP_ATOMIC kind of allocations > and as I know such allocations can't be swapped out. </verb> If it's possible for LVS to start the director to swap, is there some way to stop this? <verb> > You can try with testlvs whether the LVS uses swap. > Start the kernel with LILO option mem=8M and with large swap area. > Then check whether more than 8MB swap is used. </verb> <sect1>Real-servers In general, nothing specific is done for the real-servers. You can have any OS running on them (except VS-Tun, which runs only on Linux real-servers). You plug them in to the network, startup the services on the VIP (for VS-DR, VS-Tun) or the RIP (VS-NAT), setup the default gw (the router and the director respectively in the usual setups) and you're ready to go. Except: You have to handle the <ref id="arp_problem" name="arp problem"> with VS-DR (and possibly VS-Tun). Unfortunately this turns what would be a trivial installation into one that requires clear thinking. If you don't want to deal with the arp problem for your first installation, then setup a VS-NAT LVS. Eventually you'll want the higher throughput and lower latency of VS-DR, in which case you'll need to understand the arp problem. The simplest approach is to use the NOARP option of ifconfig to setup lo:0 on non-linux unix real-servers. For linux, <itemize> <item>2.0.x kernels: the NOARP option to ifconfig works as for other unices <item>2.2.x recent (i.e.x>12) kernels; activate arp hiding <item>2.4.x kernels; patch the kernel with Julian's "hidden patch" and then activate the arp hiding </itemize> For non-unix (ie Windows) real-servers, look below for further instructions. <sect2>Linux Real-servers If you are handling the arp problem by hiding the VIP device on the real-servers, then you need to patch the real-servers if <itemize> <item>2.2.x (where x<12) <item>2.4.x (all versions, see info about <ref id="2.4_arp" name="patching 2.4 real-servers">). </itemize> The current version(s) of 2.2.x, e.g. 2.2.19 are already patched with the arp hiding code i.e. you don't have to patch the current 2.2.x kernels. For Linux-2.0.x real-servers, you can use the NOARP option when setting up a device for the VIP. <sect2>Other unices If you are running non-Linux unix real-servers, you can handle the arp problem by configuring the device carrying the VIP with the -arp switch. This list of real-servers is from Ratz <tt/ratz@tac.ch/ About the only thing he hasn't tried yet is Plan 9. <verb> Solaris 2.5.1, 2.6, 2.7 Linux (of course): 2.0.36, 2.2.9, 2.2.10, 2.2.12 FreeBSD 3.1, 3.2, 3.3 NT (although Webserver would crash): 4.0 no SP IRIX 6.5 (Indigo2) HPUX 11 </verb> Ratz's code is now in the configure script. This part of the script has not been well tested (you might find that it doesn't setup your non-linux unix box properly yet, please contact me - Joe). Here's the information for non-Linux unices. On some Unixes you have to plumb the interface before assigning an IP. The plumb instruction is not included here. <verb> #uname : FreeBSD #uname -r : 3.2-RELEASE #<command> : ifconfig lo0 alias <VIP> netmask 0xffffffff -arp up #ifconfig -a: lo0: flags=80c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST>mtu 16837 # inet 127.0.0.1 netmask 0xff000000 # inet <VIP> netmask 0xffffffff #uname : IRIX #uname -r : 6.5 #<command> : ifconfig lo0 alias <VIP> netmask 0xffffffff -arp up #ifconfig -a: lo0: flags=18c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST,CKSUM> # inet 127.0.0.1 netmask 0xff000000 # inet <VIP> netmask 0xffffffff #uname : SunOS #uname -r : 5.7 #<command> : ifconfig lo0:1 <VIP> netmask 255.255.255.255 up #ifconfig -a: lo0: flags=849<UP,LOOPBACK,RUNNING,MULTICAST>mtu 8232 # inet 127.0.0.1 netmask ff000000 # lo0:1 flags=849<UP,LOOPBACK,RUNNING,MULTICAST>mtu 8232 # inet <VIP> netmask ffffffff #uname : HP-UX #uname -r : B.11.00 #<command> : ifconfig lan1:1 10.10.10.10 netmask 0xffffff00 -arp up #ifconfig -a: lan0: flags=842<BROADCAST,RUNNING,MULTICAST> # inet <some IP> netmask ffffff00 # lan0:1: flags=8c2<BROADCAST,RUNNING,NOARP,MULTICAST> # inet <VIP> netmask ffffff00 # </verb> Some unices aren't very cooperative and other methods (e.g. adding an extra NIC) should do it. Ratz 16 Apr 2001 <quote> in most cases (when using the NOARP option) you need alias support. Some Unices have no support for aliased interfaces or only limited, such as QNX, Aegis or Amoeba for example. Others have interface flag inheritance problems like HP-UX where it is impossible to give an aliased interface a different flag vector as for the underlying physical interface (as happens with Linux 2.2 and 2.4 - Joe). So for HP/UX you need a special setup because with the standard depicted setup for DR it will NOT work. I've done most Unices as Realserver and was negatively astonished by all the different implementation variations of the different Unix flavours. This maybe resulted from unclear statements from the RFC's. </quote> <sect2>NT (This is not handled by the configure script). Here's Wensong's recipe for setting up the lo device on a NT real-server. If you don't have MS Lookback Adapter Driver installed on your NT boxes, enter Network Control Panel, click the Adapter section, click to add a new adapter, select the MS Loopback Adapter. Your NT cdrom is needed here. Then add your VIP (Virtual IP) address on the MS Loopback Adapter, do not enter a gateway address on the Loopback Adapter. Since the netmask 255.255.255.255 is considered invalid in M$ NT, you just accept the default netmask, then enter MS-DOS prompt, remove the extra routing entry. <verb> c:route delete <VIP's network> <VIP's netmask> </verb> This will make the packets destined for this network will go through the other network interface, not this MS Loopback interface. As I remember, setting its netmask to 255.0.0.0 also works. alternatively (Jerome RICHARD <tt/jrichard@virtual-net.fr/) On Windows NT Server, you just have to install a network adapter called "MS Loopback" (Provided on the Windows NT CDROM in new network section) and then you setup the VIP on this interface. <sect2>W2K from o1004g <tt/o1004g@nbuster.com/ <enum> <item>Click Start, point to Settings, click Control Panel, and then double-click Add/Remove Hardware. <item>Click Add/Troubleshoot a device, and then click Next. <item>Click Add a new device, and then click Next. <item>Click No, I want to select the hardware from a list, and then click Next. <item>Click Network adapters, and then click Next. <item>In the Manufacturers box, click Microsoft. <item>In the Network Adapter box, click Microsoft Loopback Adapter, and then click Next. <item>Click Finish. </enum> <sect1>iptables/ipchains compatability problems<label id="ipchains_compatability"> ipchains in 2.2.x kernels has been replaced by iptables in 2.4.x kernels. For 2.4 kernels, ipchains is available for backwards compatibility. However ipchains and iptables can't be used at the same time. The ip_tables module is incompatible with ipchains. If present, the ip_tables module must be unloaded for ipchains to work. If you have ip_tables loaded, you'll get uninformative errors when you try to run ipchains commands with 2.4. Rather than saying that ipchains under 2.4 is there for compatibility, it would be more accurate to say that the ipchains commands available with 2.4 kernels will only cause you grief and it will be faster to rewrite your scripts to iptables, than to fall into all the holes you'll find using the compatibility. It won't take long before some script/program expects to run ip_tables on your 2.4 machine and as soon as that happens one or both (I don't know which) of your iptables or ipchains are hosed. <sect>Ipvsadm<label id="ipvsadm"> ipvsadm is the user interface to LVS. <sect1>Using ipvsadm You use ipvsadm from the command line (or in rc files) to setup: - <itemize> <item> services/servers that the director directs (eg http goes to all real-servers, while ftp goes only to real-server_1). <item>weighting given to each real-server - useful if some servers are faster than others. <item>scheduling algorithm <itemize> <item>round robin (rr), weighted round robin (wrr) - new connections are assigned to each real-server in turn <item>least connected (lc), weighted least connection (wlc) - new connections go to real-server with the least number of connections (this is not neccessarily the least busy machine but is a step in that direction). Note: Doug Bagley <tt/doug@deja.com/ points out that *lc schedulers will not work properly if a particular real-server is used in two different LVSs. <item>newer scheduling methods designed for webcaches (DH, destination hash from code developed by Thomas Proell <tt/proellt@gmx.de/) and multiple firewalls (SH, source hash by Henrik Nordstrom <tt/hno@safecore.se/). </itemize> </itemize> any of these will do for a test setup (rr will cycle connections to each real-server in turn, allowing you to check that all real-servers are functioning in the LVS). You use ipvsadm to <itemize> <item>add services: add a service with weight >0 <item>shutdown services: set the weight to 0. This allows current connections to continue, untill they disconnect or expire, but will not allow new connections. When there are no connectionsi remaining, you can reconfigure your service or turn off the server. <item>delete services: this stops traffic for the service, but the entry in the connection table is not deleted till it times out. This allows deletion followed shortly thereafter by adding back the service, to not affect established (but quiescent) connections. <item>save the ipsvadm settings of your lvs, with ipvsadm-save <item>restore the ipvsadm settings or your lvs, with ipvsadm-restore </itemize> <sect1>Compile ipvsadm for each new ipvs Compile and install ipvsadm on the director using the supplied Makefile. You can optionally compile ipvsadm with popt libraries, which allows ipvsadm to handle more complicated arguments on the command line. The default compile uses static popt libraries. If your libpopt.a is too old, your ipvsadm will segv. (I'm compiling with a newer dynamic libpopt). Since you compile ipvs and ipvsadm independantly and you cannot compile ipvsadm until you have patched the kernel headers, a common mistake is to compile the kernel and reboot, forgetting to compile/install ipvsadm. Unfortunately there is only rudimentary version detection code into ipvs/ipvsadm. If you have a mismatched ipvs/ipvsadm pair, many times you won't notice any problems, as any particular version of ipvsadm will work with a wide range of patched kernels. Usually with 2.2.x kernels, if the ipvs/ipvsadm versions mismatch, you'll get wierd but non-obvious errors about not being able to install your LVS. Other possibilities are that the output of ipvsadm -L will have IP's that are clearly not IPs (or not the IP's you put in) and ports that are all wrong. There was a change in the /proc file system for ipvs about 2.2.14 which caused problems for anyone with a mismatched ipvsadm/ipvs. The ipvsadm from different kernel series (2.2/2.4) do not recognise the ipvs kernel patches from the other series (they appear to not be patched for ipvs). The later 2.2.x ipvsadms know the minimum version of ipvs that they'll run on, and will complain about a mismatch. They don't know the maximum version (produced presumably some time in the future) /outsithat they will run on. This protects you against the unlikely event of installing a new 2.2.x version of ipvsadm on an older version of ipvs, but will not protect you against the more likely scenerio where you forget to compile ipvsadm after building your kernel. The ipvsadm maintainers are aware of the problem and aren't going to fix it sometime. If you didn't even apply the kernel patches for ipvs, then ipvsadm will complain about missing modules. "Ty Beede" <tt/tybeede@metrolist.net/ writes: <verb> > on a slackware 4.0 machine I went to compile ipvsadm and it gave > me an error indicating that the iphdr type was undefined and > it didn't like that when it saw the ip_fw.h header file.� I just > #included <linux/ip.h> in ipvsadm.c, which is where the iphdr > #structure is defined and everything went ok </verb> From: Doug Bagley <tt/doug@deja.com/ <verb> The reason that it fails "out of the box" is because fwp_iph's type definition (struct iphdr) was #ifdef'd out in <linux/ip_fw.h> (and not included anywhere else) since the symbol __KERNEL_ was undefined. Including <linux/ip.h> before <linux/ip_fw.h> in the .c file did the trick. </verb> <sect1>schedulers<label id="scheduler"> On receiving a connect request from a client, the director assigns a real-server to the client based on a "schedule". The scheduler type is set with ipvsadm. The schedulers available are <itemize> <item>rr, wrr: round robin, weighted round robin <item>lc, wlc: least connection, weighted least connection (the director has a table with the number of connections for each real-server). <item><ref id="persistent_connection" name="persistent connection"> <item>LBLC: a persistent memory algorythm <item>dh: destination hash <item>sh: source hash </itemize> The rr,wrr,lc,wlc schedulers should all work similarly for identical real-servers with identical services. The lc scheduler will better handle situations where machines are brought down and up again (see <ref id="thundering_herd" name="thundering herd problem">). If the real-servers are offering different services and some have clients connected for a long time while others are connected for a short time, or some are compute bound, while others are network bound, then none of the schedulers will do a good job of distributing the load between the real-servers. LVS doesn't have any load monitoring of the real-servers. Figuring out a way of doing this that will work for a range of different types of services isn't simple (see the mailing list archives for endless discussion on remote load monitoring). The LBLC code (from Julian) and the dh scheduler (from Thomas Proell) are designed for web caching real-servers (e.g. squids). For normal LVS services (eg ftp, http), the content offered by each real-server is the same and it doesn't matter which real-server the client is connected to. For a web cache, after the first fetch has been made, the web caches have different content. As more pages are fetched, the contents of the web caches will diverge. Since the web caches will be setup as peers, they can communicate by ICP (internet caching protocol) and find the cache(s) with the required page. This is faster than fetching the page from the original webserver. However, it would be better after the first fetch of a page from http://www.foo.com/* , for all subsequent clients wanting a page from http://www.foo.com/ to be connected to that real-server. The original method for handling this was to make connections to the real-servers persistent, so that all fetches from a client went to the same real-server. The -dh (destination hash) algorythm makes a hash from the target IP and all requests to that IP will be sent to the same real-server. This means that content from a URL will not be retrieved multiple times from the remote server. The real-servers (eg squids in this case) will each be retreiving content from different URLs. The -sh (source hash) scheduler is for directors with multiple firewalls. It's from Henrik Nordstrom, who is involved with developing web caches. The director hashes on the MAC address of the firewall. Henrik Nordstrom 14 Feb 2001 <quote> Here is a small patch to make LVS keep the MARK, and have return traffic inherit the mark. We use this for routing purposes on a multihomed LVS server, to have return traffic routed back the same way as from where it was received. What we do is that we set the mark in the iptables mangle chain depending on source interface, and in the routing table use this mark to have return traffic routed back in the same (opposite) direction. The patch also moves the priority of LVS INPUT hook back to infront of iptables filter hook, this to be able to filter the traffic not picked up by LVS but matchin it's service definitions. We are not (yet) interested of filtering traffic to the virtual servers, but very interested in filtering what traffic reaches the Linux LVS-box itself. (Julian) - who uses NFC_ALTERED ? Netfilter. The packet is accepted by the hook but altered (mark changed). (Julian) Give us an example (with dummy addresses) for setup that require such fwmark assignments. For a start you need a LVS setup with more than one real interface receiving client traffic for this to be of any use. Some clients (due to routing outside the LVS server) comes in on one interface, other clients on another interface. In this setup you might not want to have a equally complex routing table on the actual LVS server itself. Regarding iptables / ipvs I currently "only" have three main issues. <itemize> <item>As the "INPUT" traffic bypasses most normal routes, the iptables conntrack will get quite confused by return traffic.. <item>Sessions will be tracked twice. Both by iptables conntrack and by IPVS. <item>There is no obvious choice if IPVS LOCAL_IN sould be placed before or after iptables filter hook. Having it after enables the use of many fancy iptables options, but instead requires one to have rules in iptables for allowing ipvs traffic, and any mismatches (either in rulesets or IPVS operation) will cause the packets to actually hit the IP interface of the LVS server which in most cases is not what was intended. </itemize> </quote> From: Wensong Zhang <tt/wensong@gnuchina.org/ 16 Feb 2001 <quote> Please see "man ipvsadm" for short description of DH and SH schedulers. I think some examples to use those two schedulers. Example1: cache cluster shared by several load balancers. <verb> Internet | |------cache array | |----------------------------- | | DH DH | | Access Access Network1 Network2 </verb> The DH scheduler can keep the two load balancer redirect requests destined for the same IP address to the same cache server. If the server is dead or overloaded, the load balancer can use cache_bypass feature to send requests to the original server directly. (Make sure that the cache servers are added in the two load balancers in the same order) Note that the DH development is inspired by the consistent hashing scheduler patch from Thomas Proell <tt/proellt@gmx.de/ Example2: Firewall Load Balancing <verb> |-- FW1 --| Internet ----- SH --| |-- DH -- Protected Network |-- FW2 --| </verb> Make sure that the firewall boxes are added in the load balancers in the same order. Then, request packets of a session are sent to a firewall, e.g. FW1, the DH can forward the response packets from protected network to the FW1 too. However, I don't have enough hardware to test this setup myself. Please let me know if any of you make it work for you. :) </quote> For initial discussions on the -dh and -sh scheduler see on the mailing list under "some info for DH and SH schedulers" and "LVS with mark tracking". <sect1>does rr and lc weighting equally distribute the load? I ran the <url url="http://www.polygraph.org/" name="polygraph"> simple.pg test on a VS-NAT LVS with 4 realservers using rr scheduling. Since the responses from the real-servers should average out I would have expected the number of connection and load average on the real-servers to be equally distributed over the real-servers. Here's the output of ipvsadm shortly after the number of connections had reached steady state (about 5 mins). <verb> IP Virtual Server version 0.2.12 (size=16384) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:polygraph rr -> doc.mack.net:polygraph Masq 1 0 883 -> dopey.mack.net:polygraph Masq 1 0 924 -> bashfull.mack.net:polygraph Masq 1 0 1186 -> sneezy.mack.net:polygraph Masq 1 0 982 </verb> The servers were identical hardware. I expect (but am not sure) that the utils/software on the machines is identical (I set up doc,dopey about 6 months after sneezy,bashfull). Bashfull was running 2.2.19, while the other 3 machine were running 2.4.3 kernels. The number of connections (all in TIME_WAIT) at the real-servers was different for each (otherwise apparently identical) real-server and was in the range 450-500 for the 2.4.3 machines and 1000 for the 2.2.19 machine (measured with netstat -an | grep $polygraph_port |wc ) and varied about 10% over a long period. This run had been done immediately after another run and InActConn had not been allowed to drop to 0. Here I repeated this run, after first waiting for InActConn to drop to 0 <verb> IP Virtual Server version 0.2.12 (size=16384) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:polygraph rr -> doc.mack.net:polygraph Masq 1 0 994 -> dopey.mack.net:polygraph Masq 1 0 994 -> bashfull.mack.net:polygraph Masq 1 0 994 -> sneezy.mack.net:polygraph Masq 1 1 992 TCP lvs2.mack.net:netpipe rr </verb> Bashfull (the 2.2.19 machine) had 900 connections in TIME_WAIT while the other (2.4.3) machines were 400-600. Bashfull was also delivering about 50% more hits to the client. Repeating the run using "lc" scheduling, the InActConn remains constant. <verb> IP Virtual Server version 0.2.12 (size=16384) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:polygraph lc -> doc.mack.net:polygraph Masq 1 0 994 -> dopey.mack.net:polygraph Masq 1 0 994 -> bashfull.mack.net:polygraph Masq 1 0 994 -> sneezy.mack.net:polygraph Masq 1 0 993 </verb> The number of connections (all in TIME_WAIT) at the real-servers did not change. Joe, 14 May 2001 according to the ipvsadm man page, for "lc" scheduling, the new connections are assigned according to the number of "active connections". Is this the same as "ActConn" in the output of ipvsadm? If the number of "active connections" used to determine the scheduling is "ActConn", then for services which don't maintain connections, the scheduler won't have much information, just "0" for all real-servers? Julian <quote> The formula is: ActConn * K + InActConn where K can be 32 to 50, I don't remember the last used value. So, it is not only the active conns, this will break UDP. </quote> I've been running the polygraph simple.pg test over the weekend using rr scheduling on what (AFAIK) are 4 identical real-servers in a VS-NAT LVS. There are no ActConn and a large number of InActConn. Presumably the client makes a new connection for each request. <quote> The implicit persistence of TCP connection reuse can cause such side effects even for RR. When the setup includes small number of hosts and the used rate is big enough to reuse the client's port, the LVS detects existing connections and new connections are not created. This is the reason you can see some of the rs not to be used at all, even for such method as RR. </quote> the client is using ports from 1025-4999 (has about 2000 open at one time) and it's not going above the 4999 barrier. ipvsadm shows a constant InActConn of 990-995 for all realservers, but the number of connections on each of the real-servers (netstat -an) ranges from 400-900. So if the client is reusing ports (I thought you always incremented the port by 1 till you got to 64k and then it rolled over again), LVS won't create a new entry in the hash table if the old one hasn't expired? <quote> Yes, it seems you have (5000-1024) connections that never expire in LVS. </quote> Presumably because the director doesn't know the number of connections at the real-servers (it only has the number of entries in its tables), and because even apparently identical real-servers aren't identical (the hardware here is the same, but I set them up at different times, presumably not all the files and time outs are the same), the throughput of different real-servers may not be the same. <sect1>persistent connections<label id="persistent_connection"> Unfortunately the term "persistence" has 2 meanings in setting up an LVS. There is "persistent connection" a term used for connecting to webservers and databases and "persistent connection" used in LVS. These are quite different. You need LVS persistence if you need the client to always connect to the same real-server. You'll need this if <itemize> <item>you need session information to stay on one machine, eg SSL keys for https <item>you are using a multiport protocol. Examples are <itemize> <item>ftp using ports 20,21. (For VS-NAT, this is handled by the ip_masq_ftp module). You need persistence for VS-DR and VS-Tun <item>e-commerce sites where a customer on port 80 has to transfer to port 443 on the same machine to pay for an order. </itemize> </itemize> From: <tt/bobby.moore@worldspan.com/ wrote: > What does the term 'persistence' mean for IPVS? Persistant connection outside of LVS is described in <url url="http://www.research.compaq.com/wrl/techreports/abstracts/95.4.html" name="http persistent connection"> and is an application level protocol. In normal http (or database connection), after the server has sent it's reply, it shuts down the tcpip connection. This makes your session with the server stateless - it has no idea what you've previously done on the machine. If the exchanges are small (eg 1 packet), then as well you've gone through a lot of handshakes to exchange one packet as your payload. At connect time the client and server notify each other that they support persistent connection and the server uses an algorithm to determine when to drop the connection (timeout, needs handle...). The client can drop the connection anytime they like. The persistant (or sticky) connection of LVS is a layer 4 protocol which makes connections from client(s) on an IP (or network) go to the same real-server. This is not the same as the persistent connection described above and could alternately be described as connection affinity or port affinity. In an LVS if you are accumulting state (eg a shopping cart), then you want the client to connect to the same server on port 443 as was used for the port 80 connection. If you are doing passive ftp, then you want the second connection to come back to the same real-server. In these cases you want the connection affinity provided by the LVS persistent connection option. From: Wensong Zhang <tt/wensong@gnuchina.org/ 11 Jan 2001 The working principle of persistence in LVS is as follows: <itemize> <item> a persistent template is used to keep the persistence between the client and the server. <item> when the first connection from a client, the LVS box will select a server according to the scheduling algoriths, then create a persistent template and the connection entry. the control of the connection entry is the template. <item>The late connections from the clients will be forwarded to the same server, as long as the template doesn't expire. The control of their connection entries are the template. <item>If the template has its controlled connections, it won't expire. <item>If the template has no controlled connections, it expires in its own time. </itemize> You can trace your system in the following way. For example: <tscreen><verb> [root@kangaroo /root]# ipvsadm -ln IP Virtual Server version 1.0.3 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.26.20.118:80 wlc persistent 360 -> 172.26.20.91:80 Route 1 0 0 -> 172.26.20.90:80 Route 1 0 0 TCP 172.26.20.118:23 wlc persistent 360 -> 172.26.20.90:23 Route 1 0 0 -> 172.26.20.91:23 Route 1 0 0 [root@kangaroo /root]# ipchains -L -M -n IP masquerading entries prot expire source destination ports TCP 02:46.79 172.26.20.90 172.26.20.222 23 (23) -> 0 </verb></tscreen> Although there is no connection, the template isn't expired. So, new connections from the client 172.26.20.222 will be forwarded to the server 172.26.20.90. <sect2>persistent client connection, pcc (for kernel =<2.2.10) All connections from an IP go to same real-server. Timeout for inactive connections is 360 sec. pcc is designed for https and <ref id="cookie" name="cookie"> serving. With ppc, after the first connection (say to port 80) any subsequent connections requests from the same client but from another port (eg 443) will be sent to the same real-server. The problem with this is that about 25% of the people on the internet have the same IP (AOL customers are connected to the internet via a server in Virginnia, USA). If you have pcc set, then after the first client connects from AOL, then all subsequent connections from AOL will go to the same real-server, until the last AOL client disconnects. This effect will override attempts to distribute the load between real-servers. <sect2>persistent port connection (ppc) (for kernel >= 2.2.12)<label id="granularity"> With kernel 2.2.12, the persistent connection feature has been changed from a scheduling algorythm (you get rr|wrr|lc|wlc|pcc) to a switch (you can have persistent connection with rr|wrr|lc|wlc). If you do not select a scheduling algorithm when asking for a persistent connection, ipvsadm will default to wlc. The difference between pcc and ppc is probably of minor consequence to the LVS admin (if you want persistent connection, you have to have it and you don't care how you got it). With ppc, connections are assigned on a port by port basis. With ppc, if both port 80 and 443 were persistant, then connections from the same client would not neccessarily go to the same real-server. This solves the AOL problem. If you are handing out <ref id="cookie" name="cookies"> to a client on port 80 and they need to go to port 443 to give their credit card, you want them going to the same real-server. There is no way to make ports sticky by groups (or pairs), so for the moment you emulate the pcc connection by using port 0. Because of the way proxies can work, a client can come from one IP for one connection (eg port 80) and from another IP for the next connection (eg port 443). For this you can make a netmask of IPs sticky. (valery brasseur) > I have seen some discussion about "proxy farm" such as AOL or T-Online, (Wensong) If you want to build a persistent proxy cluster, you just need set a LVS box at the front of all proxy servers, and use the persistent port option in the ipvsadm commands. BTW, you can have a look at http://wwwcache.ja.net/JanetServices/PilotServices.html for how to build a big JANET cache cluster using LVS. If you want to build a persistent web service but some proxy farms are non-persistent at client side, then you can use the persistent granularity so that clients can be grouped, for example you use 255.255.255.0 mask, the clients from the same /24 network will go to the same server. While persistence can be used for services that require multiple ports eg ftp/ftp-data, http/https it can be useful for ssl services. Here's an example of using persistence granularity (from Ratz 3 Jan 2001). The -M 255.255.255.255 sets up /32 granularity. Here port 80 and port 443 are being linked by fwmarks. <verb> ipchains -A input -j ACCEPT -p tcp -d 192.168.1.100/32 80 -m 1 -l ipchains -A input -j ACCEPT -p tcp -d 192.168.1.100/32 443 -m 1 -l ipvsadm -A -f 1 -s wlc -p 333 -M 255.255.255.255 ipvsadm -a -f 1 -r 192.168.1.1 -g -w 1 ipvsadm -a -f 1 -r 192.168.1.2 -g -w 1 </verb> For more information on persistence granularity see the section on <ref id="fwmark_persistence_granularity" name="persistence granularity with fwmark">. It's use for fwmark is the same as for VIP. Francis Corouge wrote: I made a VS-DR lvs. All services work well, but with IE 4.1 on secured connection, pages are received randomly. when you make several requests, sometime the page is displayed, but sometimes a popup error message is displayed <verb> Internet Explorer can't open your Internet Site <url> An error occured with the secured connexion. </verb> I did not test with other versions of IE, but netscape works fine. It works when I connect directly to the real server (real-server disconnected from the LVS, and the VIP on the real-server allowed to arp). Julian: Is the https service created persistent? ipvsadm -p Why does persistence fix this problem? (also see http://www.linuxvirtualserver.org/persistence.html) I assume the problem is in the way SSL is working: cached keys, etc. Without persistence configured, the SSL connections break when they hit another real server. > what is (or might be) different about IE4 and Netscape? Maybe in the way the bugs are encoded. But I'm not sure how the SSL requests are performed. It depends on that too. Example 1. https only This is done with persistent connection. lvs_dr.conf config file excerpt SERVICE=t https ppc 192.168.1.1 output from ipvsadm <verb> ipvsadm settings IP Virtual Server version 0.9.4 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP ssl.mack.net:https wlc persistent 360 -> di.mack.net:https Route 1 0 0 </verb> Example 2. All ports sticky, timeout 30mins, wrr scheduling lvs_dr.conf config file excerpt SERVICE=t 0 wrr ppc -t 1800 192.168.1.1 which specifies tcp (t), service (all ports = 0), weighted round robin scheduling (wrr), timeout 1800 secs (-t 1800), to realserser 192.168.1.1. Here's the code generated by configure <verb> #ppc persistent connection, timeout 1800 sec /sbin/ipvsadm -A -t 192.168.1.110:0 -s wrr -p 1800 echo "adding service 0 to real-server 192.168.1.1 using connection type dr weight 1" /sbin/ipvsadm -a -t 192.168.1.110:0 -R 192.168.1.1 -g -w 1 </verb> here's the output of ipvsadm <verb> # ipvsadm IP Virtual Server version 0.9.4 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP ssl.mack.net:https wrr persistent 1800 -> di.mack.net:https Route 1 0 0 </verb> Problems: Removing persistant connections after a real-server crash From: Patrick Kormann <tt/pkormann@datacomm.ch/ I have the following problem: I have a direct routed 'cluster' of 4 proxies. My problem is that even if the proxy is taken out of the list of real servers, the persistent connection is still active, that means, that proxy is still used. <verb> Andres Reiner wrote: (...snip...) >> Now I found some strange behaviour using 'mon' for the >> high-availability. If a server goes down it is correctly removed from >> the routing table. BUT if a client did a request prior to the server's >> failure, it will still be directed to the failed server afterwards. I >> guess this got something to do with the persistent connection setting >> (which is used for the cold fusion applications/session variables). >> >> In my understanding the LVS should, if a routing entry is deleted, no >> longer direct clients to the failed server even if the persistent >> connection setting is used. >> >> Is there some option I missed or is it a bug ? </verb> Wensong Zhang wrote: <verb> > No, you didn't miss anything and it is not a bug either. :) > > In the current design of LVS, the connection won't be drastically > removed but silently drop the packet once the destination of the > connection is down, because monitering software may marks the server > temporary down when the server is too busy or the monitering software > makes some errors. When the server is up, then the connection continues. > If server is not up for a while, then the client will timeout. One thing > is gauranteed that no new connections will be assigned to a server when > it is down. When the client reestablishs the connection (e.g. press > reload/refresh in the browser), a new server will be assigned. jacob.rief@tis.at// wrote: > > Unfortunately I have the same problem as Andres (see below) > If I remove a real server from a list of persistent > virtual servers, this connection never times out. Not even > after the specified timeout has been reached. Only if I unset </verb> (Wensong) The persistent template won't timeout until all its connections timeout. After all the connections from the same client connection expires, new connections can be assigned to one of the remaining servers. You can use "ipchains -M -L -n" (or netstat -M) to check the connection table (for 2.4.x use cat /proc/net/ip_conntrack). <verb> > persisency the connection will be redirected onto the remaining > real servers. Now if I turn on persistency again, a prevoiusly > attached client does not reconnect anymore - it seems as > if LVS remembers such clients. It does not even help, if I delete > the whole virtual service and restore it immediately, in the > hope to clear the persistency tables. > (ipvsadm -D -t <VIP>; ipvsadm -A -t <VIP> -p; ipvsadm -a -t <VIP> -R <alive > real server>) > And it also does not help closing the browser and restarting it. > I run LVS in masquerading mode on a 2.2.13-kernel patched > with ipvs-0.9.5. > Would'nt it be a nice feature to flush the persistent client > connection table, and/or list all such connections? </verb> (Wensong) There are several reasons that I didn't do it in the current code. One is that it is time-consuming to search a big table (maybe one million entries) to flush the connections destined for the dead server; the other is that the template won't expire until its connection expire, the client will be assigned to the same server as long as there is a connection not expired. Anyway, I will think about better way to solve this problem. <verb> (valery brasseur) > I would like to to load balancing base on cookie and/or URL, (Wensong) Have a look at http://www.LinuxVirtualServer.org/persistence.html :-) (also see <ref id="cookie" name="cookie">) Jean-Francois Nadeau wrote: > I will use LVS to load balance web servers (Direct Routing and WRR algo). > I use persitency with a big timeout (10 minutes). > Many of our clients are behind big proxies and I fear this will > unbalance our cluster because of the persitent timeout. (Wensong) persistent virtual services may lead to the load imbalance among servers. Using some weight adapation approaches may help avoid that some servers are overloaded for a long time. When the server is overloaded, decrease its weight so that connections from new clients won't be sent to that server. When the server is underloaded, increase its weight. > Can we alter directly /proc/net/ip_masquerade ? No, it is not feasible, because directly modifying masq entries will break the established connection. </verb> <sect2>Persistent and regular services are possible on the same real-server. If you setup a 2 real-server VS-DR LVS with persistence, <verb> ipvsadm -A -t $VIP -p -s -rr ipvsadm -a -t $VIP -R $realserver1 $VS_DR -w 1 ipvsadm -a -t $VIP -R $realserver2 $VS_DR -w 1 </verb> giving the ipvsadm output <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 0.2.5 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:0 rr persistent 360 -> bashfull.mack.net:0 Route 1 0 0 -> sneezy.mack.net:0 Route 1 0 0 </verb> then (as expected) a client can connect to any service on the real-servers (always getting the same real-server). If you now add an entry for telnet to both real-servers, (you can run these next instructions before or after the 3 lines immediately above) <verb> ipvsadm -A -t $VIP:telnet -s -rr ipvsadm -a -t $VIP:telnet -R $realserver1 $VS_DR -w 1 ipvsadm -a -t $VIP:telnet -R $realserver2 $VS_DR -w 1 </verb> giving the ipvsadm output <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 0.2.5 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:0 rr persistent 360 -> bashfull.mack.net:0 Route 1 0 0 -> sneezy.mack.net:0 Route 1 0 0 TCP lvs2.mack.net:telnet rr -> sneezy.mack.net:telnet Route 1 0 0 </verb> the client will telnet to both real-servers in turn as would be expected for an LVS serving only telnet, but all other services (ie !telnet) go to the same first real-server. All services but telnet are persistent. The director will make persistent all ports except those that are explicitely set as non-persistent. These two sets of ipvsadm commands do not overwrite each other. Persistent and non-persistent connections can be made at the same time. (from Julian) This is part of the LVS design. The templates used for persistence are not inspected when scheduling packets for non-persistent connections. <sect2>Examples of persistence <itemize> <item> ftp (VS-NAT): connections to both ftp ports (for both active and passive ftp) are handled by the module ip_masq_ftp. You don't need to add persistence for ftp with VS-NAT. <item> ftp (VS-DR or VS-Tun): you need persistence on the realservers. Run the first set of commands above. <item> ftp and http (VS-NAT): persistence not needed (ip_masq_ftp handles the ftp ports for active and passive ftp). <item> ftp and http (VS-DR or VS-Tun): persistence needed to handle the two port protocol ftp. You can set this up by running the first set of commands above. A secondary consequence of this arrangement is that a client connecting to the http service of the LVS will always get the same real-server (this may not be a great problem). If you want to make the http service non-persistent but leaving all other services persistent, then run the second set of commands (above) as well. <item> http and https (all forwarding methods): Normally an https connection is made after the client has made selections on an http connection when data is stored on the real-server for the client. In this case the real-server should be made persistent (first set of commands above). Do not run ipvsadm commands for http (like the second set of commands above) as this will make the http connections non-persistent. </itemize> Note: making real-server connections persistent allows _all_ ports to be forwarded by the LVS to the real-servers. Non-persistent LVS connections are only for the nominated service. An open, persistently connected real-server is a security hazard. You should run ipchains commands on the director to block all services on the VIP except those you want forwarded to the real-servers. <sect2>Multiple proxies From: Jeremy Johnson <tt/jjohnson@real.com/ <verb> > > how does LVS handles a single client that uses > > multiple proxies... for instance aol, when an aol user attempts to connect > > to a website, each request can come from a different proxy so, how/if does > > LVS know that the request is from the same client and bind them to the same > > server? </verb> (Joe) <verb> > if this is what aol does then each request will be independant and will > not neccessarily go to the same real-server. Previous discussions about aol > have assumed that everyone from aol was coming out of the same IP. > Currently this is handled by making the connection persistant and all > connections from aol will go to one real-server. </verb> (Michael Sparks <tt/zathras@epsilon3.mcc.ac.uk/) If ISP user (eg AOL) has a proxy array/farm then the requests are _likely_ to come from two possibilities: * A single subnet (if using an L4/L7 switch that rewrites ether frames, or using several NAT based L4/L7 switches) or * A single IP (If using the common form of L4/L7 switch) The former can be handled using a subnet mask in the persistance settings, the latter is handled by normal persistance. *However* In the case of our proxy farm neither of these would work since we have 2 subnet ranges for our systems - 194.83.240/24 & 194.82.103/24, and an end user request may come out of each subnet totally defeating the persistance idea... (in fact dependent on our clients configuration of their caches, the request could appear to come from the above two subnets or the above 2 subnets and about 1000 other ones as well) Unfortunately this problem is more common that might be obvious, due to the NLANR hierarchy, so whilst persistance on IP/subnet solves a large number of problems, it can't solve all of them. <sect1>Changing weights with ipvsadm When setting up a service, you set the weight with a command like (default for -w is 1). <verb> ipvsadm -a -t $VIP:$SERVICE -r $REALSERVER_NAME:$SERVICE $FORWARDING -w 1 </verb> If you set the weight for the service to "0", then no new connections will be made to that service (see also man ipvsadm, about the -w option). Lars Marowsky-Bree <tt/lmb@suse.de/ 11 May 2001 <quote> Setting weight = 0 means that no further connections will be assigned to the machine, but current ones remain established. This allows to smoothly take a real server out of service, ie for maintenance. Removing the server hard cuts all active connections. This is the correct response to a monitoring failure, so that clients receive immediate notice that the server they are connected to died so they can reconnect. </quote> Laurent Lefoll <tt/Laurent.Lefoll@mobileway.com/ 11 May 2001 <quote> Is there a way to clear some entries in the ipvs tables ? If a server reboots or crashes, the connection entries remains in the ipvsadm table. Is there a way to remove manually some entries? I have tried to remove the real server from the service (with ipvsadm -d .... ), but the entries are still there. Joe <quote> After a service (or real-server) failure, some agent external to LVS will run ipvsadm to delete the entry for the service. Once this is done no new connections can be made to that service, but the entries are kept in the table till they timeout. (If the service is still up, you can delete the entries and then re-add the service and the client will not have been disconnected). You can't "remove" those entries, you can only change the timeout values. Any clients connected through those entries to the failed service(s) will find their connection hung or deranged in some way. We can't do anything about that. The client will have to disconnect and make a new connection. For http where the client makes a new connection almost every page fetch, this is not a problem. Someone connected to a database may find their screen has frozen. </quote> </quote> If you are going to set the weight of a connection, you need to first know the state of the LVS. If the service is not already in the ipvsadm table, you add (-a) it. If the service is already in the ipvsadm table, you edit (-a) it. There is no command to just set the weight no matter what the state. A patch exists to do this (from Horms) but Wensong doesn't want to include it. Scripts which dynamically add, delete or change weights on services will have to know the state of the LVS before making any changes, or else trap errors from running the wrong command. <sect1>experimental scheduling code This section is a bit out of date now. See the <ref id="ipvsadm"> new schedulers by Thomas Prouell for web caches and by Henrik Norstrom for firewalls. Ratz <tt/ratz@tac.ch/ has produced a scheduler which will keep activity on a particular real-server below a fixed level. For this next code write to Ty or grab the code off the list server Date: Wed, 23 Feb 2000 From: Ty Beede <tt/tybeede@metrolist.net/ This is a hack to the ip_vs_wlc.c schedualing algorithm. It is curently implemnted in a quick, ad hoc fashion. It's purpose is to support limiting the total number of connections to a real server. Currently it is implmented using the weigh value as the upper limit on the number of activeconns(connections in an established TCP state). This is a very simple implementation and only took a few minutes after reading through the source. I would like, however, to develop it further. Due to it's simple nature it will not function in several types of enviroments, those based on connectionless protocals (UDP, this uses the inactconns variable to keep track of things, simply change the activeconns varible-in the weigh check- to inactconns for UDP) and it may impose complecations when persistance is implemented. The current algorimthm simply checks that weight > activeconns before including a server in the standard wlc scheduling. This works for my enviroment, but could be changed to perhaps (weight * 50) > (activeconns * 50) + inactconns to include the inactconns but make the activeconns more important in the decison. Currently the greatest weight value a user may specify is approimalty 65000, independant of this modification. As long as the user keeps most importanly the weight values correct for the total number of connections and in porportion to one another the things should function as expected. In the event that the cluster is full, all real severs have maxed out, then it might be neccessary for overflow control, or the client's end will hang. I haven't tested this idea but it could simply be implemented by specifing the over flow server last, after the real severs using the ipvsadm tool. This will work because as each real server is added using ipvsadm it is put on a list, with the last one added being last on the list. The scheduling algorithm traverses this list linearly from start to finish and if it finds that all severs are maxed out, then the last one will be the overflow and that will be the only one to send traffic to. Anyway this is just a little hack, read the code and it should make sense. It has been included as an attachment. If you would like to test this simply replace the old ip_vs_wlc.c scheduling file in /usr/src/linux/net/ipv4 with this one. Compile it in and set the weight on the real severs to the max number of connections in an established TCP state or modifiy the source to your liking. Date: Mon, 28 Feb 2000 From: Ty Beede <tt/tybeede@metrolist.net/ I wrote a little patch and posted it a few days ago... I indicated that overflow might be accomplished by adding the overflow server to the lvs last. This statement is completely off the wall wrong. I'm not really sure why I thought that would work but it won't, first of all the linked list adds each new instance of a real sever to the start of the real servers list, not the end like I though. Also it would be impossible do distingish the overflow server from the real servers in the case that not all the real-servers were busy. I don't know where I got that idea from but I'm going to blame it on my "bushy eyed youth". In responce to needing overflow support I'm thinking about implementing "prority groups" into the lvs code. This would logically group the real severs into different groups, though with a higher priority group would fillup before those with a lower grouping. If anybody could comment on this it would be nice to hear what the rest of you think about overflow code. <sect2>Theoretical issues in developing better scheduling algorithms <verb> (Julian) > > It seems to me it would be useful in some cases to use the total number > > of connections to a real server in the load balancing calculation, in > > the case where the real server participates in servicing a number of > > different VIPs. > > (Wensong) > Yeah, it is true. Sometimes, we need tradeoff between > simplicity/performance and functionality. Let me think more about > this, and probably maximum connection scheduling together together > too. For a rather big server cluster, there may be a dedicated load > balancer for web traffic and another load balancer for mail traffic, > then the two load balancers may need exchange status periodically, it > is rather complicated. Yes, if a real server is used from two or more directors the "lc" method is useless. > Actually, I just thought that dynamic weight adaption according to > periodical load feedback of each server might solve all the above > problems. </verb> (Joe - this is part of a greater problem with LVS, we don't have good monitoring tools and we don't have a lot of information on the varying loads that real-servers have, in order to develope strategies for informed load regulation) (Julian) From my experience with real servers for web, the only useful parameters for the real server load are: <verb> - cpu idle time If you use real servers with equal CPUs (MHz) the cpu idle time in percents can be used. In other cases the MHz must be included in a expression for the weight. - free ram According to the web load the right expression must be used including the cpu idle time and the free ram. - free swap Very bad if the web is swapping. </verb> The easiest parameter to get, the Load Average is always < 5. So, it can't be used for weights in this case. May be for SMTP ? The sendmail guys use only the load average in sendmail when evaluating the load :) So, the monitoring software must send these parameters to all directors. But even now each of the directors use these weights to create connections proportionally. So, it is useful these parameters for the load to be updated in short intervals and they must be averaged for this period. It is very bad to use current value for a parameter to evaluate the weight in the director. For example, it is very useful to use something like "Average value for the cpu idle time for the last 10 seconds" and to broadcast this value to the director on each 10 seconds. If the cpu idle time is 0, the free ram must be used. It depends on which resource zeroed first: the cpu idle time or the free ram. The weight must be changed slightly :) The "*lc" algorithms help for simple setups, eg. with one director and for some of the services, eg http, https. It is difficult even for ftp and smtp to use these schedulers. When the requests are very different, the only valid information is the load in the real server. Other useful parameter is the network traffic (ftp). But again, all these parameters must be used from the director to build the weight using a complex expression. I think the complex weight for the real server based on connection number (lc) is not useful due to the different load from each of the services. May be for the "wlc" scheduling method ? I know that the users want LVS to do everything but the load balancing is very complex job. If you handle web traffic you can be happy with any of the current scheduling methods. I didn't tried to balance ftp traffic but I don't expect much help from *lc methods. The real server can be loaded, for example, if you build new Linux kernel while the server is in the cluster :) Very easy way to switch to swap mode if your load is near 100%. <sect>Fwmarks<label id="fwmark"> <sect1>Introduction The original method for settting up an LVS was to use the VIP as the target for ipvsadm commands. A more flexible method, of using firewall marks (fwmarks) was introduced by Horms in Apr 2000. Ted Pavlic then showed how to group arbitary services using fwmarks, which had previously been possible to a limited extent with LVS persistence. The fwmarks method is more flexible and simpler to administer. fwmark is used <itemize> <item> to group services together within a single LVS e.g. <itemize> <item>group 1 - port 80,443 for an e-commerce site <item>group 2 - port 20,21 for an ftp server </itemize> <item> group large numbers of IP together </itemize> Setting up an LVS on fwmarks rather than the VIP is now the method of choice for anything but a collection of simple one port non-persistent services. Fwmark should be used instead of the VIP when persistence is required or multiport services are involved. Some history (Horms) <quote> The impeteus origionally came out of a VA Linux Systems Professional Services customer who I was called onsite to help sway towards using LVS. My original proposal and implementation was to allow virtual services based on netmasks. Wensong rejected this because of some potential performance issues. I distinctly remember working on the original implementation on a train trip from the Blue Mountains to Sydney's Central Station. By the time I had to change trains go to Wynyard the code was working :) A few days latter Julian came up with the idea of using a fwmark, a feature of the ip_masq code that had been around for a while but wasn't heavily used. Wensong passed this on to me. I wrote the kernel, ipvsadm and ldirectord changes and largely have maintained them ever since. I believe they were included with ipvs-0.9.9. It is of note that as a part of the work that came out of this customer the -R and -S options to ipvsadm were suggested and implemented by myself. These were released just before the inclusion of the fwmark code. This customer was also the impetus for putting together what is now known as Ultra Monkey. All in all quite an interesting outcome for a couple of days on site. Pleasingly I believe that the customer in question is using Ultra Monkey with the fwmark support in LVS. </quote> Sample configurations/topologies for fwmarks are at <url url="http://ultramonkey.org/" name="Ultramonkey">. <sect1>single port service: telnet with fwmarks Assuming you already have setup the networks and default gw for the machines in your LVS, here's how you'd setup telnet without fwmarks (ie the "normal" method, using the VIP as the target for ipvsadm commands) on a two real-server LVS. <verb> #make a table for connections to VIP:telnet, with round robin scheduling #schedule real-server bashfull for connections to VIP:telnet, weight=1, forwarding method=DR #schedule real-server sneezy for connections to VIP:telnet, weight=1, forwarding method=DR director:# ipvsadm -A -t VIP:telnet -s rr director:# ipvsadm -a -t VIP:telnet -r bashfull:telnet -g -w 1 director:# ipvsadm -a -t VIP:telnet -r sneezy:telnet -g -w 1 </verb> Here's how to do the same thing with fwmarks. You first mark the packets with ipchains or iptables. <sect2>ipchains for 2.2.x director <verb> #flush ipchains tables #mark with value=1, tcp packets from anywhere, arriving on eth1 (holds the VIP on my setup), #with dst_addr=192.168.2.110 (the VIP) for port telnet #show ipchains tables director:# ipchains -F director:# ipchains -A input -p tcp -i eth1 -s 0.0.0.0/0 -d 192.168.2.110/32 --dport telnet -m 1 director:# ipchains -L input Chain input (policy ACCEPT): target prot opt source destination ports - tcp ------ anywhere lvs2.mack.net any -> telnet </verb> <sect2>iptables for 2.4.x director The iptables parameters are taken from an example by <url url="http://www.foursticks.com.au/~pschulz/qos/pfifo.sample" name="Paul Schulz">, which I found through <url url="http://www.google.com" name="google">. First put a mark of value=1 on tcp packets which arrive from anywhere with dst_addr=VIP:telnet (the VIP is on eth1 in my setup). <verb> #flush the mangle table #in the skb, put mark=1 on all tcp packets arriving on eth1 from anywhere, with dest=VIP:telnet #output the mangle table, just for a look director:# iptables -F -t mangle director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport telnet -j MARK --set-mark 1 director:/etc/lvs# iptables -L -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- anywhere lvs2.mack.net tcp dpt:telnet MARK set 0x1 Chain OUTPUT (policy ACCEPT) target prot opt source destination </verb> The fwmark is only associated with the packet while it is in the director skb (socket buffer). The packet which emerges from the director and is forwarded to the real-server is a normal (unmarked) packet. (You can't use the director's fwmark information on the real-server to decide on how to handle the packet.) <sect2>install the LVS with ipvsadm <verb> #setup an ipvsadm table for packets with mark=1, schedule them with round robin. #schedule real-server sneezy for connections with mark=1, forwarding method=DR, weight=1 #schedule real-server bashfull for connections with mark=1, forwarding method=DR, weight=1 director:# ipvsadm -A -f 1 -s rr director:# ipvsadm -a -f 1 -r sneezy.mack.net:telnet -g -w 1 director:# ipvsadm -a -f 1 -r bashfull.mack.net:telnet -g -w 1 </verb> Here's the output of ipvsadm <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 0.2.7 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr -> bashfull.mack.net:23 Route 1 0 0 -> sneezy.mack.net:23 Route 1 0 0 </verb> You can now telnet to the VIP. You'll get the expected round robin scheduling of your connections to bashfull and sneezy. <sect1>Grouping services: single group, active ftp(20,21) The telnet example above could equally well be done using the VIP or a fwmark as the target for ipvsadm commands. The same is true for any one port service, where connections to services are made independantly of each other. Sometimes we need to group services together, e.g. port 20,21 for an ftp server or port 80, 443 for an e-commerce site. The current method for handling this, <ref id="persistent_connection" name="persistent connections">, links all ports on the VIP, and the director will forward connections to all ports, not just the two we are interested in. For security purposes, if persistence is used to group services, then connection requests to the other ports will have to be blocked. Although workable, it's an ugly solution. For background on how the specifications for fwmarks were set to allow services to be grouped, see <ref id="fwmarks_persistence" name="Appendix 1"> for the initial discussion between Ted and the LVS developers (Horms and Julian), <ref id="fwmarks_persistence_proof_of_principle" name="Appendix 2"> where Ted let me know that he'd had it working, and <ref id="fwmarks_persistence_announcement" name="Appendix 3"> for Ted's announcement to the mailing list. <sect2>grouping using VIP and persistence Here's an example grouping ports 20,21 for ftp. This uses the VIP as the target for ipvsadm commands with persistent connection (the original, VIP way). <verb> #make a table for connections to all ports on VIP with round robin scheduling, persistence timeout=360secs #schedule real-server bashfull for connections to all ports on VIP, weight=1, forwarding method=DR #schedule real-server sneezy for connections to all ports on VIP, weight=1, forwarding method=DR director:# ipvsadm -A -t VIP:0 -s rr -p 360 director:# ipvsadm -a -t VIP:0 -r sneezy.mack.net:0 -g -w 1 director:# ipvsadm -a -t VIP:0 -r bashfull.mack.net:0 -g -w 1 </verb> Here's the output of ipvsadm <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 0.2.7 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:0 rr persistent 360 -> sneezy.mack.net:0 Route 1 0 0 -> bashfull.mack.net:0 Route 1 0 0 </verb> After the client has made the initial connection on port 21, then any subsequent connection on port 20 (within the 360sec timeout period) will go to the same real-server. The problem is that the director will forward to the same real-server, connection requests made to any port by the client. If we have port 80 and 443 on the real-server, then these services will be linked to each other (which we may want), and they will also be linked to the ftp service (which we may not want). If you telnet to the VIP, this request will be forwarded to the real-servers too (in production you'll have to block this). <sect2>grouping with fwmarks Here's how to setup an ftp server with fwmarks. First mark the packets of interest with ipchains or iptables (i.emark all tcp packets to VIP:ftp and VIP:ftp-data arriving on eth1). <sect3>ipchains for 2.2 director <verb> #flush ipchains tables #mark ftp packets #put the same mark on ftp-data packets #show ipchains tables director:# ipchains -F director:# ipchains -A input -p tcp -i eth1 -s 0.0.0.0/0 -d 192.168.2.110/32 --dport ftp -m 1 director:# ipchains -A input -p tcp -i eth1 -s 0.0.0.0/0 -d 192.168.2.110/32 --dport ftp-data -m 1 director:# ipchains -L input Chain input (policy ACCEPT): target prot opt source destination ports - tcp ------ anywhere lvs2.mack.net any -> ftp - tcp ------ anywhere lvs2.mack.net any -> ftp-data </verb> <sect3>iptables for 2.4 director <verb> #clear mangle table #mark ftp packets #put the same mark on ftp-data packets #show mangle table director:# iptables -F -t mangle director:/etc/lvs# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport ftp -j MARK --set-mark 1 director:/etc/lvs# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport ftp-data -j MARK --set-mark 1 director:/etc/lvs# iptables -L -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- anywhere lvs2.mack.net tcp dpt:ftp MARK set 0x1 MARK tcp -- anywhere lvs2.mack.net tcp dpt:ftp-data MARK set 0x1 Chain OUTPUT (policy ACCEPT) target prot opt source destination </verb> <sect3>install LVS with ipvsadm Next setup ipvsadm to schedule packets marked with fwmark=1 to your real-servers. You need persistence (here timeout set to 600secs). <verb> director:# ipvsadm -A -f 1 -s rr -p 600 director:# ipvsadm -a -f 1 -r sneezy.mack.net:0 -g -w 1 director:# ipvsadm -a -f 1 -r bashfull.mack.net:0 -g -w 1 </verb> Here's the output of ipvsadm with two current connections to the LVS and 3 expiring ones. Note they are all to the same real-server, as expected for a persistent connection. Since forwarding is by VS-NAT, the ip_vs_ftp module automatically loads. <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 0.2.7 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr persistent 600 -> bashfull.mack.net:0 Route 1 2 3 -> sneezy.mack.net:0 Route 1 0 0 </verb> A netpipe test showed the same latency and throughput for a connection based on fwmark or based on VIP. What happens now when you telnet from the client to the VIP? (pause to let you think.) The director is only forwarding packets with fwmark=1 to the LVS, so a telnet request to the VIP is accepted by the director and not forwarded to the real-servers. If telnetd is running on the director, you'll get a login prompt from the director. In production you'll have to block this too (just like you had to when setting up on a VIP). So what's the difference, you ask, between setting up an ftp server with persistence on the VIP on one hand (which requires you to block all other packets with iptables rules), and grouping 20,21 with fwmarks on the other (which requires exactly the same blocking of unwanted packets)? Not a lot. At the moment you're at least even Lars Marowsky-Br�e <tt/lmb@suse.de/ 2000-05-11 <quote> When using the LVS box as a firewall/router, the fwmark technique is a perfectly adequate solution, which doesn't cost anything. </quote> But look at the next example. <sect1>Grouping services: two groups, active ftp(20,21) and e-commerce(80,443) Setup 2 groups of services, group 1 - ftp(20,21), group 2 - ecommerce(80,443). First mark packets in 2 groups. <sect2>ipchains for 2.2 director <verb> director:# ipchains -F director:# ipchains -A input -p tcp -i eth1 -s 0.0.0.0/0 -d 192.168.2.110/32 --dport ftp -m 1 director:# ipchains -A input -p tcp -i eth1 -s 0.0.0.0/0 -d 192.168.2.110/32 --dport ftp-data -m 1 director:# ipchains -A input -p tcp -i eth1 -s 0.0.0.0/0 -d 192.168.2.110/32 --dport http -m 2 director:# ipchains -A input -p tcp -i eth1 -s 0.0.0.0/0 -d 192.168.2.110/32 --dport https -m 2 director:# ipchains -L input Chain input (policy ACCEPT): target prot opt source destination ports - tcp ------ anywhere lvs2.mack.net any -> ftp - tcp ------ anywhere lvs2.mack.net any -> ftp-data - tcp ------ anywhere lvs2.mack.net any -> www - tcp ------ anywhere lvs2.mack.net any -> https </verb> <sect2>iptables for 2.4 director <verb> director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport ftp -j MARK --set-mark 1 director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport ftp-data -j MARK --set-mark 1 director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport http -j MARK --set-mark 2 director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport https -j MARK --set-mark 2 director:/etc/lvs# iptables -L -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- anywhere lvs2.mack.net tcp dpt:ftp MARK set 0x1 MARK tcp -- anywhere lvs2.mack.net tcp dpt:ftp-data MARK set 0x1 MARK tcp -- anywhere lvs2.mack.net tcp dpt:www MARK set 0x2 MARK tcp -- anywhere lvs2.mack.net tcp dpt:https MARK set 0x2 Chain OUTPUT (policy ACCEPT) target prot opt source destination </verb> <sect2>setup LVS to schedule (with persistence) 2 groups of packets Note: The ipvs code in Apr 2001 needed a patch to get the expected behaviour. This section describes the function of LVS before and after patching. As a result of these tests, the patch will be applied to future releases. ipvs-1.0.7-2.2.19 is already patched (Apr 2001). The 2.4.3 series are not patched yet. To see if the code has been patched look in ipvs/Changelog for something like this <quote> Julian changed persistent connection template for fwmark-based service from <CIP,VIP,RIP> to <CIP,FWMARK,RIP>, so that different fwmark-based services that share the same VIP can work correctly. </quote> If your ipvs code is pre-patched, then you can skip down to the part where the behaviour after applying the patch is described. If your code isn't patched, you should just go get the patch and skip to the part where <ref id="fwmark_persistent_patch" name="the expected behaviour is described">. Otherwise here's what happened with the original code. <verb> director:# ipvsadm -A -f 1 -s rr -p 600 director:# ipvsadm -a -f 1 -r sneezy.mack.net:0 -g -w 1 director:# ipvsadm -a -f 1 -r bashfull.mack.net:0 -g -w 1 director:# ipvsadm -A -f 2 -s rr -p 600 director:# ipvsadm -a -f 2 -r sneezy.mack.net:0 -g -w 1 director:# ipvsadm -a -f 2 -r bashfull.mack.net:0 -g -w 1 </verb> <verb> IP Virtual Server version 0.2.7 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr persistent 600 -> bashfull.mack.net:0 Route 1 0 0 -> sneezy.mack.net:0 Route 1 0 0 FWM 2 rr persistent 600 -> bashfull.mack.net:0 Route 1 0 0 -> sneezy.mack.net:0 Route 1 0 0 </verb> If you ftp and http to the VIP, you'd expect the ftp connections to go to fwmark 1 (presumably to the first real-server bashfull) and the http connections to go to fwmark 2 (again presumably to bashfull). With the director running 1.0.6-2.2.19 (ipvs/kernel version), all connections (ftp, http) go to group 1. With the director 0.2.7-2.4.2, all connections go to group 2. Here's the output from ipvsadm for the 2.2.19 example immediately after downloading a webpage. You would expect the http InActConn to be associated with FWM2. <verb> IP Virtual Server version 1.0.6 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr persistent 30 -> bashfull.mack.net:0 Route 1 0 2 -> sneezy.mack.net:0 Route 1 0 0 FWM 2 rr persistent 30 -> bashfull.mack.net:0 Route 1 0 0 -> sneezy.mack.net:0 Route 1 0 0 director:/etc/lvs# </verb> It appears (Apr 2001) that the ipvs code doesn't really follow the <ref id="fwmarks_persistence" name="persistent fwmarks spec">. When there is a collision between VIP space and fwmark space (eg in these examples, where all packets are going to the same VIP), then the VIP takes precedence and the two fwmark groups are not differentiated. The collision arises because there is only one set of templates for the connection tables. The code to produce the expected behaviour requires a separate set of templates for fwmarks and VIP. The patch to do this is on <url url="http://www.linuxvirtualserver.org/~julian" name="Julian's patch page"> and has names like persistent-fwmark-0.2.8-2.4-1.diff, persistent-fwmark-1.0.5-2.2.18-1.diff. (Note: the 0.2.8 patch had DOS carriage control and wouldn't patch till I removed the ^M characters). (Note: as of ipvs-0.9.0, this patch has been applied to the source tree.) <label id="fwmark_persistent_patch"> After patching the ip_vs code to produce the new ip_vs.o module (rmmod the old one first), you get the expected fwmark behaviour. Here's the output of ipvsadm after ftp'ing and http'ing from a client. Note that the ftp connection is to fwmark=1. The InActConn is the expiring connection from the http client to fwmark=2. <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr persistent 30 -> bashfull.mack.net:0 Route 1 1 0 -> sneezy.mack.net:0 Route 1 0 0 FWM 2 rr persistent 30 -> bashfull.mack.net:0 Route 1 0 1 -> sneezy.mack.net:0 Route 1 0 0 </verb> <sect2>The original idea from Ted Pavlic Ted Pavlic <tt/tpavlic@netwalk.com/ 2000-10-08 Just another persistence option that you may or may not have thought of... LVS does support port-group sticky persistance. Before FWMARK support was added to LVS, the only types of persistance one could do were: <itemize> <item>One port persistence (all queries to 80 return to the same real server per CIP) <item>ALL port persistence (all queries to all ports return to the same RIP per CIP) </itemize> But now that FWMARK support exists in LVS, it is easy to create group-based sticky persistence. That is... It adds the option where: <itemize> <item> Only these two ports (443 and 80) return to the same RIP per CIP <item> Meanwhile, another persistence table keeps track of 20, 21, and 1024:65535 <item> Any other port is not persistent </itemize> Just have ipchains keep track of flagging the incoming packets with the correct port group identifier: <verb> ipchains -A input -D VIPNET/VIPMASK PORT -p PROTOCOL -m FWMARK </verb> And have IPVS stop looking at IPs and start look at FWMARKs: <verb> ipvsadm -A -f FWMARK ipvsadm -a -f FWMARK -r RIP:0 </verb> <sect2>ssl and cookies Ted Pavlic <tt/tpavlic@netwalk.com/ 2000-10-13 LVS DIRECTLY supports two types of persistence and INDIRECTLY supports another. If you are just asking how to make port 443 persistent so that those who receive a cookie on 443 will come back to the same real server on 443, simply: <verb> /sbin/ipvsadm -A -t 192.168.1.110:443 -p /sbin/ipvsadm -a -t 192.168.1.110:443 -R 192.168.2.1 /sbin/ipvsadm -a -t 192.168.1.110:443 -R 192.168.2.2 /sbin/ipvsadm -a -t 192.168.1.110:443 -R 192.168.2.3 ... </verb> Will setup persistence just for port 443. However, say someone gets a cookie on port 80 and gives it back on port 443 -- in that case you want to have persistence between multiple ports. Using port 0 accomplishes this: <verb> /sbin/ipvsadm -A -t 192.168.1.110:0 -p /sbin/ipvsadm -a -t 192.168.1.110:0 -R 192.168.2.1 /sbin/ipvsadm -a -t 192.168.1.110:0 -R 192.168.2.2 /sbin/ipvsadm -a -t 192.168.1.110:0 -R 192.168.2.3 ... </verb> In this setup, anyone who visits ANY service will continue to go back to the same real server. So requests which come in on 80 or 443 will continue to come in to the same real server regardless of port. This is an OK solution, but it basically makes all services persistent which might mess up scheduling. That is, this is a decent solution but sometimes not extremely desirable. If you want to simply group ports 80 and 443 together, you need to do something more intuitive. Use FWMARK... <verb> ipchains -A input -d 192.168.1.110/32 80 -p tcp -m 1 ipchains -A input -d 192.168.1.110/32 443 -p tcp -m 1 /sbin/ipvsadm -A -f 1 -p /sbin/ipvsadm -a -f 1 -R 192.168.2.1 /sbin/ipvsadm -a -f 1 -R 192.168.2.2 /sbin/ipvsadm -a -f 1 -R 192.168.2.3 ... </verb> Now only port 80 and 443 will be grouped together via persistence. Any other ipvsadm rules will be completely separate. This means that you can make 80 and 443 persistence by their own little "port group" and leave ports 25 and 110 (for example) not persistent. OR... You could group all the FTP ports together as well on a completely different persistence group... i.e. <verb> ipchains -A input -d 192.168.1.110/32 80 -p tcp -m 1 ipchains -A input -d 192.168.1.110/32 443 -p tcp -m 1 /sbin/ipvsadm -A -f 1 -p /sbin/ipvsadm -a -f 1 -R 192.168.2.1 /sbin/ipvsadm -a -f 1 -R 192.168.2.2 /sbin/ipvsadm -a -f 1 -R 192.168.2.3 # Really adding port 20 isn't needed ipchains -A input -d 192.168.1.110/32 20 -p tcp -m 2 ipchains -A input -d 192.168.1.110/32 21 -p tcp -m 2 ipchains -A input -d 192.168.1.110/32 1024:65535 -p tcp -m 2 /sbin/ipvsadm -A -f 2 -p /sbin/ipvsadm -a -f 2 -R 192.168.2.1 /sbin/ipvsadm -a -f 2 -R 192.168.2.2 /sbin/ipvsadm -a -f 2 -R 192.168.2.3 ... </verb> and again Wayne wrote <quote> Is there a easy way to relating server in both port 80 and port 443 (with VS-NAT)? Say I have two farms, each with same three servers. One farm load balancing HTTP requests and another farm load balancing HTTPS farms. To make sure the user in the persistent mode connected to the HTTP server always go to the same server for HTTPS service, we would like to have some way to relate the services between the two farms, is there a easy way to do it? </quote> ratz <tt/ratz@tac.ch/ 2001-01-03 Two possibilities to solve this with LVS <enum> <item> Use port 0 in your setup. (adv.: easy to set up and easy understand) <item> Use fwmark and group them together. (adv.: finer port granularity possible) </enum> Example (1): <verb> ipvsadm -A -t 192.168.1.100:0 -s wlc -p 333 -M 255.255.255.255 ipvsadm -a -t 192.168.1.100:0 -r 192.168.1.1 -g -w 1 ipvsadm -a -t 192.168.1.100:0 -r 192.168.1.2 -g -w 1 </verb> Example (2): <verb> ipchains -A input -j ACCEPT -p tcp -d 192.168.1.100/32 80 -m 1 -l ipchains -A input -j ACCEPT -p tcp -d 192.168.1.100/32 443 -m 1 -l ipvsadm -A -f 1 -s wlc -p 333 -M 255.255.255.255 ipvsadm -a -f 1 -r 192.168.1.1 -g -w 1 ipvsadm -a -f 1 -r 192.168.1.2 -g -w 1 </verb> <sect1>passive ftp<label id="fwmark_passive_ftp"> You can setup <ref id="passive_ftp" name="passive ftp"> with the VIP as the target using persistence. This is not a particular satisfactory solution, as connect requests to all ports will be forwarded. As well, if another service on the real-server fails (eg http), then all services have to be failed out together. Here's a solution to passive ftp from Ted Pavlic using fwmark. This allows setting up passive ftp independantly of other services. Passive ftp listens on an unknown and unpredictable high port on real-server. This is handled by forwarding requests to all high ports (it's still ugly, but at least this way, we can fail out ftp independently of other services). <sect2>test session with active ftp Here's ftp setup in active mode, as a control. <verb> director:# iptables -L -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- anywhere lvs2.mack.net tcp dpt:ftp MARK set 0x1 MARK tcp -- anywhere lvs2.mack.net tcp dpt:ftp-data MARK set 0x1 # #setup ipvsadm, making all packets with mark=1 persistent director:# ipvsadm -A -f 1 -s rr -p 600 director:# ipvsadm -a -f 1 -r sneezy:0 -g -w 1 director:# ipvsadm -a -f 1 -r bashfull:0 -g -w 1 director:# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr persistent 600 -> bashfull.mack.net:0 Route 1 0 0 -> sneezy.mack.net:0 Route 1 0 0 </verb> Here's netstat -an on the client and the real-server (bashfull) immediately after an ftp file transfer (with the client still connected). <verb> #client: client:~# netstat -an | grep 110 #110 is part of the VIP tcp 0 0 client:1176 VIP:21 ESTABLISHED #real-server bashfull:/home/ftp/pub# netstat -an | grep 254 #254 is part of the client IP tcp 0 0 VIP:20 client:1180 TIME_WAIT tcp 0 0 VIP:20 client:1178 TIME_WAIT tcp 0 0 VIP:20 client:1177 TIME_WAIT tcp 0 0 VIP:21 client:1176 ESTABLISHED </verb> Only port 20,21 are involved here. Here's the command line at the client during the active ftp transfer (all expected output). <verb> ftp> get tulip.c local: tulip.c remote: tulip.c 200 PORT command successful. 150 Opening BINARY mode data connection for tulip.c (104241 bytes). 226 Transfer complete. 104241 bytes received in 0.0232 secs (4.4e+03 Kbytes/sec) </verb> The iptables rules on the director do not allow passive ftp connection. To test this put the ftp client into passive mode. <verb> ftp> pass Passive mode on. ftp> dir 227 Entering Passive Mode (192,168,2,110,4,72) ftp: connect: Connection refused ftp> </verb> connection is not allowed. To check that the system is still functioning, put the client back into active mode. <verb> ftp> pass Passive mode off. ftp> dir 200 PORT command successful. 150 Opening ASCII mode data connection for /bin/ls. total 155178 . . -rw-r--r-- 1 root root 104241 Nov 10 1999 tulip.c 226 Transfer complete. ftp> </verb> <sect2>test session with passive ftp Here's the setup for passive ftp (2.4.x director) (you can leave ipvsadm untouched). <verb> director:# iptables -F -t mangle #mark ftp packets director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport ftp -j MARK --set-mark 1 director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport 1024: -j MARK --set-mark 1 director:# iptables -L -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- anywhere lvs2.mack.net tcp dpt:ftp MARK set 0x1 MARK tcp -- anywhere lvs2.mack.net tcp dpts:1024:65535 MARK set 0x1 </verb> Here's the command line from the ftp client still in active mode <verb> ftp> dir 200 PORT command successful. </verb> The session is hung, the server shows an established connection to port 21 and the client session has to be killed. Here's the passive session. <verb> client:~# ftp VIP Connected to VIP. 220 bashfull.mack.net FTP server (Version wu-2.4.2-academ[BETA-15](1) Wed May 20 13:45:04 CDT 1998) ready. Name (VIP:root): ftp 331 Guest login ok, send your complete e-mail address as password. Password: 230 Guest login ok, access restrictions apply. Remote system type is UNIX. Using binary mode to transfer files. ftp> pass Passive mode on. ftp> cd pub 250 CWD command successful. ftp> dir *.c 227 Entering Passive Mode (192,168,2,110,4,75) 150 Opening ASCII mode data connection for /bin/ls. -rw-r--r-- 1 root root 104241 Nov 10 1999 tulip.c 226 Transfer complete. ftp> mget *.c mget tulip.c? y 227 Entering Passive Mode (192,168,2,110,4,78) 150 Opening BINARY mode data connection for tulip.c (104241 bytes). 226 Transfer complete. 104241 bytes received in 0.0233 secs (4.4e+03 Kbytes/sec) ftp> </verb> Here's the connections at the real-server immediately after the file transfer. There is the regular connection at the ftp port (21) and a connection timing out to a high port on the real-server. <verb> bashfull:/home/ftp/pub# netstat -an | grep 254 #254 is part of the client IP tcp 0 0 VIP:1104 client:1191 TIME_WAIT tcp 0 0 VIP:21 client:1184 ESTABLISHED </verb> Here's the output from ipvsadm after connecting to the URL ftp://vip/ using a web-browser <verb> director:# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr persistent 600 -> bashfull.mack.net:0 Route 1 1 5 -> sneezy.mack.net:0 Route 1 0 0 </verb> <sect2>LVS with 2 groups: group 1 = ftp(active and passive), group 2 = http <verb> #fwmark rules director:# iptables -F -t mangle #active and passive ftp in group 1 director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport ftp -j MARK --set-mark 1 director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport ftp-data -j MARK --set-mark 1 director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport 1024: -j MARK --set-mark 1 #http as group 2 director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport http -j MARK --set-mark 2 director:/etc/lvs# iptables -L -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- anywhere lvs2.mack.net tcp dpt:ftp MARK set 0x1 MARK tcp -- anywhere lvs2.mack.net tcp dpt:ftp-data MARK set 0x1 MARK tcp -- anywhere lvs2.mack.net tcp dpts:1024:65535 MARK set 0x1 MARK tcp -- anywhere lvs2.mack.net tcp dpt:www MARK set 0x2 # #setup LVS for 2 groups director:# ipvsadm -C #ftp (active and passive) are persistent as group 1 director:# ipvsadm -A -f 1 -s rr -p 600 director:# ipvsadm -a -f 1 -r sneezy:0 -g -w 1 director:# ipvsadm -a -f 1 -r bashfull:0 -g -w 1 #http as group 2 (not persistent) director:# ipvsadm -A -f 2 -s rr director:# ipvsadm -a -f 2 -r sneezy:http -g -w 1 director:# ipvsadm -a -f 2 -r bashfull:http -g -w 1 director:# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr persistent 600 -> bashfull.mack.net:0 Route 1 0 0 -> sneezy.mack.net:0 Route 1 0 0 FWM 2 rr -> sneezy.mack.net:80 Route 1 0 0 -> bashfull.mack.net:80 Route 1 0 0 </verb> The client connected (in order) ftp://VIP/, http://VIP/ (passive ftp) and then by active (command line) ftp to VIP. Here's the ipvsadm output. <verb> director:# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr persistent 600 -> bashfull.mack.net:0 Route 1 2 3 -> sneezy.mack.net:0 Route 1 0 0 FWM 2 rr -> sneezy.mack.net:80 Route 1 2 2 -> bashfull.mack.net:80 Route 1 4 0 </verb> Here's the connections showing on the real-server. The most recent ones are at the top of the list. The connection list shows (from the bottom, i.e. in the order of connection), passive ftp, http, and active ftp. <verb> bashfull:/home/ftp/pub# netstat -an | grep 254 #254 is part of the CIP tcp 0 0 VIP:21 client:1207 ESTABLISHED tcp 0 0 VIP:80 client:1206 FIN_WAIT2 tcp 0 0 VIP:80 client:1204 FIN_WAIT2 tcp 0 0 VIP:1108 client:1202 TIME_WAIT tcp 0 0 VIP:21 client:1201 ESTABLISHED </verb> The whole point of this setup is to make ftp and http, which belonged to one persistence group when setup on a VIP, into two groups. Now you can bring the httpd and the ftpd up and down independantly (if you want to fail them out, to change the configuration or software). <sect1>fwmark with VS-NAT (based on a posting by Horms on 14 Jul 2000) Here we setup a VS-NAT LVS on a 2.4.x director. (Note: With 2.4 LVS, the masquerading is setup by the ipvs code, i.e. you don't have to masquerade the packets back from the real-servers). These examples assume that the VIP is on eth1 and your network is already setup (ie the real-servers are using the director as the default gw etc). Mark packets for the VIP and setup the LVS for telnet. (Warning: this first example is not going to get you anything you want.) <verb> # #mark packets director:# iptables -F -t mangle director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 -j MARK --set-mark 1 director:# iptables -L -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- anywhere lvs2.mack.net MARK set 0x1 # #Setup ipvsadm director:# ipvsadm -C director:# ipvsadm -A -f 1 -s rr director:# ipvsadm -a -f 1 -r sneezy.mack.net:telnet -m -w 1 director:# ipvsadm -a -f 1 -r bashfull.mack.net:telnet -m -w 1 director:# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr -> bashfull.mack.net:23 Masq 1 0 0 -> sneezy.mack.net:23 Masq 1 0 0 </verb> You can connect with telnet to the VIP and you'll be forwarded to both real-servers in the expected way. All packets from the client will be marked and processed by the ipvsadm rules. What happens if you attempt to connect to VIP:80 (pause to think)? Here's the answer. <verb> client:~# telnet VIP 80 Trying 192.168.2.110... Connected to lvs2.mack.net. Escape character is '^]'. Welcome to Linux 2.2.19. bashfull login: root Linux 2.2.19. Last login: Fri Apr 13 11:43:52 on ttyp1 from client2.mack.net. No mail. </verb> If you connect to VIP:80 with a browser, it sits there showing the watch symbol for quite a while. What happened? The explanation is that you told the director to mark all packets (i.e. from any port) from the client, rewrite them to have dest_addr=RIP:telnet and forward the rewritten packets to the real-server. So when you telnet'ed to VIP:80, the packets were forwarded to RIP:23. Just to make sure that I'd interpretted this correctly, here's the first packets seen by tcpdump running on the client and the real-server during the connect attempts. (These are from different sessions, so the ports on the client are different each time.) client: here the client is connecting to VIP:80 (lvs2.www) <verb> 12:09:44.449566 client2.1118 > lvs2.www: S 2887976275:2887976275(0) win 5840 <mss 1460,sackOK,timestamp 118456418[|tcp]> (DF) [tos 0x10] 12:09:44.450453 lvs2.www > client2.1118: S 1441372470:1441372470(0) ack 2887976276 win 32120 <mss 1460,sackOK,timestamp 117741798[|tcp]> (DF) 12:09:44.450579 client2.1118 > lvs2.www: . ack 1 win 5840 <nop,nop,timestamp 118456418 117741798> (DF) [tos 0x10] </verb> real-server (bashfull): here the real-server is receiving packets to the RIP:23 (bashfull.telnet) <verb> 11:44:28.319675 client2.1116 > bashfull.telnet: S 2722509719:2722509719(0) win 5840 <mss 1460,sackOK,timestamp 118440378[|tcp]> (DF) [tos 0x10] 11:44:28.319974 bashfull.telnet > client2.1116: S 1283414485:1283414485(0) ack 2722509720 win 32120 <mss 1460,sackOK,timestamp 117725760[|tcp]> (DF) 11:44:28.320681 client2.1116 > bashfull.telnet: . ack 1 win 5840 <nop,nop,timestamp 118440378 117725760> (DF) [tos 0x10] </verb> If you want only telnet requests to be forwarded to the real-servers, you should mark only packets for VIP:telnet. If you want both telnet and http forwarded then you should give them each their own mark. Here's how to setup VS-NAT with fwmark for both telnet and http. <verb> director:# iptables -F -t mangle #telnet packets to the VIP get fwmark=1 director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport telnet -j MARK --set-mark 1 #http packets to the VIP get fwmark=2 director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport http -j MARK --set-mark 2 director:# iptables -L -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- anywhere lvs2.mack.net tcp dpt:telnet MARK set 0x1 MARK tcp -- anywhere lvs2.mack.net tcp dpt:www MARK set 0x2 # #setup ipvsadm director:# ipvsadm -C #forward packets with mark=1 to the telnet port director:# ipvsadm -A -f 1 -s rr director:# ipvsadm -a -f 1 -r sneezy.mack.net:telnet -m -w 1 director:# ipvsadm -a -f 1 -r bashfull.mack.net:telnet -m -w 1 #forward packets with mark=2 to the httpd port director:# ipvsadm -A -f 2 -s rr director:# ipvsadm -a -f 2 -r sneezy.mack.net:http -m -w 1 director:# ipvsadm -a -f 2 -r bashfull.mack.net:http -m -w 1 director:# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr -> bashfull.mack.net:23 Masq 1 0 0 -> sneezy.mack.net:23 Masq 1 0 0 FWM 2 rr -> bashfull.mack.net:80 Masq 1 0 0 -> sneezy.mack.net:80 Masq 1 0 0 </verb> Here's the (expected) output of ipvsadm showing the client with 2 telnet sessions and having just downloaded a webpage from the LVS. <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr -> bashfull.mack.net:23 Masq 1 1 0 -> sneezy.mack.net:23 Masq 1 1 0 FWM 2 rr -> bashfull.mack.net:80 Masq 1 0 1 -> sneezy.mack.net:80 Masq 1 0 0 </verb> <sect1>Collisions between fwmark and VIP rules Since it's possible to write iptables rules that include many different types of packets, it's possible to write VIP and fwmark rules that would conflict by accepting the same packet. Here's a setup that would accept telnet by both VIP and fwmarks. <verb> director:# iptables -t mangle -A PREROUTING -i eth1 -p tcp -s 0.0.0.0/0 -d 192.168.2.110/32 --dport ftp -j MARK --set-mark 1 director:# ipvsadm -A -t lvs2.mack.net:telnet -s rr director:# ipvsadm -a -t lvs2.mack.net:telnet -r sneezy.mack.net:telnet -g -w 1 director:# ipvsadm -a -t lvs2.mack.net:telnet -r bashfull.mack.net:telnet -g -w 1 director:# ipvsadm -A -f 1 -s rr director:# ipvsadm -a -f 1 -r sneezy.mack.net:telnet -g -w 1 director:# ipvsadm -a -f 1 -r bashfull.mack.net:telnet -g -w 1 # director:# iptables -L -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- anywhere lvs2.mack.net tcp dpt:ftp MARK set 0x1 # director:# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:telnet rr -> bashfull.mack.net:telnet Route 1 0 0 -> sneezy.mack.net:telnet Route 1 0 0 FWM 1 rr -> bashfull.mack.net:telnet Route 1 0 0 -> sneezy.mack.net:telnet Route 1 0 0 </verb> Here's the ipvsadm output after 4 telnet connections from a client <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:telnet rr -> bashfull.mack.net:telnet Route 1 2 0 -> sneezy.mack.net:telnet Route 1 2 0 FWM 1 rr -> bashfull.mack.net:telnet Route 1 0 0 -> sneezy.mack.net:telnet Route 1 0 0 </verb> All connections go to the first (here VIP) entries. The same ipvsadm table and connection pattern results if you feed the VIP and fwmarks rules into ipvsadm in the reverse order. This behaviour is not part of the spec (yet). You might want to check the behaviour, if you are doing this sort of setup. <sect1>persistence granularity with fwmark<label id="fwmark_persistence_granularity"> <sect2>introduction Persistence granularity was added to LVS by Lars <tt/lmb@suse.de/ 1999-10-13 <quote> This patch adds netmasks to persistent ports, so you can adjust the granularity of the templates. It should help solve the problems created with non-persistent cache clusters on the client side." </quote> The problem being addressed is that some clients (eg AOL customers) connect to the internet via large proxy farms. The IP they present to the server will not neccessarily be the same for different sessions, even though they remain connected to their proxy machine. Persistence granularity makes all clients from a network equivalent as far as persistence is concerned. Thus a client could appear as CIP=x.x.x.13 for their http connections, but CIP=x.x.x.14 for their https connections. With persistence granularity set to /24, all CIPs from the same class C network will be sent to the same real-server. The default behaviour is for all connections from the same CIP to be sent to the one real-server but other connections from the same network will be scheduled to other real-servers (i.e. the default persistence granularity is /32). Persistence granularity is applied to the CIP and works the same whether you are using fwmark or the VIP to setup the LVS. You set the netmask (granularity) for <ref id="granularity" name="persistence granularity"> with ipvsadm. If the LVS was setup with the following command, the persistence granularity is 255.255.255.0. <verb> ipvsadm -A -t 192.168.1.100:0 -s wlc -p 333 -M 255.255.255.0 </verb> Let's say a client from a class C network (e.g. with IP=100.100.100.2) connects to the LVS. If any other client connects from 100.100.100.0/24 they will also connect to the same real-server as long as the original client's entry in the persistence table has not expired (i.e. the first client is still connected, or disconnected < 333 secs ago). <sect2>examples Here's an example VS-DR LVS set to mark packets for an IP on the outside of the director (this IP serves as the VIP in the usual LVS setup, but there's no such thing as a VIP with fwmarks) with --dport telnet. Persistence granularity is set to the default (-M 255.255.255.255). <verb> director:# ipvsadm -C director:# ipvsadm -A -f 1 -s rr -p 600 director:# ipvsadm -a -f 1 -r sneezy:0 -g -w 1 director:# ipvsadm -a -f 1 -r bashfull:0 -g -w 1 </verb> Two clients (192.168.2.254, 192.168.2.253) connect to the LVS. Each host connects to different real-servers but multiple connects from each client go to the same real-server (i.e. client A always goes to real-server A; client B always goes to real-server B, at least till the persistence timeout clears). Here both clients have connected twice. <verb> director:# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr persistent 600 -> bashfull.mack.net:0 Route 1 2 0 -> sneezy.mack.net:0 Route 1 2 0 </verb> This is the connection pattern expected if the connections were based on the CIP/32 and fwmark (ie all clients are scheduled independently). Here's the same setup with persistence granularity set to /24. <verb> director:# ipvsadm -C director:# ipvsadm -A -f 1 -s rr -p 600 -M 255.255.255.0 director:# ipvsadm -a -f 1 -r sneezy:0 -g -w 1 director:# ipvsadm -a -f 1 -r bashfull:0 -g -w 1 director:# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr persistent 600 mask 255.255.255.0 -> bashfull.mack.net:0 Route 1 0 0 -> sneezy.mack.net:0 Route 1 0 0 </verb> Here's what happens when the 2 clients, both of who belong to the same CIP/24 persistence group, connect twice - all connections go to the same real-server. <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr persistent 600 mask 255.255.255.0 -> bashfull.mack.net:0 Route 1 4 0 -> sneezy.mack.net:0 Route 1 0 0 </verb> <sect2>Discussion with Julian about persistence granularity Joe <quote> I expect if you were using persistence with fwmark, then any connection requests arriving with the same fwmark will be treated as belonging to that persistence group. Presumably any combination of client IPs and/or networks could have been used to make the rules which marks the packets. </quote> Julian Yes, it is for the same group but in one fwmark group there are many templates created. These templates are different for the client groups. The template looks like this: CIPNET:0 -> SERVICE(FWMARK/VIP):0 -> RIP:0 All ports 0 for the fwmark-based services So, for client 10.1.2.3/24 (24=persistent granularity) the template looks like this: 10.1.2.0:0 -> VIP:0 -> RIP:0 LVS patched with the persistent_fwmark patch: 10.1.2.0:0 -> FWMARK:0 -> RIP:0 So, the templates are created with CIP/GRAN in mind and the lookup uses CIPNET too. We use CIPNET = CIP & CNETMASK before creation and lookup. <quote> so if I did <verb> iptables -s 10.1.2.3 -m 1 ipvsadm -A -f 1 -s rr -p 600 -M 255.255.255.0 </verb> only packets from 10.1.2.3 will have a fwmark on them, but the director would forward all packets from 10.1.2.0/24, even those without fwmarks? </quote> The patched LVS will accept only the marked packets for this fwmark service, from the same /24 client subnet. If only one client IP sends packets that are marked then the real service will receive packets only from 10.1.2.3. The current LVS versions don't consider the service and all packets CIPNET -> VIP will be forwarded using the first created template for CIPNET:0->VIP:0, i.e. these packets will randomly hit one of the many services that accept packets for the same VIP (just like in your setup) and then may be a wrong real server. <quote> <quote> The current LVS versions don't consider the service and all packets CIPNET -> VIP </quote> but there is no VIP here, I'm using fwmark only. what does the -M 255.255.255.0 do in this case? </quote> The current LVS versions (i.e. without the persistent_fwmark patch) assume the VIP is the iphdr->daddr, i.e. the destination address in the datagram and this addresses is used to lookup/create the template. <quote> how about your persistent-patch, which I've been working with? </quote> The patch ignores this daddr when creating or looking for templates. Instead, the service fwmark values is used when the service is fwmark-based: CIPNET:0 -> FWMARK:0 -> RIP:0 The normal services use daddr as VIP when looking for or creating templates: CIPNET:0 -> daddr:0 -> RIP:0 The persistence is associated with the client address (CIP). The sequence is this: - packet comes from CIP to VIP1 - fw marking, optional - lookup for existing connection CIP:CPORT->VIP1:VPORT, if yes => forward, if not found: - lookup service => fwmark 1, persistent - try to select real service in context of the virtual service Apply the persistence granularity to the client address CIPNET = CIP & svc->netmask Now lookup for template <enum> <item> not patched: check for existing template CIPNET:0, VIP1:0 <item> patched: check for existing template CIPNET:0, 1(fwmark):0 </enum> if there is template, bind the new connection to the template's destination if there is no existing template, get one destination using the scheduler and bind it to the newly created template and the new connection. The created template is <enum> <item> CIPNET:0, VIP1:0, DEST_RIP:0 <item> CIPNET:0, 1(fwmark):0, DEST_RIP:0 </enum> - forward the packet <quote> Persistence granularity was designed for people coming in from large proxy servers (eg AOL). With fwmarks, this can be handled by iptables rules. </quote> Yes, the fact that we group the clients using this netmask is not related to the virtual service type: normal or fwmark-based. Yes, each different IP is treated as different client. When a netmask < 32 is used, the group of addresses is treated as one client when applying the persistence rules. This is not related to the packet marking and virtual service type. <sect1>fwmark allows VS-DR director to be default gw for real-servers If a VS-DR director is accepting packets by fwmarks, then it does not have a VIP. The director can then be the default gw for the real-servers (see <ref id="VS-DR_director_default_gw" name="VS-DR director is default gw for real-servers">). <sect1>Routing to director and real-servers in an LVS setup with fwmark<label id="routing_fwmark"> If you are using fwmark as the target for the LVS, the destination addresses could be any arbitary grouping of IPs. Since there is no interface on the director with any of these IPs, then some method is needed so that packets destined for the LVS get to the director. This same problem occurs with <ref id="TP_QandA" name="transparent proxy">. The solution is to configure the router to send all those packets to the director. Once the packets have been forwarded from the director, in the case of VS-DR some method is needed for the real-server to accept the packets. With VS-DR there is no routing problem; the director sends the packets to the interface with the MAC address of the RIP. However the real-server must recognise that the packets are to be processed locally. Several methods are available. <itemize> <item> The real-server can accept the packets by transparent proxy <item> an alias can be set to accept a network of addresses, without assigning an IP to the device Horms <tt/horms@vergenet.net/ 10 Apr 2000 <verb> ifconfig lo:0 192.168.1.0 netmask 255.255.255.0 </verb> You can now ping anything in the 192.168.1.0/24 network from the console. Note: You can't ping any of those IP's from a remote host (after adding a route on the remote host to this network). If you put this network onto an eth0 alias (rather than an lo alias), it doesn't reply to pings from the console - presumably the ping replies in the lo:0 case are coming from 127.0.0.1. For another example of routing to an interface without an IP, see <ref id="route_on_non_ip_interface" name="routing to real-servers from director in VS-DR">. </itemize> <sect1>fwmark simplifies configuration for large numbers of addresses If a fwmark rule accepts packets for a /24 network, then 254 IPs are configured in one instruction. The next sections are examples. <sect1>Example: firewall farm Horms <tt/horms@vergenet.net/ 2000-12-06 Assume that packets from out local network (192.168.0.0/23) are outgoing traffic. Mark all outgoing packets with fwmark 1 <verb> ipchains -A input -s 192.168.0.0/23 -m 1 # Now, set up a virtual service to act on the marked packets ipvsadm -A -f 1 ipvsadm -a -f 1 -r 192.168.1.7 ipvsadm -a -f 1 -r 192.168.1.8 ipvsadm -a -f 1 -r 192.168.1.9 </verb> Where 192.168.1.7, 192.168.1.8 and 192.168.1.9 are your firewall boxen. <sect1>Example: LVS'ing a CIDR block Matthew S. Crocker wrote: <quote> would like to put a CIDR block of addresses (/25) through my LVS server. Is there a way I can set one entry for a VIP range and then the load balancing will be handled over the entire range. </quote> Horms <tt/horms@vergenet.net/ 2001-01-13 Set up fwmark rules on the input chain to match incoming packets for the CIDR and mark them with a fwmark. e.g. <verb> ipchains -A input -d 192.168.192.0/24 -m 1 </verb> Use the fwmark (1 in this case) as the virtual service. <verb> ipvsadm -A -f 1 ipvsadm -a -f 1 -r 10.0.0.1 ipvsadm -a -f 1 -r 10.0.0.2 </verb> <sect1>Example: forwarding based on client source IP client A (from 192.x.x.x) should go to real-server 1..3, and client B (from 10.x.x.x) should go to real-server 4..6. (Julian, 10-05-2000) Write fwmark rules based on the source IP of the packets. Then create two virtual services, one for each fwmark. <sect1>Example: load balancing multiple class C networks Ian Courtney wrote: <quote> Basically here at our ISP, we tend to have 2-3 Class C's worth of hosting per server. We would like to move the the LVS, but I'm not exactly sure how I should be setting it up. </quote> Chris <tt/chris@isg.de/ 2001-01-15 You can use the fwmark option for the loadbalancing <verb> #mark the incoming packets with ipchains ipchains -A input -s 0.0.0.0/0 -d 192.168.0.0/24 -m 1 #then you can setup your LVS like ipvsadm -A -f 1 -s wlc ipvsadm -a -f 1 -r 10.10.10.15 -g ipvsadm -a -f 1 -r 10.10.10.16 -g </verb> the router should point to the director. Ian Courtney wrote back: <quote> It didn't work until I aliased all 3 class C's to my director. Do I have to do this? </quote> Julian Anastasov <tt/ja@ssi.bg/ 2001-01-16 Yes, only the packets destined for local addresses/networks are accepted. The others are dropped or forwarded to another box. <quote> the next project involves redoing our standard linux web space, which so far consists of about 8 webservers, each hosting atleast 2 class C's worth of hosting. I some how don't think Linux will take nicely to have 16 or more class C's aliased to it. </quote> If possible use netmask <24. I assume you execute (replace with the right Class C nets): <verb> ifconfig lo:1 207.228.79.0 netmask 255.255.254.0 ifconfig lo:2 207.148.155.0 netmask 255.255.255.0 ifconfig lo:3 207.148.151.0 netmask 255.255.255.0 </verb> on the director and on each real server and solve the arp problem using: <verb> echo 1 > /proc/sys/net/ipv4/conf/all/hidden echo 1 > /proc/sys/net/ipv4/conf/lo/hidden </verb> in the real servers. If you don't want to advertise these addresses using ARP to the Cisco LAN, you can execute the above two commands in the director too. <sect1>Example: proxy server Thomas Proell, 16 Aug 2000 <quote> How do you use fwmark if you want the director to accept packets for a wide range of addresses, for which is doesn't have IPs. </quote> (Horms) Here's a setup I used... <verb> Internet | Router 192.168.128.1 "client" Linux Director | va2-------------------------va3-----------------+--------- proxy (va4) 192.168.16.3 192.168.16.1 192.168.128.2 192.168.128.5 </verb> I have used 192.168/16, but these could be real addresses too. I have only put one proxy server in the diagram but I did test it with 2 <verb> Client: default gw va3 (192.168.16.1) Linux Director: eth0: 192.168.128.2 (internet/proxy side) eth1: 192.168.16.1 (client side) Default gw: Router ,192.168.128.1 IPV4 forwarding enabled. Ipvsadm rules - these can be translated into ldirectord configuration. ipvsadm -A -f 1 -s wlc ipvsadm -a -f 1 -r 192.168.128.3:0 -g -w 1 ... add additonal proxy servers </verb> Interestingly enough if you add a proxy that just forwads traffic then it will end up going direct. This may be useful as a failback server if the proxy servers fail. <verb> ipchains -A input -s 0.0.0.0/0.0.0.0 -d 127.0.0.1/255.255.255.255 -j ACCEPT ipchains -A input -s 0.0.0.0/0.0.0.0 -d 192.168.128.2/255.255.255.255 -j ACCEPT ipchains -A input -s 0.0.0.0/0.0.0.0 -d 192.168.16.2/255.255.255.255 -j ACCEPT ipchains -A input -s 0.0.0.0/0.0.0.0 -d 0.0.0.0/0.0.0.0 80 -p tcp -j REDIRECT 80 -m 1 </verb> The -m 1 means that IPVS will regognise packets patched by this filter as belonging to the virtual service as long as it sees the packets as local. -j REDIRECT 80 makes the packets appear as local. It is of note that the port you redirect to is _ignored_ because of the way IPVS works - paickets using fwmark are sent to the port they arrived on. This means that packets will be sent to proxy servers as port 80 traffic. <verb> Proxy: eth0: 192.168.128.5 Default gw: 192.168.128.1 (router) IPV4 forwarding enabled. ipchains -A input -s 0.0.0.0/0.0.0.0 -d 127.0.0.1/255.255.255.255 -j ACCEPT ipchains -A input -s 0.0.0.0/0.0.0.0 -d 192.168.128.5/255.255.255.255 -j ACCEPT ipchains -A input -s 0.0.0.0/0.0.0.0 -d 0.0.0.0/0.0.0.0 80:80 -p 6 -j REDIRECT +8080 </verb> Note, this is where the redirection to port 8080 takes place. <sect1>Example: transparent web cache On Mon, May 08, 2000 at 11:18:08AM +0700, Pongsit@yahoo.samart.co.th wrote: <verb> > If i would like to use LVS to balance 3 transparent proxy is this how i do it ? > > > Internet > | > | > ------------------------------------------- hub 1 > | | | > |eth0 | | proxy1 ,2 and 3 set as a > proxy1 proxy2 proxy3 transparent proxy with firewall > |eth1 | | where eth0 connect to internet > | | | and eth1 to the internal network > ___________________________________________ > | | | | | hub 2 > | | | | | > LVS/DR client machines | > | > | > ___________________________________________ hub 3 if i have more internel > users > </verb> Horms <tt/horms@vergenet.net/ 2000-05-08 If you want to do transparent proxying then I would suggest a topology more along the lines of: <verb> Internet | | ------------------------------------------------ hub 1 | | LVS/DR | | ________________________________________________ | | | | | | | hub 2 | | | | | | | proxy1 proxy2 proxy3 client machines | | | _________________________________________________ hub 3 if i have more internel users </verb> Use IP chains mark all outgoing port 80 traffic, other than from the 3 proxy servers with firewall mark 1 (ipchains -m 1...). Set up a IPVS virtual service matching of fwmark 1 (ipvsadm -A -f 1...). The proxy servers will need to be set up to recognise all port 80 traffic forwarded to them as local. This way all outgoing traffic hits the LVS box. If it is for port 80 and isn't from one of the proxy servers then it gets load balanced and forwarded to one of the proxy servers. You may want to consider a hot standby LVS/DR host to eliminate a single point of failure on your network. I haven't tested this but I think it should work. <sect1>Example: Dynamically generated images in webpages<label id="dynamic"> One of the assumptions of setting up an LVS is that the content presented on the real-servers is identical. This is required because the client can be sent to any of the real-servers. This requirement is not handled if the client fills in a form which produces a gif on the real-server. Alois Treindl <tt/alois@astro.ch/ 30 Apr 2001 <quote> If a page is created by a CGI and contains dynamically created GIFs, the requests for these gifs will land on a different realserver than the one where the cgi runs. Will I need persistence? I am running an astrology site; a typical request is to a CGI which creates an astrological drawing, based on some form data; this drawing is stored as a temporary GIF file on the server. A html page is output by the CGI which contains a reference to this GIF. The browser receives the html, and then requests the GIF file from the server. It will mostly hit a different server than the one who created the GIF. So either we make sure that the new client request for the GIF hits the same realserver which ran the CGI (i.e. have persistence) or we must create the GIF on a shared directory, so that each realserver sees it. I have not tested it yet (not ported the CGIs yet to the new LVS box) but I think things are not so simple. In a 'rr' scheduling configuration, for example, the scheduler could play dirty, depending on the number of http requests for the given page, and the number of realservers. Both could be incommensurable in a way that the http request for the GIF never reaches the same realserver as the one which ran the CGI request. I had already decided that I need shared directories between all real servers for our CGI environment which does computationally expensive things all the time. Some CGIs create also data files which are used by later CGIs. It is either shared directories for such files, or a shared database (which we also use). These temp files will be sitting in the RAM cache of the NFS server, so that only network bandwidth between the realservers and the NFS server is the limiting factor. This is why I give the NFS server 2 gb of RAM, the max it will physically take, and this is why I chose 2.2.19 as the kernel because it contains NFS-3, which is said to be faster than NFS-2. </quote> (Joe) I tested it here on a page which generates a gif for the client. I found that I could never get the gif. Presumably after downloading the page containing the reference to the gif, the round robin scheduler sends the request for the gif to another real-server. Presumably even page counters will have this problem. Writing to a shared directory should work. Here's a solution with persistent fwmark using ip_tables to setup on a 2.4.x kernel. (Note: for page counters, this method will increment for each real-server, and not for the total page count over all the real-servers as would happen with a shared directory.) <verb> #put fwmark=1 on all tcp packets for VIP:http arriving on eth0 director:# iptables -t mangle -A PREROUTING -i eth0 -p tcp -s 0.0.0.0/0 -d 192.168.1.110/32 --dport http -j MARK --set-mark 1 #setup a 2 real-server LVS to persistently forward packets with fwmark=1 using rr scheduling. director:# ipvsadm -A -f 1 -s rr -p 600 director:# -a -f 1 -r sneezy.mack.net:0 -g -w 1 director:# -a -f 1 -r bashfull.mack.net:0 -g -w 1 #output setup director:# iptables -L -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- anywhere lvs.mack.net tcp dpt:http MARK set 0x1 director:# ipvsadm IP Virtual Server version 0.2.11 (size=16384) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr persistent 600 -> bashfull.mack.net:0 Route 1 0 0 -> sneezy.mack.net:0 Route 1 0 0 </verb> Here's the output of ipvsadm after the successful generation and display of the dynamically generated gif. Note all connections went to one real-server. <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 0.2.11 (size=16384) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr persistent 600 -> bashfull.mack.net:0 Route 1 5 3 -> sneezy.mack.net:0 Route 1 0 0 </verb> <sect1>Appendix 1: Specificiations for grouping of services with fwmarks<label id="fwmarks_persistence"> Here are the discussions that has resulted in the current specifications for handling of persistence with fwmarks in LVS. Ted Pavlic Jul 14, 2000 <quote> What I was asking about would be something like this: <verb> virtual=192.168.6.2-192.168.6.30:80 real=192.168.6.240:80 gate service=http request="index.html" receive="Test Page" scheduler=rr </verb> I have 1029 virtual servers -- that is I have 1029 hosts which need to be load balanced. </quote> Horms <tt/horms@vergenet.net/ 2000-07-14 (fwmark) has the advantage of simplfying the amount of _kernel_ configuration that has to be done which is a big win, even if this is automated by a user space application. The basic idea is that this provides a means for LVS to have virtual services that have more than one host/port/protocol triplet. In your situation this means that you can have a single virtual service that handles many virtual IP addresses and all ports and protocols (UDP, TCP and You should take a look at <url url="http://ultramonkey.sourceforge.net/" name="ultramonkey"> (note from Joe, UM is now 1.0.2, look for examples there). My understanding is that this is quite similar to how your LVS topology will be set up, though I understand you will be having more than one of these configured. Basically what happens is that you set up LVS to consider any packets like other LVS virtual services other than that no VIP is specified. e.g. <verb> ipvsadm -A -f 1 -s rr ipvsadm -a -f 1 -r 192.168.6.3:80 -m ipvsadm -a -f 1 -r 192.168.6.2:80 -m ipvsadm -L -n IP Virtual Server version 0.9.11 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 rr -> 192.168.6.3:80 Masq 1 0 0 -> 192.168.6.2:80 Masq 1 0 0 </verb> The other half of the equation is that ipchains is used to match incoming traffic for virtual IP addresses and mark them with fwmark 1. Say you have 8 contiguous class C's of virtual addresses beginning at 192.168.0.0/24. The ipchains command to set up matching of these packets would be: <verb> ipchains -A input -d 192.168.0.0/21 -m 1 </verb> You also need to set up a silent interface so that the LVS box sees traffic for the VIPs as local. To do this use: <verb> ifconfig lo:0 192.168.0.0 netmask 255.255.248.0 mtu 1500 echo 1 > /proc/sys/net/ipv4/conf/all/hidden echo 1 > /proc/sys/net/ipv4/conf/lo/hidden </verb> Now, as long as 192.168.0.0/21 is routed to the LVS box, or more particularly the floating IP address of the LVS box brought up by heartbeat, traffic for the VIPs will be routed to the LVS box, the ipchains rules will mark it with fwmark 1 and LVS will see this fwmark and consider the traffic as destined for a virtual service. Ted Jul 14, 2000 <quote> for me to enable persistent connections to every port using direct routing, would this work? <verb> ipvsadm -A -f 1 -s rr -p 1800 ipvsadm -a -f 1 -r 216.69.192.201:0 -g ipvsadm -a -f 1 -r 216.69.192.202:0 -g </verb> </quote> Horms Yes, that would work. The port in the "ipvsadm -a" commands is ignored if the real servers are being added to a fwmark service. Connections will be sent to the port on the real server that they will be recieved on the virtual server. So port 80 traffic will go to port 80, port 443 traffic will go to port 443 etc... As a caveat you should really make sure that your ipchains statments catch all traffic for the given addresses including ICMP traffic so ICMP traffic is handled correctly by LVS. (Julian on catching ICMP traffic) IIRC, this is already not a requirement in the last LVS versions. If we look in skb->fwmark for ICMP packets it is impossible to use normal and fwmark virtual services to same VIP because we can't create such ipchains rules. The good news is that in 2.4 (0.0.3) the virtual service lookup (the fwmark field) is used only for the new connections. In 2.2 the service is looked up even for existing entries but we don't want to break the MASQ code entirely Ted Pavlic <tt/tpavlic@netwalk.com/ 19 Jul 2000 When using fwmark to assign real servers to virtual servers, how is scheduling and persistence handled? In my particular example, I have: 216.69.196.0/22 (ie 4 class C networks) all marked with a fwmark of 1. ipvsadm setup is <verb> Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 lc persistent 600 -> nw01:0 Route 1 0 0 -> nw02:0 Route 1 0 0 </verb> Say someone connects to 216.69.196.1 and the connection is assigned to nw01. At this point ipvsadm shows <verb> Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 lc persistent 600 -> nw01:0 Route 1 1 0 -> nw02:0 Route 1 0 0 </verb> A new person connects to another IP in 216.69.196.0/22 (say 216.69.196.2). Will this new connection to 216.69.196.2 go to nw02 because it has the least number of TOTAL connections, or will it go to nw01 because for that PARTICULAR IP, both have 0 connections? Now then say that the person who just connected to 216.69.196.1 makes a connection (within the 600 persistence seconds) to 216.69.196.3. Will this new connection go to nw01 because it's being persistent? Or will it go to either server depending on the number of connections? Here's what I think would be the best way to do things... If multiple IPs are marked with FWMARK 1, LVS should consider them all one entry in its active/inactive table. I don't believe that's how things are currently being handled. (Julian) The templates are not accounted in the active/inactive counters. <quote> (Joe, almost a year later - Julian, what do you mean here?) (Julian 13 Apr 2001) Ted here thinks that the templates are accounted in the inactive/active counters. And before the persistent-fwmark patch we can have many templates for one fwmark-based service: <verb> CIPNET:0 -> VIP1:0 -> RIP_A:0 CIPNET:0 -> VIP2:0 -> RIP_B:0 </verb> where VIP1 and VIP2 are marked with same fwmark. Ted recommends these two templates to be replaced with one, i.e. just like in the persistent-fwmark patch: CIPNET:0 -> FWMARK:0 -> RIP1:0 We can't see the templates (which are normal connection entries with some reserved values in the connections structure fields) accounted in the inactive/active counters. The reason for this is that the inactive/active counters are used to represent the real server load but our templates don't lead to any load in the real servers, we use them only to maintain the persistence. </quote> When a service is marked persistent all connections from CIP to VIP go to same RIP for the specified period. Even for the fwmark based services. This works for many independent VIPs. The other case is fwmark service covering a DNS name. I expect comments from users with SSL problems and persistent fwmark service. Is there a problem or may be not? I agree, may be the both cases can be useful: <verb> 1. CIP->VIP 2. CIP->FWMARK </verb> Any examples where/why (2) is needed? But switching the LVS code always to use (2) for the persistent fwmark services is possible. (Ted) In my opinion, here are some pros and cons of case 2: Pros: Improves scheduling, I think, and true load balancing. If someone is using [W]RR or [W]LC, the LVS box will actually look at the real servers as a whole rather than separate real server entries for EACH VIP. Does that make sense? For example, in my particular configuration I have over one thousand VIPs which are load balanced onto four RIPs. When I configure the LVS server to use LC scheduling, I'd like it to look at how many TOTAL connections are being made to each RIP not how many connections are being made to each RIP PER VIP. I would like to load balance all one thousand VIPs as a WHOLE onto the four RIPs rather than load balance EACH VIP. That is, in some of my less active sites, most of their traffic will probably hit one VIP just because not much traffic will need to be load balanced. However, more active sites will hit both servers. The load will then not be distributed equally among the servers as one server will probably get not only the active traffic but also the less active traffic and the other server will only get the more active traffic (in the case of having two RIPs). Cons: One person on the Internet will keep connecting to the same RIP for many different VIPs if persistence is turned on. If this causes a problem, the LVS administrator can do one of two different things: 1) Rather than load balancing a fwmark template, go back to load balancing specific VIPs. The scheduling will then be unique for those particular VIPs. 2) Create multiple fwmark templates. The scheduling for each template will be unique. In my opinion if you group a bunch of IPs together by marking them with an fwmark, that you say that you want to load balance all of those COLLECTIVELY -- almost like load balancing one site. I'm just saying, are there any examples where CIP->FWMARK is not needed? As far as the LVS is concerned, if someone connects to a VIP marked with fwmark 1, it should treat it just like every other VIP marked with fwmark 1 -- as if they were all one VIP. But today on my LVS (where I have a ten minute persistence setup) I connected to one virtual server marked with fwmark 1 and got a certain real server. I then expected to connect to another virtual server also marked with fwmark 1 and get that same real server. I did not, however. If what you're telling me is correct, the persistence should have connected me to the same real server as long as I was connecting within that ten minute window. Now in this particular example -- connecting to DIFFERENT virtual servers -- it isn't so necessary for persistence to be carried through PER virtual server. I'm just worried that least connection scheduling and round-robin scheduling aren't working at the fwmark level -- I'm worried that they are working at the VIP level as if I had setup hundreds of explicit VIP rules inside IPVSADM. (Julian) > In my opinion, here are some pros and cons of case 2: I hope this feature (2) will be implemented in the next LVS version (if Wensong don't see any problems). I.e. the templates can be changed to case (2) for the persistent fwmark services. For now we (I and Horms) don't see any problems after this change. Then connections from one client IP to different VIPs (from the same fwmark service) will go to the same real server (only for the persistent fwmark services). (Julian) <verb> > Do you see any reason why enabling CIP->FWMARK for all cases would be a bad > thing? > > That is, not only using case 2 for persistent fwmark, but just whenever > fwmark was used. Personally, I cannot ever forsee a scenerio when a person > would setup an fwmark for load balancing and want each VIP associated with > that fwmark to act independently. <?> </verb> Web cluster for independent domains (VIPs). fwmark service is used only to reduce the amount of work for configuration. <verb> > I've always thought that the scheduling algorithms should look directly at > the real servers rather than the real server stats for each particular > virtual server. That is, least connection scheduling would look at the total > number of connections on a real server, not just the connections from that > particular VIP. Round-robin would go round-robin from real server to real > server based on the last connection from ANY VIP to the real servers... > However, before fwmark I realized that this would probably very difficult to > do especially in cases where an LVS administrator was load balancing to a > number of different real server clusters that may overlap. </verb> This is a job for the user space tools: WRR scheduling method + weights derived from the real server load. Yes, one real server can be loaded from: <itemize> <item>many directors <item>many virtual services <item>other processes not part from the real service </itemize> In this case the director's opinion (for each virtual service) about the real server load is wrong. The only way to handle such case properly is to use WRR method. In the other cases WLC, LC and RR can do their job. <quote> fwmark, to me, just by causing all VIPs marked with a particular fwmark to look like one big VIP makes it possible to do basically that which I just described. I don't see why anyone would not want such functionality with the fwmark services. If one did want such functionality, he would probably partition the VIPs associated with his fwmark into separate fwmarks or even explicit VIP entries anyway. </quote> Yes. IMO, this can be a problem only for the balancing but I don't think so. The problems will come when one real server dies and the client can't access any VIP part from the fwmark service for a period of time. <sect1>Appendix 2: Demonstration of grouping services with fwmarks<label id="fwmarks_persistence_proof_of_principle"> Here's the original e-mail between Ted <tt/tpavlic@netwalk.com/ 3 Aug 2000 and Joe One of the things it fwmarks lets me do is make ports sticky by groups. Basically I setup ipchains rules that say all packets to ports 80/tcp and 443/tcp mark with a 1. All packets to ports 20/tcp and 21/tcp as well as 1024:65535/tcp mark with a 2. Voila... I just made ports stick by groups. I then go into IPVS and setup my real servers under FWMARK1 and FWMARK2. Ports 80 and 443 are now persistent as a group just as 20 and 21 and 1024:65535 are persistent as a group. If my HTTP goes down on one of my real servers, I do not have to take my FTP down as well. I only have to remove the real server from the FWMARK1 group. It's great! Joe <verb> > most people don't program their own on-line transaction processing program > and the point of an LVS is for the real-servers to be running the same code > as when they're stand alone. </verb> My users run PHP scripts as well as ASPs that keep session information. That session information is unique per server and usually is stored in a local /tmp directory. Users are handed cookies which tie them to their session information. If they go to the wrong real server, that session information won't exist and a number of things could go wrong. most of my real servers run a lot of services... HTTP, HTTPS, FTP, SMTP, POP3, IMAP, DNS, And when one of them went down (with persistence set up), I would have to take the entire real server down. Several problems: *) One little thing goes down... POP3, for example. Now the load increases a great deal on all my other real servers... Perhaps causing the load to become so high that sendmail starts rejecting connections... and then THAT real server also is taken COMPLETELY down... domino effect. If I could have just taken POP3 down off of that server, it would have been perfect. *) Say something horrible happens causing sendmail to go down on all the servers... or HTTP... or POP3... any one service -- just as long as it goes down on all servers. Rather than just causing that service to be affected, ALL of my services go down because every real server was taken completely off-line until that ONE service is fixed. :( But I figured that those two problems wouldn't be that big of a deal... I could probably put such a system in production. Well -- I put such a system in production and those problems weren't that big of a deal... Except for a COUPLE of times when all services went down and caused a BIG hassle. So my superiors wanted something better -- needed in fact. So at first I came up with the interim idea of separating persistent services and non-persistent services by IP. All of my persistent services were basically on one supernet and all of my non-persistent services were on another subnet. Consequently, I could tie the one supernet to one FWMARK and the other subnet to another FWMARK. Now if a persistent service went down, it would bring down only all of the persistent services. Also, if a non-persistent service went down, it would only bring down all of the non-persistent services. This was definitely an interim solution because it required a lot more IPs that any one administrator should need, and it still was far from perfect.. BUT... I started to realize that just as I could mark different supernets and subnets with different FWMARKs, I could go farther down the TCP/IP layers and mark things at their protocol and port level. That's where I realized that we COULD do persistents by port group just with a little help from ipchains. <verb> > I asked Horms if there was any point in having multiple fwmarks. > His only example was if you had duplicate sets of realservers. Eg the > paying customers get the fast servers, while the people coming into the > free site get the 486 with 16M. </verb> Similar idea here... except rather than setting up your policies like: <verb> * Paying customers -> fast server * Free -> slow server </verb> You have: <verb> * SMTP -> a real-server * POP3 -> another real-server * HTTP/HTTPS -> yet another real-server * FTP -> and another real-server </verb> The key of it all is the fact that you can group by about any parameter that ipchains can see. If ipchains can segregate it, you can group it. Anything that ipchains can do IPVS can then add onto itself. > Have you solved passive ftp without using persistance? I really don't think there's any way to get around it... In order to get passive FTP to work, you need to make TCP port 21 persistent with every TCP port above 1024. I mean -- how else could you do it without putting some big brother software inside of LVS which would keep an eye on FTP and see what port it tells the end-user to connect to. Still, putting 21 and 1024:65535 together is a lot better than putting everything together. Personally I only plan on load balancing things in the < 1024 range anyway, so I have no problem including that huge group above 1024. This is my setup <verb> FWMARK1 => HTTP/HTTPS (persistent) FWMARK2 => FTP (persistent) FWMARK3 => SMTP FWMARK4 => POP3 FWMARK5 => DOMAIN FWMARK6 => IMAP FWMARK7 => ICMP (for kicks) ================ IP Virtual Server version 0.9.12 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 1 lc persistent 600 -> nw04:0 Route 1 58 121 -> nw03:0 Route 1 49 76 -> nw02:0 Route 1 60 98 -> nw01:0 Route 1 61 44 FWM 2 lc persistent 600 -> nw04:0 Route 1 0 2 -> nw03:0 Route 1 0 2 -> nw02:0 Route 1 1 13 -> nw01:0 Route 1 1 0 FWM 3 lc -> nw04:0 Route 1 4 11 -> nw03:0 Route 1 4 12 -> nw02:0 Route 1 3 20 -> nw01:0 Route 1 3 16 FWM 4 lc -> nw04:0 Route 1 3 54 -> nw03:0 Route 1 1 74 -> nw02:0 Route 1 3 51 -> nw01:0 Route 1 2 73 FWM 5 lc -> nw03:0 Route 1 0 46 -> nw01:0 Route 1 0 44 -> nw02:0 Route 1 0 45 -> nw04:0 Route 1 0 45 FWM 6 lc -> nw04:0 Route 1 0 0 -> nw03:0 Route 1 0 0 -> nw02:0 Route 1 1 0 -> nw01:0 Route 1 0 0 FWM 7 lc -> nw04:0 Route 1 0 0 -> nw03:0 Route 1 0 0 -> nw02:0 Route 1 0 0 -> nw01:0 Route 1 0 0 ============== </verb> Is this anything new? <verb> > It's new to me and Horms didn't have any other ideas for multiple > fwmarks 3 weeks ago, so I expect it will be new to him. </verb> I've been thinking of ways of combining different programs which already exist out there to get L7 scheduling working. For exmaple -- you have some program (sorta like policy routing but one more layer up) that filters packets at the application layer and does something to them... routes them to a particular IP... something like that... and then have ipchains mark each one of those packets with a particular mark... and have LVS work from there. You see -- using multiple fwmarks makes me think that you can do a lot more with LVS. We could probably borrow some of the ideas used for some of the dynamic routing protocols, like BGP or RIP. A master could advertise its IPVS hash table. If it didn't advertise within a given interval of time, other LVS's could take over. During the failover, rather than trading an IP like we were talking about, all LVSs could know which one is the active one and ICMP redirect to that LVS or something like that. Right now I'm routing every virtual server through the active LVS. This lets me do a lot of nifty things (for me at least): * Very little has to happen on the LVS during failovers. They basically just trade an IP. In fact -- I COULD do the failover right at the router before the LVS's -- just have it route to another IP. * I do not have to bring every IP up on my real-servers -- I just have to bring the network that they're on up on a hidden loopback device. When you route an entire network to a loopback device, the loopback device answers every IP on that subnet automatically. So even with 1024+ IPs, I have to setup very few interfaces/aliases because a great deal of them are on the same subnet. <sect1>Appendix 3: Announcement of grouping services with fwmarks<label id="fwmarks_persistence_announcement"> Ted Pavlic <tt/tpavlic@netwalk.com/ 4 Aug 2000 Periodically the issue comes up regarding wanting to do persistence by groups of ports. Until now, an LVS administrator could make a single port persistent or all ports persistent. Single port persistence was nice for quite a few things. However, things like HTTP and HTTPS caused complications with it. Someone who connected to a webpage on HTTP and started a session tied to them with a cookie would want to return to that same real server when they went to the HTTPS version of that site. FTP would also cause a problem with single port persistence as someone who wanted to use passive FTP wouldn't be gauranteed the same server when they returned on a random TCP port above 1024. There are other examples as well. So the solution to these problems would be to make every port persistent. This works pretty well, but now anytime a user of a large network behind a firewall would connect to a real server on ANY service, everyone behind that firewall would hit that same real server. Plus, if an administrator wanted to stop scheduling a single service to a single real server, he would have to take all services down on that single real server. This causes many problems as well... especially if one small service dies on every real server -- brings down every service on every real server. So there has been the need for persistence by port GROUPS. Rather than saying all ports are persistent, it would be nice to tell LVS to tie just 80/tcp and 443/tcp together or just 21/tcp and 1024:65535/tcp together. Before the wonderful FWMARK additions to LVS, this was not possible. But now that LVS listens to FWMARKs, it becomes possible to group ports together inside ipchains with different FWMARKs and then tell LVS to listen to those FWMARKs. For example, one can setup a rule inside FWMARK to do this... <verb> 80/tcp, 443/tcp --> FWMARK1 21/tcp, 1024:65535/tcp --> FWMARK2 25/tcp --> FWMARK3 110/tcp --> FWMARK4 </verb> Then inside LVS (assume on this setup all of these services are served by the same real server cluster), say: <verb> FWMARK1 -> PERSISTENT -> real1,real2,real3,real4 FWMARK2 -> PERSISTENT -> real1, real2, real3, real4 FWMARK3 -> real1, real2, real3, real4 FWMARK4 -> real1, real2, real3, real4 </verb> Not only have you now setup persistence by port groups, but you've also split your services back up into autonomous services that will not bring EVERY server down for the sake of persistence. If FTP goes down on real1, real1 only needs to be stopped scheduling for FTP. <sect2>another explanation Ted Pavlic <tt/tpavlic@netwalk.com/ 2000-09-15 Using fwmark, you can setup something which used to be a big desire in LVS, persistence by port groups. For example... Say you were serving HTTP and HTTPS. In this case, you would probably want calls to one HTTP server to end up hitting the same HTTPS server. This way session information and such would be accessable no matter how the end-user was accessing the website. Say you also wanted all forms of FTP to work... You would need persistence there, but not necessarily the same persistence as HTTP/HTTPS. And other protocols do not need to be persistent. Back in the olden days before fwmark, to do any of this you would have to make ALL ports persistent. You couldn't simply say "Group 80 and 443 together and make them persistent and then make 21, 20, AND 1024:65535 persistent." If one service went down, you would have to bring down ALL services. Some sort of persistence by port groups would allow you to only need to take down whatever went down and the affected server could still serve other services. FWMARK allows you to do this by way of setting up multiple FWMARKs. That is -- you can use ipchains to say that: <verb> HTTP,HTTPS --> FWMARK1 FTP --> FWMARK2 SMTP --> FWMARK3 POP --> FWMARK4 </verb> Then in LVS, setup: <verb> FWMARK1 --> WLC Persistent 600 FWMARK2 --> WLC Persistent 300 FWMARK3 --> WLC FWMARK4 --> WLC </verb> And if FTP went down, all you'd have to do is stop scheduling FTP rather than stop scheduling EVERYTHING. Also note that FWMARK makes setting up MASS VIPs really easy (of course because of recent ARIN policy changes, this probably won't be done much more anymore). That is, if you wanted to load balance 1000 VIPs, it might be easy to setup one single rule in ipchains to cover them all, where it would be 1000 rules for EACH real server in ipvsadm. It makes me think that if there was a utility already out there that could sit on a director and figure out where name-based packets were going it might be able to mark each name-based host with a different FWMARK and pass that right back to LVS... Then LVS wouldn't have to worry about handling name-based stuff ITSELF. Of course the name-based challenge is even more challenging considering how much data needs to be looked at to figure out if a TCP stream is a name-based HTTP session going to specific name X.... but that's a completely other argument... Just food for thought. <sect>Configure tools<label id="configure"> There are some tools to help you configure an LVS. <itemize> <item>Gui for VS-DR lvs-gui http://www.us.vergenet.net/linux/lvs-gui. This only sets up a VS-DR LVS and is awaiting further work. <item>configure script for all LVS types which does sanity checking on setup but which doesn't handle director failover <item>Ultra Monkey which handles director failover, but has to be setup by hand. </itemize> A recent survey of production LVS's (by Lorn Kay <tt/lorn_kay@hotmail.com/, results on the <url url="http://www.linuxvirtualserver.org/survey/" name="LVS website">) showed that 10 out of 19 were setup by methods described as "other", ie they didn't use any of the scripts on the LVS website. Clearly the above scripts are not regarded as production quality by our users. <sect1>Configure (The section in the LVS-mini-HOWTO may be more recent than this.) The configure script that reads a conf file and sets up VS-DR, VS-NAT, VS-Tun, with single or multi-NIC machines using regular ethernet devices or transparent proxy to accept packets. Elementary checks are done on routing and connectivity and the script has always produced a working LVS when no errors are reported. (I haven't tried it with any deliberately pathological cases). The output is a well commented rc.lvs and rc.mon file to run at bootup (or when you're setting up an LVS). The script is in the Software section of the website and will have a name like configure-x.x-lvs.tar.gz. The script does not handle director failover. The configure script detects the kernel version of the director (for old versions of ipvs/kernels) and is run on the director. The resulting rc.lvs (and rc.mon) files are run on the director and then the real-servers (the configure files/directory nfs exported to the real-servers, where I run the same rc.lvs file to configure the real-servers). The rc.lvs file knows whether it's running on a director or real-server. After setup, the network connections neccessary to export this directory can be removed. Running the script on the real-servers after it is run on the director allows the rc.lvs script to check the connections to the now configured director. Running the rc.lvs script on the real-server first will achieve the same result, but you will get error messages, and won't know if it succeeded even if it did. Pick an appropriate template conf file (lvs_nat.conf, lvs_tun.conf or lvs_dr.conf) for the mode you are operating the LVS under (VS-NAT, VS-Tun or VS-DR respectively) and edit the IPs and services for your situation. The files come preconfigured for telnet as the service. This is the simplest service for initial tests: the client is readily available and on connection, the login prompt will tell you which server you have connected to. Use round robin scheduling, so that you will connect to each server in turn, confirming that they are all working. Other services can be added by uncommenting/adding or editing the entries in the example *.conf files. The *.conf files give suggested IP's (192.168.1.x/24, 10.1.1.x/24) for the various machines. For those that don't like to debug disasters from trivial mistakes, the configure script will use either the name (eg telnet) or port (eg 23) of the service in the *.conf file, using the /etc/services file to translate one to another. Likely additions needed to your /etc/services will be <verb> ftp-data 20/tcp ssh 22/tcp domain 53/tcp nameserver dns #the string "dns" needs to be added here domain 53/ucp nameserver dns #and here shell 514/tcp cmd rsh #add rsh here https 443/tcp https 443/udp printer 515/tcp spooler lpd #add the string "lpd" here nfs 2049/udp nfs mysql 3306/tcp mysql 3306/udp netpipe 5002/tcp </verb> In several instances, a machine will need multiple IPs. You can put multiple IP's on a single NIC with IP aliasing (an option when building the kernel) - just enter the aliased interface (eg eth0:12) for that IP into the *.conf file. run the configure script $ ./configure.pl lvs_nat.conf This produces an rc.lvs_xxx script (eg rc.lvs_nat, rc.lvs_tun, rc.lvs_dr), and a mon_xxx.cf script. (After testing put rc.lvs_xxx into /etc/rc.d or /etc/init.d and put mon_xxx.cf in /etc/mon) Run the rc.lvs script on the director and then the real-servers with the command $ . ./rc.lvs_dr or possibly (if you get weird errors, eg it doesn't detect correctly whether it's running on a director or a real-server - this happens on my 2.0.36 real-server) $sh rc.lvs_dr The rc.lvs script - <itemize> <item>adds ethernet devices and routes to director, real-servers (including non-Linux) <item>checks connections with fping. <item>runs ipchains <item>turns ipforwarding on (VS-NAT) or off (VS-DR, VS-Tun) <item>turns off icmp redirects (VS-NAT) <item>adds services with ipvsadm. </itemize> The rc.lvs script does not - <itemize> <item> know about director failover. It assumes there is only one director. </itemize> Check the output from ipvsadm, ifconfig -a and netstat -rn on the director and then the real-server(s), to see that the services/IP's are correct. If not re-edit and re-run the script(s). <sect2>Test with telnet Telnet is a non-persistent service (in the http sense, rather than the LVS sense) and will allow you to check that all real-servers are present (you'll get the login prompt from each real-server in turn). Check that the service(s) are running on each server at the IP of the VIP (use netstat -an). Some services (eg telnet listen to 0.0.0.0, ie to all IPs on a machine). If there are no errors on running the rc.lvs_xxx script, then telnet from the client to the VIP (here 192.168.1.110). You will be routed through to one of the real-servers and you will get the login prompt (and real name) of a real-server and can login. On the director look at the output of ipvsadm, you should see a connection to a real-server on port:23. On the real-server do $ netstat -an | grep 23 and look for connections to the telnet port. Logout and telnet again to the VIP. You will get the login prompt from the next real-server. <sect2>Test with something else eg http Edit lvs_nat.conf activating http, rerun configure.pl and rerun the new rc.lvs_nat files. http: Point your browser to the VIP http://192.168.1.110. You will get the DocumentRoot of one of the real-servers. Open another copy of the browser and connect again. You should get the other server (this will be easier to see if the webpages are different). Look at the output of ipvsadm on the director for connections to the httpd ports, and on the server look at the output from netstat -an | grep 80 (or 8080 if you are running VS-NAT and have remapped the ports) for connections. If your httpd is persistent (in the http sense), you will/may connect to the same website each time. <sect>Services In principle setting up a service on an LVS is simple - you run the service on the real-server and forward the packets from the director. The simplest service to LVS is telnet: the client types a string of characters and the server returns a string of characters. In practice some services interact more with their environment. Ftp needs another port. With http, the server needs to know its name (it will have the IP of real-server, but will need to proclaim to the client that it has the VIP). https is not listening to an IP, but to requests to a nodename. This section shows the steps needed to get the common services working. When trying something new on an LVS, always have the service telnet LVS'ed. If something is not working with your service, check how telnet is doing. Telnet has the advantages <itemize> <item>telnetd listens on 0.0.0.0 on the real-server (at least under inetd) <item>the exchange between the client and server is simple, well documented, <item>the connection is non-persistence (new sessions initiated from a client will make a new connection with the LVS) unencrypted and in ascii (you can follow it with tcpdump) <item>the telnet client is available on most OS's </itemize> <sect1>setting up a new service When setting up an LVS on a new service, the client-server semantics are maintained <itemize> <item>the client thinks it is connecting directly to a server <item>the real-server thinks it is being contacted directly by the client </itemize> Example: nfs over LVS, real-server exports its disk, client mounts disk from LVS (taken from <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance data for single real-server LVS">), real-server:/etc/exportfs (real-server exports disk to client, here a host called client2) <verb> / client2(rw,insecure,link_absolute,no_root_squash) </verb> The client mounts the disk from the VIP. Here's client2:/etc/fstab (client mounts disk from machine with an /etc/hosts entry of VIP=lvs). <verb> lvs:/ /mnt nfs rsize=8192,wsize=8192,timeo=14,intr 0 0 </verb> The client makes requests to VIP:nfs. The director must forward these packets to the real-servers. Here's the conf file for the director. <verb> #lvs_dr.conf for nfs on realserver1 . . VIP=eth1:110 lvs 255.255.255.255 192.168.1.110 DIP=eth0 dip 192.168.1.0 255.255.255.0 192.168.1.255 DIRECTOR_DEFAULT_GW=client2 SERVICE=t telnet rr realserver1 realserver2 #for sanity check on LVS #to call NFS the name "nfs" put the following in /etc/services #nfs 2049/udp #note the 'u' for service type in the next line SERVICE=u nfs rr realserver1 #the service of interest SERVER_VIP_DEVICE=lo:0 SERVER_NET_DEVICE=eth0 SERVER_DEFAULT_GW=client #----------end lvs_dr.conf------------------------------------ </verb> <sect1>services must be setup for forwarding type The services must be setup to listen on the correct IP. With telnet, this is easy (telnetd listens on 0.0.0.0 under inetd), but most other services need to be configured to listen to an IP. For VS-NAT, the packets will arrive with dst_addr=RIP, i.e. the service will be listening to the IP of the real-server. When the real-server replies, then name of the machine returned will be the real-server, but the src_addr will be rewritten to be the VIP. With VS-DR and VS-NAT the packets will arrive with dst_addr=VIP, i.e. the service will be listening to an IP which is NOT the IP of the real-server. Configuring the httpd to listen to the RIP rather than the VIP is a common cause of problems for people setting up http/https. In both cases, in production, you will need to make the name of the machine given by the real-server to be the name associated with the VIP. Note: if the real-server is Linux 2.4 and is accepting packets by transparent proxy, then see the section on <ref id="TP" name="TP"> for the IP the service should listen on. <sect1>ftp general ftp is a 2 port service in both active and passive modes. In general multiport services or services which need to run together on the one real-server (eg http/https), can be handled by persistance or by Ted Pavlic's adaption of fwmark (see <ref id="fwmark_passive_ftp" name="fwmark for passive ftp">). ftp comes in 2 flavors active/passive <sect1>ftp (active) - the classic command line ftp This is a 2 port service. <itemize> <item>port 20 - data (the files you want) <item>port 21 - commands (eg ls) </itemize> <sect2>ip_vs_ftp/ip_masq_ftp module helpers As part of the ip_vs build, the modules ip_masq_ftp (2.2.x) and ip_vs_ftp (2.4.x) are produced. The ip_masq_ftp module is a patched version of the file which allowed ftp through a NAT box. This patch stopped its original function (at least in early kernels, don't know whether it still does or not). The 2.2.x ftp module is only available as a module (i.e. it can't be built into the kernel). Juri Haberland <tt/juri@koschikode.com/ 30 Apr 2001 <quote> AFAIK the IP_MASQ_* parts can only be built as modules. They are automagically selected if you select CONFIG_IP_MASQUERADE. </quote> Julian Anastasov May 01, 2001 <quote> Starting from 2.2.19 the following module parameter is required: <verb> modprobe ip_masq_ftp in_ports=21 </verb> Joe <quote> I don't see it in /usr/src/linux/Documentation, ipvs-1.0.7-2.2.19/Changelog, google or dejanews. Is this an ip_vs feature or is it a new kernel feature? </quote> I see info only in the source. This is a new 2.2.19 feature. ratz <quote> It's /usr/src/linux/net/ipv4/ip_masq_ftp.c: <verb> * Multiple Port Support * The helper can be made to handle up to MAX_MASQ_APP_PORTS (normally 12) * with the port numbers being defined at module load time. The module * uses the symbol "ports" to define a list of monitored ports, which can * be specified on the insmod command line as * ports=x1,x2,x3... * where x[n] are integer port numbers. This option can be put into * /etc/conf.modules (or /etc/modules.conf depending on your config) * where modload will pick it up should you use modload to load your * modules. * Additional portfw Port Support * Module parameter "in_ports" specifies the list of forwarded ports * at firewall (portfw and friends) that must be hooked to allow * PASV connections to inside servers. * Same as before: * in_ports=fw1,fw2,... * Eg: * ipmasqadm portfw -a -P tcp -L a.b.c.d 2021 -R 192.168.1.1 21 * ipmasqadm portfw -a -P tcp -L a.b.c.d 8021 -R 192.168.1.1 21 * modprobe ip_masq_ftp in_ports=2021,8021 </verb> And it is a new kernel feature, not LVS feature. </quote> </quote> what are these modules for: from ipvsadm(8) (ipvs 0.2.11) <quote> If a virtual service is to handle FTP connections then persistence must be set for the virtual service if Direct Routing or Tunnelling is used as the forwarding mechanism. If Masquerading is used in conjunction with an FTP service than persistence is not necessary, but the ip_vs_ftp kernel module must be used. This module may be manually inserted into the kernel using insmod(8) </quote> From Julian 3 May 2001, the modules are required for <itemize> <item>VS-NAT <item>recommended for active ftp and mandatory for passive ftp, if persistence tricks are not used when setting up the LVS. </itemize> The modules are NOT used for VS-DR or VS-Tun: in these cases persistence is used (or fwmarks version of persistence). <sect2>VS-NAT, 2.2.x director I found that ftp worked just fine without the module for 2.2.x (1.0.3-2.2.18 kernel). <sect2>VS-NAT, 2.4.x director For 2.4.x you can connect with ftp without any extra modules, but you can't "ls" the contents of the ftp directory. For that you need to load the ip_vs_ftp module. Without this module, your client's screen won't lock up, it just does nothing. If you then load the module, you can list the contents of the directory. <sect2>VS-DR, VS-Tun For VS-DR, VS-Tun active ftp needs persistence. Otherwise it does not work, with or without ip_masq_ftp loaded. You can login, but attempting to do a `ls` will lockup the client screen. Checking the real-server, shows connections on ports 20,21 to paired ports on the client. <sect1>ftp (passive)<label id="passive_ftp"> Passive ftp is used by netscape to get files from an ftp url like ftp://ftp.domain.com/pub/ . Here's an explanation of passive ftp from http://www.tm.net.my/learning/technotes/960513-36.html <quote> If you can't open connections from Netscape Navigator through a firewall to ftp servers outside your site, then try configuring the firewall to allow outgoing connections on high-numbered ports. Usually, ftp'ing involves opening a connection to an ftp server and then accepting a connection from the ftp server back to your computer on a randomly-chosen high-numbered telnet port. the connection from your computer is called the "control" connection, and the one from the ftp server is known as the "data" connection. All commands you send and the ftp server's responses to those commands will go over the control connection, but any data sent back (such as "ls" directory lists or actual file data in either direction) will go over the data connection. However, this approach usually doesn't work through a firewall, which typically doesn't let any connections come in at all; In this case you might see your ftp connection appear to work, but then as soon as you do an "ls" or a "dir" or a "get", the connection will appear to hang. Netscape Navigator uses a different method, known as "PASV" ("passive ftp"), to retrieve files from an ftp site. This means it opens a control connection to the ftp server, tells the ftp server to expect a control connection to the ftp server, tells the ftp server to expect a second connection, then opens the data connection to the ftp server itself on a randomly-chosen high-numbered port. This works with most firewalls, unless your firewall retricts outgoing connections on high-numbered ports too, in which case you're out of luck (and you should tell your sysadmins about this). "Passive FTP" is described as part of the ftp protocol specification in RFC 959 ("http://www.cis.ohio-state.edu/htbin/rfc/rfc959.html"). </quote> If you are setting up an LVS ftp farm, it is likely that users will retrieve files with a browser and you will need to setup the LVS to handle passive ftp. You will either need <ref id="persistent_connection" name="persistence"> (also see on the LVS website under documentation; persistence handling in LVS) or <ref id="fwmark_passive_ftp" name="fwmark persistent connection for ftp">. For passive ftp, the ftpd sets up a listener on a high port for the data transfer. This problem for LVS is that the IP for the listener is the RIP and not the VIP. Wenzhuo Zhang 1 May 2001 <quote> I've been using 2.2.19 on my dialup masquerading box for quite some time. It doesn't seem to me that the option is required, whether in PASV or PORT mode. We can actually get ftp to work in NAT mode without using the ip_masq_ftp module. The trick is to tell the real ftp servers to use the VIP as the passive address for connections from outside; e.g. in wu-ftpd, add the following lines to the /etc/ftpaccess: <verb> passive address RIP <localnet> passive address 127.0.0.1 127.0.0.0/8 passive address VIP 0.0.0.0/0 </verb> Of course, the ftp virtual service has to be persistent port 0. </quote> On Thu, 3 May 2001, Alois Treindl wrote: <quote> I found (with kernel 2.2.19) that I needed the command <verb> modprobe ip_masq_ftp in_ports=21 </verb> so that (passive mode) ftp from Netscape would work. Julian Anastasov <tt/ja@ssi.bg/ 03 May 2001 <quote> Yes, it seems this option is not useful for the active FTP transfers because if the data connection is not created while the client's PORT command is detected in the command stream then it is created later when the internal real server creates normal in->out connection to the client. So, it is not a fatal problem for active FTP to avoid this option. The only problem is that these two connections are independent and the command connection can die before the data connection, for long transfers. With the in_ports option used this can not happen. The fatal problems come for the passive transfers when the data connection from the client must hit the LVS service. For this, the ip_masq_ftp module must detect the 227 response from the real server in the in->out packets and to open a hole for the client's data connection. And the "good" news is that this works only with in_ports/in_mark options used. </quote> without the in_ports=21 it did not work. I am using proftpd as ftp server, which does not seem to have on option so that I could configure on the server that it gives the VIP to clients making a PASV request; it always gives the realserver IP address in replies to such requests. <quote> Bad ftpd :) It seems the follwing rules are valid: - active ftp always works through stupid balancers (for external clients) that have minimum support for masquerading, with some drops in the command connection - passive ftp always works through stupid masq boxes (for internal clients) The passive ftp setup is useful because the data connection can be mairked as a slave to the command connection and in this way avoid connection reconnects. </quote> </quote> <sect1>ftp is difficult to secure Roberto Nibali <tt/ratz@tac.ch/ 06 May 2001 We have multiple choices if we want to narrow down the input ipchains rules on the front interface of director <itemize> <item>Use ftp via LVS. (this is not a solution actually, we still need special input rules on the EXT_IF for 1024:65535) <item>Use ftp without LVS but with SNAT. (difficult to setup) <item>Use SuSE ftp proxy suite. <item>Use 2.4 kernel and ip_conntrack_ftp (don't know much about this, ask Rusty) <item>Don't use ftp at all (this is what we want) </itemize> The biggest problem is with the ip_masq_ftp module. It should create an ip_fw entry in the masq_table for the PORT port. It doesn't do this and we have to open the whole port range. For PASV we have to DNAT the range. <verb> ipchains -A forward -i $EXT_IF -s $INTERNAL_NET $UNPRIV_PORTS -d $DEP -j MASQ </verb> FTP is made up of two connections, the Control- and the Data- Connection. <itemize> <item>ftp Control Connection The Client contacts the Servers port 21 from an UNPRIV Port. No trouble, standard, plain, vanilla TCP-Connection, we all love it. Over this connection the client sends commands to the server. We will see examples later. <item>FTP Data Connection "Data" can be either the content of a file (sent as e.g. the result of a "get" or "put" command) or the content of a directory-listing (i.e. the result of a "ls" or "dir" command). The data connection is where the trouble starts. To transfert data, a second connection is opened. Usually the client opens this second connection to the server. But for active ftp, the server opens this second connection, using the well-known port 20 (called ftp-data) as sourceport. But which port on the client should he connect to? The client announces the port via a "port"-command over the control connection. This is nasty: Ports are negotiated on application-level where L4 switches like LVS can see what's going on. For passive ftp, the server announces the port the client should connect to in its reply to the client's "pasv"-command (this command starts passive FTP, active is the default). The client then opens the data-connection to the server. The port that the server listens on is an unprivileged port (rather than a privileged port as is normal for internet services). A passive ftp transfer then requires that connections be allowed between all 63000 unprivileged ports on both the client and real-servers rather than just one. A passive ftp server is difficult to secure with packet filter rules. </itemize> If we have to protect a client, we would like to only allow passive ftp, because then we do not have to allow incoming connections. If we have to protect a server, we would like to only allow active ftp, because then we only have to allow the incoming control-connection. This is a deadlock. <label id="netcat"> Example ftp sessions with <url url="http://l0pht.com/~weld/netcat/nc110.tgz" name="netcat">. (Note this URL now refers through to "atstake.com" which has the NT binary of netcat, but not the unix source. If you know where it is now or want the original code, let me know.) We need 2 xterms (x1, x2), netcat and an ftp-server (here "zar" 172.23.2.30). First passive mode (because it is conceptionally easier) <verb> #x1: Open the control-connection to the server, #and sent the command "pasv" to the server. $ netcat zar 21 220 zar.terreactive.ch FTP server (Version 6.4/OpenBSD/Linux-ftpd-0.16) ready. user ftp 331 Guest login ok, send your complete e-mail address as password. pass ftp 230 Guest login ok, access restrictions apply. pasv 227 Entering Passive Mode (172,23,2,30,169,29) </verb> The server replied with 6 numbers: <itemize> <item> 172,23,2,30 is the IP I have to connect to <item> (169*256+29=43293) is the Port </itemize> In x2 I open a second connection with a second netcat <verb> $ netcat 172.23.2.30 43293 # x2 will now display output from this connection </verb> Now in x1 (the control-connection) <verb> $ list list 150 Opening ASCII mode data connection for '/bin/ls'. 226 Transfer complete. </verb> and in x2 the listing appears. Active ftp I use the same control-connection in x1 as above, but I want the server to open a connection. Therefore I first need a listener. I do it with netcat in x2: <verb> $ netcat -l -p 2560 </verb> Now I tell the server on the control connection to connect (2560=10*256+0) <verb> port 172,23,2,8,10,0 200 PORT command successful. <verb> Now you see, why I used port 2560. 172.23.2.8 is, of course, my own IP-address. And now, using x1, I ask for a directory-listing with the list command, and it appears in x2. For completeness sake, here is the the full in/output. First the xterm 1: <verb> netcat zar 21 220 zar.terreactive.ch FTP server (Version 6.4/OpenBSD/Linux-ftpd-0.16) ready. user ftp 331 Guest login ok, send your complete e-mail address as password. pass ftp 230 Guest login ok, access restrictions apply. pasv 227 Entering Passive Mode (172,23,2,30,169,29) list 150 Opening ASCII mode data connection for '/bin/ls'. 226 Transfer complete. port 172,23,2,8,10,0 200 PORT command successful. list 150 Opening ASCII mode data connection for '/bin/ls'. 226 Transfer complete. quit 221 Goodbye. </verb> xterm 2: <verb> netcat 172.23.2.30 43293 total 7 dr-x--x--x 2 root root 1024 Jul 26 2000 bin drwxr-xr-x 2 root root 1024 Jul 26 2000 dev dr-x--x--x 2 root root 1024 Aug 20 2000 etc drwxr-xr-x 2 root root 1024 Jul 26 2000 lib drwxr-xr-x 2 root root 1024 Jul 26 2000 msgs dr-xr-xr-x 11 root root 1024 Mar 15 14:26 pub drwxr-xr-x 3 root root 1024 Mar 11 2000 usr </verb> <verb> netcat -l -p 2560 total 7 dr-x--x--x 2 root root 1024 Jul 26 2000 bin drwxr-xr-x 2 root root 1024 Jul 26 2000 dev dr-x--x--x 2 root root 1024 Aug 20 2000 etc drwxr-xr-x 2 root root 1024 Jul 26 2000 lib drwxr-xr-x 2 root root 1024 Jul 26 2000 msgs dr-xr-xr-x 11 root root 1024 Mar 15 14:26 pub drwxr-xr-x 3 root root 1024 Mar 11 2000 usr </verb> <sect1>evaluation of SuSE ftp proxy Roberto Nibali <tt/ratz@tac.ch/ 08 May 2001 There has been some talks about ftp, security and LVS recently and different opinions appeared. I wasn't aware of the fact the people still heavily use the ftp protocol through a firewall, rather then putting a completely secluded box in a corner. Back here at terreActive we have been fighting with the ftp problem since 4 years already and we have not yet the ultimate solution. As such we also evaluated <url url="http://www.suse.de/en/support/proxy_suite/" name="the SuSE FTP proxy"> What follows is an evaluation mostly done by one of our coworkers Martin Trampler and me (ratz). We're not yet finished with testing everything and all possible setups (NAT, non-NAT, client behind a firewall, etc.) but the result looks rather good in terms of improving security. A better paper will probably follow, but we're too busy right now and ftp is anyway not allowed in our company policy unless the customer has a special SLA. <sect2>Motivated by <itemize> <item>the general problems related to allowing FTP-traffic through a non-stateful packetfilter <item>the special problems we encountered with the Firewall-1 software on one of our packetfilter which in the end forced us to deinstall FW1 <item>the broken Linux FTP-masquerading module </itemize> I started in March/01 with a search for a FTP Proxy-software, which could be used as a drop-in on pab1/2-machines to increase the security of the machines (clients and servers) behind the packetfilters. Since it was a requirement that this Software should be able to transparently proxy external clients (i.e. the clients don't realize that there is a proxy inbetween), there was only one package which deserved a closer look: The FTP-Proxy from the SuSE Proxy Suite (which actually consists of nothing but this FTP-Proxy). This Proxy now includes support for transparent proxying mode. <sect2>Mode of operation The Proxy consists of a single binary (ftp-proxy, stripped about 50k). All configuration-options can be set in a single configuration-file which by default is named ftp-proxy.conf and searched for in whatever directory was given with the configure-option --sysconfdir=. If this option was not given it is searched for in /usr/local/proxy-suite/etc/, which shows SuSEs BSD-heritage. Best way is, to give the config-file at runtime with the -f cmdline-option. It is very useful to compile debugging-support into the binary during evaluation and to run it with the cmdline-option -v 4 for maximum debugging. Debugging output is then appended to /tmp/ftp-proxy.debug. It can be run from (x)inetd or in standalone-mode as daemon. I only evaluated the daemon. It reads the config-file (which must exist) and binds to some local port (e.g. 3129, which is IANA-unassigned and squid+1). The Packetfilter has to be configured to redirect all packets which come in on port 21 to this port (more later). As soon as it gets a request it handles it by first replying to the client only. After the initial USER <username> command it connects to the server (or, more exactly, to port 21 of the host whose IP was the destination of the redirected package). The configfile may contain user-specific sections which direct special users to special servers. This feature may be very useful but was not evaluated either. It then continues as an agent between client and server; checking either side's communication for correct syntax, as a good application-proxy should. As soon as the client prepares a data connection (either by sending a PASV or a PORT command, it acknowledges it and, in case of a requested passive connection, establishes a listener which binds to the server's IP (!!). This came rather as a surprise to me. It actually works and means, that the data connection is transparent as well for the client. The range of port on which it listens is configureable as well as the range of ports it uses for outgoing connection (to either the server or to the client in the active-ftp case). As soon as the client actually wants to retrieve data, the connection to the server is established and the data is shuffeled around. Since the connection to the server is completely seperate from the client connection, its mode doesn't have to be the one the client requests (also by default it is). The data-connection to the server may also be configured to always be active or passive. Here it is clearly desireable to always use passive mode to avoid opening another listener on the packetfilter. <sect2>Evaluation After initially having some minor problems to compile the proxy (it has to be configured --with-regex) and to get it running (by default it thinks it is started by inetd, i.e. standalone-mode is not the default) it ran without problems and also wrote informative messages into the debugging file. Almost everything can be configured in the configuration file (also not everything is documented unambigously) but in general the quality of the documentation, the logging and the debugging messages seems quite high. The proxy is, as already mentioned, completely transparent for the client and of course intransparent for the server (i.e. the server sees the connections coming from the client). First more extensive stress-testing and code-review performed on 04/Apr/01 showed the following irregularities: <itemize> <item>Using active ftp, the data-connection on the server-side often failed. This problem could be circumvented by forcing the Proxy to always use passive mode on the server side (which should be used anyways, see below) <item>The control-connection on the server side often could not be established. <item>When running with debugging-output enabled, we found strange characters in the debugging as well as in the logging-output. </itemize> The failed connections result from portnumbers being reused where they should be increased. I think, this problem would also be found on the client-side of the connection if the stress-test would issue more than 1 data-retrieving command. It may help to undefine the Destination[Min|Max]Port configuration directive to get a port assigned by the system. The - attack-behaviour vanished after disabling debugging output but may nevertheless being an issue. We found a questionable use of a static char* in a formatting routine. <sect2>Integration into a firewall suite <sect3>Packetfilter-ruleset First the Packetfilterport on which the proxy listens must be closed. It is well possible to bind the proxy to e.g. localhost, but the ipchains ... -j REDIRECT (see below) only allows the specification of a port, not of port+IP. If the proxy is bound to an IP it doesn't get the packets. It has to be universally bound and therefore its port must be closed. In the following I use: <itemize> <item> PASV_PORTS: Configuration-directives "Passive[Min|Max]DataPort"; Ports on which Listeners for passive connections (to Clients) may be installed <item> SC_PORTS: Configuration-directives "Destination[Min|Max]Port"; Ports which are used for connecting to the server (control-conn., data-conn. if server in passive mode) or on which the proxy listens for the server's data (if server in active mode, not documented). For client-connections it is necessary that: <itemize> <item> In the input-chain (for the IF on which client packets arrive) there must be a redirect for packets from the client to the server, port 21 to the port on which the proxy listens. <item>In the input-Chain (for the IF on which client packets arrive) connections from the client to the server on port (UNPRIV->21) and (UNPRIV->PASV_PORTS) are accepted (as well as the reply-packets in the output-chain). The redirect-rule already acts as accepting rule for the incoming packets for port 21. Furthermore in the output-chain connections from the proxy (but with IP-adress of the server!), port 20, to the client (UNPRIV) must be allowed (as well as the corresponging reply-packets in the input-chain). </itemize> For server-connections it is necessary that: <itemize> <item>In the output-chain of the server-side IF connections from the proxy (SC_PORTS) to port 21 of the server are allowed (and the reply packets in the input-chain) <item>In the output-chain of the server-side IF connections from the proxy (SC_PORTS) to the server (UNPRIV) are allowed (and the reply packets in the input-chain) </itemize> </itemize> The latter pair of rules covers the case that all data connections from the proxy to the server are passive. Note, that no rules for the forward-chain are necessary at all. This diagram shows the control-connection. <verb> +--------------------------+ | | Proxy | | | |3129____________| | +--------+ tcp/21 |----- ^ | -----| tcp/21 +--------+ | Client |----------->|eth0| | +->|eth1|---------->| Server | +--------+ to server |--------+redirect -----| +--------+ |Packetfilter | +--------------------------+ </verb> The obvious problem is, to formulate a fw-ftp-proxy script which accepts 2 NEs (external client, internal server) as input and does not generate redundant rules. Because the server-side connection is completely independent from the client, its rules must only be added once for each server while the client-side rules (including the redirect) are dependent of both server and client. Probably the best way would be to add a script which only handles the client-side and to add each ftp-server seperately with a "tcp@fw"-rule. Since tcp@fw does not allow specification of source-ports, this rule would then be wider as necessary. In the client-side script, the portrange used in the proxy-configfile would then have to be hardwired. It would be necessary to verify, that for every server used as target in a client-side script there is at least (or even better exactly one) tcp@fw as described above. <sect2>Security considerations We had a swift look at the code and it looks rather clean and well documented to me. Unfortunately some features are incorrectly documented or not documented at all while some features are already documented but not yet implemented. As already mentioned, the port on which the proxy listens has to be closed. The servers should only be driven in passive mode, which should be possible for any server. The PASV_PORTS should be restricted to a dozen or so (depending on the load). For the maintainers of the servers, the major drawback is the proxy's intransparency. <sect2>Features not yet evaluated <itemize> <item>User-specific configurations <item>Use with LDAP and the TCP-Wrapper library (configure-options --with-ldap and --with-libwrap <item>Limiting the set of allowed commands <item>Proc-Filesystem Interface (module) <item>chroot of forked processes </itemize> <sect2>Conclusion <sect3>General Aspects Given the current situation, where we shoot huge holes in the firewall to fully enable (passive) ftp connections to servers located inside, the use of this proxy would greatly increase the security of these systems. Prior to deployment I think the code should be reviewed more closely (remember that the proxy opens listeners on the PF!) and some more efforts should be undertaken to find a configuration which is as tight as possible by providing the required functionality (cf. the section above). <sect3>Extensions It should, in general, be possible to have a second proxy running for inside clients. There we still have the problem, that we have to open the whole UNPRIV-Range for connections coming from sourceport 20. Basically I think, that this problem should be handled differently: Providing the functionality is the business of the server (hence the name) . The FTP-Protocol provides passive mode exactly for this case (firewalled client). So we should in general not allow clients behind our Firewalls/Packetfilters to make active FTP connections. We found out that it should not be too difficult to enable "bidirectional transparency". <sect1>telnet Simple one port service. Use this (or <ref id="netcat" name="netcat">) for initial testing of your LVS. It is a simpler client/service than http (is not persistent) and a connection shows up as an ActConn in the ipvsadm output. <sect1>ssh surprisingly (considering that it negotiates a secure connection) nothing special either. You do not need persistent port/client connection for this. some stuff from Wensong about ssh (also see above for persistance) <tt/jeremy@xxedgexx.com/ wrote: <verb> > I'm using ipvs > to balance ssh connections but for some reason ipvs is only using one real > server and persists to use that server until I delete its arp entry from > the ipvs machine and remove the virtual loopback on the real server. > Also, I noticed that connections do not register which this behavior. </verb> (Wensong) do you use the persistent port for your VIP:22? If so, the default timeout of persistent port is 360 seconds, once the ssh session finishes, it takes 360 seconds to expire the persistent session. (In ipvs-0.9.1, you can flexibly set the timeout for the persistent port.) There is no need to use persistent port for ssh service, because the RSA keys are exchanged in each ssh session, and each session is not related. <sect1>dns This is from Ted Pavlic. Two (independant) connections, tcp and udp to port 53 are needed. (from the IPCHAINS-HOWO) DNS doesn't always use UDP; if the reply from the server exceeds 512 bytes, the client uses a TCP connection to port number 53, to get the data. Usually this is for a zone transfer. Here is part of an lvs.conf file which has dns on two real-servers. <verb> #dns, note: need both udp and tcp #A real-server must be able to determine its own name. #(log onto machine from console and use nslookup # to see if it knows who it is) # and to do DNS on the VIP and name associated with the VIP #To test a running LVS, on client machine, run nslookup and set server = VIP. SERVICE=t dns wlc 192.168.1.1 192.168.1.8 SERVICE=u dns wlc 192.168.1.1 192.168.1.8 </verb> If the LVS is run without mon, then any setup that allows the real-servers to resolve names is fine (ie if you can sit at the console of each real-server and run nslookup, you're OK). If the LVS is run with mon (eg for production), then dns needs to be setup in a way that dns.monitor can tell if the LVS'ed form of dns is working. When dns.monitor tests a real-server for valid dns service, it first asks for the zone serial number from the authoritative (SOA) nameserver of the virtualserver's domain. This is compared with the serialnumber for the zone returned from the real-server. If these match then dns.monitor declares that the real-server's dns is working. The simplest way of setting up an LVS dns server is for the real-servers to be secondaries (writing their secondary zone info to local files, so that you can look at the date and contents of the files) and some other machine (eg the director) to be the authoritative nameserver. Any changes to the authoritative nameserver (say the director) will have to be propagated to the secondaries (here the real-servers) (delete the secondary's zone files and HUP named on the real-servers). After the HUP, new files will be created on the secondary nameservers (the real-servers) with the time of the HUP and with the new serial numbers. If the files on the secondary nameservers are not deleted before the HUP, then they will not be updated till the refresh/expire time in the zonefile and the secondary nameservers will appear to dns.monitor to not be working. There is no reason to create an LVS to do DNS. DNS has its own cacheing and hierachial method of loadbalancing. However if you have an LVS already running serving http, ftp... then it's simple to throw in dns as well (Ted). <sect1>sendmail/smtp/pop3/qmail For mail which is being passed through, LVS is a good solution. If the mail is being delivered to the real-server, then the mail will arrive randomly at any one of the real-servers and write to the different filesystems. Since you probably want your mail to arrive at one place only, the only way of handling this right now is to have the /home directory nfs mounted on all the real-servers from a backend fileserver which is not part of the LVS. (an nfs.monitor is in the works.) Each real-server will have to be configured to accept mail for the virtual server DNS name (say lvs.domain.com). It should be possible to use Coda (http://www.coda.cs.cmu.edu/) to keep /home directories synchronised, or inter-mezzo or gfs all of which look nice, but we haven't tested. To maintain user passwds on the real-servers - (Gabriel Neagoe <tt/Gabriel.Neagoe@snt.ro/) for syncing the passwords - IF THE ENVIRONMENT IS SAFE- you could use NIS or rdist <sect2>identd (auth) problems You will not be explicitely configuring identd in an LVS. However <ref id="authd" name="identd"> is used by sendmail and tcpwrappers and will cause problems. Sendmail can't use identd when running on an LVS (see <ref id="identd_and_sendmail" name="identd and sendmail">). Running identd as an LVS service doesn't fix this. <verb> for sendmail: in sendmail.cf file set the value Timeout.ident=0 </verb> (see http://www.sendmail.org/faq/section3.html - Why do connections to the smtp port take such a long time?) for <url url="http://www.qmail.org" name="qmail">: From: Martin Lichtin <tt/lichtin@bivio.com/ If invoked with tcp-env in inetd.conf - use the -R option if spawned using svc and DJ's daemontools packages - <verb> > /usr/local/bin/tcpserver -u 1002 -g 1001 -c 500 -v 0 smtp /var/qmail/bin/qmail-smtpd </verb> tcpserver is the recommended method of running qmail, where you use the -R option for tcpserver -R: Do not attempt to obtain $TCPREMOTEINFO from the remote host. To avoid loops, you must use this option for servers on TCP ports 53 and 113. To test an LVS'ed smtp server (connect to lvs:smtp from the client) <verb> client:~# telnet lvs.cluster.org trying 192.168.1.110... Connected to lvs.cluster.org Escape character is '^]'. 220 lvs.cluster.org ESMTP Sendmail 8.9.1a/8.9.0; Sat 6 Nov 1999 13:16:30 GMT HELO client.cluster.org 250 client.cluster.org Hello root@client.cluster.org [192.168.1.12], pleased to meet you quit 221 client.cluster.org closing connection </verb> check that you can access each real-server in turn (here 192.168.1.12 was accessed). pop3 - as for smtp. The mail agents must see the same /home file system, so /home should be mounted on all real-servers from a single file server. <sect2>Thoughts about sendmail/pop (another variation on the many reader/many writer problem) From: Rob Thomas <tt/rob@rpi.net.au/ <tt/loc@indochinanet.com/ wrote: <verb> > I need this to convince my boss that LVS is THE SOLUTION for very > Scalable and High Available Mail/POP server. </verb> This is about the hardest clustering thing you'll ever do. Because of the constant read/write access's you -will- have problems with locking, and file corruption.. The 'best' way to do this is (IMHO): <enum> <item>NetCache Filer as the NFS disk server. <item>Several SMTP clients using NFS v3 to the NFS server. <item>Several POP/IMAP clients using NFS v3 to the NFS server. <item>At least one dedicated machine for sending mail out (smarthost) <item> LinuxDirector box in front of 2 and 3 firing requests off </enum> Now, items 1 2 -and- 3 can be replaced by Linux boxes, but, NFS v3 is still in Alpha on linux. I -believe- that NetBSD (FreeBSD? One of them) has a fully functional NFS v3 implementation, so you can use that. The reason why I emphasize NFSv3 is that it -finally- has 'real' locking support. You -must- have atomic locks to the file server, otherwise you -will- get corruption. And it's not something that'll happen occasionally. Picture this: <verb> [client] -- [ l.d ] -- [external host] | [smtp server]-+-[pop3 server] | [filesrv] </verb> Whilst [client] is reading mail (via [pop3 server]), [external host] sends an email to his mailbox. the pop3 client has a file handle on the mail spool, and suddenly data is appended to this. Now the problem is, the pop3 client has a copy of (what it thinks) is the mail spool in memory, and when the user deletes a file, the mail that's just been received will be deleted, because the pop3 client doesn't know about it. This is actually rather a simplification, as just about every pop3 client understands this, and will let go of the file handle.. But, the same thing will happen if a message comes in -whilst the pop3d is deleting mail-. <verb> POP Client SMTP Client I want to lock this file <-- I want to lock this file <-- You can lock the file --> You can lock the file --> Consider it locked <-- File is locked --> Consider it locked <-- Ooh, I can't lock it --> </verb> The issue with NFS v1 and v2 is that whilst it has locking support, it's not atomic. NFS v3 can do this: <verb> POP Client SMTP Client I want to lock this file <-- I want to lock this file <-- File is locked --> Ooh, I can't lock it --> </verb> That's why you want NFSv3. Plus, it's faster, and it works over TCP, rather than UDP 8-) From: Stefan Stefanov <tt/sstefanov@orbitel.bg/ > This is about the hardest clustering thing you'll ever do. Because of I think this might be not-so-hardly achieved with CODA and Qmail. Coda (http://www.coda.cs.cmu.edu) allows "clustering" of file system space. Qmail's (http://www.qmail.org) default mailbox format is Maildir, which is very lock safe format (even on NFS without lockd). (I haven't implemented this, it's just a suggestion.) <sect1>Mail farms<label id="mail_farm"> Peter Mueller <tt/pmueller@sidestep.com/ 10 May 2001 <quote> what open source mail programs have you guys used for SMTP mail farm with LVS? I'm thinking about Qmail or Sendmail? Michael Brown <tt/Michael_E_Brown@Dell.com/, Joe and Greg Cope <tt/gjjc@rubberplant.freeserve.co.uk/ 10 May 2001 <quote> You can do load balancing against multiple mail servers without LVS. Use multiple MX records to load balance, and mailing list management software (Mailman, maybe?). DNS responds with all MX records for a request. The MTA should then choose one at random from the same piority. (A cache DNS will also return all MX records.) You don't get persistent use of one MX record. If the chosen MX record points to a machine that's down, the MTA will choose another MX record. </quote> </quote> Wensong <quote> I think that central load balancing is more efficient in resource utilization than randomly picking up servers by clients, basic queuing theory can prove this. For example, if there are two mail servers grouped by multiple DNS MX records, it is quite possible that a mail server of load near to 1 still receiving new connections (QoS is bad here), in the mean while the other mail server just has load 0.1. If the central load balancing can keep the load of two server around 0.7 respectively, the resource utilization and QoS is better than that of the above case. :) </quote> Michael Brown <tt/Michael_E_Brown@Dell.com/ 15 May 2001 <quote> I agree, but... :-) <enum> <item> You can configure most mail programs to start refusing connections when load rises above a certain limit. The protocol itself has built-in redundancy and error-recovery. Connections will automatically fail-over to the secondary server when the primary refuses connections. Mail will _automatically_ spool on the sender's side if the server experiences temporary outage. <item> Mail service is a special case. The protocol/RFC itself specified application-level load balancing, no extra software required. <item> Central load balancer adds complexity/layers that can fail. </enum> I maintain that mail serving (smtp only, pop/imap is another case entirely) is a special case that does not need the extra complexity of LVS. Basic Queuing theory aside, the protocol itself specifies load-balancing, failover, and error-recovery which has been proven with years of real-world use. LVS is great for protocols that do not have the built-in protocol-level load-balancing and error recovery that SMTP inherently has (HTTP being a great example). All I am saying is use the right tool for the job. </quote> Note this discussion applies to mail which is being forwarded by the MTA. The final target machine has the single-writer, many-reader problem as before. Joe <quote> How would someone like AOL handle the mail farm problem? How do users get to their mail? Does everyone in AOL get their mail off one machine (or replicated copies of it) or is each person directed to one of many smaller machines to get their mail? </quote> Michael Brown <quote> Tough question... AOL has a system of inbound mail relays to receive all their user's mail. Take a look: <verb> [mebrown@blap opt]$ nslookup Default Server: ausdhcprr501.us.dell.com Address: 143.166.227.254 > set type=mx > aol.com Server: ausdhcprr501.us.dell.com Address: 143.166.227.254 aol.com preference = 15, mail exchanger = mailin-03.mx.aol.com aol.com preference = 15, mail exchanger = mailin-04.mx.aol.com aol.com preference = 15, mail exchanger = mailin-01.mx.aol.com aol.com preference = 15, mail exchanger = mailin-02.mx.aol.com aol.com nameserver = dns-01.ns.aol.com aol.com nameserver = dns-02.ns.aol.com mailin-03.mx.aol.com internet address = 152.163.224.88 mailin-03.mx.aol.com internet address = 64.12.136.153 mailin-03.mx.aol.com internet address = 205.188.156.186 mailin-04.mx.aol.com internet address = 152.163.224.122 mailin-04.mx.aol.com internet address = 205.188.158.25 mailin-04.mx.aol.com internet address = 205.188.156.249 mailin-01.mx.aol.com internet address = 152.163.224.26 mailin-01.mx.aol.com internet address = 64.12.136.57 mailin-01.mx.aol.com internet address = 205.188.156.122 mailin-01.mx.aol.com internet address = 205.188.157.25 mailin-02.mx.aol.com internet address = 64.12.136.89 mailin-02.mx.aol.com internet address = 205.188.156.154 mailin-02.mx.aol.com internet address = 64.12.136.121 dns-01.ns.aol.com internet address = 152.163.159.232 dns-02.ns.aol.com internet address = 205.188.157.232 </verb> So that is on the recieve side. On the actual user reading their mail side, things are much different. AOL doesn't use normal SMTP mail. They have their own proprietary system, which interfaces to the normal internet SMTP system through gateways. I don't know how AOL does their internal, proprietary stuff, but I would guess it would be massively distributed system. Basically, you can break down your mail-farm problem into two, possibly three, areas. <verb> 1) Mail receipt (from the internet) 2) Users reading their mail 3) Mail sending (to the internet) </verb> Items 1 and 3 can normally be hosted on the same set of machines, but it is important to realize that these are separate functions, and can be split up, if need be. For item #1, the listing above showing what AOL does is probably a good example of how to set up a super-high-traffic mail gateway system. I normally prefer to add one more layer of protection on top of this: a super low-priority MX at an offsite location. (example: aol.com preference = 100, mail exchanger = disaster-recovery.offsite.aol.com ) For item #2, that is going to be a site policy, and can be handled many different ways depending on what mail software you use (imap, pop, etc). The good IMAP software has LDAP integration. This means you can separate groups of users onto separate IMAP servers. The mail client then can get the correct server from LDAP and contact it with standard protocols (IMAP/POP/etc). For item #3, you will solve this differently depending on what software you have for #2. If the client software wants to send mail directly to a smart gateway, you are probably going to DNS round-robin between several hosts. If the client expects it's server (from #2) to handle sending email, then things will be handled differently. </quote> Wenzhuo Zhang <tt/wenzhuo@zhmail.com/ Here's an <url url="http://lw.itworld.com/linuxworld/lw-1999-09/lw-09-sendmail.html" name="article on paralleling mail servers"> by Derek Balling. Shain Miley 25 May 2001 <quote> I am planning on setting up an LVS IMAP cluster. I read some posts that talk about file locking problems with NFS that might cause mailbox corruption. Do you think NFS will do the trick or is there a better (faster, journaling) file system out there that will work in a production environment. </quote> Matthew S. Crocker <tt/matthew@crocker.com/ 25 May 2001 NFS will do the trick but you will have locking problems if you use mbox format e-mail. You *must* use MailDir instead of mbox to avoid the locking issues. You can also use GFS (www.globalfilesystem.org) which has a fault tolerant shared disk solution. Don Hinshaw <tt/dwh@openrecording.com/ I do this. I use Qmail as it stores the email in Maildir format, which uses one file per message as opposed to mbox which keeps all messages in a single file. On a cluster this is an advantage since one server may have a file locked for writing while another is trying to write. Since they are locking two different files it eases the problems with NFS file locking. Courier also supports Maildir format as I believe does Postfix. I use Qmail+(many patches) for SMTP, Vpopmail for a single UID mail sandbox (shell accounts my ass, not on this rig), and Courier-Imap. Vpopmail is configured to store userinfo in MySQL and Courier-Imap auths out of Vpopmail's tables. Everything is installed in /var/qmail/* and that /var/qmail/ is an NFS v3 export from a RAID server. All servers connect to a dedicated MySQL server that also stores it's databases on another NFS share from the RAID. Also each server mounts /www from the RAID. Each realserver runs all services, smtpd, imapd, httpd and dns. I use TWIG as a webmail imap client, which is configured to connect to imapd on localhost (each server does this). Incoming smtp, httpd and dns requests are load balanced, but not imapd, since they are local connections on each server. Each server stores it's logs locally, then they are combined with a cron script and moved to the raid. It's been working very well in a devel environment for over a year (round- robin dns, not lvs). I've recently begun the project to rebuild the system and scale it up into a commercially viable system, which is quite a task since most of the software packages are at least a year old, and I'll be using a pair of LVS directors instead of the RRDNS. <quote> Users will also be using some sort of webmail (IMP/HORDE) to get their mail when they are off site...other than that standard Eudora/Netscape will be used for retrieval. </quote> I settled on TWIG mainly because of it's vhost support. With Vpopmail, I can execute <verb> # /var/qmail/vpopmail/bin/vadddomain somenewdomain.com <postmaster passwd> </verb> and add that domain to dns and begin adding users and serving it up. I had to tweak TWIG just a bit to get it to properly deal with the "user@domain" style login that Vpopmail requires, but otherwise it works great. Each vhost domain can have it's own config file, but there is only one copy of TWIG in /www. TWIG uses MySQL, and though it doesn't require it, I also create a seperate database for each vhost domain. IMP's development runs along at about Mach 0.00000000004 and I got tired of waiting for them to get a working addressbook. That plus it doesn't vhost all that well. SquirrelMail is very nice, but again not much vhost support. Plus TWIG includes the kitchen sink. Email, contacts, schedule, notes, todo, bookmarks and even USENET (which I don't use), each module can be enabled/disabled in the config file for that domain, and it's got a very complete advanced security module (which I also don't use). It's all PHP and using mod_gzip is pretty fast. I tested the APC optimizer for PHP, but every time I made a change to a script I had to reload Apache. Not very handy for a devel system, but it did add some noticable speed increases, until I unloaded it. <quote> The real servers would need access to both the users home directories as well as the /var/mail directory. I am not too familiar with the actual locking problems...I understand the basics but I also hear that NFS V3 was supposed to fix some of the locking issues with V2...I also saw some links to GFS,AFS,etc not too sure how they would work... </quote> Just a quick note: About a week ago I tried compiling a kernel that had been patched by SGI for XFS. The kernel (2.4.2) compiled fine, but choked once the LVS patches had been applied. Not having a lot of time to play around with it, I simply moved to 2.4.4+lvs 0.9 and decided not to bother with XFS on the director boxes. <quote> also I thought about samba and only found one post from last year where someone was going to try it but there was no more info there. </quote> Well, there's how I do it. I've tried damned near every combination of GPL software available over about the last 2 years to finally arrive at my current setup. Now if I could just load balance MySQL... (Greg Cope) <quote> MySQL connections / data transfere work much faster (20% ish) when on local host - so how about running mysql on each host, which is a select only system, and each localhost uses replication to a mster DB that is used for inserts and updates ? </quote> Ultimately I think I'll have to. After I get done rebuilding the system to use kernel 2.4 and LVS and everything is stabilized, then I'll be looking very hard at just this sort of thing. <sect1>authd/identd (port 113) and tcpwrappers (tcpd) You do not explicitely set authd (==identd) as an LVS service. It is used with some services (eg sendmail, services running inside tcpwrappers). authd initiates calls from the real-servers to the client. LVS is designed for services which receive connect requests from clients. LVS does not allow authd to work anymore and this must be taken into account when running services that cooperate with authd. The inability of authd to work with LVS is important enough that there is a separate <ref id="authd" name="section on authd">. <sect1>http name and IP-based (with VS-DR or VS-Tun) http with name- and ip-based http is a simple one port service. Your httpd must be listening to the VIP which will be on lo:0 or tunl0:0. The httpd can be listening on the RIP too (on eth0) for mon, but for the LVS you need the httpd listening to the VIP as well. Thanks to Doug Bagley <tt/doug@deja.com/ for getting this info on ip and name based http into the HOWTO. Both ip-based and name-based webserving in an LVS are simple. In ip-based (HTTP/1.0) webserving, the client sends a request to a hostname which resolves to an IP (the VIP on the director). The director sends the request to the httpd on a real-server. The httpd looks up its httpd.conf to determine how to handle the request (e.g. which DOCUMENTROOT). In named-based (HTTP/1.1) webserving, the client passes the HOST: header to the httpd. The httpd looks up the httpd.conf file and directs the request to the appropriate DOCUMENTROOT. In this case all URL's on the webserver can have the same IP. The difference between ip- and name-based web support is handled by the httpd running on the real-servers. LVS operates at the IP level and has no knowledge of ip- or name-based httpd and has no need to know how the URLs are being handled. For the definitive word on ip-based and name-based web support see http://www.apache.org/docs/vhosts/index.html Here are some excerpts. The original (HTTP/1.0) form of http was IP-based, ie the httpd accepted a call to an IP:port pair, eg 192.168.1.110:80. In the single server case, the machine name (www.foo.com) resolves to this IP and the httpd listens to calls to this IP. Here's the lines from httpd.conf <tscreen><verb> Listen 192.168.1.110:80 <VirtualHost 192.168.1.110> ServerName lvs.mack.net DocumentRoot /usr/local/etc/httpd/htdocs ServerAdmin root@chuck.mack.net ErrorLog logs/error_log TransferLog logs/access_log &etago;VirtualHost> </verb></tscreen> To make an LVS with IP-based httpds, this IP is used as the VIP for the LVS and if you are using VS-DR/VS-Tun, then you set up multiple real-servers, each with the httpd listening to the VIP (ie its own VIP). If you are running an LVS for 2 urls (www.foo.com, www.bar.com), then you have 2 VIPs on the LVS and the httpd on each real-server listens to 2 IPs. The problem with ip-based virtual hosts is that an IP is needed for each url and ISPs charge for IPs. (Doug Bagley <tt/doug@deja.com/) With HTTP/1.1, a client Name based virtual hosting uses the HTTP/1.1 "Host:" header, which HTTP/1.1 clients send. This allows the server to know what host/domain, the client thinks it is connecting to. A normal HTTP request line only has the request path in it, no hostname, hence the new header. IP-based virtual hosting works for older browsers that use HTTP/1.0 and don't send the "Host:" header, and requires the server to use a separate IP for each virtual domain. The httpd.conf file then has <tscreen> <verb> NameVirtualHost 192.168.1.110 <VirtualHost 192.168.1.110> ServerName www.foo.com DocumentRoot /www.foo.com/ .. &etago;VirtualHost 192.168.1.110> <VirtualHost 192.168.1.110> ServerName www.bar.com DocumentRoot /www.bar.com/ .. &etago;VirtualHost 192.168.1.110> </verb></tscreen> DNS for both hostnames resolves to 192.168.1.110 and the httpd determines the hostname to accept the connection from the "Host:" header. Old (HTTP/1.0) browsers will be served the webpages from the first VirtualHost in the httpd.conf. For LVS again nothing special has to be done. All the hostnames resolve to the VIP and on the real-servers, VirtualHost directives are setup as if the machine was a standalone. From Ted Pavlic <tt/pavlic@netwalk.com/. Note that in 2000, <url url="http:/www.arin.net/announcements/" name="ARIN"> (look for "name based web hosting" announcements, the link changes occasionally) announced that IP based webserving would be phased out in favor of name based webserving for ISPs who have more that 256 hosts. This will only require one IP for each webserver. (There are exceptions, ftp, ssl, frontpage...) <sect1>http with VS-NAT Summary: make sure the httpd on the real-server is listening on the RIP not the VIP (this is the opposite of what was needed for VS-DR or VS-Tun). (Remember, there is no VIP on the real-server with VS-NAT). tc lewis had an (ip-based) non-working http VS-NAT setup. The VIP was a routable IP, while the real-servers were virtual hosts on the non-routable 192.168.1.0/24 network. >From: Michael Sparks <tt/michael.sparks@mcc.ac.uk/ What's happening is a consequence of using NAT. Your LVS is accepting packets for the VIP, and re-writing them to either 192.168.123.3 or 192.168.123.2. The packets therefore arrive at those two servers marked for address 192.168.123.2 or 192.168.123.3, not the VIP. As a result when apache sees this: <tscreen><verb> <VirtualHost w1.bungalow.intra> ... &etago;VirtualHost> </verb></tscreen> It notices that the packets are arriving on either 192.168.123.2 or 192.168.123.3 and not w1.bungalow.intra, hence your problem. Solutions <itemize> <item> If this is the only website being serviced by these two servers, change the config so the default doc root is the one you want. <item> If they're servicing many websites, map a realworld IP to an aliases on the real-servers and use that to do the work. IMO this is messy, and could cause you major headaches. <item> Use VS-DR or VS-Tun - that way the above config could be used without problems since the VS address is a local address as well. This'd be my choice. </itemize> Joe 10 May 2001 It just occured to me that a real-server in a VS-NAT LVS is listening on the RIP. The client is sending to the VIP. In an HTTP 1.1 or name based httpd, doesn't the server get a request with the URL (which will have the VIP) in the payload of the packet (where an L4 switch doesn't see it)? Won't the server be unhappy about this? This has come up before with name based service like <ref id="https" name="https"> and for <ref id="indexing" name="indexing of webpages">. Does anyone know how to force an HTTP 1.1 connection (or to check whether the connection was HTTP 1.0 or 1.1) so we can check this? Paul Baker <tt/pbaker@where2getit.com/ 10 May 2001 <quote> The HTTP 1.1 request (and also 1.0 requests from any modern browser) contain a Host: header which specifies the hostname of the server. As long as the webservers on the real-servers are aware that they are serving this hostname. There should be no issue with 1.1 vs 1.0 http requests. </quote> so both virtualHost and servername should be the reverse dns of the VIP? <quote> Yes. Your Servername should be the reverse dns of the VIP and you need to have a Virtualhost entry for it as well. In the event that you are serving more than one domain on that VIP, then you need to have a VirtualHost entry for each domain as well. </quote> what if instead of the name of the VIP, I surf to the actual IP? There is no device with the VIP on the VS-NAT real-server. Does there need to be one? Will an entry in /etc/hosts that maps the VIP to the public name do? Ilker Gokhan <tt/IlkerG@sumerbank.com.tr/ <quote> If you write URL with IP address such as http://123.123.123.123/, the Host: header is filled with this IP address, not hostname. You can see it using any network monitor program (tcpdump). </quote> <sect1>httpd normally closes connections If you look with ipvsadm to see the activity on an LVS serving httpd, you won't see much. A non-persistent httpd on the real-server closes the connection after sending the packets. Here's the output from ipvsadm, immediately after retrieving a gif filled webpage from a 2 real-server LVS. <verb> director:# ipvsadm IP Virtual Server version 0.2.5 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:www rr -> bashfull.mack.net:www Masq 1 2 12 -> sneezy.mack.net:www Masq 1 1 11 </verb> The InActConn are showing the connections that transferred hits that have been closed and are in the FIN state waiting to timeout. You may see "0" in the InActConn column, leading you to think that you are not getting the packets via the LVS. <sect1>Persistence with http; browser opens many connections to httpd With the first version of the http protocol, HTTP/1.0, a client would request a hit/page from the httpd. After the transfer, the connection was dropped. It is expensive to setup a tcp connection just to transfer a small number of packets, when it is likely that the client will be making several more requests immediately afterwards (e.g. if the client downloads a page with references to gif images in it, then after parsing the html page, it will issue requests to fetch the gifs). With HTTP/1.1 persistent connection was possible. The client/server pair negotiate to see if persistent connection is available. The httpd will keep the connection open for a period (KeepAliveTimeout, 15sec usually) after a transfer in case further transfers are requested. The client can drop the connection any time it wants to (i.e. when it has got all the hits on a page). Alois Treindl <tt/alois@astro.ch/ 30 Apr 2001 <quote> when I reload a page on the client, the browser makes several http hits on the server for the graphics in the page. These hits are load balanced between the real servers. I presume this is normal for HTTP/1.0 protocol, though I would have expected Netscape 4.77 to use HTTP/1.1 with one connection for all parts of a page. </quote> Joe Here's the output of ipvsadm after downloading a test page consisting of 80 different gifs (80 lines of <img src="foo.gif">. <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 1.0.7 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs.mack.net:http rr -> bashfull.mack.net:http Route 1 2 0 -> sneezy.mack.net:http Route 1 2 0 </verb> It would appear that the browser has made 4 connections which are left open. The client shows (netstat -an) 4 connections which are ESTABLISHED, while the real-servers show 2 connections each in FIN_WAIT2. Presumably each connection was used to transfer an average of 20 requests. If the client-server pair were using persistent connection, I would expect only one connection to have been used. Andreas J. Koenig <tt/andreas.koenig@anima.de/ 02 May 2001 <quote> Netscape just doesn't use a single connection, and not only Netscape. All major browsers fire mercilessly a whole lot of connections at the server. They just don't form a single line, they try to queue up on several ports simultaneously... ...and that is why you should never set KeepAliveTimeout to 15 unless you want to burn your money. You keep several gates open for a single user who doesn't use them most of the time while you lock others out. </quote> (Julian) <quote> Hm, I think the browsers fetch the objects by creating 3-4 connections (not sure how many exactly). If there is a KeepAlive option in the httpd.conf you can expect small number of inactive connections after the page download is completed. Without this option the client is forced to create new connections after each object is downloaded and the HTTP connections are not reused. The browsers reuse the connection but there are more than one connections. KeepAlive Off can be useful for banner serving but a short KeepAlive period has its advantages in some cases with long rtt where the connection setups costs time and because the modern browsers are limited to the number of connections they open. Of course, the default period can be reduced but its value depends on the served content, whether the client is expected to open many connections for short period or just one. </quote> Peter Mueller <tt/pmueller@sidestep.com/ 01 May 2001 <quote> I was searching around on the web and found the following relevant links.. <verb> http://thingy.kcilink.com/modperlguide/performance/KeepAlive.html http://httpd.apache.org/docs/keepalive.html -- not that useful http://www.apache.gamma.ru/docs/misc/fin_wait_2.html -- old but interesting </verb> </quote> Andreas J. Koenig <tt/andreas.koenig@anima.de/ 02 May 2001 <quote> If you have 5 servers with 15 secs KeepAliveTimeout, then you can serve 60*60*24*5/15 = 28800 requests per day Joe <quote> don't you actually have MaxClients=150 servers available and this can be increased to several thousand presumably? </quote> Peter Mueller <quote> I think a factor of 64'000 is forgotten here (number of possible reply ports), plus the fact that most http connections do seem to terminate immediately, despite the KeepAlive. </quote> Sure, and people do this and buy lots of RAM for them. But many of them servers are just in 'K' state, waiting for more data on these KeepAlive connections. Moreover, they do not compile the status module into their servers and never notice. Let's rewrite the above formula: MaxClients / KeepAliveTimeout denotes the number of requests that can be satisfied if all clients *send* a keepalive header (I think that's "Connection: keepalive") but *do not actually use* the kept-alive line. If they actually use the kept-alive line, you can serve more, of course. Try this: start apache with the -X flag, so it will not fork children and set the keepalivetimeout to 60. Then load a page from it with Netscape that contains many images. You will notice that many pictures arive quickly and a few pictures arive after a long, long, long, looooong time. When the browser parses the incoming HTML stream and sees the first IMG tag it will fire off the first IMG request. It will do likewise for the next IMG tag. At some point it will reach an IMG tag and be able to re-use an open keepalive connection. This is good and does save time. But if a whole second has passed after a keepalive request it becomes very unlikely that this connection will be re-used ever, so 15 seconds is braindead. One or two seconds is OK. In the above experiment my Netscape loaded 14 images immediately after the HTML page was loaded, but it took about a minute for each of the remaining 4 images which happened to be the first in the HTML stream. </quote> Joe Here's the output of ipvsadm after downloading the same 80 gif page with the -X option on apache (only one httpd is seen with ps, rather than the 5 I usually have). <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 0.2.11 (size=16384) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs.mack.net:http rr -> bashfull.mack.net:http Route 1 1 1 -> sneezy.mack.net:http Route 1 0 2 </verb> The page shows a lot of loading at the status line, then stops showing 100% of 30k. However the downloaded page is blank. A few seconds later the gifs are displayed. The client shows 4 connections in CLOSE_WAIT and the real-servers each show 2 connections in FIN_WAIT2. Paul J. Baker <tt/pbaker@where2getit.com/ 02 May 2001 <quote> The KeepAliveTimeout value is NOT the connection time out. It says how long Apache will keep an active connection open waiting for a new request to come on the SAME connection after it has fulfilled a request. Setting this to 15 seconds does not mean apache cuts all connections after 15 seconds. I write server load-testing software so I have do quiet a bit of research in the behaviour of each browser. If Netscape hits a page with a lot of images on it, it will usually open about 8 connections. It will use these 8 connections to download things as quickly as it can. If the server cuts each connection after 1 request is fullfilled, then Netscape browser has to keep reconnecting. This costs a lot of time. KeepAlive is a GOOD THING. Netscape does close the connections when it is done with them which will be well before the 15 seconds since the last request expire. Think of KeepAliveTimeout as being like an Idle Timeout in FTP. Imagine it being set to 15 seconds. </quote> <sect1>Dynamically generated images on web pages Solutions are to generate the image on a shared directory or to use fwmark to setup the LVS. Both methods are described in the section using fwmark for <ref id="dynamic" name="dynamically generated images">. <sect1>other considerations with http: logs, shutting down httpd, cookies, mod_proxy, indexing programs <sect2>Logs <verb> >From: Emmanuel Anne <tt/emanne@absysteme.fr/ >.. the problem about the logs. Apparently the best is to have >each web server process its log file on a local disk, and then >to make stats on both all files for the same period... >It can become quite complex to handle, is there not a way to >have only one log file for all the servers </verb> (Joe) (hasn't been tested): log to a common nfs mounted disk? I don't know whether you can have httpds running on separate machines writing to the same file. I notice (using truss on Solaris) that apache does write locking on files while it is running. Possibly it write-locks the log files. Normally multiple forked httpds are running. Presumably each of them writes to the log files and presumably each of them locks the log files for writing. <sect2>Shutting down http You need to shut down httpd gracefully, by bringing the weight to 0 and letting connections drop, or you will not be able to bind to port 80 when you restart httpd. If you want to do on the fly modifications to your httpd, and keep all real-servers in the same state, you may have problems. Date: Fri, 05 Jan 2001 08:12:05 -0800 From: Thornton Prime <tt/thornton@jalan.com/ <verb> > I have been having some problems restarting apache on servers that are > using LVS-NAT and was hoping someone had some insight or a workaround. > > Basically, when I make a configuration change to my webservers and I try > to restart them (either with a complete shutdown or even just a graceful > restart), Apache tries to close all the current connections and re-bind > to the port. The problem is that invariably it takes several minutes for > all the current connections to clear even if I kill apache, and the > server won't start as long as any socket is open on port 80, even if it > is in a 'CLOSING' state. > > Michael E Brown wrote: > > > > Catch-22. I think the proper way to do something like this is to take the > > affected server out of the LVS table _before_ making any configuration > > changes to the machine. Wait until all connections are closed, then make > > your change and restart apache. You should run into less problems this > > way. After the server has restarted, then add it back into the pool. > > I thought of that, but unfortunately I need to make sure that the > servers in the cluster remain in a near identical state, so the > reconfiguration time should be minimal. </verb> Julian wrote Hm, I don't have such problems with Apache. I use the default configuration-time settings, may be with higher process limit only. Are you sure you use the latest 2.2 kernels in the real servers? <verb> > I'm guessing that my problem is that I am using LVS persistent > connections, and combined with apache's lingering close this makes it > difficult for apache to know the difference between a slow connection > and a dead connection when it tries to close down, so the time it takes > to clear some of the sockets approaches my LVS persistence time. > > I haven't tried turning off persistence, and I haven't tried > re-compiling apache without lingering-close. This is a production > cluster with rather heavy traffic and I don't have a test cluster to > play with. In the end rebooting the machine has been faster than waiting > for the ports to clear so I can restart apache, but this seems really > dumb, and doesn't work well because then my cluster machines have > different configuration states. </verb> One reason your servers to block can be a very low value for the client number. You can build apache in this way: CFLAGS=-DHARD_SERVER_LIMIT=2048 ./configure ... and then to increase MaxClients (up to the above limit). Try with different values. And don't play too much with the MinSpareServers and MaxSpareServers. Values near the default are preferred. Is your kernel compiled with higher value for the number of processes: /usr/src/linux/include/linux/tasks.h <verb> > Is there any way anyone knows of to kill the sockets on the webserver > other than simply wait for them to clear out or rebooting the machine? > (I tried also taking the interface down and bringing it up again ... > that didn't work either.) > > Is there any way to 'reset' the MASQ table on the LVS machine to force a > reset? </verb> No way! The masq follows the TCP protocol and it is transparent to the both ends. The expiration timeouts in the LVS/MASQ box are high enough to allow the connection termination to complete. Do you remove the real servers from the LVS configuration before stopping the apaches? This can block the traffic and can delay the shutdown. It seems the fastest way to restart the apache is apachectl graceful, of course, if you don't change anything in apachectl (in the httpd args). <sect2>Cookies see <ref id="cookie" name="cookie"> <sect2>URL parsing <verb> Date: Wed, 13 Dec 2000 16:45:46 -0500 (EST) From: John Cronin <tt/jsc3@havoc.gtf.org/ > Is there any way to do URL parsing for http requests (ie send cgi-bin > requests to one server group, static to another group?) </verb> Probably the best way to do this is to do it in the html code itself; make all the cgis <verb>hrefs to cgi.<your-domain-here>.com</verb>. Similarly, you can make images hrefs to <verb>image.<your-domain-here>.com</verb>. You then set these up as additional virtual servers, in addition to your www virtual server. That is going to be a lot easier than parsing URLs; this is how they have done it at some of the places I have done consulting for; some of those places were using Extreme Networks load balancers, or Resonate, or something like that, using dozens of Sun and Linux servers, in multiple hosting facilities. from Horms What you are after is a layer-7 switch, that is something that can inspect HTTP packets and make decisions bassed on that information. You can use squid to do this, there are other options. A post was made to this list about doing this a while back. Try hunting through the archives. LVS on the other hand is a layer-4 switch, the only information that it has available to it is IP address and port and protocol (TCP/IP or UDP/IP). It cannot inspect the data segment and see even understand that the request is an HTTP request, let alone that the URL requested is /cgi-bin or whatever. There has been talk of doing this, but to be honest it is a different problem to that which LVS solves and arguably should live in user space rather than kernel space as a _lot_ more proccessing is required. <sect2>mod_proxy <verb> From: Atif Ghaffar <tt/atif@4unet.net/ > Michael E Brown wrote: > > On Mon, 25 Dec 2000, Sean wrote: > > > Hi, > > > > I need to forward request using the Direct Routing method to a server. > > However I determine which server to send the request to depending on the > > file it has requested in the HTTP GET not based on it's load. > Use LVS to balance the load among several servers set up to reverse-proxy > your real-servers, set up the proxy servers to load-balance to > real-servers based upon content. > -- </verb> On the LVS servers you can run apache with mod_proxy compiled in, then redirect traffic with it. <verb> Example ProxyPass /files/downloads/ http://internaldownloadserver/ftp/ ProxyPass /images/ http://internalimagesserver/images/ more on Proxy pass: http://www.linuxfocus.org/English/March2000/article147.html or you can use mod_rewrite, in that case, your REAL servers should be reachable from the net. there is also a transparent proxy module for apache http://www.stevek.com/projects/mod_tproxy/ </verb> <sect2>Running indexing programs (eg htdig) on the LVS<label id="indexing"> (From Ted I think) Setup - real-servers are node1.foobar.com, node2.foobar.com... nodeN.foobar.com, director has VIP=lvs.foobar.com (all real-servers appear as lvs.foobar.com to users). Problem - if you run the indexing program on one of the (identical) real-servers, the urls of the indexed files will be http://nodeX.foobar.com/filename These urls will be unuseable by clients out in internetland since the real-servers are not individually accessable by clients. If instead you run the indexing program from outside the LVS (as a user), you will get the correct urls for the files, but you will have to move/copy your index back to the real-servers. Solution (from Ted Pavlic, edited by Joe). On the indexing node, if you are using VS-NAT add a non-arping device (eg lo:0, tunl0, ppp0, slip0 or dummy) with IP=VIP as if you were setting up VS-DR (or VS-Tun). With VS-DR/VS-Tun this device with the VIP is already setup. The VIP is associated in dns with the name lvs.foobar.com. To index, on the indexing node, start indexing from http://lvs.foobar.com and the real-server will index itself giving the URLs appropriate for the user in the index. Alternately (for VS-NAT), on the indexing node, add the following line to /etc/hosts 127.0.0.1 localhost lvs.foobar.com make sure your resolver looks to /etc/hosts before it looks to dns and then run your indexing program. This is a less general solution, since if the name of lvs.foobar.com was changed to lvs.bazbar.com, or if lvs.foobar.com is changed to be a CNAME, then you would have to edit all your hosts files. The solution with the VIP on every machine would be handled by dns. There is no need to fool with anything unless you are running VS-NAT. <sect1>https<label id="https"> http is an IP based protocol, while https is a name based protol. http: you can test an httpd from the console by configuring it to listen on the RIP of the real-server. Then when you bring up the LVS you can re-configure it to listen on the VIP. https: requires a certificate with the official (DNS) name of the server as the client sees it (the DNS name of the LVS cluster which is associated with the VIP). The https on the real-server then must be setup as if it had the name of the LVS cluster. To do this, activate the VIP on a device on the real-server (it can be non-arping or arping - make sure there are no other machines with the VIP on the network or disconnect your real-server), make sure that the real-server can resolve the DNS name of the LVS to the VIP (by dns or /etc/hosts), setup the certificate and conf file for https and startup the httpd. Check that a netscape client running on the real-server (so that it connects to the real-server's VIP and not to the arping VIP on the director) can connect to https://lvs.clustername.org Do this for all the real-servers, then use ipvsadm on the director to forward https requests to each of the RIPs. The scheduling method for https must be persistent for keys to remain valid. <quote> When compiling in Apache.. What kind of certificate should I create for a real application with Thawte? </quote> Alexandre Cassen <tt/Alexandre.Cassen@wanadoo.fr/ When you generate your CSR use the CN (Common Name) of the DNS entry of your VIP. <sect1>Databases<label id="databases"> Normal databaseds (eg mysqld, i.e. anything but Oracle's parallel database server for several 100k$) running under LVS suffer the same restrictions of single writer/many readers as does any other service (eg smtp) where the user can write to files on the real-server. Databases running independantly on several real-servers have to be kept synchronised for content, just as do webservers. If the database files are read-only as far as the LVS clients are concerned, and the LVS administrator can update each copy of the database on the real-servers at regular intervals (eg a script running at 3am) then you can run a copy of the databased on each real-server, reading the files which you are keeping synchronised. Online transaction processing requires that LVS clients be able to write to the database. If you try to do this by setting up an LVS where each real-server has a databased and its own database files, then writes from a particular user will go to only one of the real-servers. The database files on the other real-servers will not be updated and subsequent LVS users will be presented with inconsistent copies of the database files. The Linux Scalable Database project http://lsdproject.sourceforge.net/ is working on code to serialise client writes so that they can be written to all real-servers by an intermediate agent. Their code is experimental at the moment, but is a good prospect in the long term for setting up multiple databased and file systems on separate real-servers. (Note: May 2001, the <ref id="replication" name="replication feature of mysql"> is functionally equivelant.) Currently most databased are deployed in a multi-tier setup. The clients are out in internet land; they connect to a web-server which has clients for the database; the web-server database client connects to a single databased. In this arrangement the LVS should balance the webservers/database clients and not balance the database directly. Production LVS databases, eg the service implemented by Ryan Hulsker <tt/RHulsker@ServiceIntelligence.com/ (sample load data at http://www.secretshopnet.com/mrtg/) have the LVS users connect to database clients (perl scripts running under a webpage) on each real-server. These database clients connect to a single databased running on a backend machine that the LVS user can't access. The databased isn't being LVS'ed - instead the user connects to LVS'ed database clients on the real-server(s) which handle intermediate dataprocessing, increasing your throughput. The approach of having databaseds on each real-server accessing a common filesystem on a back-end server, fails. Tests with mysqld running on each of two real-servers working off the same database files mounted from a backend machine, showed that reads were OK, but writes from any real-server either weren't seen by the other mysqld or corrupted the database files. Presumably each mysqld thinks it owns the database files and keeps copies of locks and pointers. If another mysqld is updating the filesystem at the same time then these first set of locks and pointers are invalid. Presumably any setup in which multiple databaseds were writing to one file system (whether NFS'ed, GFS'ed, coda, intermezzo...) would fail for the same reason. In an early attempt to setup this sort of LVS jake buchholz <tt/jake@execpc.com/ setup an LVS'ed mysql database with a webinterface. LVS was to serve http and each real-server to connect to the mysqld running on itself. Jake wanted the mysql service to be lvs'ed as well and for each real-server to be a mysql client. The solution was to have 2 VIPs on the director, one for http and the other for mysqld. Each http real-server makes a mysql request to the myqslVIP. In this case no real-server is allowed to have both a mysqld and an httpd. A single copy of the database is nfs'ed from a fileserver. This works for reads. <label id="replication"> <url url="http://www.mysql.com" name="MySQL"> (and most other databases) supports replication of databases. Ted Pavlic <tt/tpavlic@netwalk.com/ on Fri, 23 Mar 2001 <quote> When used with LVS, a replicated database is still a single database. The MySQL service is not load balanced. HOWEVER, it is possible to put some of your databases on one server and others on another. Replicate each SET of databases to the OTHER server and only access them from the other server when needed (at an application or at some fail-over level). </quote> Doug Sisk <tt/sisk@coolpagehosting.com/ 9 May 2001 <quote> An <url url="http://www.phpbuilder.com/columns/tanoviceanu20000912.php3" name="article on mysql's built in replication facility"> </quote> <sect1>Cookies<label id="cookie"> Cookies are not a service. Cookies are a mechanism for maintaining state for a client when using the stateless http/https protocols. Other methods for maintaining state involve passing information to the client in the URL. (This can be done with e.g. <url url="http://www.php.org/" name="php">.) Cookies are passed between servers and clients which have http, https and/or database services and need to be considered when setting up an LVS. For the cookie specification see <url url="http://home.netscape.com/newsref/std/cookie_spec.html" name="netscape site">. Being a layer 4 switch, LVS doesn't inspect the content of packets and doesn't know what's in them. A cookie is contained in a packet and the packet looks just like any other packet to an LVS. <verb> Eric Brown wrote: > Can LVS in any of its modes be configured to support cookie based persistent > sessions? Date: Wed, 3 Jan 2001 19:40:58 -0800 From: Horms <tt/horms@vergenet.net/ No. This would require inspection of the TCP data secion, and infact an understanding of HTTP. LVS has access only to the TCP headers. </verb> Roberto Nibali <tt/ratz@tac.ch/ 19 Apr 2001 LVS is a Layer4 load balancer and can't do content based (L7) load balancing. You shouldn't try to solve this problem by changing the TCP Layer to provide a solution which should be handled by the Application Layer. You should never touch/tweak TCP settings out of the boundaries given in the various RFC's and their implementations. If your application passes a cookie to the client, these are the general approaches: <itemize> <item>buy an L7 load balancer (and don't use LVS). <item>Set a very high persistency timeout and hope it is higher than the period a client will wait to come back after he found his credit card, or look at other sites, or have a cup of coffee. This is not a good solution. <itemize> <item>Increased persistency timeout increases the number of concurrent connections possible, which increases the amount of memory required to hold the connection table. A persistency timeout of 30min, with clients connecting at 500 connections/s you would need a memory pool of at least: 30*60*128*500/(1024*1024) = 109 MBytes. With the standard timeout of 300 seconds, you'd only need 109/6 = 18 Mbytes. <item> Long persistency times are incompatible with the DoS defense strategies employeed by <ref id="DoS" name="secure_tcp">. </itemize> <item> Have a 2-tier architecture where you have the application directly on the webserver itself and maybe helped by a database. The problem of the cookies storage is not solved however. You have to deal with the replication problem. Imagine following setup: <verb> ----> Web1/App --> / \ Clients ----> director -> Web2/App ---> DB Server \ / ----> Web3/App --> </verb> Cookies are generated and stored locally on each WebX server. But if you have a persistency timeout of 300s (default LVS setting) and the client had his cup of coffee while entering his visa numbers, he would get to a new server. This new server whould then ask the client to reauthenticate. There are solutions to this e.g <itemize> <item>NFS export a dedicated cookie directory over the back-interfaces. Cookies are quickly distributed among the servers. <item>the application is written to handle cookie replication and propagation between the WebX servers (you have at least 299 seconds time to replicate the cookie on all web servers. This should be enough even for distributing over serial line and do a crosscheck :) This does not work (well) for geographically distributed webserver. </itemize> <item>3-Tier architecture <verb> --> Web1 -- / \ Clients ----> LVS ----> Web2 ----> Application Server <---> DB Server \ / --> Web3 --> </verb> The cookies are generated by the application server and either stored there or on the database server. If a request comes in, the LVS assigns the request f.e to Web1 and sets the persistency timeout. Web1 does a short message exchange with the application server which generates the sessionID as a cookie and stores it. The webserver sends the cookie back and now we are safe. Again this whole procedure has t_pers_timeout (300 seconds normally) amout of time. Let's assume the client times out (has gone for a cup of coffee). When he comes back normally on a Layer4 load balancer he will be forwarded to a new server, (say Web2). The CGI script on Web2 does the same as happened originally on Web1: it generates a cookie as sessionID. But the application server will tell the script that there is already a cookie for this client and will pass it to Web2. In this way we have unlimited persistency based on cookies but limited persistency for TCP. Advantages <itemize> <item>set your own persistency timeout values <item>TCP state timeout values are not changed. <item>table lookup is faster <item>it's cheaper than buying an L7 load balancer </itemize> Disadvantages: <itemize> <item>more complex setup, more hardware <item>you have to write some software </itemize> <item>If a separate database is running on each webserver, use replication to copy the cookie between servers. (You have 300 secs to do this). This was also mentioned by Ted Pavlic in connection with <ref id="databases" name="databases">. </itemize> <sect1>r commands; rsh, rcp, and their ssh replacements<label id="rshd"> An example of using rsh to copy files is in <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance data for single real-server LVS"> Sect 5.2, Caution: The matter of rsh came up in a private e-mail exchange. The person had found that rshd, operating as an LVS'ed service, initiated a call (rsh client request) to the rshd running on the LVS client. (See Stevens "Unix Network Programming" Chapter 14, which explains rsh.) This call will come from the RIP rather than the VIP. This will require rsh to be run under VS-NAT or else the real-servers must be able to contact the client directly. Similar requests from the <ref id="authd" name="identd"> client and <ref id="passive_ftp" name="passive ftp"> on real-servers cause problems for LVS. <sect1>nfs It is possible with LVS to export directories from real-servers to a client, making an nfs fileserver (see <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance data for single real-server LVS">), near the end). This is all fine and dandy except that there is no easy way to fail-out the nfs service. Joseph Mack wrote: <quote> One of the problems with running NFS as an LVS'ed service (ie to make an LVS fileserver), that has come up on this mailing list is that a filehandle is generated from disk geometry and file location data. In general then the identical copies of the same file that are on different real-servers will have different file handles. When a real-server is failed out (e.g. for maintenance) and the client is swapped over to a new machine (which he is not supposed to be able to detect), he will now have an invalid file handle. Is our understanding of the matter correct? </quote> Dave Higgen <tt/dhiggen@valinux.com/ 14 Nov 2000 In principle. The file handle actually contains a 'dev', indicating the filesystem, the inode number of the file, and a generation number used to avoid confusion if the file is deleted and the inode reused for another file. You could arrange things so that the secondary server has the same FS dev... but there is no guarantee that equivalent files will have the same inode number; (depends on order of file creation etc.) And finally the kicker is that the generation number on any given system will almost certainly be different on equivalent files, since it's created from a random seed. <quote> If so is it possible to generate a filehandle only on the path/name of the file say? </quote> Well, as I explained, the file handle doesn't contain anything explicitly related to the pathname. (File handles aren't big enough for that; only 32 bytes in NFS2, up to 64 in NFS3.) Trying to change the way file handles are generated would be a MASSIVE redesign project in the NFS code, I'm afraid... In fact, you would really need some kind of "universal invariant file ID" which would have to be supported by the underlying local filesystem, so it would ramify heavily into other parts of the system too... NFS just doesn't lend itself to replication of 'live' filesystems in this manner. It was never a design consideration when it was being developed (over 15 years ago, now!) There HAVE been a number of heroic (and doomed!) efforts to do this kind of thing; for example, Auspex had a project called 'serverguard' a few years ago into which they poured millions in resources... and never got it working properly... :-( Sorry. Not the answer you were hoping for, I guess... <sect1>RealNetworks streaming protocols Jerry Glomph Black <tt/black@real.com/ August 25, 2000 RealNetworks' streaming protocols are <itemize> <item> PNM (TCP on port 7070, UDP from server -> player on ports 6970-7170). PNM was the original protocol in version 1 through 5. It's now mostly legacy. <item> RTSP (TCP on port 554, similar UDP as above, but often on multiple ports) With the G2 release, we adopted the RTSP delivery standard. The current version, RealPlayer 8 came out about two weeks ago. A free one is available to run on just about any platform in common use today. The Linux versions are great. <item> There's also a HTTP/TCP-only fallback mode which is (usually) on port 8080. </itemize> The server configuration can be altered to run on any port, but the above numbers are the customary, and almost universally-used ones. Mark Winter, a network/system engineer in my group wrote up the following detailed recipe on how we do it with LVS: add IP binding in the G2 server config file <verb> <List Name="IPBindings"> <Var Address_1="<real ip address>"/> <Var Address_2="127.0.0.1"/> <Var Address_3="<virtual ip address>"/> </List> On the LVS side ./ipvsadm -A -u <VIP>:0 -p ./ipvsadm -A -t <VIP>:554 -p ./ipvsadm -A -t <VIP>:7070 -p ./ipvsadm -A -t <VIP>:8080 -p ./ipvsadm -a -u <VIP>:0 -r <REAL IP ADDRESS> ./ipvsadm -a -t <VIP>:554 -r <REAL IP ADDRESS> ./ipvsadm -a -t <VIP>:7070 -r <REAL IP ADDRESS> ./ipvsadm -a -t <VIP>:8080 -r <REAL IP ADDRESS> </verb> (Ted) I just wanted to add that if you use FWMARK, you might be able to make it a little simpler and not have to worry about forwarding EVERY UDP port. <verb> # Mark packets with FWMARK1 ipchains -A input -d <VIP>/32 7070 -p tcp -m 1 ipchains -A input -d <VIP>/32 554 -p tcp -m 1 ipchains -A input -d <VIP>/32 8080 -p tcp -m 1 ipchains -A input -d <VIP>/32 6970:7170 -p udp -m 1 # Setup the LVS to listen to FWMARK1 ipvsadm -A -f 1 -p # Setup the real server ipvsadm -a -f 1 -r <RIP> </verb> Not only is this only six lines rather than eight, but now you've setup a persistent port grouping. You do not have to forward EVERY UDP port, and you're still free to setup non-persistent services (or other persistent services that are persistent based on other ports). When you want to remove a real server, you now do not have to remove FOUR real servers, you just remove one. Same thing with adding. Plus, if you want to change what's forwarded to each real server, you can do so with ipchains and not bother with taking up and down the LVS. ALSO... if you have an entire network of VIPs, you can setup IPCHAINS rules which will forward the entire network automatically rather than each VIP one by one. -------------------------- <sect>VS-NAT <sect1>Introduction VS-NAT is based on cisco's LocalDirector. This method was used for the first LVS. (If you want to set up a test LVS, this is still probably the simplest setup.) With VS-NAT, the incoming packets are rewritten by the director to have the destinatation address of one of the real-servers and then forwarded to the real-server. The replies from the real-server are sent to the director where they are rewritten have the source address of the VIP. Unlike the other two methods of forwarding used in an LVS (VS-DR and VS-Tun) the real-server only needs a functioning tcpip stack (eg a networked printer). I.e. the real-server can have any operatining system and no modifications are made to the configuration of the real-servers (except setting their route tables). <sect1>Example Two Network VS-NAT (VIP and RIPs on different network) Here the client is on the same network as the VIP (in a production LVS, the client will be coming in from an external network via a router). The director can have 1 or 2 NICs (two NICs will allow higher throughput of packets, since the traffic on the real-server network will be separated from the traffic on the client network). <verb> Machine IP client CIP=192.168.1.254 director VIP VIP=192.168.1.110 (the IP for the LVS) director internal interface DIP=10.1.1.1 real-server1 RIP1=10.1.1.2 real-server2 RIP2=10.1.1.3 real-server3 RIP3=10.1.1.4 . . real-serverN RIPn=10.1.1.n+1 dip DIP=10.1.1.9 (director interface on the VS-NAT network) </verb> <verb> ________ | | | client | |________| CIP=192.168.1.254 | (router) | __________ | | | | VIP=192.168.1.110 (eth0:110) | director |---| |__________| | DIP=10.1.1.9 (eth0:9) | | ----------------------------------- | | | | | | RIP1=10.1.1.2 RIP2=10.1.1.3 RIP3=10.1.1.4 (all eth0) _____________ _____________ _____________ | | | | | | | real-server | | real-server | | real-server | |_____________| |_____________| |_____________| </verb> here's the lvs.conf file for this setup <verb> LVS_TYPE=VS_NAT INITIAL_STATE=on VIP=eth0:110 lvs 255.255.255.0 192.168.1.255 DIP=eth0 dip 192.168.1.0 255.255.255.0 192.168.1.255 DIRECTOR_DEFAULT_GW=client SERVICE=t telnet rr real-server1:telnet real-server2:telnet real-server3:telnet SERVER_NET_DEVICE=eth0 SERVER_DEFAULT_GW=dip #----------end lvs_nat.conf------------------------------------ </verb> The VIP is the only IP known to the client. The RIPs are on a different network to the VIP (although with only 1 NIC on the director, the VIP and the RIPs are on the same wire). In normal NAT, masquerading is the rewriting of packets originating behind the NAT box. With VS-NAT, the incoming packet (src=CIP,dst=VIP, abbreviated to CIP->VIP) is rewritten by the director (becoming CIP->RIP). The action of the LVS director is called demasquerading. The demasqueraded packet is forwarded to the real-server. The reply packet (RIP->CIP) is generated by the real-server. <sect1>All packets from the real-server to the outside world must go through the director<label id="NAT_default_gw"> For VS-NAT to work <itemize> <item> The default gw for the real-servers must be the director. <item> That's not enough: all packets from the real-servers must go through the director. </itemize> This is the single most common cause of problems setting up a VS-NAT LVS. In a normal server farm, the default gw of the real-server would be the router to the internet and the packet RIP->CIP would be sent directly to the client. In a VS-NAT LVS, the default gw of the real-servers must be the director. The director masquerades the packet from the real-server (rewrites it to VIP->CIP) and the client receives a rewritten packet with the expected source IP of the VIP. Note: the packet must be routed via the director, there must be no other path to the client. A packet arriving at the client directly from the real-server will not be seen as a reply to the client's request and the connection will hang. If the director is not the default gw for the real-servers, then if you use tcpdump on the director to watch an attempt to telnet from the client to the VIP (run tcpdump with `tcpdump port telnet`), you will see the request packet (CIP->VIP), the rewritten packet (CIP->RIP) and the reply packet (RIP->CIP). You will not see the rewritten reply packet (VIP->CIP). (Remember if you have a switch on the real-server's network, rather than a hub, then each node only sees the packets to/from it. tcpdump won't see packets to between other nodes on the same network.) Part of the setup of VS-NAT then is to make sure that the reply packet goes via the director, where it will be rewritten to have the addresses (VIP->CIP). In some cases (e.g. <ref id="one_network" name="1 net NS-NAT">) icmp redirects have to be turned off on the director so that the real-server doesn't get a redirect to forward packets directly to the client. In a production system, a router would prevent a machine on the outside exchanging packets with machines on the RIP network. As well, the real-servers will be on a private network (eg 192.168.x.x/24) and will not be routable. In a test setup (no router), these safeguards don't exist. All machines (client, director, real-servers) are on the same piece of wire and if routing information is added to the hosts, the client can connect to the real-servers independantly of the LVS. This will stop VS-NAT from working (your connection will hang), or it may appear to work (you'll be connecting directly to the real-server). In a test setup, traceroute from the real-server to the client should go through the director (2 hops). The configure script will test that the director's gw is 2 hops from the real-server and that the route to the director's gw is via the director, hopefully to prevent this type of error. In production you should _not_ be able to ping from the real-servers to the client. The real-servers should not know about any other network than their own (here 10.1.1.0). The connection from the real-servers to the client is through ipchains (for 2.2.x kernels) and VS-NAT tables setup by the director. In my first attempt at VS-NAT setup, I had all machines on a 192.168.1.0 network and added a 10.1.1.0 private network for the real-servers/director, without removing the 192.168.1.0 network on the real-servers. All replies from the servers were routed onto the 192.168.1.0 network rather than back through VS-NAT and the client didn't get any packets back. The VS-NAT setup can have a separate NIC for the DIP and the VIP putting the real-server network and the LAN for the VIP on different wires (the director could be a firewall for the real-servers). This should prevent real-servers routing packets directly to the client (at least it has for me). Here's the general setup I use for testing. The client (192.168.2.254) connects to the VIP on the director. (The VIP on the real-server is present only for VS-DR and VS-Tun.) For VS-DR, the default gw for the real-servers is 192.168.1.254. For VS-NAT, the default gw for the real-servers is 192.168.1.9. <verb> ____________ | |192.168.1.254 (eth1) | client |---------------------- |____________| | CIP=192.168.2.254 (eth0) | | | | | VIP=192.168.2.110 (eth0) | ____________ | | | | | director | | |____________| | DIP=192.168.1.9 (eth1, arps) | | | (switch)------------------------ | RIP=192.168.1.2 (eth0) VIP=192.168.2.110 (for VS-DR, lo:0, no_arp) _____________ | | | real-server | |_____________| </verb> This setup works for both VS-NAT and VS-DR. Here's the routing table for one of the real-servers as in the VS-NAT setup. <verb> bashfull:# netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 192.168.1.0 0.0.0.0 255.255.255.0 U 40 0 0 eth0 127.0.0.0 0.0.0.0 255.0.0.0 U 40 0 0 lo 0.0.0.0 192.168.1.9 0.0.0.0 UG 40 0 0 eth0 </verb> Here's a traceroute from the real-server to the client showing 2 hops. <verb> traceroute to client2.mack.net (192.168.2.254), 30 hops max, 40 byte packets 1 director.mack.net (192.168.1.9) 1.089 ms 1.046 ms 0.799 ms 2 client2.mack.net (192.168.2.254) 1.019 ms 1.16 ms 1.135 ms </verb> icmp redirects are on at the director, but the director doesn't issue a redirect (see <ref id="icmp_redirects">) because the packet RIP->CIP from the real-server emerges from a different NIC on the director than it arrived on (and with different source IP). The client machine doesn't send a redirect since it is not forwarding packets, it's the endpoint of the connection. <sect1>Run configure Use lvs_nat.conf as a template (sample here will setup VS-NAT in the diagram above assuming the real-servers are already on the network and using the DIP as the default gw). <verb> #--------------lvs_nat.conf---------------------- LVS_TYPE=VS_NAT INITIAL_STATE=on #director setup: VIP=eth0:110 192.168.1.110 255.255.255.0 192.168.1.255 DIP=eth0:10 10.1.1.10 10.1.1.0 255.255.255.0 10.1.1.255 #Services on real-servers: #telnet to 10.1.1.2 SERVICE=t telnet wlc 10.1.1.2:telnet #http to a 10.1.1.2 (with weight 2) and to high port on 10.1.1.3 SERVICE=t 80 wlc 10.1.1.2:http,2 10.1.1.3:8080 10.1.1.4 #real-server setup (nothing to be done for VS-NAT) #----------end lvs_nat.conf------------------------------------ </verb> The output is a commented rc.lvs_nat file. Run the rc.lvs_nat file on the director and then the real-servers (the script knows whether it is running on a director or real-server). The configure script will setup up masquerading, forwarding on the director and the default gw for the real-servers. <sect1>Setting up demasquerading on the director; 2.4.x and 2.2.x The packets coming in from the client are being demasqueraded by the director. In 2.2.x you need to masquerade the replies. Here's the masquerading code in rc.lvs_nat produced by configure.pl <verb> echo "turning on masquerading " #setup masquerading echo "1" >/proc/sys/net/ipv4/ip_forward echo "installing ipchain rules" /sbin/ipchains -A forward -j MASQ -s 10.1.1.2 http -d 0.0.0.0/0 #repeated for each real-server and service .. .. echo "ipchain rules " /sbin/ipchains -L </verb> In this example, http is being masqueraded by the director, allowing the real-server to reply to the telnet requests from the director being demasqueraded by the director as part of the 2.2.x LVS code. In 2.4.x masquerading of LVS'ed services is done explicitely by the LVS code and no extra masquerading (ipchains/iptables) commands need be run. <sect1>masquerading services from the real-servers You may want to allow clients on the real-servers to connect to servers on the internet. Setting this up is quite independant of the LVS and you can do this with a VS-NAT or VS-DR director. For VS-NAT this is simple as the default gw is already the director. For VS-DR, you have to <ref id="NAT_with_VS-DR" name="route the packets from clients on the real-servers through the director">, while packets from the VIP on the real-server goes is routed elsewhere. Telnet requests initiated by the real-server will go out through the director (the default gw for the real-server) without any masquerading (or demasquerading). Since the real-servers for a VS-NAT LVS will most likely have private IP's, telnet'ing may not get you very far. If you did want telnet to be masqueraded by the director, so that you can get to the outside world, then on the director you need to run a command like <verb> /sbin/ipchains -A forward -j MASQ -s 10.1.1.2 telnet -d 0.0.0.0/0 </verb> Telnet will be now masqueraded out by the director, quite independently of the LVS setup. With VS-NAT, the ports can be re-mapped. A request to port 80 on the director can be sent to port 8000 on a real-server. This is possible because the source and destination of the packets are already being rewritten and no extra overhead is required to rewrite the port numbers. The rewriting is slow (60usec/packet on a pentium classic) and limits the throughput of VS-NAT (for 536byte packets, this is 72Mbit/sec or about 100BaseT). While VS-NAT throughput does not scale well with the number of real-servers, the advantage of VS-NAT is that real-servers can have any OS, no modifications are needed to the real-server to run it in an LVS, and the real-server can have services not found on Linux boxes. For the earlier versions of VS-NAT (with 2.0.36 kernels) the timeouts were set by linux/include/net/ip_masq.h, the default values of masquerading timeouts are: <verb> #define MASQUERADE_EXPIRE_TCP 15*16*Hz #define MASQUERADE_EXPIRE_TCP_FIN 2*16*Hz #define MASQUERADE_EXPIRE_UDP 5*16*Hz </verb> <sect1>How VS-NAT works ipvsadm does the following <verb> #setup connection for telnet, using round robin /sbin/ipvsadm -A -t 192.168.1.110:23 -s rr #connections to x.x.x.110:telnet are sent to # real-server 10.1.1.2:telnet #using VS-NAT (the -m) with weight 1 /sbin/ipvsadm -a -t 192.168.1.110:23 -r 10.1.1.2:23 -m -w 1 #and to real-server 10.1.1.3 #using VS-NAT with weight 2 /sbin/ipvsadm -a -t 192.168.1.110:23 -r 10.1.1.3:23 -m -w 2 </verb> (if the service was http, the webserver on the realhost could be listening on port 8000 instead of 80) Example: client requests a connection to 192.168.1.110:23 director chooses real server 10.1.1.2:23, updates connection tables, then <verb> packet source dest incoming CIP:3456 VIP:23 inbound rewriting CIP:3456 RIP1:23 reply (routed to DIP) RIP1:23 CIP:3456 outbound rewriting VIP:23 CIP:3456 </verb> The client gets back a packet with the source_address = VIP. For the verbally oriented... The request packet is sent to the VIP. The director looks up its tables and sends the connection to real-server1. The packet is rewritten with a new destination (in this case with the same port, but the port could be changed too) and sent to RIP1. The real-server replies, sending back a packet to the client. The default gw for the real-server is the director. The director accepts the packet and rewrites the packet to have source=VIP and sends the rewritten packet to the client. Why isn't the source of the incoming packet rewritten to be the DIP or VIP? (Wensong) ...changing the source of the packet to the VIP sounds good too, it doesn't require that default route rule, but requires additional code to handle it. <sect1>In VS-NAT, how do packets get back to the client, or how does the director choose the VIP as the source_address for the outgoing packets? (with Julian) In normal NAT, where a bunch of machines are sitting behind a NAT box, all outward going packets are given the IP on the outside of the NAT box. What if there are several IPs facing the outside world? For NAT it doesn't really matter as long as the same IP is used for all packets. The default value is usually the first interface address (eg eth0). With VS-NAT you want the outgoing packets to have the source of the VIP (probably on eth0:1) rather than the IP on the main device on the director (eth0). With a single real-server VS-NAT LVS serving telnet, the client will send a packet CIP:high_port -> VIP:telnet This will be masqueraded by the director to CIP:high_port -> RIP:telnet The real-server generates a reply RIP:telnet -> CIP:high_port which arrives on the director (being sent there because the director is the default gw for the real-server). To get the packet from the director to the client, you have to reverse the masquerading done by the LVS. To do this, on the director you add an ipchains rule <verb> director:# ipchains -A forward -p tcp -j MASQ -s realserver1 telnet -d 0.0.0.0/0 </verb> If the director has multiple IPs facing the outside world (eg eth0=192.168.2.1 the regular IP for the director and eth0:1=192.168.2.110 the VIP), the masquerading code has to choose the correct IP for the outgoing packet. Only the packet with src_addr=VIP will be accepted by the client. A packet with any other scr_addr will be dropped. The normal default for masquerading (eth0) should not be used in this case. The required m_addr (masquerade address) is the VIP. > Does LVS fiddle with the ipchains tables to do this? No, ipchains only delivers packets to the masquerading code. It doesn't matter how the packets are selected in the ipchains rule. The m_addr (masqueraded_address) is assigned when the first packet is seen (the connect request from the client to the VIP). LVS sees the first packet in the LOCAL_IN chain when it comes from the client. LVS assigns the VIP as maddr. The MASQ code sees the first packet in the FORWARD chain when there is a -j MASQ target in the ipchains rule. The routing selects the maddr. If the connection already exists the packets are masqueraded. The LVS can see packets in the FORWARD chain but they are for already created connections, so no maddr is assigned and the packets are masqueraded with the address saved in the connections structure (the VIP) when it was created. Here's another version of the same conversation Julian Anastasov <tt/ja@ssi.bg/ 01 May 2001 There are 3 common cases: <enum> <item>The connection is created as response to packet. <item>The connection is created as response to packet to another connection. <item>The connection is already created </enum> Case (1) can happen in the plain masquerading case where the in->out packets hit the masquerading rule. In this case when nobody recommends the saddr for the packets going to the external side of the MASQ, the masq code uses the routing to select the maddr for this new connection. This address is not always the DIP, it can be the preferred source address for the used route, for example, address from another device. Case (1) happens also for LVS but in this case we know: <itemize> <item>the client address/port (from the received datagram) <item>the virtual server address/port (from the received datagram) <item>the real server address/port (from the LVS scheduler) </itemize> But this is on out->in packet and we are talking about in->out packets Case (2) happens for related connections where the new connection can be created when all addresses and ports are known or when the protocol requires some wildcard address/port matching, for example, ftp. In this case we expect the first packet for the connection after some period of time. It seems you are interested how case (3) works. The answer is that the NAT code remembers all these addresses and ports in a connection structure with these components <itemize> <item>external address/port (LVS: client) <item>masquerading address/port (LVS: virtual server) <item>internal address/port (LVS: real server) <item>protocol <item>etc </itemize> LVS and the masquerading code simply hook in the packet path and they perform the header/data mangling. In this process they use the information from the connection table(s). The rule is simple: when a packet is already for established connection we must remember all addresses and ports and always to use same values when mangling the packet header. If we select each time different addresses or ports we simply break the connection. After the packet is mangled the routing is called to select the next hop. Of course, you can expect problems if there are fatal route changes. So, the short answer is: the LVS knows what maddr to use when a packet from the real server is received because the connection is already created and we know what addresses to use. Only in the masquerading case (where LVS os not involved) connections can be created and a masquerading address to be selected without using rule for this. In all other cases there is a rule that recommends what addresses to be used at creation time. After creation the same values are used. <sect1>Performance of VS-NAT The throughput of VS-NAT is limited by the time taken by the director to rewrite a packet. The limit for a pentium classic 75MHz is about 80Mbit/sec (100baseT). Increasing the number of real-servers does not increase the throughput. The <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance page"> shows a slightly higher latency with VS-NAT compared to VS-DR or VS-Tun, but the same maximum throughput. The load average on the director is high (>5) at maximum throughput, and the keyboard and mouse are quite sluggish. The same director box operating at the same throughput under VS_DR or VS-Tun has no perceptable load as measured by top or by mouse/keyboard responsiveness. <sect1>One network VS-NAT<label id="one_network"> The disadvantage of the 2 network VS-NAT is that the real-servers are not able to connect to machines in the network of the VIP. You couldn't make a VS-NAT setup out of machines already on your LAN, which were also required for other purposes to stay on the LAN network. Here's a one network VS-NAT LVS. <verb> ________ | | | client | |________| CIP=192.168.1.254 | | __________ | | | | VIP=192.168.1.110 (eth0:110) | director |---| |__________| | DIP=192.168.1.9 (eth0:9) | | ------------------------------------ | | | | | | RIP1=192.168.1.2 RIP2=192.168.1.3 RIP3=192.168.1.4 (all eth0) _____________ _____________ _____________ | | | | | | | real-server | | real-server | | real-server | |_____________| |_____________| |_____________| </verb> The problem: A return packet from the real-server (with address RIP->CIP) will be sent to the real-server's default gw (the director). ICMP redirects will be sent from the director telling the real-server of the better route directly to the client. The real-server will then send the packet directly to the client and it will not be demasqueraded by the director. The client will get a reply from the RIP rather than the VIP and the connection will hang. The cure: In the previous HOWTO I said that initial attempts to handle this by turning off redirects had not worked. The problem appears now to be solved. Thanks to <tt/michael_e_brown@dell.com/ and Julian <tt/ja@ssi.bg/ for help sorting this out. To get a VS-NAT LVS to work on one network - 1. On the director, turn off icmp redirects <verb> director:/etc/lvs# echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects director:/etc/lvs# echo 0 > /proc/sys/net/ipv4/conf/default/send_redirects director:/etc/lvs# echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects </verb> (Note: eth0 may be eth1 etc, on your machine). 2. Make the director the default and only route for outgoing packets. You will probably have set the routing on the real-server up like this <verb> realserver:/etc/lvs# netstat -r Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 0.0.0.0 director 0.0.0.0 UG 0 0 0 eth0 </verb> Note the route to 192.168.1.0/24. This allows the real-server to send packets to the client by just putting them out on eth0, where the client will pick them up directly (without being demasqueraded) and the LVS will not work. Remove the route to 192.168.1.0/24. realserver:/etc/lvs#route del -net 192.168.1.0 netmask 255.255.255.0 dev eth0 This will leave you with <verb> realserver:/etc/lvs# netstat -r Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 0.0.0.0 director 0.0.0.0 UG 0 0 0 eth0 </verb> The VS-NAT LVS now works. If LVS is forwarding telnet, you can telnet from the client to the VIP and connect to the real-server. You can ping from the client to the real-server. You can also connect _directly_ to services on the real-server _NOT_ being forwarded by LVS (in this case e.g. ftp). You can no longer connect directly to the real-server for services being forwarded by the LVS. (In the example here, telnet ports are not being rewritten by the LVS, ie telnet->telnet). <verb> client:~# telnet realserver Trying 192.168.1.11... ^C (i.e. connection hangs) </verb> Here's tcpdump on the director. Since the network is switched the director can't see packets between the client and real-server. The client initiates telnet. `netstat -a` on the client shows a SYN_SENT from port 4121. <tscreen><verb> director:/etc/lvs# tcpdump tcpdump: listening on eth0 16:37:04.655036 realserver.telnet > client.4121: S 354934654:354934654(0) ack 1183118745 win 32120 <mss 1460,sackOK,timestamp 111425176[|tcp]> (DF) 16:37:04.655284 director > realserver: icmp: client tcp port 4 121 unreachable [tos 0xc0] </verb></tscreen> (repeats every second until I kill telnet on client) I don't see the connect request from client->real-server. The first packet I see is the ack from the real-server, which will be forwarded via the director. The director will rewrite the ack to be from the director. The client will not accept an ack to port 4121 from director:telnet. <sect1>Various debugging techniques for routes (with Julian) The routes added with the route command go into the kernel FIB (Forwarding information base) route table. The contents are displayed with the route (or netstat -a) command. Following an icmp redirect, the route updates go into the kernel's route cache (route -C). You can flush the route cache with <verb> echo 1 > /proc/sys/net/ipv4/route/flush or ip route flush cache </verb> Here's the route cache on the real-server before any packets are sent. <verb> realserver:/etc/rc.d# route -C Kernel IP routing cache Source Destination Gateway Flags Metric Ref Use Iface realserver director director 0 1 0 eth0 director realserver realserver il 0 0 9 lo </verb> With icmp redirects enabled on the director, repeatedly running traceroute to the client shows the routes changing from 2 hops to 1 hop. This indicates that the real-server has received an icmp redirect packet telling it of a better route to the client. <verb> realserver:/etc/rc.d# traceroute client traceroute to client (192.168.1.254), 30 hops max, 40 byte packets 1 director (192.168.1.9) 0.932 ms 0.562 ms 0.503 ms 2 client (192.168.1.254) 1.174 ms 0.597 ms 0.571 ms realserver:/etc/rc.d# traceroute client traceroute to client (192.168.1.254), 30 hops max, 40 byte packets 1 director (192.168.1.9) 0.72 ms 0.581 ms 0.532 ms 2 client (192.168.1.254) 0.845 ms 0.559 ms 0.5 ms realserver:/etc/rc.d# traceroute client traceroute to client (192.168.1.254), 30 hops max, 40 byte packets 1 client (192.168.1.254) 0.69 ms * 0.579 ms </verb> Although the route command shows no change in the FIB, the route cache has changed. (The new route of interest is bracketted by >< signs.) <verb> realserver:/etc/rc.d# route -C Kernel IP routing cache Source Destination Gateway Flags Metric Ref Use Iface client realserver realserver l 0 0 8 lo realserver realserver realserver l 0 0 1038 lo realserver director director 0 1 138 eth0 >realserver client client 0 0 6 eth0< director realserver realserver l 0 0 9 lo director realserver realserver l 0 0 168 lo </verb> Packets to the client now go directly to the client instead of via the director (which you don't want). It takes about 10mins for the client's route cache to expire (experimental result). The timeouts may be in /proc/sys/net/ipv4/route/gc_*, but their location and values are well encrypted in the sources :) (some more info from Alexey at <url url="http://marc.theaimsgroup.com/?l=linux-kernel&ero;m=91754108723334&ero;w=2" name="LVS archives"> ) Here's the route cache after 10mins. <tscreen><verb> realserver:/etc/rc.d# route -C Kernel IP routing cache Source Destination Gateway Flags Metric Ref Use Iface realserver realserver realserver l 0 0 1049 lo realserver director director 0 1 139 eth0 director realserver realserver l 0 0 0 lo director realserver realserver l 0 0 236 lo </verb></tscreen> There are no routes to the client anymore. Checking with traceroute, shows that 2 hops are initially required to get to the client (i.e. the routing cache has reverted to using the director as the route to the client). After 2 iterations, icmp redirects route the packets directly to the client again. <tscreen><verb> realserver:/etc/rc.d# traceroute client traceroute to client (192.168.1.254), 30 hops max, 40 byte packets 1 director (192.168.1.9) 0.908 ms 0.572 ms 0.537 ms 2 client (192.168.1.254) 1.179 ms 0.6 ms 0.577 ms realserver:/etc/rc.d# traceroute client traceroute to client (192.168.1.254), 30 hops max, 40 byte packets 1 director (192.168.1.9) 0.695 ms 0.552 ms 0.492 ms 2 client (192.168.1.254) 0.804 ms 0.55 ms 0.502 ms realserver:/etc/rc.d# traceroute client traceroute to client (192.168.1.254), 30 hops max, 40 byte packets 1 client (192.168.1.254) 0.686 ms 0.533 ms * </verb></tscreen> If you now turn off icmp redirects on the director. <tscreen><verb> director:/etc/lvs# echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects director:/etc/lvs# echo 0 > /proc/sys/net/ipv4/conf/default/send_redirects director:/etc/lvs# echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects </verb></tscreen> Checking routes on the real-server - <tscreen><verb> realserver:/etc/lvs# netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 0.0.0.0 director 0.0.0.0 UG 0 0 0 eth0 </verb></tscreen> nothing has changed here. Flush the kernel routing table and show the kernel routing table - <tscreen><verb> realserver:/etc/lvs# ip route flush cache realserver:/etc/lvs# route -C Kernel IP routing cache Source Destination Gateway Flags Metric Ref Use Iface realserver director director 0 1 0 eth0 director realserver realserver l 0 0 1 lo </verb></tscreen> There are no routes to the client. Now when you send packet to the client, the route stays via the director needing 2 hops to get to the client. There are no one hop packets to the client. <tscreen><verb> realserver:/etc/rc.d# traceroute client traceroute to client (192.168.1.254), 30 hops max, 40 byte packets 1 director (192.168.1.9) 0.951 ms 0.56 ms 0.491 ms 2 client (192.168.1.254) 0.76 ms 0.599 ms 0.574 ms realserver:/etc/rc.d# traceroute client traceroute to client (192.168.1.254), 30 hops max, 40 byte packets 1 director (192.168.1.9) 0.696 ms 0.562 ms 0.583 ms 2 client (192.168.1.254) 0.62 ms 0.603 ms 0.576 ms realserver:/etc/rc.d# traceroute client traceroute to client (192.168.1.254), 30 hops max, 40 byte packets 1 director (192.168.1.9) 0.692 ms * 0.599 ms 2 client (192.168.1.254) 0.667 ms 0.603 ms 0.579 ms realserver:/etc/rc.d# traceroute client traceroute to client (192.168.1.254), 30 hops max, 40 byte packets 1 director (192.168.1.9) 0.689 ms 0.558 ms 0.487 ms 2 client (192.168.1.254) 0.61 ms 0.63 ms 0.567 ms realserver:/etc/rc.d# traceroute client traceroute to client (192.168.1.254), 30 hops max, 40 byte packets 1 director (192.168.1.9) 0.705 ms 0.563 ms 0.526 ms 2 client (192.168.1.254) 0.611 ms 0.595 ms * realserver:/etc/rc.d# traceroute client traceroute to client (192.168.1.254), 30 hops max, 40 byte packets 1 director (192.168.1.9) 0.706 ms 0.558 ms 0.535 ms 2 client (192.168.1.254) 0.614 ms 0.593 ms 0.573 ms </verb></tscreen> The kernel route cache <tscreen><verb> realserver:/etc/rc.d# route -C Kernel IP routing cache Source Destination Gateway Flags Metric Ref Use Iface client realserver realserver l 0 0 17 lo realserver realserver realserver l 0 0 2 lo realserver director director 0 1 0 eth0 >realserver client director 0 0 35 eth0< director realserver realserver l 0 0 16 lo director realserver realserver l 0 0 63 lo </verb></tscreen> shows the the only route to the client (labelled with >< ) is via the director. > For send_redirects, what's the difference between all, default and eth0? from Julian see the <url url="http://marc.theaimsgroup.com/?l=linux-virtual-server&ero;m=97932487110806&ero;w=2" name="LVS archives"> When the kernel needs to check for one feature (send_redirects for example) it uses such calls: if (IN_DEV_TX_REDIRECTS(in_dev)) ... These macros are defined in /usr/src/linux/include/linux/inetdevice.h The macro returns a value using expression from <verb>all/<var> and <dev>/<var></verb> So, these macros check for example for: <verb> all/send_redirects || eth0/send_redirects </verb> or <verb> all/hidden && eth0/hidden </verb> when you create eth0 for first time using ifconfig eth0 ... up default/send_redirects is copied to eth0/send_redirects from the kernel, internally. I.e. default/ contains the initial values the device inherits when it is created. This is the safest way a device to appear with correct <verb>conf/<dev>/</verb> values. When we put value in <verb>all/<var></verb> you can assume that we set the <verb><var></verb> When we put value in <verb>all/<var></verb> you can assume that we set the <verb><var></verb> for all devices in this way: <verb> all/<var> the macro returns: for && 0 0 for && 1 the value from <dev>/<var> for || 0 the value from <dev>/<var> for || 1 1 </verb> This scheme allows the different devices to have different values for their vars. For example, if we set 0 to all/send_redirects, the 3th line applies to the values, i.e. the result from the macro is the real value in <dev>/send_redirects. If we set 1 to all/send_redirects according to the 4th line, the macro always returns 1 regardless of the <dev>/send_redirects > ...how to debug/understand TCP/IP packets? The RFC documents are your friends: http://www.ietf.cnri.reston.va.us/rfc.html The numbers you need: <verb> 793 TRANSMISSION CONTROL PROTOCOL 1122 Requirements for Internet Hosts -- Communication Layers 1812 Requirements for IP Version 4 Routers 826 An Ethernet Address Resolution Protocol </verb> for tcpdump, see man tcpdump. from <tt/Steve.Gonczi@networkengines.com/, for Micrsoft NT _server_, there is a uSoft supplied packet capture utility as well. also -W. Richard Stevens: TCP-IP Illustrated, Vol 1, a good intro into packet layouts and protocol basics. (anything by Stevens is good - Joe). From: Ivan Figueredo <tt/idf@weewannabe.com/ for windump - http://netgroup-serv.polito.it/windump/ Here's an untested solution from Julian for a one network VS-NAT <verb> >put the client in the external logical network. By this way the >client, the director and the real server(s) are on same physical network >but the client can't be on the masqueraded logical network. So, change the >client from 192.168.1.80 to 166.84.192.80 (or something else). Don't add >interface 192.168.1.0/24 in the client. The path to 192.168.1.0/24 must be >through DIP (I don't see such IP for the Director). Why in your setup >DIP==VIP ? If you add DIP (166.84.192.33 for example) in the director you >can later add path for 192.168.1.0/24 through 166.84.192.33. There is no >need to use masquerading with 2 NICs. Just remove the client from the >internal logical network used by the LVS cluster. </verb> A different working solution from Ray Bellis <tt/rpb@community.net.uk/ <verb> >On my system the VIP and RIP aren't on the same *physical* network, just on >the same *logical* subnet. >I still have a dual-ethernet box acting as a director, and the VIP is >installed as an alias interface on the external side of the director, even >though the IP address it has is in fact assigned from the same subnet as the >machines on the internal side of the director. </verb> <sect1>Postings from the mailing list Ray Bellis <tt/rpb@community.net.uk/ has used a 2 NIC director to have the RIPs on the same logical network as the VIP (ie RIP and VIP numbers are from the same subnet), although they are in different physical networks. <tt/frederic.defferrard@ansf.alcatel.fr/ would be possible to use LVS-NAT to load-balance virtual-IPs to ssh-forwarded real-IPs? Ssh can also be used to create a local access that is forwarded to a remote access throught the ssh protocol. For example you can use ssh to securely map a local acces to a remote POP server: local:localport ==> local:ssh ~~~~~ ssh port forwarding ~~~~~ remote:ssh ==> remote:pop And when you connect to local:localip you are transparently/securely connected to remote:pop The main idea is to allow RS in differents LANs with RS that are non-Linux (precluding VS-Tun). <verb> Example: - VS:81 ---- ssh ---- RS:80 / INTERNET - - - - > VS:80 (NAT)-- VS:82 ---- ssh ---- RS:80 \ - VS:83 ---- ssh ---- RS:80 </verb> (Wensong) you can use VPN (or CIPE) to map some external real servers into your private cluster network. If you use LVS-NAT, make sure the routing on the real server must be configuration properly so that the response packets will go through the load balancer to the clients. <verb> > I think that it isn't necessery to have the default router to the load > balancer when using ssh because when the RS address is the same that the > VS address (differents ports) > > > > > Wensong > > > > > Example: > > > - VS:81 ---- ssh ---- RS:80 > > > / > > > INTERNET - - - - > VS:80 (NAT)-- VS:82 ---- ssh ---- RS:80 > > > \ > > > - VS:83 ---- ssh ---- RS:80 > > > > > > The main idea is to allow RS in differents LANs. > > > </verb> With the NAT method, your example won't work because the LVS/NAT treats packets as local ones and forward to the upper layers without any change. However, your example give me an idea that we can dynamically redirect the port 80 to port 81, 82 and 83 respectively for different connections, then your example can work. However, the performance won't be good, because lots of works are done in the application level, and the overhead of copying from kernel to user-space is high. Another thought is that we might be able to setup LVS/DR with real server in different LANs by using of CIPE/VPN stuff. For example, we use CIPE to establish tunnels from the load balancer to real servers like <verb> 10.0.0.1================10.0.1.1 realserer1 10.0.0.2================10.0.1.2 realserer2 --- Load Balancer 10.0.0.3================10.0.1.3 realserer3 10.0.0.4================10.0.1.4 realserer4 10.0.0.5================10.0.1.5 realserer5 </verb> Then, you can add LVS/DR configuration commands as: <verb> ipvsadm -A -t VIP:www ipvsadm -a -t VIP:www -r 10.0.1.1 -g ipvsadm -a -t VIP:www -r 10.0.1.2 -g ipvsadm -a -t VIP:www -r 10.0.1.3 -g ipvsadm -a -t VIP:www -r 10.0.1.4 -g ipvsadm -a -t VIP:www -r 10.0.1.5 -g </verb> I haven't tested it. Please let me know the result if anyone tests this configuration. <sect>VS-DR<label id="VS-DR"> VS-DR is based on IBM's NetDispatcher. The NetDispatcher sits in front of a set of webservers, which appear as one webserver to the clients. The NetDispatcher served http for the Atlanta and the Sydney Olympic games and for the chess match between Kasparov and Deep Blue. Here's an example set of IPs for a VS-DR setup. Note that in this example the RIPs are on the same network as the VIP. If the RIPs were on a different network (eg 192.168.2.0/24) to the VIP && the router (here the client) was not allowed to route to RIP network (ie the real-servers do not receive the arp requests from the router asking "who has VIP tell router"), then the arp problem is solved without requiring a patch on the director's kernel. In this example, for (my) convenience, the servers are on the same network as the client and you have to handle the arp problem (I used the arp -f /etc/ethers approach). <verb> Host IP client CIP=192.168.1.254 director DIP=192.168.1.1 virtual IP (VIP) VIP=192.168.1.110 (arpable, IP clients connect to) real-server1 RIP1=192.168.1.2, VIP=192.168.1.110 (lo:0, not arpable) real-server2 RIP2=192.168.1.3, VIP=192.168.1.110 (lo:0, not arpable) real-server3 RIP3=192.168.1.4, VIP=192.168.1.110 (lo:0, not arpable) . . real-server-n 192.168.1.n+1 </verb> <verb> #lvs_dr.conf LVS_TYPE=VS_DR INITIAL_STATE=on VIP=eth0:110 lvs 255.255.255.255 192.168.1.110 DIP=eth0 dip 192.168.1.0 255.255.255.0 192.168.1.255 DIRECTOR_DEFAULT_GW=client SERVICE=t telnet rr real-server1 real-server2 real-server3 SERVER_VIP_DEVICE=lo:0 SERVER_NET_DEVICE=eth0 SERVER_DEFAULT_GW=client #----------end lvs_dr.conf------------------------------------ </verb> <verb> ________ | | | client | |________| CIP=192.168.1.254 | (router) | __________ | | | | VIP=192.168.1.110 (eth0:1, arps) | director |--- DIP=192.168.1.1 (eth0) |__________| | | | ------------------------------------- | | | | | | RIP1=192.168.1.2 RIP2=192.168.1.3 RIP3=192.168.1.4 (eth0) VIP=192.168.1.110 VIP=192.168.1.110 VIP=192.168.1.110 (all lo:0, non-arping) _____________ _____________ _____________ | | | | | | | real-server | | real-server | | real-server | |_____________| |_____________| |_____________| | | | (router) (router) (router) | | | ----------------------------------------------> to client (or router in front of director) </verb> Here's the lvs_dr.conf file <verb> #--------------------lvs_dr.conf LVS_TYPE=VS_DR INITIAL_STATE=on #director setup VIP=eth0:12 192.168.1.110 255.255.255.255 192.168.1.110 DIP=eth0 192.168.1.10 192.168.1.0 255.255.255.0 192.168.1.255 #service setup, one service at a time SERVICE=t telnet rr 192.168.1.1 192.168.1.8 127.0.0.1 #real-server setup SERVER_LVS_DEVICE=lo0:1 SERVER_NET_DEVICE=eth0 #----------end lvs_dr.conf------------------------------------ </verb> Direct routing setup and testing is the same as VS-Tun except that all machines within the VS-DR LVS (ie the director and real-servers) must be able to arp each other. This means that they have to be on the same network without any forwarding devices between them. This means that they are using the same piece of transport layer hardware ("wire"), eg RJ-45, coax, fibre. There can be hub(s) or switch(es) in this mix. Communication within the LVS is by link-layer, using MAC addresses rather than IP's. All machines in the LVS have the VIP, only the VIP on the director replies to arp requests, the VIP on the real-servers must be on a non-arping device (eg lo:0, dummy). The restrictions for VS-DR are <itemize> <item> The client must be able to connect to the VIP on the director <item>Real-servers and the director must be on the same piece of wire (they must be able to arp each other) as packets are sent by link-layer from the director to the real-servers. <item>The route from the real-servers to the client _cannot_ go through the director, i.e. the director cannot be the default gw for the real-servers. (Note: the client does not need to connect to the the real-servers for the LVS to function. The real-servers could be behind a firewall, but the real-servers must be able to send packets to the client). The return packets, from the real-servers to the client, go directly from the real-servers to the client and _do_not_ go back through the director. For high throughput, each real-server can have its own router/connection to the client/internet and return packets need not go through the router feeding the director. (for more info see e-mail postings about VS-DR topologies in the section <ref id="topology" name="More on the arp problem and topologies of VS-DR and VS-Tun LVS's">). To allow the director to be the default gw for the real-servers (e.g. when the director is the firewall), see <ref id="martian" name="Julian's martian modification">. </itemize> Note for VS-DR (and VS-Tun), the services on the real-servers are listening to the VIP. You can have the service listening to the RIP as well, but the LVS needs the service to be listening to the VIP. This is not an issue with services like telnet which listen to all local IPs (ie 0.0.0.0), but httpd is set up to listen to only the IPs that you tell it. Normally for VS-DR, the client is on a different network to the director/server(s), and each real-server has its own route to the outside world. In the simple test case below, where all machines are on the 192.168.1.0 network, no routers are required, and the return packets, instead of going out (the router(s)) at the bottom of the diagram, would return to the client via the network device on 192.168.1.0 (presumably eth0). <sect1>How VS-DR works Here's part of the rc.lvs_dr script which configures the real-server with RIP=192.168.1.8 <verb> #setup servers for telnet, VS-DR /sbin/ipvsadm -A -t 192.168.1.110:23 -s rr echo "adding service 23 to real-server 192.168.1.6 " /sbin/ipvsadm -a -t 192.168.1.110:23 -R 192.168.1.6 -g -w 1 </verb> With VS-DR, the target port numbers of incoming packets cannot be remapped (unlike VS-NAT). A request to port 23 (telnet) on the VIP will be forwarded to port 23 on a real-server, thus the RIP entry has no accompanying port. Here's the packet headers as the request is processed by the LVS. <verb> packet source dest data 1. request from client CIP:3456 VIP:23 - 2. ipvsadm table: director chooses server=RIP1, creates link-layer packet MAC of DIP MAC of RIP1 IP datagram source=CIP:3456, dest=VIP:23, data= - 3. real-server recovers IP datagram CIP:3456 VIP:23 - 4. real-server looks up routing table, finds VIP is local, processes request locally, generates reply VIP:23 CIP:3456 "login:" 5. packet leaves real-server via its default gw, not via DIP. </verb> For the verbally oriented... A packet arrives from the client for the VIP (CIP:3456->VIP:23). The director looks up its tables and decides to send the connection to real-server_1. The director arps for the MAC address of RIP1 and sends a link-layer packet to that MAC containing an IP datagram with CIP:3456->VIP:23. This is the same src:dst as the incoming packet and the tcpip layer see this as a forwarded packet. To allow this packet to be sent to the real-server, it is not neccessary for forwarding must be on in the director (it is off by default in 2.2.x, 2.4.x kernels, this is handled by the configure script). The packet arrives at real-server_1. The real-server recovers the IP datagram, looks up its routing table, finds that the VIP (on an otherwise unused, non-arping and nonfunctional device) is local. I'm not sure what exactly happens next, but I believe the Linux tcpip stack then delivers the packet to the socket listeners, rather than to the device with the VIP, but I'm out of my depth now. The real-server now has a packet CIP:3456->VIP:23, processes it locally, constructs a reply, VIP:23->CIP:3456. The real-server looks up its routing table and sends the reply out its default gw to the internet (or client). The reply does not go through the director. The role of VS-DR is to allow the director to deliver a packet with dst=VIP (the only arp'ing VIP being on the director), not to itself, but to some machine that (as far as the director knows) doesn't have the VIP address at all. The only difference between VS-DR and VS-Tun is that instead of putting the IP datagram inside a link-layer packet with dst=MAC of the RIP, for VS-Tun the IPdatagram from the client CIP->VIP is put inside another IPdatagram DIP->RIP. The use of the non-arping lo:0 and tunl0 to hold the VIP for VS-DR and VS-Tun (respectively) is to allow the real-server's routing table to have an entry for a local device with IP=VIP _AND_ that so that other machines can't see this IP (ie it doesn't reply to arp requests). There is nothing particularly loopback about the lo:0 device that is required to make VS-DR work anymore than there is anything tunnelling about a tunl0 device. For 2.0.x kernels, a tunnel packet is de-capsulated because it is marked type=IPIP, and will be decapsulated if delivered to an lo device just as well as if delivered to a tunl device. The 2.2.x kernels are more particular and need a tunl device (see "Properties of devices for VIP"). <sect1>Handling the arp problem for VS-DR <sect2>VIP on lo:0 The VIP on the real-servers must not reply to arp requests from the client (or the router between the client and the director). <sect3>Real-servers with Linux 2.2.x kernels The loopback device does not arp by default for all OS's except Linux 2.2.x,2.4.x kernels (even when you use -noarp with ifconfig). You may need to do something if you are running a real-server with a 2.2.x or 2.4.x kernel (see the <ref id="arp_problem" name="arp problem">). <sect2>Hiding the VIP on the real-servers by putting them on a separate network (Lars' method). Lars set this up first on VS-Tun. Here it is for VS-DR. The director has 2 NICs and the real-servers are on a different network (10.1.1.0/24) to the VIP (192.168.1.0/24). All IPs reply to arps. The router/client cannot route to the real-server network and the real-server IPs do not need to be internet routable. The only change to the lvs_dr.conf file is to change the DIP to eth1 (you also don't have to handle the arp problem). <verb> ________ | | | client | |________| CIP=192.168.1.254 | (router) | VIP=192.168.1.110 (eth0, arps) __________ | | | director | |__________| DIP=10.1.1.1 (eth1, arps) | | ------------------------------------- | | | | | | RIP1=10.1.1.2 RIP2=10.1.1.3 RIP3=10.1.1.4 (eth0) VIP=192.168.1.110 VIP=192.168.1.110 VIP=192.168.1.110 (all lo:0, can arp) _____________ _____________ _____________ | | | | | | | real-server | | real-server | | real-server | |_____________| |_____________| |_____________| | | | (router) (router) (router) | | | ----------------------------------------------> to client </verb> <sect2>Transparent Proxy (TP or Horms' method) - not having the VIP on the real-server at all. This subject has it's <ref id="TP" name="own section">. <sect1>VS-DR scales well Performance tests (75MHz pentium classics, on 100Mbps network) with VS-DR on the <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance page"> showed that the limit for VS-DR is the rate at which the director can forward packets to the real-servers. LVS doesn't add any detectable latency or change the throughput of forwarding. There is little load on the director operating at high throughput in VS-DR mode. Apparently little computation is involved in forwarding. <sect1>VS-DR director is default gw for real-servers<label id="VS-DR_director_default_gw"> In the case where the director is the firewall for the real-server network, the director has to be the default gw for the real-servers. If the reply packet from the real-server to the client (VIP->CIP) goes through the director (which has a device with IP=VIP), the director is being asked to route a packet with a src address that is on the director. <verb> >From: Horms <tt/horms@vergenet.net/> > >The problem is that with Direct routing the reply from the real >server has the vip as the source address. As this is an address >of one of the interfaces on the director it will drop it if you >try and forward it through the director. It appears from >experimentation with /proc/sys/net/ipv4/conf/*/rp_filter >that at least on 2.2.14, there is no way to turn this behaviour >off. </verb> This type of packet is called a "source martian" and is dropped by the director. martians can be logged with <verb> # echo 1 >/proc/sys/net/ipv4/conf/all/log_martians. </verb> There are 3 solutions to this; 2 by Julian and 1 by Horms. <sect2>Director has 1 NIC, accepts packets via transparent proxy. If the director accepts packets for the VIP via transparent proxy, then the director doesn't have the VIP and the return packets are processed normally. (Note: transparent proxy only works on the director for 2.2.x kernels). Here's Julian's posting <verb> Clients | ISP |eth0/ppp0/... Router/Firewall/Director (LVS box) |eth1 +----------+------------+ |eth0 |eth0 Real 1 Real2 </verb> Router: transparent proxy for VIP (or all served VIPs). The ISP must feed your Director with packets for your subnet 199.199.199.0/24 VS-DR mode (Yes, VS-DR, this is not a mistake). eth1: 199.199.199.2. default gw is ISP. Real server(s): nothing special. VIP on hidden device or via transparent proxy. eth0: 199.199.199.3. default gateway is 199.199.199.2 (the Director) This is a minimum required config. You can add internal subnets yourself using the same physical network (one NIC) or by adding additional NICs, etc. They are not needed for this test. Packets from the real servers with saddr=VIP will be forwarded from the director because VIP is not configured in the Director. We expect that this setup is faster than VS/NAT. <sect2>Julian's martian modification<label id="martian"> This is a kernel patch, director has 2 NICs, VIP is on outside NIC. It doesn't work with one NIC. The patch (below) has been tested against 2.2.15pre9 (Joe) and 2.2.13 (Stephen Zander <tt/gibreel@pobox.com/). The kernel code is not changing very fast for these files. If patching other 2.2 kernels produces no rejects (HUNK FAILED) then the patch is probably OK. The patch for 2.4.x kernels is at <url url="http://www.linuxvirtualserver.org/~julian" name="Julian's patch page"> and in the text below. After applying this patch, for a test, use the default values for */rp_filter(=0). This allows real servers to send packets with saddr=VIP and daddr=client through the Director. If this patch is applied and external_eth/rp_filter is 0 (which is the default) the real servers can receive packets with saddr=any_director_ip and dst=any_RIP_or_VIP which is not very good. On the external net, set rp_filter=1 for better security. Here's the test setup (Joe) <verb> ____________ | | | client | |____________| | | 192.168.2.0/24 _____|______ | | | director | VS-DR director has 2 NICs |____________| | eth0 192.168.1.9 | eth0:12 192.168.1.1 | | 192.168.1.0/24 _____|____________________ | | _____|__________ | | | real-server(s) | default gw=192.168.1.1 |________________| </verb> 192.168.1.1 is the normal router. For the test it was put on the director instead (as an alias). The director has 2 NICs, with forwarding=on (client and real-servers can ping each other). Director runs linux-0.9.8-2.2.15pre9 unpatched or with Julian's patch. LVS is setup using the configure script in the HOWTO, redirecting telnet, with rr scheduling to 3 real-servers. The real-servers were running 2.0.36 (1) or 2.2.14 (2). The arp problem was handled for the 2.2.14 real-servers by permanently installing in the client's arp table, the MAC address of the NIC on the outside of the director, using the command `arp -f /etc/ethers` The director was booted 4 times, into unpatched, patched, unpatched and patched. After each reboot the lvs scripts were run on the director and the real-servers, then the functioning of the LVS tested by telnet'ing multiple times from the client to the VIP. For the unpatched kernel, the client connection hung and inactive connections acccumulated for each real-server. For the patched kernel, the client telnet'ed to the VIP connecting with each real-server in turn. The configure script will set up the modified VS-DR (it will warn you that you need the patch to work). Setup details are in <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance page"> <sect3>Martian modification performance Performance has similar latency to VS-NAT but the load is low on the director at high throughput of VS-DR (see the <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance page">). <sect3>Martian modification patch for 2.2.x <tscreen> <code> --- linux/include/net/ip_fib.h.orig Wed Feb 23 16:54:27 2000 +++ linux/include/net/ip_fib.h Wed Mar 15 13:46:22 2000 @@ -200,7 +200,7 @@ extern int inet_rtm_getroute(struct sk_buff *skb, struct nlmsghdr* nlh, void *arg); extern int inet_dump_fib(struct sk_buff *skb, struct netlink_callback *cb); extern int fib_validate_source(u32 src, u32 dst, u8 tos, int oif, - struct device *dev, u32 *spec_dst, u32 *itag); + struct device *dev, u32 *spec_dst, u32 *itag, int our); extern void fib_select_multipath(const struct rt_key *key, struct fib_result *res); /* Exported by fib_semantics.c */ --- linux/net/ipv4/fib_frontend.c.orig Wed Feb 23 16:54:27 2000 +++ linux/net/ipv4/fib_frontend.c Wed Mar 15 14:44:45 2000 @@ -189,7 +189,7 @@ */ int fib_validate_source(u32 src, u32 dst, u8 tos, int oif, - struct device *dev, u32 *spec_dst, u32 *itag) + struct device *dev, u32 *spec_dst, u32 *itag, int our) { struct in_device *in_dev = dev->ip_ptr; struct rt_key key; @@ -206,7 +206,8 @@ return -EINVAL; if (fib_lookup(&key, &res)) goto last_resort; - if (res.type != RTN_UNICAST) + if ((res.type != RTN_UNICAST) && + ((res.type != RTN_LOCAL) || our)) return -EINVAL; *spec_dst = FIB_RES_PREFSRC(res); if (itag) @@ -216,13 +217,20 @@ #else if (FIB_RES_DEV(res) == dev) #endif + { + if (res.type == RTN_LOCAL) { + *itag = 0; + return -EINVAL; + } return FIB_RES_NH(res).nh_scope >= RT_SCOPE_HOST; + } if (in_dev->ifa_list == NULL) goto last_resort; if (IN_DEV_RPFILTER(in_dev)) return -EINVAL; key.oif = dev->ifindex; + if (res.type == RTN_LOCAL) key.iif = loopback_dev.ifindex; if (fib_lookup(&key, &res) == 0 && res.type == RTN_UNICAST) { *spec_dst = FIB_RES_PREFSRC(res); return FIB_RES_NH(res).nh_scope >= RT_SCOPE_HOST; --- linux/net/ipv4/route.c.orig Wed Feb 23 17:00:07 2000 +++ linux/net/ipv4/route.c Wed Mar 15 13:07:28 2000 @@ -1037,7 +1037,7 @@ if (!LOCAL_MCAST(daddr)) return -EINVAL; spec_dst = inet_select_addr(dev, 0, RT_SCOPE_LINK); - } else if (fib_validate_source(saddr, 0, tos, 0, dev, &spec_dst, &itag) < 0) + } else if (fib_validate_source(saddr, 0, tos, 0, dev, &spec_dst, &itag, our) < 0) return -EINVAL; rth = dst_alloc(sizeof(struct rtable), &ipv4_dst_ops); @@ -1181,7 +1181,7 @@ if (res.type == RTN_LOCAL) { int result; result = fib_validate_source(saddr, daddr, tos, loopback_dev.ifindex, - dev, &spec_dst, &itag); + dev, &spec_dst, &itag, 1); if (result < 0) goto martian_source; if (result) @@ -1206,7 +1206,7 @@ return -EINVAL; } - err = fib_validate_source(saddr, daddr, tos, FIB_RES_OIF(res), dev, &spec_dst, &itag); + err = fib_validate_source(saddr, daddr, tos, FIB_RES_OIF(res), dev, &spec_dst, &itag, 0); if (err < 0) goto martian_source; @@ -1279,7 +1279,7 @@ if (ZERONET(saddr)) { spec_dst = inet_select_addr(dev, 0, RT_SCOPE_LINK); } else { - err = fib_validate_source(saddr, 0, tos, 0, dev, &spec_dst, &itag); + err = fib_validate_source(saddr, 0, tos, 0, dev, &spec_dst, &itag, 1); if (err < 0) goto martian_source; if (err) </code> </tscreen> <sect3>Martian modification patch for 2.4.0 <tscreen> <code> --- linux-2.4.0/include/net/ip_fib.h~ Sat Jan 20 13:16:50 2001 +++ linux/include/net/ip_fib.h Sun Mar 11 11:07:22 2001 @@ -203,7 +203,7 @@ extern int inet_rtm_getroute(struct sk_buff *skb, struct nlmsghdr* nlh, void *arg); extern int inet_dump_fib(struct sk_buff *skb, struct netlink_callback *cb); extern int fib_validate_source(u32 src, u32 dst, u8 tos, int oif, - struct net_device *dev, u32 *spec_dst, u32 *itag); + struct net_device *dev, u32 *spec_dst, u32 *itag, int our); extern void fib_select_multipath(const struct rt_key *key, struct fib_result *res); /* Exported by fib_semantics.c */ --- linux-2.4.0/net/ipv4/fib_frontend.c~ Fri Jun 2 07:25:21 2000 +++ linux/net/ipv4/fib_frontend.c Sun Mar 11 11:07:22 2001 @@ -204,7 +204,8 @@ */ int fib_validate_source(u32 src, u32 dst, u8 tos, int oif, - struct net_device *dev, u32 *spec_dst, u32 *itag) + struct net_device *dev, u32 *spec_dst, u32 *itag, + int our) { struct in_device *in_dev; struct rt_key key; @@ -233,7 +234,8 @@ if (fib_lookup(&key, &res)) goto last_resort; - if (res.type != RTN_UNICAST) + if ((res.type != RTN_UNICAST) && + ((res.type != RTN_LOCAL) || our)) goto e_inval_res; *spec_dst = FIB_RES_PREFSRC(res); if (itag) @@ -244,6 +246,10 @@ if (FIB_RES_DEV(res) == dev) #endif { + if (res.type == RTN_LOCAL) { + *itag = 0; + goto e_inval_res; + } ret = FIB_RES_NH(res).nh_scope >= RT_SCOPE_HOST; fib_res_put(&res); return ret; @@ -254,6 +260,7 @@ if (rpf) goto e_inval; key.oif = dev->ifindex; + if (res.type == RTN_LOCAL) key.iif = loopback_dev.ifindex; ret = 0; if (fib_lookup(&key, &res) == 0) { --- linux-2.4.0/net/ipv4/route.c~ Sun Nov 5 23:10:48 2000 +++ linux/net/ipv4/route.c Sun Mar 11 11:07:22 2001 @@ -1177,7 +1177,7 @@ if (!LOCAL_MCAST(daddr)) goto e_inval; spec_dst = inet_select_addr(dev, 0, RT_SCOPE_LINK); - } else if (fib_validate_source(saddr, 0, tos, 0, dev, &spec_dst, &itag) < 0) + } else if (fib_validate_source(saddr, 0, tos, 0, dev, &spec_dst, &itag, our) < 0) goto e_inval; rth = dst_alloc(&ipv4_dst_ops); @@ -1339,7 +1339,7 @@ if (res.type == RTN_LOCAL) { int result; result = fib_validate_source(saddr, daddr, tos, loopback_dev.ifindex, - dev, &spec_dst, &itag); + dev, &spec_dst, &itag, 1); if (result < 0) goto martian_source; if (result) @@ -1364,7 +1364,7 @@ goto e_inval; } - err = fib_validate_source(saddr, daddr, tos, FIB_RES_OIF(res), dev, &spec_dst, &itag); + err = fib_validate_source(saddr, daddr, tos, FIB_RES_OIF(res), dev, &spec_dst, &itag, 0); if (err < 0) goto martian_source; @@ -1447,7 +1447,7 @@ if (ZERONET(saddr)) { spec_dst = inet_select_addr(dev, 0, RT_SCOPE_LINK); } else { - err = fib_validate_source(saddr, 0, tos, 0, dev, &spec_dst, &itag); + err = fib_validate_source(saddr, 0, tos, 0, dev, &spec_dst, &itag, 1); if (err < 0) goto martian_source; if (err) </code> </tscreen> <sect3>Performance of Martian Modification See Performance page on website. <sect2>Accepting packets on VS-DR director by fwmarks Horms <ref id="fwmark" name="fwmarks"> allows the director to accept packets by fwmark. There is no VIP required on the director. <sect1>default gw(s) and routing with VS-DR/VS-Tun<label id="Pearthree"> The material here came from listening to a talk by Herbie Pearthree of IBM (posting 2000-10-10) and from a posting by TC Lewis (lost it). In normal IP communication between two hosts, the setup is symmetrical: each end of the link has an ethernet devices with an IPs and a route to the other machine. Packets are transmitted in pairs (an outgoing packet and a reply, often just an ACK). In VS-DR or VS-Tun the roles of the two machines are split between 3 machines. Here is a two network test setup, with the client in the position normally occupied by the router. In production, the client will have a public IP and connect via a router. (This is my test setup. A big trap is that services which make calls from the RIP, eg <ref id="authd" name="identd"> and <ref id="rshd" name="rshd"> will work in my setup, but fail in a production setup as the RIP will not be a routable IP). <verb> ____________ | |192.168.1.254 (eth0) | client |---------------------- |____________| <- | CIP=192.168.2.254 (eth1) | | | | V | | VIP=192.168.2.110 (eth0) | ____________ | | | | | director | | |____________| | DIP=192.168.1.1 (eth1, arps) | | | | V |---------------------------- | -> RIP=192.168.1.2 (eth0) VIP=192.168.2.110 (lo:0, no_arp) _____________ | | | real-server | |_____________| </verb> <sect2>Director's default gw The client sends a packet to the VIP on the director. The director's response is a packet to the MAC address of the RIP. Normally the director would send a packet to the client (on the internet). In an LVS the director doesn't need a default gw as it never sends packets back to the client (outside world), it only sends packets to the real-servers. A default gw for the director is not needed for the functioning of the LVS. Having a default gw would only allow the director to reply to packets from the internet, such as port scans, creating a security hazard. The director shouldn't have a default gw. <sect2>Real-server's default gw, route to real-server from router The real-server doesn't reply to the director, instead it sends its reply to the client. The real-server requires a default gw (here 192.168.1.154), but the client/router never replies to the real-server, the client/router sends its replies to the director. So the client/router doesn't need a route to the real-server network. To have one would be a security hazard. The real-server now can't ping its default gw (since there's no route for the reply packet), but the LVS still works. The flow of packets around the VS-DR LVS is shown by the ascii arrows. When an attacker tries to access the nodes on the LVS, it can only connect to the LVS services on the director. It can't connect to the real-server network, as there is no routing to the real-servers (even if they get access to the router). Presumably the real-servers are not accessable from the outside as they'll be on private networks anyhow. Note that for Julian's <ref id="martian" name="martian modification">, the director will need a default gw. <sect1>routing to real-server from director<label id="route_on_non_ip_interface"> If you are only using the link between the director and real-server for VS-DR packets (i.e. you aren't telnet or ssh'ing from the real-server to the director for your admin, and you aren't copying logs from one machine to another), then you don't need an IP on the interface on the director which connects to the real-server(s). tc lewis <tt/tcl@bunzy.net/ 12 Jul 2000 (paraphrased) <quote> I would like to send packets from the VS-DR director to the real-servers by a separate interface (eth2), but not assign an IP to this interface. Normally I put a 192.168.100.x ip on eth2, but without it, route add -net 192.168.100.0 netmask 255.255.255.0 dev eth2 just gives me an error about eth2 not existing. I just want to save an extra IP. What i'm asking is: does the director's eth2 need an ip on 192.168.100.0/24, or can i just somehow add that route to that interface to tell the machine to send packets that way? With lvs, the real servers are never going to care about the director's interface ip, since there's no direct tcp/ip connections or anything there, but it looks like it still needs an ip anyway. If all that that interface is doing is forwarding outgoing packets from the director via the dr method, then i don't see why it needs an ip address. </quote> Ted Pavlic <tt/tpavlic@netwalk.com/ You basically want to do device routing. There's nothing special about this -- many routers do it... NT even does it. So does Linux. Your original route command should work <verb> route add -net 192.168.100.0 netmask 255.255.255.0 dev eth2 </verb> as long as you've brought up eth2. Now tricking Linux into bringing up eth2 without an address might be the hard part. Try this: <verb> ifconfig eth2 0.0.0.0 up or ifconfig eth2 0 up </verb> tc lewis <tt/tcl@bunzy.net/ <verb> ifconfig eth0 0.0.0.0 up </verb> then the route did work. I tried that before with a netmask but it didn't work. Ted Pavlic <tt/tpavlic@netwalk.com/ Remember that IP=0 actually is IP=0.0.0.0, which is another name for the default route. The reason why IP=0 is 0.0.0.0 ... Remember that each IP address is simply a 4-byte unsigned integer, right? Well... the easiest way to envision this is to imagine that an IP is just like a base-256 number. For example: <verb> 216.69.192.12 (my mail server) would be: 12 + 192 * 256 + 69 * 256 * 256 + 216 * 256 * 256 * 256 </verb> Which is equal to 3628449804. So... telnet 216.69.192.12 25 is the same as: telnet 3628449804 25 0.0.0.0 is just a special system address which is the same as the default route. Making a route from 0.0.0.0 to some gateway will set your default route equal to that gateway. That's all "route add default gw ..." does. Don't believe me? Do a route -n. So when I told TC to put 0 on his IP-less NIC, I was just choosing a system IP that I knew would not ever need to be transmitted on. Linux wanted an IP to create the interface... so I gave it one -- the IP of the default gateway. Packets would never need to leave the system going to 0.0.0.0, and Linux has to listen to this address ANYWAY, so you might as well explicitly put it on an interface. What would have also worked (and might have been a better idea) would be to put 127.0.0.1 on that interface. That is another system address that Linux will listen to anyway if loopback has been turned on... and it should never transmit anything away from itself with that as the destination address, so it's safe to put it on more than one interface. The only reason I chose 0 over 127.0.0.1 is because 0 is easy... It's small... It's quick. Whenever I want to telnet to my localhost's port blah I just do a: telnet 0 blah because I'm lazy.. (Linux sees 0, interprets 0.0.0.0, sees an address it listens to, and basically treats 0 like a loopback) Also you'll notice that if you give an interface 0.0.0.0 as an IP address and do an ifconfig to get stats on that interface, it will still retain no IP address. Another perquesite of using 0.0.0.0 in TC's particular situation. It may actually cause less confusion in the end. <sect1>Setting up NAT clients on VS-DR real-servers<label id="NAT_with_VS-DR"> Here's Julian's recipe (25 Sep 2000) for setting up NAT for clients on real-servers in a VS-DR LVS. Settings for the real server(s), send all packets from the RIP network (RIPN) to the DIP (an IP on the director in the RIPN). <verb> real-server: #ip rule add prio 100 from RIPN/24 table 100 real-server: #ip route add table 100 0/0 via DIP dev eth0 </verb> The director has to to listen on DIP (if it doesn't already), not send ICMP redirects from the DIP ethernet device and masquerade packets from the RIPN. <verb> director: #ifconfig eth0:1 DIP netmask 255.255.255.0 director: #echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects director: #echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects director: #ipchains -A forward -s RIPN/24 -j MASQ </verb> <sect>VS-Tun VS-Tun is an LVS original. It is based on VS-DR and has the same high scalability/throughput of VS-DR. VS-Tun can be used with real-servers that can tunnel (==IPIP encapsulation). The director encapsulates the request packet inside an IPIP packet before sending it to the real-server. The real-server must be able to decapsulate the IPIP packet (currently linux only). Unlike VS-DR, with VS-Tun the real-servers can be on a network remote from the director, and can each be on separate networks. Thus the real-servers could be in different countries (eg a set of ftp mirror sites for a project). If this is the case the real-servers will be generating reply packets with VIP->CIP. Not being on the VIP network, the routers for the real-servers will have to be programmed to accept outgoing packets with source=VIP. Routers normally drop these packets as an anti-spoofing measure. If the real-servers are on the same network as the director, then VS-DR and VS-Tun are equivalent in terms of performance and ease of setup (see the <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance page">) With VS-DR, the real-servers can have almost any OS. Here's an example set of IPs for a VS-Tun setup. For (my) convenience the servers are on the same network as the client. The only restrictions for VS-Tun with remote hosts are that the client must be able to route to the director and that the real-servers must be able to route to the client (the return packets to the client come directly from the real-servers and do not go back through the director). Normally for VS-Tun,the client is on a different network to the director/server(s), and each server has its own route to the outside world. In the simple test case below where all machines are on the 192.168.1.0 network there would be no default route for the servers, and routing for packets from the servers to the client would use the device on the 192.168.1.0 network (presumably eth0). In reallife, the real-servers would have their own router/connection to the internet and packets returning to the client would go through this router. In any case reply packets do not go back through the director. <verb> Machine IP client CIP=192.168.1.254 director DIP=192.168.1.1 VIP=192.168.1.110 (arps, IP clients connect to) real-server-1 RIP1=192.168.1.2, VIP (tunl0, non-arping, 192.168.1.110) real-server-2 RIP2=192.168.1.3, VIP (tunl0, non-arping, 192.168.1.110) real-server-3 RIP3=192.168.1.4, VIP (tunl0, non-arping, 192.168.1.110) . . real-server-n RIPn=192.168.1.n+1, VIP (tunl0, non-arping, 192.168.1.110) </verb> <verb> #lvs_tun.conf LVS_TYPE=VS_TUN INITIAL_STATE=on VIP=eth0:110 192.168.1.110 255.255.255.255 192.168.1.110 DIP=eth0 192.168.1.9 192.168.1.0 255.255.255.0 192.168.1.255 DIRECTOR_DEFAULT_GW=client SERVICE=t telnet rr real-server1 real-server2 SERVER_VIP_DEVICE=tunl0 SERVER_NET_DEVICE=eth0 SERVER_DEFAULT_GW=client #----------end lvs_tun.conf------------------------------------ </verb> <verb> ________ | | | client | |________| CIP=192.168.1.254 | | VIP=192.168.1.110 (eth0:1, arps) __________ | | | director | |__________| DIP=192.168.1.1 (eth0) | | ----------------------------------- | | | | | | RIP1=192.168.1.2 RIP2=192.168.1.3 RIP3=192.168.1.4 (eth0) _____________ _____________ _____________ | | | | | | | real-server | | real-server | | real-server | |_____________| |_____________| |_____________| VIP=192.168.1.110 VIP=192.168.1.110 VIP=192.168.1.110 (all tunl0,non-arping) | | | (router) (router) (router) | | | ----------------------------------------------> to client </verb> <sect1>How VS-Tun works Here's part of the rc.lvs_dr script which configures the real-server with RIP=192.168.1.8 <verb> #setup servers for telnet /sbin/ipvsadm -A -t 192.168.1.110:23 -s rr /sbin/ipvsadm -a -t 192.168.1.110:23 -R 192.168.1.1 -i -w 1 </verb> With VS-Tun, the target port numbers of incoming packets cannot be remapped. A request to port 23 on the VIP will be forwarded to port 23 on a real-server, thus no port number is used for setting up the IP of the real-server. Here's the packet headers as the request is processed by VS-Tun. <verb> packet source dest data 1. request from client CIP:3456 VIP:23 - 2. ipvsadm table: director chooses server=RIP1, encapsulates into IPIP packet DIP RIP1 IP datagram source=CIP:3456, dest=VIP:23, data= - 3. real-server recovers IP datagram CIP:3456 VIP:23 - 4. real-server looks up routing table, finds VIP is local, processes request locally, generates reply VIP:23 CIP:3456 "login: " 5. packet leaves real-server via default gw, not via DIP. </verb> For the verbally oriented... A packet arrives for the VIP. The director looks up its tables and decides to send the connection to real-server_1. The director encapsulates the request packet in an IPIP datagram with DIP->RIP. The packet arrives at real-server1, the real-server recovers the original IP datagram, looks up its routing table, finds that the VIP (on an otherwise unused, non-arping and nonfunctional device) is local and processes the packet locally. A reply packet is generated with VIP:23->CIP:3456. The real-server looks up its routing table and finds that a packet to CIP goes out its default gw (not to the DIP). The tunl0 device does not arp with 2.0.36 kernels, but does with 2.2.x kernels. Go look up the section on the <ref id="arp_problem" name="arp problem"> to see if you need to patch the kernel on the real-server. <sect1>Configure VS-Tun Edit the template lvs_tun.conf and run configure. $ ./configure_lvs.pl lvs_nat.conf Load the the parameters into the director and then the real-servers with the command $ . ./etc/rc.d/rc.lvs_tun (the script knows whether it is running on a real-server or the director). (later put in /etc/rc.d or /etc/init.d and put mon_xxx.cf in /etc/mon) check the output from ipvsadm, ifconfig -a and netstat -rn, to see that the services/IP's are correct. If not re-edit and re-run the script(s). <sect1>Real-servers on different network(s) to director If the real-server is on a different network to the director, then it will be replying with packets having a source address of the VIP. In this case the router will have to be programmed to pass packets in the outward direction with the source address of the VIP. This may be a problem with some firewalls as they will be forwarding packets with a source address not in their network. <sect1>VS-Tun Questions <verb> > How does a packet get to a tunl device from a remote machine > without a MAC address? </verb> (Julian) tunl, lo and dummy are used just to configure the VIP. We don't send any packets through these devices. The requests are delivered to the real servers using their real IP addresses. The director asks only about their real IP addresses using the VS configuration. Only their gateway asks about VIP but only the director must reply. When the packet is received in the real server it is delivered locally (not forwarded or dropped) due to configured VIP. This is the only role of these "dummy" interfaces: the kernel to treat the received packet as it is destined to our host (the real server). Nothing more. No IPIP encapsulations (for tunl), no MAC address definitions, nothing more. When we answer the request we use eth0. The tunl/lo/dummy is not selected as device for the outgoing packets. We have routes for eth0 (default gateway) which we use for the outgoing traffic. This is for DROUTE and TUNNEL mode. <verb> >If two linux boxes (not in an LVS) are joined by an IPIP tunnel and >there is no MAC address associated with the tunl0 devices at each >end of the link, then how do the packets get from one machine to >the other? </verb> Julian The packets are encapsulated via IPIP and sent to the tunnel ends real IP where they are decapsulated again and appear on the tunl interface. You don't need a MAC address for point-to-point links, or logical interfaces like tunnels. <sect>localnode<label id="localnode"> (We rarely hear of anyone using this.) The director machine can be a real-server too. This is convenient when only a small number of machines are available as servers. To use this feature, add a real-server with IP 127.0.0.1 (or a local IP on your director) to your lvs_xxx.conf file, rerun the configure script, and reinstall the rc.lvs_xxx files. You will need to setup the service to listen to the VIP on the director. Some services, eg telnet listen on all IP's on the machine and you won't have to do anything special for them. Other services, eg http have to be specifically configured to listen to each IP. You are _not_ connecting to a service on 127.0.0.1 despite what this instruction might look like. LocalNode operates independantly of NAT,TUN or DR modules. <sect1>Rewriting ports with localnode Paul Monaghan wrote: <verb> > Okay, perhaps what I am trying to do can't work for what ever reason but > here it is. I've setup ipvs rules as follows: > > TCP 209.226.95.146:8080 wlc > -> 10.0.0.1:8001 Local 1 1 0 > -> 10.0.0.1:8000 Local 1 0 0 </verb> (Wensong) <quote> For the LocalNode feature, the load balancer just lets the packets pass to the upper layer (up to service daemon) to process the request. So, the port number of the local service must be equal to that of the virtual service, otherwise it won't work. Maybe I should add some code to check whether the port number of local service is equal to that of virtual service, if not, reject it to avoid such an error. </quote> i.e. you can't use localnode with VS-NAT and have it rewrite ports. You can't have a request coming to port VIP:80 and have it serviced by a demon listening on 127.0.0.1:81. If you want to do this, you could try instead <verb> $ipchains -A input -j REDIRECT 81 -d 192.168.1.12 80 -p tcp </verb> Requests to 192.168.1.12:80 will go to 192.168.1.12:81. <sect1>Testing LocalNode If you want to explore installing localnode by hand, try this. First make sure scheduling is turned on at the director (this command adds round robin scheduling and direct routing) $ipvsadm -A -t 192.168.1.110:23 -s rr With an httpd listening on the VIP (192.168.1.110:80) of the director (192.168.1.1) AND with _no_ entries in the ipvsadm table, the director appears as a normal non-LVS node and you can connect to this service at 192.168.1.110:80 from an outside client. If you then add an external real-server to the ipvsadm table in the normal manner with /sbin/ipvsadm -a -t 192.168.1.110:80 -r 192.168.1.2 then connecting to 192.168.1.110:80 will display the webpage at the real-server 192.168.1.2:80 and not the director. This is easier to see if the pages are different (eg put the real IP of each machine at the top of the webpage). Now comes the LocalNode part - You can now add the director back into the ipvsadm table with /sbin/ipvsadm -a -t 192.168.1.110:80 -r 127.0.0.1 (or replace 127.0.0.1 by another IP on the director) Note, the port is the same for LocalNode. LocalNode is independant of the LVS mode (VS-NAT/Tun/DR) that you are using for the other IP:ports. Shift-reloading the webpage at 192.168.1.110:80 will alternately display the wepages at the server 192.168.1.2 and the director at 192.168.1.1 (if the scheduling is unweighted round robin). If you remove the (external) server with /sbin/ipvsadm -d -t 192.168.1.110:80 -r 192.168.1.2 you will connect to the LVS only at the directors port. The ipvsadm table will then look like <verb> Protocol Local Addr:Port ==> Remote Addr Weight ActiveConns TotalConns ... TCP 192.168.1.110:80 ==> 127.0.0.1 2 3 3 </verb> From the client, you cannot tell whether you are connecting directly to the 192.168.1.110:80 socket or through the LVS code. <sect>Transparent proxy (TP or Horms' method)<label id="TP"> Transparent proxy is a piece of Linux kernel code (2.2.x, 2.4.x) which allows a packet destined for an IP _not_ on the host, to be accepted locally, as if the IP was on the host. Note: web caches (proxies) can operate in transparent mode, when they cache all IP's on the internet. In this mode, requests are received and transmitted without changing the port numbers (ie port 80 in and port 80 out). In a normal web cache, the clients are asked to reconfigure their browsers to use the proxy, some_IP:3128. It is difficult to get clients to do this, and the solution is transparent caching. This is more difficult to setup, but all clients will then use the cache. In the web caching world, transparent caching is often called "transparent proxy" because it is implemented with transparent proxy. In the future, it is conceivable that transparent web caching will be implemented by another feature of the tcpip layer and it would be nice if functionality of transparent web caching had a name separate from the command that is used to implement it. <sect1>General This is Horms' (<tt/horms@vergenet.net/) method (now called the transparent proxy or TP method). It uses the transparent proxy feature of ipchains to accept packets with dst=VIP by the host (director or real-servers) when it doesn't have the IP (eg the VIP) on a device. It can be used on the real-servers (where it handles the <ref id="arp_problem" name="arp problem">) or the director to accept packets for the VIP. When used on the director, TP allows the director to be the default gw for VS-DR (see <ref id="martian" name="Julian's martian modification">). Unfortunately the 2.2 and 2.4 versions of transparent proxy are as different as chalk and cheese in an LVS. Presumably the functionality has been maintained for for transparent web caching but the effect on LVS has not been considered. You can use transparent proxy for <itemize> <item>2.2.x, director and real-servers <item>2.4.x, real-servers only </itemize> (Historical note from Horms:) From memory I was getting a cluster ready for a demo at Inetnet World, New York which was held in October 1999. The cluster was to demo all sorts of services that Linux could run that were relevant to ISPs. Apache, Sendmail, Squid, Bind and Radius I believe. As part of this I was playing with LVS-DR and spotted that the real servers coulnd't accept traffic for the VIP. I had used Transparent Proxying in the past so I tried it and it worked. That cluster was pretty cool, it took me a week to put it together and it was an ISP in an albeit very large box. Transparent proxy is only implemented in Linux. <itemize> <item> 2.2.x you need IP masquerading, transparent proxing and IP firewalls turned on. <item> 2.4.x, TP is a standard part of the kernel build, there is no separate TP option. In the netfilter options, there are the options under "Full NAT (NEW)" MASQUERADE, REDIRECT. I suspect you need all these. </itemize> <sect1>How you use TP This is a demonstration of TP using 2 machines: a server (which will accept packets by TP) and a client. On the server: ipv4 forwarding must be on. <verb> echo "1" > /proc/sys/net/ipv4/ip_forward </verb> You want your server to accept telnet requests on an IP that is not on the network (say 192.168.1.111). Here's the result of commands run at the server console before running the TP code, confirming that you can't ping or telnet to the IP. <verb> server:# ping 192.168.1.111 PING 192.168.1.111 (192.168.1.111) from 192.168.1.11 : 56(84) bytes of data. From server.mack.net (192.168.1.11): Destination Host Unreachable server:# telnet 192.168.1.111 Trying 192.168.1.111... telnet: Unable to connect to remote host: No route to host </verb> so add a route and try again (lo works here, eth0 doesn't) <verb> server:# route add -host 192.168.1.111 lo server:# telnet 192.168.1.111 Trying 192.168.1.111... Connected to 192.168.1.111. Escape character is '^]'. Welcome to Linux 2.2.16. server login: </verb> This shows that you can connect to the new IP from the localhost. No transparent proxy involved yet. If you go to another machine on the same network and add a route to the new IP. <verb> client:# route add -host 192.168.1.111 gw 192.168.1.11 client:# netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 192.168.1.111 192.168.1.11 255.255.255.255 UGH 0 0 0 eth0 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo </verb> raw sockets work between the client and server - <verb> client:# traceroute 192.168.1.111 traceroute to 192.168.1.111 (192.168.1.111), 30 hops max, 40 byte packets 1 server.mack.net (192.168.1.11) 0.634 ms 0.433 ms 0.561 ms </verb> however you can't ping (i.e. icmp doesn't work) or telnet to that IP from the other machine. <verb> client:# ping 192.168.1.111 PING 192.168.1.111 (192.168.1.111) from 192.168.1.9 : 56(84) bytes of data. From server.mack.net (192.168.1.11): Time to live exceeded client:# telnet 192.168.1.111 Trying 192.168.1.111... telnet: Unable to connect to remote host: No route to host </verb> Here's the output of tcpdump running on the target host <verb> 14:09:09.789132 client.mack.net.1101 > tip.mack.net.telnet: S 1088013012:1088013012(0) win 32120 <mss 1460,sackOK,timestamp 7632700[|tcp]> (DF) [tos 0x10] 14:09:09.791205 server.mack.net > client.mack.net: icmp: time exceeded in-transit [tos 0xd0] </verb> (Anyone have an explanation for this, apart from the fact that icmp is not working? Is the lack of icmp the only thing stopping the telnet connect?) The route to 192.168.1.111 is not needed for the next part. <verb> server:# route del -host 192.168.1.111 </verb> Now add transparent proxy to the server to allow the server to accept connects to 192.168.1.111:telnet This is the command for 2.2.x kernels <verb> server:# ipchains -A input -j REDIRECT telnet -d 192.168.1.111 telnet -p tcp server:# ipchains -L Chain input (policy ACCEPT): target prot opt source destination ports REDIRECT tcp ------ anywhere 192.168.1.111 any -> telnet => telnet Chain forward (policy ACCEPT): Chain output (policy ACCEPT): </verb> For 2.4.x kernels <verb> server:# iptables -t nat -A PREROUTING -p tcp -d 192.168.1.111 --dport telnet -j REDIRECT server:# iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination REDIRECT tcp -- anywhere 192.168.1.111 tcp dpt:telnet Chain POSTROUTING (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination </verb> You still can't ping the transparent proxy IP on the server from the client <verb> client:# ping 192.168.1.111 PING 192.168.1.111 (192.168.1.111) from 192.168.1.9 : 56(84) bytes of data. From server.mack.net (192.168.1.11): Time to live exceeded </verb> The transparent proxy IP on the server will accept telnet connects <verb> client:# telnet 192.168.1.111 Trying 192.168.1.111... Connected to 192.168.1.111. Escape character is '^]'. Welcome to Linux 2.2.16. server login: </verb> but not requests to other services <verb> client:# ftp 192.168.1.111 ftp: connect: No route to host ftp> </verb> Conclusion: The new IP will only accept packets for the specified service. It won't ping and it won't accept packets for other services. (Julian) (an answer from the 2.2.x kernel days) Transparent proxy support calls ip_local_deliver from where the LVS code is reached. One of the advantages of this method is that it is easy for a director and real-server to <ref id="promote" name="exchange roles"> in a failover setup. <verb> ________ | | | client | |________| CIP=192.168.1.254 | (router) | VIP=192.168.1.110 (eth0, arps) __________ | | | director | |__________| DIP=192.168.1.1 (eth1, arps) | | ------------------------------------- | | | RIP1=192.168.1.2 RIP2=192.168.1.3 RIP3=192.168.1.4 (eth0) _____________ _____________ _____________ | | | | | | | real-server | | real-server | | real-server | |_____________| |_____________| |_____________| | | | (router) (router) (router) | | | ----------------------------------------------> to client </verb> Here's a script to run on 2.2.x real-servers/directors to setup Horms' method. This is incorporated into the configure script. <verb> #!/bin/sh #rc.horms #script by Joseph Mack and Horms (C) 1999, released under GPL. #Joseph Mack jmack@wm7d.net, Horms horms@vergenet.net #This code is part of the Linux Virtual Server project #http://www.linuxvirtualserver.org # # #Horm's method for solving the LVS arp problem for a VS-DR LVS. #Uses ipchains to redirect a packet destined for an external #machine (in this case the VIP) to the local device. #----------------------------------------------------- #Instructions: # #1. Director: Setup normally (eg turn on LVS services there with ipvsadm). #2. Real-servers: Must be running 2.2.x kernel. # 2.1 recompile the kernel (and reboot) after turning on the following under "Networking options" # Network firewalls # IP: firewalling # IP: transparent proxy support # IP: masquerading # 2.2 Setup the real-server as if it were a regular leaf node on the network, # ie with the same gateway and IP as if it were in the LVS, but DO NOT # put the VIP on the real-server. The real-server will only have its regular IP # (called the RIP in the HOWTO). #3. Edit "user configurable" stuff below" #4. Run this script #----------------------------------------------------- #user configurable stuff IPCHAINS="/sbin/ipchains" VIP="192.168.1.110" #services can be represented by their name (in /etc/services) or a number #SERVICES is a quote list of space separated strings # eg SERVICES="telnet" # SERVICES="telnet 80" # SERVICES="telnet http" #Since the service is redirected to the local device, #make sure you have SERVICE listening on 127.0.0.1 # SERVICES="telnet http" # #---------------------------------------------------- #main: #turn on IP forwarding (off by default in 2.2.x kernels) echo "1" > /proc/sys/net/ipv4/ip_forward #flush ipchains table $IPCHAINS -F input #install SERVICES for SERVICE in $SERVICES do { echo "redirecting ${VIP}:${SERVICE} to local:${SERVICE}" $IPCHAINS -A input -j REDIRECT $SERVICE -d $VIP $SERVICE -p tcp } done #list ipchain rules $IPCHAINS -L input #rc.horms---------------------------------------------- </verb> Here's the conf file for a VS-DR LVS using TP on both the director and the real-servers. This is for a 2.2.x kernel director. (For a 2.4.x director, the VIP device can't be TP - TP doesn't work on a 2.4.x director). <verb> #------------------------------------- #lvs_dr.conf for TP on director and real-server #you will have to add a host route or equivelent on the client/router #so that packets for the VIP are routed to the director LVS_TYPE=VS_DR INITIAL_STATE=on #note director VIP device is TP VIP=TP lvs 255.255.255.255 lvs DIP=eth0 dip 192.168.1.0 255.255.255.0 192.168.1.255 DIRECTOR_DEFAULT_GW=client SERVICE=t telnet rr real-server1 real-server2 #note real-server VIP device is TP SERVER_VIP_DEVICE=TP SERVER_NET_DEVICE=eth0 SERVER_DEFAULT_GW=client #----------end lvs_dr.conf------------------------------------ </verb> Here's the output from ipchains -L showing the redirects for just the 2.2.x director <verb> Chain input (policy ACCEPT): target prot opt source destination ports REDIRECT tcp ------ anywhere lvs2.mack.net any -> telnet => telnet REDIRECT tcp ------ anywhere lvs2.mack.net any -> telnet => telnet REDIRECT tcp ------ anywhere lvs2.mack.net any -> www => www REDIRECT tcp ------ anywhere lvs2.mack.net any -> www => www Chain forward (policy ACCEPT): Chain output (policy ACCEPT): </verb> <sect1>Transparent proxy for 2.4.x Transparent proxy for 2.2.x kernels is installed with ipchains. For 2.4.x kernels transparent proxy is built on netfilter and is installed with iptables (you need iptables support in the kernel and the ip_tables module must be loaded). The ip_tables module is incompatible with the ipchains module (which in 2.4.x is available for compatibility with scripts written for 2.2.x kernels). If present, the ipchains module must be unloaded. The command for installing transparent proxy with iptables for 2.4.x came from looking in Daniel Kiracofe's <tt/drk@unxsoft.com/ <url url="http://www.unxsoft.com/TransparentProxy.html" name="Transparent Proxy with Squid mini-HOWTO"> and guessing the likely command. It turns out to be <verb> iptables -t nat -A PREROUTING [-i $SERVER_NET_DEVICE] -d $VIP -p tcp --dport $SERVICE -j REDIRECT </verb> (where $SERVICE = telnet, $SERVER_NET_DEVICE = eth0). Here's the result of installing the VIP by transparent proxy on one of the real-servers. <verb> realserver:~# iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination REDIRECT tcp -- anywhere lvs2.mack.net tcp dpt:telnet REDIRECT tcp -- anywhere lvs2.mack.net tcp dpt:http Chain POSTROUTING (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination </verb> This works fine for the real-server allowing it to accept packets for the VIP, without having the VIP on an ethernet device (eg lo, eth0). If you do the same on the director, setup for an LVS with (say) telnet forwarded in the ipvsadm tables, then the telnet connect request from the client is accepted by the director, rather than forwarded by ipvs to the real-servers (tcpdump sees a normal telnet login to the director). Apparently ipchains is sending the packets to a place that ipvs can't get at them. (Joe) <verb> > I have got TP to work on a VS-DR telnet 2.4 real-server with the command > > #iptables -t nat -A PREROUTING -p tcp -d $VIP --dport telnet -j REDIRECT > > When I put the VIP onto the director this way, the LVS doesn't work. > I connect to the director instead of the real-servers. > ipvsadm doesn't show any connections (active or otherwise) > > If I run the same command on the director, with ipvsadm blank > (ie no LVS configured), then I connect to the director from the > client (as expected) getting the director's telnet login. > > I presume that I'm coming in at the wrong place in the input chain > of the director and ipvsadm is not seeing the packets? </verb> (Julian) I haven't tried tproxy in 2.4 but in theory it can't work. The problem is that netfilter implements tproxy by mangling the destination address in the prerouting. LVS requires from the tproxy implementation only to deliver the packet locally and not to alter the header. So, I assume LVS detects the packets with daddr=local_addr and refuses to work. Netfilter maintains a sockopt SO_ORIGINAL_DST that can be used from the user processes to obtain the original dest addr/port before they are mangled in the pre routing nat place. This can be used from the squids, for example, to obtain these original values. If LVS wants to support this broken tproxy in netfilter we must make a lookup in netfilter to receive the original dst and then again to mangle (for 2nd time) the dst addr/port. IMO, this is very bad and requires LVS always to require netfilter nat because it will always depend on netfilter: LVS will be compiled to call netfilter functions from its modules. So, the only alternative remains to receive packets with advanced routing with fwmark rules. There is one problem in 2.2 and 2.4 when the tproxy setups must return ICMP to the clients (they are internal in such setup), for example, when there is no real server LVS returns ICMP DEST_UNREACH:PORT_UNREACH. In this case both kernels mute and don't return the ICMP. icmp_send() drops it. I contacted Alexey Kuznetsov, the net maintainer, but he claims there are more such places that must be fixed and "ip route add table 100 local 0/0 dev lo" is not a good command to use. But in my tests I don't have any problems, only the problem with dropped ICMP replies from the director. So, for TP, I'm not sure if we can support it in the director. May be it can work for the real servers and even when the packet is mangled I don't expect peformance problems but who knows. <sect1>Transparent proxy Q&A<label id="TP_QandA"> Q. How does a packet get to the host which is accepting packets for the VIP if the host doesn't have the VIP on an eth device to respond to arp requests? A. You have to send it there. On the real-servers in a VS-DR LVS this is already handled for you. In VS-DR when the director wants to send a packet to a real-server, it looks up the MAC address for the RIP and sends the packet with addresses CIP->VIP by linklayer to the MAC address of the real-server. Once the packet arrives at the real-server, processes listening to VIP:port pick up the packet. As pointed out by Julian, you can put a transparent proxy IP onto any host, so you can use transparent proxy to put the VIP onto the director if you like (for 2.2.x kernels). To receive the packet for the VIP, the director has to be the default gw for the network in the incoming router's table (or the router needs a host route for the VIP to the director). The same problem exists when <ref id="routing_fwmark" name="accepting packets with fwmark">. Q. Some demons listen only to specific IPs. What IP is the telnet/httpd listening to when it accepts a connection by transparent proxy? A. It depends where you are when you make the connect request (this is for 2.2.x kernels). example: You are on the console of a host and add x.x.x.111:http by transparent proxy and setup the httpd to listen to x.x.x.111:80. You cannot ping x.x.x.111. To connect to x.x.x.111:http you need to # route add -host 192.168.1.111 lo (adding a route to eth0 does not work). If you go to an outside machine, you still cannot ping x.x.x.111 and you cannot connect to x.x.x.111:http unless you make the target box the default gw or add a host route to x.x.x.111. If you now go back to the console of the transparent proxy machine and change the httpd to listen to 127.0.0.1:http (and not to x.x.x.111:http) you can still connect to x.x.x.111:http even though nothing is listening to that IP:port (linux tcpip does local delivery to local IPs). (You can also connect to 127.0.0.1:http, but this is not concerned with transparent proxy.) Returning to the outside machine, you cannot connect to x.x.x.111:http. The connections from the outside machine model connections to the director with the VIP by transparent proxy, while the connections from the console model the real-server which has a packet delivered from the director. On the real-server you could have your services listening to 127.0.0.1 rather than the VIP. You may run into DNS problems (see <ref id="indexing" name="Running indexing programs">) if the process listening to 127.0.0.1 doesn't know that it's answering to lvs.domain.org. <sect1>Identd doesn't delay connection with when packets are received by TP on 2.4.x real-servers<label id="identd_with_2.4_TP"> Here's the data showing that TP behaves differently for 2.2 and 2.4 kernels. If you want to <ref id="2.4_TP_IP" name="skip ahead">, the piece of information you need is that the IP of the packet when it arrives on the target machine by TP, is different for 2.2 and 2.4 TP. As we shall see, for 2.2.x the TP'ed packets arrive on the VIP, while for 2.4.x, the TP'ed packets arrive on the RIP. <sect2>Real-server, Linux 2.4.2 kernel accepting packets for VIP on lo:110, is delayed Here's the tcpdump on the real-server (bashfull) for a telnet request delayed by authd (the normal result for LVS). Real-server 2.4.2 with Julian's hidden patch, director 0.2.5-2.4.1. The VIP on the real-server is on lo:110. Note: all packets on the real-server are originating and arriving on the VIP (lvs2) as expected for a VS-DR LVS. <verb> initial telnet request 21:04:46.602568 client2.1174 > lvs2.telnet: S 461063207:461063207(0) win 32120 <mss 1460,sackOK,timestamp 17832675[|tcp]> (DF) [tos 0x10] 21:04:46.611841 lvs2.telnet > client2.1174: S 3724125196:3724125196(0) ack 461063208 win 5792 <mss 1460,sackOK,timestamp 514409[|tcp]> (DF) 21:04:46.612272 client2.1174 > lvs2.telnet: . ack 1 win 32120 <nop,nop,timestamp 17832676 514409> (DF) [tos 0x10] 21:04:46.613965 client2.1174 > lvs2.telnet: P 1:28(27) ack 1 win 32120 <nop,nop,timestamp 17832676 514409> (DF) [tos 0x10] 21:04:46.614225 lvs2.telnet > client2.1174: . ack 28 win 5792 <nop,nop,timestamp 514409 17832676> (DF) real-server makes authd request to client 21:04:46.651500 lvs2.1061 > client2.auth: S 3738365114:3738365114(0) win 5840 <mss 1460,sackOK,timestamp 514413[|tcp]> (DF) 21:04:49.651162 lvs2.1061 > client2.auth: S 3738365114:3738365114(0) win 5840 <mss 1460,sackOK,timestamp 514713[|tcp]> (DF) 21:04:55.651924 lvs2.1061 > client2.auth: S 3738365114:3738365114(0) win 5840 <mss 1460,sackOK,timestamp 515313[|tcp]> (DF) after delay of 10secs, telnet request continues 21:04:56.687334 lvs2.telnet > client2.1174: P 1:13(12) ack 28 win 5792 <nop,nop,timestamp 515416 17832676> (DF) 21:04:56.687796 client2.1174 > lvs2.telnet: . ack 13 win 32120 <nop,nop,timestamp 17833684 515416> (DF) [tos 0x10] </verb> <sect2>Real-server, Linux 2.4.2, accepting packets for VIP by TP, is not delayed Here's the tcpdump on the real-server (bashfull) for a telnet request which connects immediately. This is not the normal result for LVS. Real-server 2.4.2 with Julian's hidden patch (not used), director 0.2.5-2.4.1. Packets on the VIP are being accepted by TP rather than on lo:0 (the only difference). Note: some packets on the real-server (bashfull) are arriving and originating on the VIP (lvs2) and some on the RIP (bashfull). In particular all telnet packets from the CIP are arriving on the RIP, while all telnet packets from the real-server are originating on the VIP. For authd, all packets to and from the real-server are using the RIP. <verb> initial telnet request 20:56:43.638602 client2.1169 > bashfull.telnet: S 4245054245:4245054245(0) win 32120 <mss 1460,sackOK,timestamp 17784379[|tcp]> (DF) [tos 0x10] 20:56:43.639209 lvs2.telnet > client2.1169: S 3234171121:3234171121(0) ack 4245054246 win 5792 <mss 1460,sackOK,timestamp 466118[|tcp]> (DF) 20:56:43.639654 client2.1169 > bashfull.telnet: . ack 3234171122 win 32120 <nop,nop,timestamp 17784380 466118> (DF) [tos 0x10] 20:56:43.641370 client2.1169 > bashfull.telnet: P 0:27(27) ack 1 win 32120 <nop,nop,timestamp 17784380 466118> (DF) [tos 0x10] 20:56:43.641740 lvs2.telnet > client2.1169: . ack 28 win 5792 <nop,nop,timestamp 466118 17784380> (DF) real-server makes authd request to client 20:56:43.690523 bashfull.1057 > client2.auth: S 3231319041:3231319041(0) win 5840 <mss 1460,sackOK,timestamp 466123[|tcp]> (DF) 20:56:43.690785 client2.auth > bashfull.1057: S 4243940839:4243940839(0) ack 3231319042 win 32120 <mss 1460,sackOK,timestamp 17784385[|tcp]> (DF) 20:56:43.691125 bashfull.1057 > client2.auth: . ack 1 win 5840 <nop,nop,timestamp 466123 17784385> (DF) 20:56:43.692638 bashfull.1057 > client2.auth: P 1:10(9) ack 1 win 5840 <nop,nop,timestamp 466123 17784385> (DF) 20:56:43.692904 client2.auth > bashfull.1057: . ack 10 win 32120 <nop,nop,timestamp 17784385 466123> (DF) 20:56:43.797085 client2.auth > bashfull.1057: P 1:30(29) ack 10 win 32120 <nop,nop,timestamp 17784395 466123> (DF) 20:56:43.797453 client2.auth > bashfull.1057: F 30:30(0) ack 10 win 32120 <nop,nop,timestamp 17784395 466123> (DF) 20:56:43.798336 bashfull.1057 > client2.auth: . ack 30 win 5840 <nop,nop,timestamp 466134 17784395> (DF) 20:56:43.799519 bashfull.1057 > client2.auth: F 10:10(0) ack 31 win 5840 <nop,nop,timestamp 466134 17784395> (DF) 20:56:43.799738 client2.auth > bashfull.1057: . ack 11 win 32120 <nop,nop,timestamp 17784396 466134> (DF) telnet connect continues, no delay 20:56:43.835153 lvs2.telnet > client2.1169: P 1:13(12) ack 28 win 5792 <nop,nop,timestamp 466137 17784380> (DF) 20:56:43.835587 client2.1169 > bashfull.telnet: . ack 13 win 32120 <nop,nop,timestamp 17784399 466137> (DF) [tos 0x10] </verb> Evidently TP on the real-server is making the real-server think that the packets arrived on the RIP, hence the authd call is made from the RIP. As it happens in my test setup, the client can connect directly to the RIP. (In a VS-DR LVS, the client doesn't exchange packets with the RIP, so I haven't blocked this connection. In production, the router would not allow these packets to pass). Since the authd packets are between the RIP and CIP, the authd exchange can proceed to completion. <sect2>Real-server, Linux 2.2.14, accepting packets for VIP by TP, is delayed Here's the tcpdump on the real-server (bashfull) for a telnet request which connects immediately. This is not the normal result for LVS. Real-server 2.2.14, director 0.2.5-2.4.1. Packets on the VIP are being accepted by TP rather than on lo:0. Note: TP is different in 2.2 and 2.4 kernels. Unlike the case for the 2.4.2 real-server, the packets all arrive at the RIP. <verb> initial telnet request 22:16:23.407607 client2.1177 > lvs2.telnet: S 707028448:707028448(0) win 32120 <mss 1460,sackOK,timestamp 18262396[|tcp]> (DF) [tos 0x10] 22:16:23.407955 lvs2.telnet > client2.1177: S 3961823491:3961823491(0) ack 707028449 win 32120 <mss 1460,sackOK,timestamp 21648[|tcp]> (DF) 22:16:23.408385 client2.1177 > lvs2.telnet: . ack 1 win 32120 <nop,nop,timestamp 18262396 21648> (DF) [tos 0x10] 22:16:23.410096 client2.1177 > lvs2.telnet: P 1:28(27) ack 1 win 32120 <nop,nop,timestamp 18262396 21648> (DF) [tos 0x10] 22:16:23.410343 lvs2.telnet > client2.1177: . ack 28 win 32120 <nop,nop,timestamp 21648 18262396> (DF) authd request from real-server 22:16:23.446286 lvs2.1028 > client2.auth: S 3966896438:3966896438(0) win 32120 <mss 1460,sackOK,timestamp 21652[|tcp]> (DF) 22:16:26.445701 lvs2.1028 > client2.auth: S 3966896438:3966896438(0) win 32120 <mss 1460,sackOK,timestamp 21952[|tcp]> (DF) 22:16:32.446212 lvs2.1028 > client2.auth: S 3966896438:3966896438(0) win 32120 <mss 1460,sackOK,timestamp 22552[|tcp]> (DF) after delay of 10secs, telnet proceeds 22:16:33.481936 lvs2.telnet > client2.1177: P 1:13(12) ack 28 win 32120 <nop,nop,timestamp 22655 18262396> (DF) 22:16:33.482414 client2.1177 > lvs2.telnet: . ack 13 win 32120 <nop,nop,timestamp 18263404 22655> (DF) [tos 0x10] </verb> <sect1>What IP TP packets arriving on?<label id="2.4_TP_IP"> Note: for TP, there is no VIP on the real-servers as seen by ifconfig. Since telnetd on the real-servers listens on 0.0.0.0, we can't tell which IP the packets have on the real-server after being TP'ed. tcpdump only tells you the src_addr after the packets have left the sending host. Here's the setup for the test. The IP of the packets after arriving by TP was tested by varying the IP (localhost, RIP or VIP) that the httpd listens to on the real-servers. At the same time the base address of the web page was changed to be the same as the IP that the httpd was listening to. The nodes on each network link can route to and ping each other (eg 192.168.1.254 and 192.168.1.12). <verb> ____________ | |192.168.1.254 (eth1) | client |---------------------- |____________| | CIP=192.168.2.254 (eth0) | | | | | VIP=192.168.2.110 (eth0) | ____________ | | | | | director | | |____________| | DIP=192.168.1.9 (eth1, arps) | | | (switch)------------------------ | RIP=192.168.1.12 (eth0) VIP=192.168.2.110 (VS-DR, lo:0, hidden) _____________ | | | real-server | |_____________| </verb> The results (VS-DR LVS) are For 2.2.x realservers <itemize> <item>the httpd can bind to the VIP, RIP and localhost. <item>LVS client gets webpage if real-server is listening to RIP or VIP. <item>LVS client does not get webpage if real-server is listening to localhost. </itemize> For 2.4.x realservers <itemize> <item>httpd can bind to the RIP and localhost. <item>httpd cannot bind to the VIP. <item>LVS client gets webpage if real-server is listening to RIP. <item>LVS client does not get webpage if real-server is listening to localhost. </itemize> During tests, the browser says "connecting to VIP", then says "transferring from..." <itemize> <item>VS-DR, VIP on TP, kernel 2.4.2, "transferring data from RIP" <item>VS-DR, VIP on TP, kernel 2.2.14, "transferring data from VIP" (or RIP) <item>VS-DR VIP on lo:0, httpd listening to VIP, "transferring data from VIP" <item>VS-Tun VIP on tunl0:0, httpd listening on VIP, "transferring from VIP" <item>VS-NAT, httpd listening on RIP, "transferring data from realserver1" (or realserver2) </itemize> Some of these connections are problematic. The client in a VS-DR LVS isn't supposed to be getting packets from the RIP. What is happening is <itemize> <item>the httpd on the real-server is listening on the RIP <item>the base address of the webpage is the RIP <item>an incoming request from the client to the VIP will retrieve a webpage with references to gif etc that are at the RIP <item>the client will then ask for the gifs from the RIP. <item>in the above setup that I use for testing, the client does not request packets from the RIP. <item>in the above setup, the client can connect to the RIP directly (this will not be allowed in a production server, either the router will prevent the connection, or the RIP will be a non-routable IP). <item>the client retrieves the gifs and the rest of the page directly from the real-server </itemize> The way to prevent this is to remove the route on the client to the RIP network (eg see <ref id="Pearthree" name="removing routes not needed for VS-DR">). Doing so when the httpd is listening to the RIP and the base address is the RIP causes the browser on the client to hang. This shows that the client is really retrieving packets directly from the RIP. Changing the base address of the webpage back to the VIP allows the webpage to be delivered to the client, showing that the client is now retrieving packets by making requests to the VIP via the director. It would seem then that with 2.4 TP, the real-server is receiving packets on the RIP, rather than the VIP as it does with 2.2 TP. With a service listening to only 1 port (eg httpd) then the httpd has to <itemize> <item>listen on the RIP <item>the addresses on the webpage have to be for the VIP </itemize> The client will then ask for the webpage at the VIP. The real-server will accept this request on the RIP and return a webpage full of references to the VIP (eg gifs). The client will then ask for the gifs from the VIP. The real-server will accept the requests on the RIP and return the gifs. <sect1>Take home lesson for setting up TP on real-servers <sect2>2.2.x Let httpd listen on VIP or RIP, return pages with references to VIP <sect2>2.4.x Let httpd listen on RIP, return pages with references to VIP. <sect1>Handling identd requests from 2.4.x VS-DR real-servers using TP<label id="identd_with_TP_real-servers"> Since the identd request is coming from the RIP (rather than the VIP) on the real-server, you can use Julian's <ref id="NAT_with_VS-DR" name="method for NAT'ing client requests from real-servers">. <sect1>Performance of Transparent Proxy Using transparent proxy instead of a regular ethernet device has slightly higher latency, but the same maximum throughput. For performance of transparent proxy compared to accepting packets on an ethernet device see the <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance page">. Transparent proxy requires reprocessing of incoming packets, and could have a similar speed penalty as VS-NAT. However only the incoming packets are reprocessed. Initial results (before the performance tests above) were initially not encouraging. From: Doug Bagley <tt/doug@deja.com/ <verb> > Subject: [lvs-users] chosen arp problem solution can apparently affect performance > > I was interested in seeing if the linux/ipchains workaround for the > arp problem would perform just as well as the arp_invisible kernel > patch. It is apparently much worse. > > I ran a test with one client running ab ("apache benchmark"), one > director, and one real-server running Apache. They are all various > levels of pentium desktop machines running 2.2.13. > > Using the arp_invisible patch/dummy0 interface, I get 226 HTTP > requests/second. Using the ipchains redirect method, I get 70 > requests per second. All other things remained the same during the > test. </verb> See the <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance page"> for discussion and sample graphs of hits/sec for http servers. Hits/sec can increase to high levels as the payload decreases in size. While large numbers for hits/sec may be impressive, they only indicate one aspect of a web server's performance. If large (> 1 packet) files are transferred/hit or computation is involved, then hits/sec is not a useful measure of web performance. Here's the current explanation for decreased latency of transparent proxy. Kyle Sparger <tt/ksparger@dialtoneinternet.net/ <verb> > Logically, it's just a function of the way the redirect code operates. > > Without redirect: > Ethernet -> TCP/IP -> Application -> TCP/IP -> Ethernet > > With redirect: > Ethernet -> TCP/IP -> Firewall/Redirect Code -> TCP/IP -> Application -> > TCP/IP -> Ethernet > > That would definitely explain the slowdown, since _every single packet_ > received is going to go through these extra steps. </verb> Other people are happy with TP (Jerry Glomph Black <tt/black@real.com/) The revival of Horms' posting, which I overlooked a month ago, was a lifesaver for us. We had a monster load distribution problem, and spread 4 virtual IP numbers across 10 'real' boxes (running Roxen, a fantastic web platform). The ipchains-REDIRECT feature works perfectly, without any of that arp aggravation! A PII_450 held up just fine at 20 megabits/s of HTTP -REQUEST- TRAFFIC! <sect>Authd/Identd <label id="authd"> If initial connection to your service (telnet, ftp, sendmail...) is delayed by 10secs..5mins, but after you connect everything is fine, then you have problems with identd. Since problems with identd affect many aspects of an LVS, there are references to identd in several places in this document. I haven't worked out a way of collecting them all together yet, so best search with your editor for authd/identd. There are two parts to identd on your servers <itemize> <item>Identd runs on your real-servers. You don't care about this. Identd is for clients on your real-servers connecting to services on remote machines. You aren't using this when setting up an LVS. (However if you telnet from your real-servers for some other reason, you'll need to think about what identd is doing.) <item>your LVS'ed services may ask the identd client on your server to connect to the identd on the client machine and ask for the identity of the person connecting to the service on the real-server. You don't want this. In general there is no way for the reply to return to the real-server in an LVS. </itemize> The problem is in the second part, i.e. if the LVS'ed service on the real-server asks for the ident client to connect to the identd on the client. (This may be a little confusing, machines can be clients and servers at the same time.) Here's a example telnet connection through a director to a real-server where telnetd is running inside tcpwrappers. tcpwrappers uses the ident client on the remote host (the one with the telnetd) to connect to the identd on the local (telnet client) host. <verb> client:/director/usr/src/arch# telnet lvs2 Trying 192.168.2.110... Connected to lvs2.mack.net. Escape character is '^]'. (delay) Welcome to Linux 2.2.19. bashfull login: (successful login) </verb> <sect1>What is authd/identd? identd is a demon run under inetd. Other services running on the server can use identd to ask the client machine for the identity of the user making the request. When a request arrives at a server for such a service (e.g. telnet, sendmail), the auth client will connect from a high port to client:auth asking "who is the owner of the process requesting this service". If the client's authd replies with a <tt/username@nodename/, the reply will be optionally logged on the server (eg to syslog) and the connection request will be handed back to telnetd (or whichever service). If the reply is "<tt/root@nodename/", or some null reply, or there is no authd on the client, then the server's authd will wait till a timeout before allowing connection. The delay is about 10secs for Slackware and 2mins for RedHat7.0. There is no checking of the validity of the reply and since the reply is under control of the client machine, the reply <tt/username@nodename/ could be bogus. The authd is a security feature. However it doesn't get the server very much (you don't know who has made the connect request, only what they told you), while clients that fail are delayed. This may only be a nuisance for people telnet'ing in (provided they understand what's happening), but will bring mail delivery to a crawl. If you setup an LVS with real-servers that have services running inside identd, you will have to deal with identd. Any service in inetd running under tcpwrappers (probably just about every service, if tcpwrappers is installed) and sendmail (see section on sendmail) use it. <sect1>comp.os.linux.security FAQ on identd <verb> comp.os.linux.security FAQ Daniel Swan <tt/swan_daniel@hotmail.com/ v0.1 - Last updated: April 20, 2000 </verb> 4.5) What is Identd? Can I disable it? Identd identifies the username of a process owning a specific TCP/IP connection. It is usually run via inetd and listens on port 113. Identd should not be used as a method of authentication - anyone with root access can alter their identd response. Indeed, on many systems (such as FreeBSD and Windows) even a non-privledged user can specify whatever identd response they want. The protocol is most useful on multiuser systems as a method of tracking down problem users. If one of your users is causing problems on another system, that system's admin can inform you of the username of the specific user causing problems, saving you a lot of legwork. Should you run identd? That's really a judgement call. On systems with many users, the benefits could be great, but it doesn't serve any particular purpose on a single user box. Not running identd may limit your ability to connect to certain servers - many IRC and some FTP servers don't allow, or severly restrict, non-identd'd connections, for example. However, running it means leaving a service open to the outside world, with all the security risks that entails. Another thing to consider is that identd can allow attackers to find out valuable information about your system, such as whether a certain service is running as root, the operating system you are running, and the usernames of your users. Consider running identd with the -n flag, which sends userid numbers instead of usernames. See the identd manpage and /etc/identd.conf for more information about the available options. You can block access to identd by shutting it off entirely (usually done via inetd, see section on disabling services), or by using tcpwrappers and/or firewalling software to disable/restrict access. If you need identd enabled in order to connect to a certain server, you might want to consider allowing access to it only from that server. If you do choose to firewall the identd port, strongly consider using a reject policy rather than deny. Using deny may greatly increase the time it takes you to connect to servers that utilize identd, as they will wait for a response of some type before allowing you to connect. <sect1>Why identd is a problem for LVS The problem is that the identd/authd client makes a callback from the VIP (for VS-DR, VS-Tun) or the RIP (for VS-NAT) and the LVS doesn't know about it. For the simple case where clients call from the RIP see the section on <ref id="clients_on_real-servers" name="running clients on real-servers">. There the client is independant of the LVS. There you will see solutions when there are easy solutions. The case of clients on the real-servers making call backs triggered by an LVS client's requests to an LVS'ed service is more difficult as the result has to get back to the LVS'ed service. Normally in an LVS, the director in an LVS responds to connect requests by handing them to an arbitrary real-server. The corrollary of this is that replies to a client request initiated on a real-server, to the outside world, will not return to the real-server unless something is done to handle it. (The only solutions we have are those in the section on <ref id="clients_on_real-servers" name="running clients on real-servers">.) <itemize> <item>replies from the LVS client machine, arriving at the director are not connect requests, and will not belong to an established connection. They will be dropped. <item>even if the director could forward these replies to a real-server, they could go to any real-server, and not neccessarily to the real-server which originated the request. </itemize> The result is that the client request will hang or timeout. <sect1>tcpdumps of connections delayed by identd Here's the tcpdump of the client telnet'ing to a VS-DR LVS. Telnet on the real-server is running inside tcpwrappers, client and real-servers cannot connect directly i.e. they have no routing to each other. seen from client: <verb> telnet connect request 12:56:05.427252 client2.1038 > lvs.telnet: S 1170880662:1170880662(0) win 32120 <mss 1460,sackOK,timestamp 6539901[|tcp]> (DF) [tos 0x10] 12:56:05.427949 client2.1038 > lvs.telnet: . ack 416490630 win 32120 <nop,nop,timestamp 6539901 161874539> (DF) [tos 0x10] 12:56:05.431752 client2.1038 > lvs.telnet: P 0:27(27) ack 1 win 32120 <nop,nop,timestamp 6539902 161874539> (DF) [tos 0x10] client replying to real-server's auth request 12:56:05.465152 client2.auth > lvs.1377: S 1159930752:1159930752(0) ack 417813448 win 32120 <mss 1460,sackOK,timestamp 6539905[|tcp]> (DF) 12:56:05.465405 lvs.1377 > client2.auth: R 417813448:417813448(0) win 0 12:56:08.464671 client2.auth > lvs.1377: S 1162930275:1162930275(0) ack 417813448 win 32120 <mss 1460,sackOK,timestamp 6540205[|tcp]> (DF) 12:56:08.464901 lvs.1377 > client2.auth: R 417813448:417813448(0) win 0 6 second delay then trying again 12:56:14.466048 client2.auth > lvs.1377: S 1168931649:1168931649(0) ack 417813448 win 32120 <mss 1460,sackOK,timestamp 6540805[|tcp]> (DF) 12:56:14.466275 lvs.1377 > client2.auth: R 417813448:417813448(0) win 0 client login to LVS 12:56:15.501272 client2.1038 > lvs.telnet: . ack 13 win 32120 <nop,nop,timestamp 6540908 161875546> (DF) [tos 0x10] 12:56:15.503946 client2.1038 > lvs.telnet: P 27:125(98) ack 52 win 32120 <nop,nop,timestamp 6540909 161875546> (DF) [tos 0x10] 12:56:15.509024 client2.1038 > lvs.telnet: P 125:128(3) ack 55 win 32120 <nop,nop,timestamp 6540909 161875547> (DF) [tos 0x10] 12:56:15.538816 client2.1038 > lvs.telnet: P 128:131(3) ack 88 win 32120 <nop,nop,timestamp 6540912 161875550> (DF) [tos 0x10] 12:56:15.551836 client2.1038 > lvs.telnet: . ack 90 win 32120 <nop,nop,timestamp 6540914 161875550> (DF) [tos 0x10] 12:56:15.571837 client2.1038 > lvs.telnet: . ack 106 win 32120 <nop,nop,timestamp 6540916 161875551> (DF) [tos 0x10] </verb> Here's what it looks like on the real-server (this is a different connection from the above sample, so the times are not the same). <verb> real-server receives telnet request on VIP 12:50:58.049909 client2.1040 > lvs.telnet: S 1605709966:1605709966(0) win 32120 <mss 1460,sackOK,timestamp 6580274[|tcp]> (DF) [tos 0x10] 12:50:58.051263 lvs.telnet > client2.1040: S 862075007:862075007(0) ack 1605709967 win 32120 <mss 1460,sackOK,timestamp 161914907[|tcp]> (DF) 12:50:58.051661 client2.1040 > lvs.telnet: . ack 1 win 32120 <nop,nop,timestamp 6580274 161914907> (DF) [tos 0x10] 12:50:58.052819 client2.1040 > lvs.telnet: P 1:28(27) ack 1 win 32120 <nop,nop,timestamp 6580274 161914907> (DF) [tos 0x10] 12:50:58.053036 lvs.telnet > client2.1040: . ack 28 win 32120 <nop,nop,timestamp 161914907 6580274> (DF) real-server initiates auth request from VIP to client:auth 12:50:58.088510 lvs.1379 > client2.auth: S 852509908:852509908(0) win 32120 <mss 1460,sackOK,timestamp 161914911[|tcp]> (DF) 12:51:01.083659 lvs.1379 > client2.auth: S 852509908:852509908(0) win 32120 <mss 1460,sackOK,timestamp 161915211[|tcp]> (DF) real-server waits for timeout (about 8secs), sends final request to client:auth 12:51:07.083164 lvs.1379 > client2.auth: S 852509908:852509908(0) win 32120 <mss 1460,sackOK,timestamp 161915811[|tcp]> (DF) telnet replies from real-server continue, login occurs 12:51:08.117727 lvs.telnet > client2.1040: P 1:13(12) ack 28 win 32120 <nop,nop,timestamp 161915914 6580274> (DF) 12:51:08.118142 client2.1040 > lvs.telnet: . ack 13 win 32120 <nop,nop,timestamp 6581281 161915914> (DF) [tos 0x10] </verb> <sect1>There are solutions to identd problem in some cases <sect2>Director is VS-NAT In an LVS, authd on the real-server will be able to connect to the client if - VS-NAT, the real-servers are on public IPs (not likely, since you usually hide the real-servers from public view and they'll be on 192.168.x.x or 10.x.x.x networks) VS-NAT, and high ports are nat'ed out with a command like <verb> director:/etc/lvs# ipchains -A forward -j MASQ -s 192.168.1.0/24 -d 0.0.0.0/0 </verb> You usually don't want to blanket masquerade all ports. You really only want to masquerade ports that are being LVS'ed (so you can still get to the other services) in which case, for each service being LVS'ed, you to use ipchains rules like <verb> director:# ipchains -A forward -p tcp -j MASQ -s realserver1 telnet -d 0.0.0.0/0 </verb> Since the auth client (on your telnet server) is connecting from a high port on the server, a better ipchains rule which will allow auth to work when the real-servers are on private IPs. <verb> director:# ipchains -A forward -p tcp -j MASQ -s realserver1 1024:65535 -d 0.0.0.0/0 </verb> <sect2>VS-DR, VS-Tun, 2.2.x kernel directors There is no solution for VS-DR for 2.2.x directors. The auth client on the real-server initiates the connection from the VIP. There is no way for a packet from VIP:high port to get a reply through the LVS because FIXME <itemize> <item> the incoming packet from the client on the internet is destined for a non-LVS'ed high port <item> the incoming packet is not a connect request. <item> the incoming packet is not associated with an established connection. </itemize> The reply from the LVS client will be dropped. <sect2>VS-DR, VS-Tun, 2.4.x kernel directors Transparent proxy in 2.4 is different to 2.2 (see section on <ref id="identd_with_2.4_TP" name="identd with 2.4 TP" >). You should be able to <ref id="identd_with_TP_real-servers" name="masquerade the identd client's request on the real-server">. <sect1>Turn off tcpwrappers inetd.conf will have a line like <verb> telnet stream tcp nowait root /usr/sbin/tcpd in.telnetd </verb> change this to <verb> telnet stream tcp nowait root /usr/sbin/in.telnetd in.telnetd </verb> and re-HUP inetd. <sect1>Identd and smtp/pop/qmail<label id="identd_and_sendmail"> (This is older stuff from the mailing list when the problem first came up) Problem: In the case of identd, the smtpd on a real-server says to identd "give me the name of the owner of the process on IP:port that is asking me to accept mail". If identd thinks it is running on the RIP rather than the VIP and, as is most likely, RIP is not routable from the outside world, mail on the real-server will hang. If identd is running on the VIP, then replies will probably return to another real-server and mail will still hang. The converse case, of sending mail from the LVS, has the smtp server out in internetland asking the LVS for the name of the owner of the process running on the VIP sending him mail. If identd is clustered, then the request will in all probability go to another real-server. This seems equally intractable at the moment. Originally the problem was raised by <verb> From: Chris Kennedy <tt/ckennedy@iland.net/ Subject: SMTP, POP3 using Qmail and Ident, also using Solaris as real servers </verb> I have setup a virtual server using the Linux 2.2 patch and 3 Sun Ultras as the actual servers. It has crashed twice, though possibly from running bind on the Virtual Server, since it was right when I started it up (bind) that the virtual server would crash. The major problem I am having is a timeout for Ident requests on POP3 and SMTP ports which seem to be confused. When looking at the problem with tcpdump on the virtual server and the real servers the vserver seems to do the following: <verb> 13:41:48.635985 10.0.0.1.4658 > vserver.net.smtp: . ack 2764990963 win 8760 13:41:48.636030 10.0.0.1.4658 > vserver.net.smtp: . ack 1 win 8760 13:41:48.658875 10.0.0.1.auth > vserver.net.48981: R 0:0(0) ack 2765099549 win 0 <<<<<< 13:41:52.143790 10.0.0.1.auth > vserver.net.48981: R 0:0(0) ack 1 win 0 <<<<<< 13:41:58.144210 10.0.0.1.auth > vserver.net.48981: R 0:0(0) ack 1 win 0 <<<<<< </verb> The Ident, or auth port on the client machine trying to connect back to the vserver is where it will pause for about 10-15 seconds then connect just fine. I believe this may be qmail specific since a server funning sendmail will not have this problem and ident seems to be used by qmail more than it or something. No-one answered, then months later... <verb> (Ted) > I currently have HTTP loadbalanced just fine with the LinuxDirector. > I've setup SMTP in the same fashion, and I don't have as much luck. </verb> (Lars) Check if your system (tcpwrapper or sendmail) is doing a NS lookup before accepting the connection or trying to connect to ident. <verb> From: Chris Kennedy <tt/ckennedy@iland.net/ Subject: Re: SMTP -- very slow connection </verb> I had the same problem with the Direct Routing and SMTP and POP3. It looked like a problem with the Ident lookup to the server by the client, it was what always was occurring during that time out period. I saw this while doing tcpdumps on the virtual server where the client would just keep asking for Ident lookups to the Virtual IP address which are from the client port 113 to a random port above 1023 on the virtual server. I can see how this is tricky with the direct routing method since this traffic should be sent on to the real server but is not. I sort of gave up on Direct Routing for now since this looks pretty hard to fix if it really is Ident and the client requirements getting in the way. <verb> (Ted) > > I'm connecting directly to the IP. But just to be sure, I'll add an entry to > the nameserver for that particular IP -- in both forward and reverse > lookups...........done..... > > And it still does the same thing. :( > > Understand that the thirty seconds are *AFTER* the connection... Telnet > connects, gives me the escape character, and sits. If it was a nameservice > thing, I'd imagine it'd sit before it connected. > > <?> I'd actually be happier if it wasn't connecting. :) Then I'd know there > was definitely something I needed to fix between me and the real machine. > But when it connects and THEN has trouble... I'm lost. :( </verb> From: Michael Baird <tt/mike@tc3net.com/ <verb> > Sound's like an issue with ident lookup's, you probably aren't > clustering IDENT, you can 1) cluster identd, or edit your sendmail.cf > file and set the value > 0 Timeout.ident=0 </verb> (Ted) Great idea. That was it. I turned off ident in sendmail and things worked fine. However, I don't want to turn of ident in sendmail, and I figure other things might want ident too... so I want to cluster ident. Clustering ident didn't help. I clustered tcp port 113 to both servers. (I even tried "loadbalancing" 25 and 113 just to ONE server -- that way it'd always hit the same server)... And that didn't work. I got the same results -- telnet to port 25... connect... thirty seconds... and then sendmail would enter command mode. Any ideas? Do I have to loadbalance anything else besides tcp 113 for identd to work? Would like to add more info here... Why is identd run with smtp (any other reason other than wanting to know who is sending me the mail?) do you have to turn identd off in smtp to get LVS smtp to work? has anyone LVS'ed identd? (I'd imagine you wouldnt neccessarily get the ident from the same machine running the process for which you want the ident) anything else? <sect>Squid Real-Servers (poor man's L7 switch) One of the first uses for LVS was to increase throughput of webcaches. A scheduler (-dh = destination hash) specially <ref id="ipvsadm" name="designed for webcaches"> is in LVS derived from code posted to the LVS mailing list by Thomas Proell (about Oct 2000). This section was written by <tt/andreas.koenig@anima.de/ Andreas J. Koenig and was posted to the mailing list. An often lamented shortcoming of LVS clusters is that the realservers have to be configured to work identically. Thus if you want to build up a service with many servers that need to be configured differently for some reason, you cannot take advantage of the powerful LVS. The following describes an LVS topology where not all servers in the pool of available servers are configured identical and where loadbalancing is content-based. The goal is achieved by combining the features of Squid and LVS. The workhorses are running Apache, but any HTTP server would do. <sect1>Terminology Before we start we need to introduce a bit of Squid terminology. A redirector (http://www.squid-cache.org/Doc/FAQ/FAQ-15.html) is a director that examines the URL and request method of an HTTP and is enabled to change the URL in any way it needs. An accelerator (http://www.squid-cache.org/Doc/FAQ/FAQ-20.html) plays the role os a buffer and cache. The accelerator handles a relatively big amount of slow connections to the clients on the internet with a relativly small amount of memory. It passes requests through to any number of back-end servers. It can be configured to cache the results of the back-end servers according to the HTTP headers. <sect1>Preview In the following example installation we will realize this configuration (real IP addresses anonymized): <verb> VIP 10.0.0.62 LVSdirector 10.0.0.2 LVS_hot_standby 10.0.0.3 Squid1 10.0.0.5:80 Squid2 10.0.0.7:80 # Same box as Webserver2 Webserver1 10.0.0.6:81 Webserver2 10.0.0.7:81 # Same box as Squid2 Webserver3 10.0.0.8:81 Webserver4 10.0.0.9:81 </verb> Note that a squid and a webserver can coexist in a single box, that's why we have put Squid2 and Webserver7 into a single machine. Note also that squids can cache webservers' output and thus reduce the work for them. We dedicate 24 GB disk to caching in Squid1 and 6 GB disk in Squid2. And finally note that several squids can exchange digest information about cached data if they want. We haven't yet configured for this. Strictly speaking, a single squid can take the role of an LVSdirector, but only for HTTP. It's slower, but it works. By accessing one of the squids in our setup directly, this can be easily demonstrated. <sect1>Let's start assembling I'd suggest, the first thing to do is to setup the four apache on Webserver1..4. These servers are the working horses for the whole cluster. They are not what LVS terminology calls realservers though. The realservers according to LVS are the Squids. We configure the apaches completely stardard. The only deviation from a standard installation here is that we specify Port 81 in the httpd.conf. Everything else is the default configuration file that comes with apache. In the choice of the port we are, of course, free to choose any port we like. It's an old habit of mine to select 81 if a squid is around to act as accelerator. We finish this round of assembling with tests that only try to access Webserver1..4 on port 81 directly. For later testing, I recommend to activate the printenv CGI program that comes with Apache: <verb> chmod 755 /usr/local/apache/cgi-bin/printenv </verb> This program shows us, on which server the script is running (SERVER_ADDR) and which server appears as the requesting site (REMOTE_ADDR). <sect1>One squid Next we should configure one Squid box. The second one will mostly be a replication of the first, so let's first nail that first one down. When we compile the squid 2.3-STABLE4, we need already decide about compilation options. Personally I like the features associated with this configuration: <verb> ./configure --enable-heap-replacement --disable-http-violations \ --enable-cache-digests --enable-delay-pools </verb> We can build and install squid with these settings. But before we start squid, we must go through a 2700 lines configuration file and set lots of options. The following is a collection of diffs between the squid.conf.default and my squid.conf with comments in between. <verb> --- squid.conf.default Mon Aug 14 12:04:33 2000 +++ squid.conf Mon Aug 14 14:34:35 2000 @@ -47 +47 @@ -#http_port 3128 +http_port 80 </verb> Yes, we want this squid on port 80 because from outside it looks like a normal HTTP server. <verb> @@ -54 +54 @@ -#icp_port 3130 +icp_port 0 </verb> In the demo installation I turned ICP off, but I'll turn it on again later. ICP is the protocol that the squids can use to exchange sibling information about what they have on their disks. <verb> @@ -373 +373 @@ -#cache_mem 8 MB +cache_mem 700 MB </verb> This is the memory reserved for holding cache data. We have 1 GB total physical memory and 24 GB disk cache. To manage the disk cache, squid needs about 150 MB of memory (estimate 6 MB per GB for an average object size of 13kB). Once you're running, you can use squid's statistics to find out *your* average object size. I usually leave 1/6 of the memory for the operating system, but at least 100 MB. <verb> @@ -389,2 +389,2 @@ -#cache_swap_low 90 -#cache_swap_high 95 +#cache_swap_low 94 +#cache_swap_high 96 @@ -404 +404 @@ -#maximum_object_size 4096 KB +maximum_object_size 8192 KB </verb> Please refer to squid's docs for these values. <verb> @@ -463,0 +464,5 @@ +cache_dir ufs /var/squid01 5600 16 256 +cache_dir ufs /var/squid02 5600 16 256 +cache_dir ufs /var/squid03 5600 16 256 +cache_dir ufs /var/squid04 5600 16 256 + </verb> You do not need bigger disks, you need many disks to speed up squid. Join the squid mailing list to find out about the efficiency of filesystem tuning like "noatime" or Reiser FS. <verb> @@ -660 +665 @@ -#redirect_program none +redirect_program /usr/local/squid/etc/redirector.pl </verb> This is the meat of our usage of squid. This program can be as simple as you want or as powerful as you want. It can be implemented in any language and it will be run within a pool of daemons. My program is written in perl and looks something like the following: <verb> $|=1; while (<>) { chomp; my($url,$host,$ident,$method) = split; my @redir = $url =~ /\bh=([\d,]+);?/ ? split(/,/,$1) : (6,7,8,9); # last components of our IP numbers my $redir = $redir[int rand scalar @redir]; $url =~ s/PLACEHOLDER:81/10.0.0.$redir\:81/i; print STDOUT "$url\n"; } </verb> This is ideal for testing, because it allows me to request a single backend server or a set of backend servers to choose from via the CGI querystring. A request like http://10.0.0.62/cgi-bin/printenv?h=6 will then be served by backend apache 10.0.0.6. <verb> @@ -668 +673 @@ -#redirect_children 5 +redirect_children 10 </verb> The more complex the redirector program is, the more processes should be allocated to run it. <verb> @@ -674 +679 @@ -#redirect_rewrites_host_header on +redirect_rewrites_host_header off @@ -879 +884 @@ -#replacement_policy LFUDA +replacement_policy LFUDA @@ -1168 +1173 @@ -acl Safe_ports port 80 21 443 563 70 210 1025-65535 +acl Safe_ports port 80 81 21 443 563 70 210 1025-65535 @@ -1204 +1209 @@ -http_access deny all +http_access allow all </verb> For all of the above changes, please refer to the squid.conf.default. <verb> @@ -1370,2 +1375,3 @@ -#httpd_accel_host hostname -#httpd_accel_port port +# we will replace www.meta-list.net:81 with our host of choice +httpd_accel_host PLACEHOLDER +httpd_accel_port 81 </verb> As we are redirecting everything through the redirector, we can fill in anything we want. No real hostname, no real port is needed. The redirector program will have to know what we chose here. <verb> @@ -1377 +1383 @@ -#httpd_accel_with_proxy off +httpd_accel_with_proxy on </verb> If we want ICP working (and we said, we would like to get it working), we need this turned on. We're done with our first squid, we can start it and test it. If you send a request to this squid, one of the backend servers will answer according to the redirect policy of the redirector program. Basically, at this point in time we have a fully working content based redirector. As already mentioned, we do not really need LVS to accomplish this. But the downside of this approach is: - we are comparatively slow: squid is not famous for speed. - we do not scale well: if the bottleneck is a the squid, we want LVS to scale up. <sect1>Another squid So the next step in our demo is to build another squid. This is very trivial given that we have already one. We just copy the whole configuration and adjust a few parameters if there are any differences in the hardware. <sect1>Combining pieces with LVS The rest of the story is to read the appropriate docs for LVS. I have used Horms's Ultra Monkey docs and there's nothing to be added for this kind of setup. Keep in mind that only the squids are to be known by the LVS box. They are the "realservers" in LVS terminology. The apache back end servers are only known to the squids' redirector program. <sect1>Problems It has been said that LVS is fast and squid is slow, so people believe, they must implement a level 7 switch in LVS to have it faster. This remains to be proofed. Squid is really slow compared to some of the HTTP servers that are tuned for speed. If you're serving static content with a hernel HTTP daemon, you definitely do not want to lose the speed by running it through a squid. If you want persistent connections, you need to implemented them in your redirector. If you want to take dead servers out of the pool, you must implement it in your redirector. If you have a complicated redirector, you need more of them and thus need more ressources. In the above setup, ldirectord monitors just the two squids. A failure of one of the apaches might go by unnoticed, so you need to do something about this. If you have not many cacheable data like SSL or things that need to expire immediately or a high fraction of POST requests, the squid seems like a waste of resources. I'd say, in that case you just give it less disk space and memory. Sites that prove unviewable through Squid are a real problem (Joe Cooper reports there is a stock ticker that doesn't work through squid). If you have contents that cannot be served through a squid, you're in big trouble--and as it seems, on your own. <sect>Details of LVS operation <sect1>Throughput statistics<label id="statistics"> Counters for connections, bytes and packets is available (in hexadecimal) with <verb> $ cat /proc/net/ip_vs_stats </verb> Julian 22 May 2001 <quote> Conns is a counter and is incremented when a new connection is created. It is not incremented when a client re-uses a port to make a new connection. </quote> Here's ipvsadm <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 0.2.12 (size=16384) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:0 rr persistent 360 -> bashfull.mack.net:0 Route 1 0 0 -> sneezy.mack.net:0 Route 1 0 0 TCP lvs2.mack.net:telnet rr -> bashfull.mack.net:telnet Route 1 0 0 -> sneezy.mack.net:telnet Route 1 0 0 </verb> Here's the statistics <verb> director:# more /proc/net/ip_vs_stats TotalConns InPkts OutPkts InBytes OutBytes 98F9 13945999 13609E49 613AD3B2F 4F90FE6F9E Virtual Service Pro VirtService Conns InPkts OutPkts InBytes OutBytes TCP C0A8026E:0000 4 12 0 00000043B 000000000 TCP C0A8026E:0017 7 3A9 0 00000C3A5 000000000 Real Service Pro VirtService RealService Conns InPkts OutPkts InBytes OutBytes TCP C0A8026E:0000 C0A8010C:0000 4 14 0 0000004B4 000000000 TCP C0A8026E:0000 C0A8010B:0000 1 3 0 0000000B4 000000000 TCP C0A8026E:0017 C0A8010C:0017 4 A2 0 00000232A 000000000 TCP C0A8026E:0017 C0A8010B:0017 4 32A 0 00000A827 000000000 </verb> <sect1>Director Hash Table<label id="Hash_Table"> The director maintains a hash table of connections marked with <verb> <CIP, CPort, VIP, VPort, RIP, RPORT> </verb> where <itemize> <item> CIP: Client IP address <item> CPort: Client Port number <item> VIP: Virtual IP address <item> VPort: Virtual Port number <item> RIP: RealServer IP address <item> RPort: RealServer Port number. </itemize> The hash table speeds up the connection lookup and keeps state so that packets belonging to a connection from the client will be sent to the allocated real-server. Horms <tt/horms@vergenet.net/ said: <verb> > When a connection is recieved by an IPVS server and forwarded > (by whatever means) to a back-end server at what stage is > this connection entered into the IPVS table. It is before or > as the packet is sent to the back-end server or delayed > until after the 3 way handshake is complete. </verb> (Lars) The first packet is when the connection is assigned to a real server, thus it must be entered into the table then, otherwise the 3 way handshake would likely hit 3 different real servers. <verb> > It has been alleged that IBMs Net Director waits until > the completion of the three way handshake to avoid the > table being filled up in the case of a SYN flood. To > my mind the existing SYN flood protection in Linux should > protect the IPVS table in any case and the connection > needs to be in the IPVS table to enable the 3 way handshake > to be completed. </verb> (Wensong) There is state management in connection entries in the IPVS table. The connection in different states has different timeout value, for example, the timeout of the SYN_RECV state is 1 minute, the timeout of the ESTABLISHED state is 15 minutes (the default). Each connection entry occupy 128 bytes effective memory. Supposing that there is 128 Mbytes free memory, the box can have 1 million connection entries. The over 16,667 packet/second rate SYN flood can make the box run out of memory, and the syn-flooding attacker probably need to allocate T3 link or more to perform the attack. It is difficult to syn-flood a IPVS box. It would be much more difficult to attach a box with more memory. <verb> > I assume that the timeout is tunable, though reducing the > timeout could have implications for prematurely > dropping connections. Is there a possibility of implementing > random SYN drops if too many SYN are received as I believe > is implemented in the kernel TCP stack. </verb> Yup, I should implement random early drop of SYN entries long time ago as Alan Cox suggested. Actually, it would be simple to add this feature into the existing IPVS code, because the slow timer handler is activated every second to collect stale entries. I just need to some code to that handler, if over 90% (or 95%) memory is used, run drop_random_entry to randomly tranverse 10% (or 5%) entries and drop the SYN-RCV entries in them. <verb> > A second, related question is if a packet is forwarded to > a server, and this server has failed and is sunsequently > removed from the available pool using something like > ldirectord. Is there a window where the packet > can be retransmitted to a second server. This would > only really work if the packet was a new connection. </verb> Yes, it is true. If the primary load balaner fails over, all the established connections will be lost after the backup takes over. We probably need to investigate how to exchange the state (connection entries) periodically between the primary and the backup without too much performance degradation. <verb> > If persistent connections are being used and a client is > cached but doesn't have any active connections does > this count as a connection as far as load balancing, > particularly lc and wlc is concerned. I am thinking > no. This being the case, is the memory requirement for each > client that is cached but has no connections 128bytes as > per the memory required for a connection. </verb> The reason that the existing code uses one template and creates different entries for different connections from the same client is to manage the state of different connections from the same client, and it is easy to seemlessly add into existing IP Masquerading code. If only one template is used for all the connections from the same client, the box receives a RST packet and it is impossible to identify from which connection. Date: 24 Dec 2000 From: Julian Anastasov <tt/ja@ssi.bg/ <verb> > We using Hash Table to record an established network connection. > How do we know the data transmission by one conection is over > and when should we delete it from the Hash Table? </verb> OK, here we'll analyze the LVS and mostly the MASQ transition tables from net/ipv4/ip_masq.c. LVS support adds some extensions to the original MASQ code but the handling is same. First, we have three protocols handled: TCP, UDP and ICMP. The first one (TCP) has many states and with different timeout values, most of them set to reasonable values corresponding to the recommendations from some TCP related rfc* documents. For UDP and ICMP there are other timeout values that try to keep the both ends connected for reasonable time without creating many connection entries for each packet. There are some rules that keep the things working: - when a packet is received for an existing connection or when a new connection is created a timer is started/restarted for this connection. The timeout used is selected according to the connection state. If a packet is received for this connection (from one of the both ends) the timer is restarted again (and may be after a state change). If no packet is received during the selected period, the masq_expire() function is called to try to release the connection entry. It is possible masq_expire() to restart the timer again for this connection if it is used from other entries. This is the case for the templates used to implement the persistent timeout. They occupy one entry with timer set to the value of the persistent time interval. There are other cases, mostly used from the MASQ code, where helper connections are used and masq_expire() can't release the expired connection because it is used from others. - according to the direction of the packet we distinguish two cases: INPUT where the packet comes in demasq direction (from the world) and OUTPUT where the packet comes from internal host in masq direction. masq. What does "masq direction" mean for packets that are not translated using NAT (masquerading), for example, for Direct Routing or Tunneling? The short answer is: there is no masq direction for these two forwarding methods. It is explained in the LVS docs. In short, we have packets in both directions when NAT is used and packets only in one direction (INPUT) when DR or TUN are used. The packets are not demasqueraded for DR and TUN method. LVS just hooks the LOCAL_IN chain as the MASQ code is privileged in Linux 2.2 to inspect the incoming traffic when the routing decides that the traffic must be delivered locally. After some hacking, the demasquerading is avoided for these two methods, of course, after some changes in the packet and in its next destination - the real servers. Don't forget that without LVS or MASQ rules, these packets hit the local socket listeners. How are the connection states changed? Let's analyze for example the masq_tcp_states table (we analyze the TCP states here, UDP and ICMP are trivial). The columns specify the current state. The rows explain the TCP flag used to select the next TCP state and its timeout. The TCP flag is selected from masq_tcp_state_idx(). This function analyzes the TCP header and decides which flag (if many are set) is meaningful for the transition. The row (flag index) in the state table is returned. masq_tcp_state() is called to change ms->state according to the current ms->state and the TCP flag looking in the transition table. The transition table is selected according to the packet direction: INPUT, OUTPUT. This helps us to react differently when the packets come from different directions. This is explained later, but in short the transitions are separated in such way (between INPUT and OUTPUT) that transitions to states with longer timeouts are avoided, when they are caused from packets coming from the world. Everyone understands the reason for this: the world can flood us with many packets that can eat all the memory in our box. This is the reason for this complex scheme of states and transitions. The ideal case is when there is no different timeouts for the different states and when we use one timeout value for all TCP states as in UDP and ICMP. Why not one for all these protocols? The world is not ideal. We try to give more time for the established connections and if they are active (i.e. they don't expire in the 15 mins we give them by default) they can live forever (at least to the next kernel crash^H^H^H^H^Hupgrade). How does LVS extend this scheme? For the DR and TUN method we have packets coming from the world only. We don't use the OUTPUT table to select the next state (the director doesn't see packets returning from the internal hosts). We need to relax our INPUT rules and to switch to the state required by the external hosts :( We can't derive our transitions from the trusted internal hosts. We change the state only based on the packets coming from the the clients. When we use the INPUT_ONLY table (for DR and TUN) the LVS expects a SYN packet and then an ACK packet from the client to enter the established state. The director enters the established state after a two packet sequence from the client without knowing what happens in the real server, which can drop the packets (if they are invalid) or establish a connection. When an attacket sends SYN and ACK packets to flood a VS-DR or VS-Tun director, many connections are established state. Each each established connection will allocate resources (memory) for 15 mins by default. If the attacker uses many different source addresses for this attack the director will run out of memory. For these two methods LVS introduces one more transition table: the INPUT_ONLY table which is used for the connections created for the DR and TUN forwarding methods. The main goal: don't enter established state too easily - make it harder. Oh, maybe you're just reading the TCP specifications. There are sequence numbers that the both ends attach to each TCP packet. And you don't see the masq or LVS code to try to filter the packets according to the sequence numbers. This can be fatal for some connections as the attacker can cause state change by hitting a connection with RST packet, for example (ES->CL). The only info needed for this kind of attack is the source and destination IP addresses and ports. Such kind of attacks are possible but not always fatal for the active connections. The MASQ code tries to avoid such attacks by selecting minimal timeouts that are enough for the active connections to resurrect. For example, if the connection is hit by TCP RST packet from attacker, this connection has 10 seconds to give an evidence for its existance by passing an ACK packet through the masq box. To make the things complex and harder for the attacker to block a masq box with many established connections, LVS extends the NAT mode (INPUT and OUTPUT tables) by introducing internal server driven state transitions: the secure_tcp defense strategy. When enabled, the TCP flags in the client's packet can't trigger switching to established state without acknowledgement from the internal end of this connection. secure_tcp changes the transition tables and the state timeouts to achieve this goal. The mechanism is simple: keep the connection is SR state with timeout 10 seconds instead of the default 60 seconds when the secure_tcp is not enabled. This trick depends on the different defense power in the real servers. If they don't implement SYN cookies and so sometimes don't send SYN+ACK (because the incoming SYN is dropped from their full backlog queue), the connection expires in LVS after 10 seconds. This action assumes that this is a connection created from attacker, since one SYN packet is not followed by another, as part from the retransmissions provided from the client's TCP stack. We give 10 seconds to the real server to reply with SYN+ACK (even 2 are enough). If the real server implements SYN cookies the SYN+ACK reply follows the SYN request immediatelly. But if there are no SYN cookies implemented the SYN requests are dropped when the backlog queue length is exceeded. So secure_tcp is by default useful for real servers that don't implement SYN cookies. In this case the LVS expires the connections in SYN state in a short time, releasing the memory resources allocated from them. In any case, secure_tcp does not allow switching to established state by looking in the clients packets. We expect ACK from the real-server to allow this transition to EST state. The main goal of the defense strategies is to keep the LVS box with more free memory for other connections. The defense for the real servers can be build in the real servers. But may be I'll propose to Wensong to add a per-connection packet rate limit. This will help against attacks that create small number of connections but send many packets and by this way load the real servers dramatically. May be two values: rate limit for all incoming packets and rate limit per connection. The good news is that all these timeout values can be changed in the LVS setup, but only when the secure_tcp strategy is enabled. An SR timeout of 2 seconds is a good value for LVS clusters when real-servers don't implement SYN cookies: if there is no SYN+ACK from the real-server then drop the entry at the director. The bad news is of course, for the DR and TUN methods. The director doesn't see the packets returning from the real-servers and VS-DR and VS-Tun forwarding can't use the internal server driven mechanism. There are other defense strategies that help when using these methods. All these defense strategies keep the director with memory free for more new connections. There is no known way to pass only valid requests to the internal servers. This is because the real-servers don't provide information to the director and we don't know which packet is dropped or accepted from the socket listener. We can know this only by receiving an ACK packet from the internal server when the three-way handshake is completed and the client is identified from the internal server as a valid client, not as spoofed one. This is possible only for the NAT method. Julian Anastasov <tt/ja@ssi.bg/ <tt/ksparger@dialtoneinternet.net/ (29 Jan 2001) rephrases this by saying the VS-NAT is layer-3 aware. For example, NAT can 'see' if a real server responds to a packet it's been sent or not, since it's watching all of the traffic anyway. If the server doesn't respond within a certain period of time, the director can automatically route that packet to another server. LVS doesn't support this right now, but, NAT would be the more likely candidate to support it in the future, as NAT understands all of the IP layer concepts, and DR doesn't necessarily. (Julian) Someone must put back the real server when it is alive. This sounds like a user space job. The traffic will not start until we send requests. We have to send L4 probes to the real server (from the user space) or to probe it with requests (LVS from kernel space)? <sect2>Hash Table size Yasser Nabi wrote <quote> IP Virtual Server version 0.9.0 (size=16777216) </quote> Julian Anastasov <tt/ja@ssi.bg/ 25 May 2001 Too much, it takes 128MB for the table only. Use 16 bits for example. <quote> Is this a hidden/undocumented problem with IPVS or it's just an observation of memory waste ? (we use 18 bits in production) </quote> <verb> empty hash tables: 18 bits occupy 2MB RAM 24 bits occupy 128MB RAM </verb> If the box has 128MB and the bits are 24 the kernel crash is mandatory, soon or later. And this is a good reason the virtual service not to be hit. Expect funny things to happen on box with low memory <quote> I forgot that not anyone uses 256Mb or more RAM on directors :) </quote> Yes, 256MB in real situation is ~1,500,000 connections, 128 bytes each, 64MB for other things ... until someone experiments with SYN attack <quote> However, for me it makes sense to use up to 66% of total memory for LVS, especially on high-traffic directors (in the idea that the directors doesn't run all the desktop garbage that comes with most distros). </quote> If the used bits are 24, an empty hash table is 128MB. For the rest 128MB you can allocate 1048576 entries, 128 bytes each ... after the kernel killed all processes. Some calcs considering the magic value 16 as average bucket length and for 256MB memory: For 17 bits: 2^17=131072 => 1MB for empty hash table 131072*16=2097152 entries=256MB for connections For 18 bits: 2^18=262144 => 2MB for empty hash table for each MB for hash table we lose space for 8192 entries but we speedup the lookup. So, even for 1GB directors, 19 or 20 is the recommended value. Anything above is a waste of memory for hash table. In 128MB we can put 1048576 entries. In the 24-bit case they are allocated for d-linked list heads. <sect1>Port range limitations Wayne <tt/wayne@compute-aid.com/ 14 May 2000, <quote> If running a load balancer tester, say the one from IXIA to issue connections to 100 powerful web servers, would all the parameters in Julian's description need to be changed, or it should not be a problem for having many many connections from a single tester? </quote> (Julian) There is no limit for the connections from the internal hosts. Currently, the masquerading allows one internal host to create 40960 TCP connections. But the limit of 4096 connections to one external service is still valid. If 10 internal hosts try to connect to one external service, each internal host can create 4096/10 => ~409 connections. For UDP the problem is sometimes worse. It depends on the /proc/sys/net/ipv4/ip_masq_udp_dloose value. (Joe - which is internal and which is external here? The client, the real-servers?) This is a plain masquerading so internal and external refer to masquerading. These limits are not for the LVS connections, they are only for the 2.2 masquerading. <verb> / 65095 Internal Servers External Server:PORT - ... MADDR -------------------- \ 61000 </verb> When many internal clients try to connect to same external real service, the total number of TCP connections from one MADDR to this remote service can be 4096 because the masq uses only 4096 masq ports by default. This is a normal TCP limit, we distinguish the TCP connections by the fact they use different ports, nothing more. And the masq code is restricted by default to use the above range of 4096 ports. In the whole masquerading table there is a space only for 40960 TCP, 40960 UDP and 40960 ICMP connections. These values can be tuned by changing ip_masq.c:PORT_MASQ_MUL. The PORT_MASQ_MUL value simply determines the recommended length of one row in the masq hash table for connections but in fact it is involved in the above connection limits. It is recommended that the busy masq routers must increase this value. May be the 4096 masq port range too. This involves squid servers behind masq router. LVS uses another table without limits. For LVS setups the same TCP restrictions apply but for the external clients: <verb> 4999 \ Client - VIP:VPORT LVS Director 1024 / </verb> The limit of client connections to one VIP:VPORT is limited to the number of used client ports from same Client IP. The same restrictions apply to UDP. UDP has the same port ranges. But for UDP the 2.2 kernel can apply different restrictions. They are caused from some optimizations that try to create one UDP entry for many connections. The reason for this is the fact that one UDP client can connect to many UDP servers while this is not common for TCP. <sect1>DoS<label id="DoS"> LVS is vunerable to DoS by an attacker making repeated connection requests. Eventually the director will run out of memory. This will take a while but an attacker has plenty of time if you are asleep. As well with VS-DR and VS-Tun, the director doesn't have access to the TCPIP tables in the real-server(s) which show whether a connection has closed (see <ref id="Hash_Table" name="director hash table">). The director can only guess that the connection has really closed, and does so using timeouts. For information on DoS strategies for LVS see <url url="http://www.linuxvirtualserver.org/defense.html" name="DoS page">. On Wed, 14 Feb 2001, Laurent Lefoll wrote: From: Laurent Lefoll <tt/Laurent.Lefoll@mobileway.com/ <verb> > If I am not misunderstanding something, the variable > /proc/sys/net/ipv4/vs/timeout_established gives the time a TCP connection can be > idle and after that the entry corresponding to this connection is cleared. My > problem is that it seems that sometimes it's not the case. For example I have a > system (2.2.16 and ipvs 0.9.15) with /proc/sys/net/ipv4/vs/timeout_established > = 480 but the entries are created with a real timeout of 120 ! On another system </verb> From: Julian Anastasov <tt/ja@ssi.bg/ Read <url url="http://www.linuxvirtualserver.org/defense.html" name="The secure_tcp defense strategy"> where the timeouts are explained. They are valid for the defense strategies only. For TCP EST state you need to read the ipchains man page. For more explanation of the secure_tcp strategy also see the <ref id="Hash_Table" name="explanation of the director's hash table">. <verb> > when I play with "ipchains -M -S > [value] 0 0" > the variable /proc/sys/net/ipv4/vs/timeout_established is modified > even when /proc/sys/net/ipv4/vs/secure_tcp is set to 0, > so I'm not using the secure TCP defense. > The "real" timeout is of course set to [value] when a new TCP connection appears. > So should I understand that timeout_established, timeout_udp,... are always > modified by "ipchains -M -S ...." whatever I use or not secure TCP defense but > if secure-tcp is set to 0, other variables give the timeouts to use ? If so, are > these variable accessible or how to check their value ? </verb> ipchains -M -S modifies the two TCP and the UDP timeouts in the two secure_tcp modes: off and on. So, ipchains changes the three timeout_XXX vars. When you change the timeout_* vars you change them for secure_tcp=on only. Think for the timeouts as you have two sets: for the two secure_tcp modes: on and off. ipchains changes the 3 vars in the both sets. While secure_tcp is off changing timeout_* does not affect the connection timeouts. They are used when secure_tcp is on. (Joe: `ipchains 0 value 0`, where value=10 does not change the timeout values or number of entries seen in InActConn or seen with netstat -M, or ipchains -M -L -n). LVS has its own tcpip state table, when in secure_tcp mode. carl.huang <quote> what are the vs_tcp_states[ ] and vs_tcp_states_dos[ ] elements in the in ip_vs_conn structure for? </quote> Roberto Nibali <tt/ratz@tac.ch/ 16 Apr 2001 The vs_tcp_states[] table is the modified state transition table for the TCP state machine. The vs_tcp_states_dos[] is a yet again modified state table in case we are under attack and secure_tcp is enabled. It is tigher but not conforming to the RFC anymore. Let's take an example how you can read it: <verb> static struct vs_tcp_states_t vs_tcp_states [] = { /* INPUT */ /* sNO, sES, sSS, sSR, sFW, sTW, sCL, sCW, sLA, sLI, sSA */ /*syn*/ {{sSR, sES, sES, sSR, sSR, sSR, sSR, sSR, sSR, sSR, sSR }}, /*fin*/ {{sCL, sCW, sSS, sTW, sTW, sTW, sCL, sCW, sLA, sLI, sTW }}, /*ack*/ {{sCL, sES, sSS, sES, sFW, sTW, sCL, sCW, sCL, sLI, sES }}, /*rst*/ {{sCL, sCL, sCL, sSR, sCL, sCL, sCL, sCL, sLA, sLI, sSR }}, </verb> The elements 'sXX' mean state XX, so for example, sFW means TCP state FIN_WAIT, sSR means TCP state SYN_RECV and so on. Now the table describes the state transition of the TCP state machine from one TCP state to another one after a state event occured. For example: Take row 2 starting with sES and ending with sCL. At the first, commentary row, you see the incoming TCP flags (syn,fin,ack,rst) which are important for the state transition. So the rest is easy. Let's say, you're in row 2 and get a fin so you go from sES to sCW, which should by conforming to RFC and Stevens. Short illustration: <verb> /* , sES, /*syn*/ {{ , , /*fin*/ {{ , sCW, </verb> It was some months ago last year when Wensong, Julian and me discussed about a security enhancement for the TCP state transition and after some heavy discussion they implemented it. So the second table vs_tcp_states_dos[] was born. (look in the mailing list in early 2000). <sect1>Writing Filter Rules iptables (2.4 kernels) is different to ipchains (2.2 kernels). For one thing there is no "iptables -C" to check your rules (at least yet). Ratz <quote> If you're dealing with netfilter, packets don't travel through all chains anymore. Julian once wrote something about it: packets coming from outside to the LVS do: <verb> PRE_ROUTING -> LOCAL_IN(LVS in) -> POST_ROUTING </verb> packets leaving the LVS travel: <verb> PRE_ROUTING -> FORWARD(LVS out) -> POST_ROUTING </verb> From the iptables howto: COMPATIBILITY WITH IPCHAINS <quote> This iptables is very similar to ipchains by Rusty Rus� sell. The main difference is that the chains INPUT and OUTPUT are only traversed for packets coming into the local host and originating from the local host respec� tively. Hence every packet only passes through one of the three chains; previously a forwarded packet would pass through all three. </quote> I don't yet understand all of iptables yet, but it helped making the whole filtering and NAPT stuff more smooth and more flexible. You have more possibilies to manage the traffic. If it is better has to be proven in reality, so far there are not many setups with complex firewall settings, like merging different advanced routing aspects with QoS and own Targets over different networks with all kind of non-TCP/UDP traffic and an IPV6 connection. Julian <quote> Packets coming into the director (out->in): NAT: INPUT -> input routing -> local: LVS/DEMASQ -> input routing -> forwarding -> OUTPUT DR/TUN: INPUT -> input routing -> local: LVS -> output routing -> OUTPUT packets leaving the LVS travel (in->out): NAT only: INPUT -> input routing -> FORWARD (-j MASQ) -> LVS/MASQ -> OUTPUT </quote> How does it go with a 2.2 director? <quote> INPUT in 2.2 is similar as PRE_ROUTING in 2.4, i.e. INPUT, OUTPUT and FORWARD are the 2.2 firewall chains <verb> input routing: ip_route_input() output routing: ip_route_output() forwarding: ip_forward() local: ip_local_deliver() </verb> </quote> </quote> <sect1>Throughput Statistics these are available in /proc/net/ip_masq/vs_stats (for 2.2) and /proc/net/ip_vs_stats (for 2.4). Joe Can I zero out these counters if I want to get rates, or should I store the last count? Ratz, May 2001 <quote> There was a recent (2 months ago) talk about zeroing in-kernel counters and I'm not so sure if all the kernel hacker gurus agreed but: You must not zero a counter in the kernel! I didn't really understand the arguments against or pro zeroing counters so I'm not a big help here, but if others agree we certainly can add this feature. It would be ipvsadm -Z as an analogy to ip{chains|tables}. BTW, we are proud of haveing 64bit counters in the kernel :) Storing ... there are different approaches to this (complexity order): <itemize> <item> Use a script that extracts the info and writes it flat to a file <item>Use mrtg or rrdtool since I reckon you wanted to use the stats to generate some graphics anyway. These tools handle the problem for you. mrtg requires SNMP, but you can have a slightely modified snmpd.conf and execute a script that parses /proc/net/ip_masq/vs_stats and writes it into a file. The advantage of this over the first one is, that you can write the current number into one file and mrtg will know how to draw the graph. I give you an example: We have a customer named plx. Now he has only one service and 2 realserver. We extended the snmpd.conf with following lines: <verb> exec lbsessions /bin/sh /opt/tac/snmp/lbsessions exec lbsessions.plx.total /bin/sh /opt/tac/snmp/lbsessions.plx.total exec lbsessions.plx.web-web1 /bin/sh /opt/tac/snmp/lbsessions.plx.web-web1 exec lbsessions.plx.web-web2 /bin/sh /opt/tac/snmp/lbsessions.plx.web-web2 </verb> The scripts are awk scripts that get the information accordingly to the service or the realserver. You can then do a table walk of the OID 1.3.6.1.4.1.2021.8 to see what your values are: snmpwalk $IP $COMMUNITY .1.3.6.1.4.1.2021.8 Example output if everything is ok: <verb> enterprises.ucdavis.extTable.extEntry.extNames.1 = lbsessions enterprises.ucdavis.extTable.extEntry.extNames.2 = lbsessions.plx.total enterprises.ucdavis.extTable.extEntry.extNames.3 = lbsessions.plx.web-web1 enterprises.ucdavis.extTable.extEntry.extNames.4 = lbsessions.plx.web-web2 enterprises.ucdavis.extTable.extEntry.extCommand.1 = /bin/sh /opt/tac/snmp/lbsessions enterprises.ucdavis.extTable.extEntry.extCommand.2 = /bin/sh /opt/tac/snmp/lbsessions.plx.total enterprises.ucdavis.extTable.extEntry.extCommand.3 = /bin/sh /opt/tac/snmp/lbsessions.plx.web-web1 enterprises.ucdavis.extTable.extEntry.extCommand.4 = /bin/sh /opt/tac/snmp/lbsessions.plx.web-web2 enterprises.ucdavis.extTable.extEntry.extResult.1 = 0 enterprises.ucdavis.extTable.extEntry.extResult.2 = 0 enterprises.ucdavis.extTable.extEntry.extResult.3 = 0 enterprises.ucdavis.extTable.extEntry.extResult.4 = 0 enterprises.ucdavis.extTable.extEntry.extOutput.1 = 292 enterprises.ucdavis.extTable.extEntry.extOutput.2 = -1 enterprises.ucdavis.extTable.extEntry.extOutput.3 = -1 enterprises.ucdavis.extTable.extEntry.extOutput.4 = -1 </verb> Here you see that the total amount of sessions of the load balancer serving about 8 customers is 292 currently and that customer plx has no connections so far. <label id="MIB"> <item>Write a MIB for LVS stats. </itemize> </quote> <sect1>Active/Inactive connnection<label id="ActConn"> The output of ipsvadm lists connections, either as <itemize> <item>ActConn - in ESTABLISHED state <item>InActConn - any other state </itemize> Entries in the ActConn column come from <itemize> <item> service with an established connection. Examples of services which hold connections in the ESTABLISHED state long enough to see with ipvsadm are telnet and ftp (port 21). </itemize> Entries in the InActConn column come from <itemize> <item>Normal operation <itemize> <item>Services like http (in non-persistent i.e. HTTP /1.0 mode) or ftp-data(port 20) which close the connections as soon as the hit/data (html page, or gif etc) has been retrieved (<1sec). You're unlikely to see anything in the ActConn column with these LVS'ed services. You'll see an entry in the InActConn column untill the connection times out. If you're getting 1000connections/sec and it takes 60secs for the connection to time out (the normal timeout), then you'll have 60,000 InActConns. This number of InActConn is quite normal. If you are running an e-commerce site with 300secs of persistence, you'll have 300,000 InActConn entries. Each entry takes 128bytes (300,000 entries is about 40M of memory, make sure you have enough RAM for your application). The number of ActConn might be very small. </itemize> <item>Pathological Conditions (i.e. your LVS is not setup properly) <itemize> <item>identd delayed connections: The 3 way handshake to establish a connection takes only 3 exchanges of packets (i.e. it's quick on any normal network) and you won't be quick enough with ipvsadm to see the connection in the states before it becomes ESTABLISHED. However if the service on the real-server is under <ref id="authd" name="identd">, you'll see an InActConn entry during the delay period. <item>Incorrect routing (usually the wrong default gw for the real-servers): In this case the 3 way handshake will never complete, the connection will hang, and there'll be an entry in the InActConn column. </itemize> </itemize> Usually the number of InActConn will be larger or very much larger than the number of ActConn. Here's a VS-DR LVS, setup for ftp, telnet and http, after telnetting from the client (the client command line is at the telnet prompt). <verb> director:# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:www rr -> bashfull.mack.net:www Route 1 0 0 -> sneezy.mack.net:www Route 1 0 0 TCP lvs2.mack.net:0 rr persistent 360 -> sneezy.mack.net:0 Route 1 0 0 TCP lvs2.mack.net:telnet rr -> bashfull.mack.net:telnet Route 1 1 0 -> sneezy.mack.net:telnet Route 1 0 0 </verb> showing the ESTABLISHED telnet connection (here to real-server bashfull). Here's the output of netstat -an | grep (appropriate IP) for the client and the real-server, showing that the connection is in the ESTABLISHED state. <verb> client:# netstat -an | grep VIP tcp 0 0 client:1229 VIP:23 ESTABLISHED real-server:# netstat -an | grep CIP tcp 0 0 VIP:23 client:1229 ESTABLISHED <verb> Here's immediately after the client logs out from the telnet session. <verb> director# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:www rr -> bashfull.mack.net:www Route 1 0 0 -> sneezy.mack.net:www Route 1 0 0 TCP lvs2.mack.net:0 rr persistent 360 -> sneezy.mack.net:0 Route 1 0 0 TCP lvs2.mack.net:telnet rr -> bashfull.mack.net:telnet Route 1 0 0 -> sneezy.mack.net:telnet Route 1 0 0 client:# netstat -an | grep VIP #ie nothing, the client has closed the connection #the real-server has closed the session in response #to the client's request to close out the session. #The telnet server has entered the TIME_WAIT state. real-server:/home/ftp/pub# netstat -an | grep 254 tcp 0 0 VIP:23 CIP:1236 TIME_WAIT #a minute later, the entry for the connection at the real-server is gone. </verb> Here's the output after ftp'ing from the client and logging in, but before running any commands (like `dir` or `get filename`). <verb> director:# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:www rr -> bashfull.mack.net:www Route 1 0 0 -> sneezy.mack.net:www Route 1 0 0 TCP lvs2.mack.net:0 rr persistent 360 -> sneezy.mack.net:0 Route 1 1 1 TCP lvs2.mack.net:telnet rr -> bashfull.mack.net:telnet Route 1 0 0 -> sneezy.mack.net:telnet Route 1 0 0 client:# netstat -an | grep VIP tcp 0 0 CIP:1230 VIP:21 TIME_WAIT tcp 0 0 CIP:1233 VIP:21 ESTABLISHED real-server:# netstat -an | grep 254 tcp 0 0 VIP:21 CIP:1233 ESTABLISHED </verb> The client opens 2 connections to the ftpd and leaves one open (the ftp prompt). The other connection, used to transfer the user/passwd information, is closed down after the login. The entry in the ipvsadm table corresponding to the TIME_WAIT state at the real-server is listed as InActConn. If nothing else is done at the client's ftp prompt, the connection will expire in 900 secs. Here's the real-server during this 900 secs. <verb> real-server:# netstat -an | grep CIP tcp 0 0 VIP:21 CIP:1233 ESTABLISHED real-server:# netstat -an | grep CIP tcp 0 57 VIP:21 CIP:1233 FIN_WAIT1 real-server:# netstat -an | grep CIP #ie nothing, the connection has dropped #if you then go to the client, you'll find it has timed out. ftp> dir 421 Timeout (900 seconds): closing control connection. </verb> http 1.0 connections are closed immediately after retrieving the URL (i.e. you won't see any ActConn in the ipvsadm table immediately after the URL has been fetched). Here's the outputs after retreiving a webpage from the LVS. <verb> director:# ipvsadm IP Virtual Server version 0.2.8 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:www rr -> bashfull.mack.net:www Route 1 0 1 -> sneezy.mack.net:www Route 1 0 0 client:~# netstat -an | grep VIP bashfull:/home/ftp/pub# netstat -an | grep CIP tcp 0 0 VIP:80 CIP:1238 TIME_WAIT </verb> <sect2>Q&A from the mailing list Ty Beede wrote: <verb> > I am curious about the implementation of the inactconns and > activeconns variables in the lvs source code. </verb> (Julian) <verb> Info about LVS <= 0.9.7 TCP active: all connections in ESTABLISHED state inactive: all connections not in ESTABLISHED state UDP active: 0 (none) (LVS <= 0.9.7) inactive: all (LVS <= 0.9.7) active + inactive = all </verb> Look in this table for the used timeouts for each protocol/state: /usr/src/linux/net/ipv4/ip_masq.c, masq_timeout_table For VS/TUNNEL and VS/DR the TCP states are changed checking only the TCP flags from the incoming packets. For these methods UDP entries can expire (5 minutes?) if only the real servers sends packets and there are no packets from the client. For info about the TCP states: - /usr/src/linux/net/ipv4/tcp.c - rfc793.txt From: Jean-francois Nadeau <tt/jf.nadeau@videotron.ca/ Done some testing (netmon) on this and here's my observations : 1. A connection becomes active when LVS sees the ACK flag in the TCP header incoming in the cluster : i.e when the socket gets established on the real server. 2. A connection becomes inactive when LVS sees the ACK-FIN flag in the TCP header incoming in the cluster. This does NOT corespond to the socket closing on the real server. Example with my Apache Web server. <verb> Client <--> Server A client request an object on the web server on port 80 : SYN REQUEST ----> SYN ACK <---- ACK -----> *** AcitveConn=1 and 1 ESTABLISHED socket on real server. HTTP get -----> *** The client request the object HTTP response <----- *** The server sends the object APACHE closes the socket : *** AcitveConn=1 and 0 ESTABLISHED socket on real server The CLIENT receives the object. (took 15 seconds in my test) ACK-FIN -----> *** AcitveConn=0 and 0 ESTABLISHED socket on real server </verb> Conclusion : ActiveConn is the active number of CLIENT connections..... not on the server in the case of short transmissions like objects on a web page. Its hard to calculate a server's capacity based on this because slower clients makes ActiveConn greater than whats the server is really processing. You wont be able to reproduce that effect on a LAN because the client receives the segment too fast. (Julian) In the LVS mailing list many people explained that the correct way to balance the connections is to use monitoring software. The weights must be evaluated using values from the real server. In VS/DR and VS/TUN the Director can be easily fooled with invalid packets for some period and this can be enough to inbalance the cluster when using "*lc" schedulers. I reproduce the effect connecting at 9600 bps and getting a 100k gif from Apache while monitoring established sockets on port 80 on the real server and ipvsadm on the cluster. (Julian) You are probably using VS/DR or VS/TUN in your test. Right? Using these methods the LVS is changing the TCP state based on the incoming packets, i.e. from the clients. This is the reason that the Director can't see the FIN packet from the real server. This is the reason that LVS can be easily SYN flooded, even flooded with ACK following the SYN packet. The LVS can't change the TCP state according to the state in the real server. This is possible only for VS/NAT mode. So, in some situations you can have invalid entries in ESTABLISHED state not corresponding to the connections in the real server which effectively ignores these SYN packets using cookies. The VS/NAT looks the betters solution against the SYN flood attacks. Of course, the ESTABLISHED timeout can be changed to 5 minutes for example. Currently, the max timeout interval (excluding the ESTABLISHED state) is 2 minutes. If you think that you can serve the clients using smaller timeout for the ESTABLISHED state when under "ACK after SYN" attack you can change it with ipchains. You don't need to change it under 2 minutes in LVS 0.9.7. In the last LVS version SYN+FIN switches the state to TW which can't be controlled using ipchains. In other cases you can change the timeout for the ESTABLISHED and FIN-WAIT states. But you can change it only down to 1 minute. If this can't help by 2GB RAM or more for the Director. One thing that can be done but this is may be paranoia: change the INPUT_ONLY table: <verb> from: FIN SR ---> TW to: FIN SR ---> FW </verb> OK, this is incorrect interpretation of the TCP states but this is a hack which allows the min state timeout to be 1 minute. Now using ipchains we can set the timeout to all TCP states to 1 minute. If this is changed you can now set ESTABLISHED and FIN-WAIT timeouts down to 1 minute. In current LVS version the min effective timeout for ESTABLISHED and FINWAIT state is 2 minutes. From: Jean-Francois Nadeau <tt/jf.nadeau@videotron.ca/ I'm using DR on the cluster with 2 real servers. I'm trying to control the number of connections to acheive this : The cluster in normal mode balances requests on the 2 real servers. If the real servers reaches a point where they can't serve clients fast enough, a new entry with a weight of 10000 is entered in LVS to send the ovry with a weight of 10000 is entered in LVS to send the ovry with a weight of 10000 is entered in LVS to send the overflow locally on a web server with a static web page saying "we're too busy". It's a cgi that intercept 'deep links' in our site and return a predefined page. A 600 seconds persistency ensure that already connected clients stays on the server they began to browse. The client only have to hit refresh until the number of AciveConns (I hoped) on the real servers gets lower and the overflow entry gets deleted. Got the idea... Load balancing with overflow control. (Julian) Good idea. But the LVS can't help you. When the clients are redirected to the Director they stay there for 600 seconds. But when we activate the local redirection of requests due to overflow, ActiveConn continues to grow in LVS, while Inactconn decreases as expected. So the load on the real server gets OK... but LVS doesnt sees it and doesnt let new clients in. (it takes 12 minutes before ActiveConns decreases enough to reopen the site) I need a way, a value to check at that says the server is overloaded, begin redirecing locally and the opposite. I know that seems a little complicated.... (Julian) What about trying to: - use persistent timeout 1 second for the virtual service. If you have one entry for this client you have all entries from this client to the same real server. I didn't tested it but may be a client will load the whole web page. If the server is overloaded the next web page will be "we're too busy". - switch the weight for the Director between 0 and 10000. Don't delete the Director as real server. Weight 0 means "No new connections to the server". You have to play with the weight for the Director, for example: - if your real servers are loaded near 99% set the weight to 10000 - if you real servers are loaded before 95% set the weight to 0 (From: Jean-Francois Nadeau <tt/jf.nadeau@videotron.ca/) Will a weight of 0 redirect traffic to the other real servers (persistency remains ?) (Julian) If the persistent timeout is small, I think. I can't get rid of the 600 seconds persistency because we run a transactionnal engine. i.e. if a client begins on a real server, he must complete the transaction on that server or get an error (transactionnal contexts are stored locally). Such timeout can't help to redirect the clients back to the real servers. You can check the free ram or the cpu idle time for the real servers. By this way you can correctly set the weights for the real servers and to switch the weight for the Director. These recommendations can be completely wrong. I've never tested them. If they can't help try to set httpd.conf:MaxClients to some reasonable value. Why not to put the Director as real server permanently. With 3 real servers is better. (Jean) Those are already optimized, bottleneck is when 1500 clients tries our site in less than 5 minutes..... One of ours has suggested that the real servers check their own state (via TCP in use given by sockstat) and command the director to redirect traffic when needed. Can you explain more in details why the number of ActiveConn on real server continue to grow while redirecting traffic locally with a weight of 10000 (and Inactonn on that real server decreasing normally). (Julian) Only the new clients are redirected to the Director at this moment. Where the active connections continue to grow, in the real servers or in the Director (weight=10000)? <sect1>Creating large numbers of InActConn with testlvs; testing DoS strategies<label id="InActConn"> testlvs (by Julian <tt/ja@ssi.bg/) is available on <url url="http://www.linuxvirtualserver.org/~julian" name="Julian's software page">. It sends a stream of SYN packets (SYN flood) from a range of addresses (default starting at 10.0.0.1) simulating connect requests from many clients. Running testlvs from a client will occupy most of the resources of your director and the director's screen/mouse/keyboard will/may lock up for the period of the test. To run testlvs, I export the testlvs directory (from my director) to the real-servers and the client and run everything off this exported directory. <sect2>configure real-server The real-server is configured to reject packets with src_address 10.0.0.0/8. Here's my modified version of Julian's show_traffic.sh <label id="show_traffic">, which is run on the real-server to measure throughput. Start this on the real-server before running testlvs on the client. For your interest you can look on the real-server terminal to see what's happening during a test. <verb> #!/bin/sh #show_traffic.sh #by Julian Anastasov ja@ssi.bg #modified by Joseph Mack jmack@wm7d.net # #run this on the real-server before starting testlvs on the client #when finished, exit with ^C. # #suggested parameters for testlvs #testlvs VIP:port -tcp -packets 20000 #where # VIP:port - target IP:port for test # #packets are sent at about 10000 packets/sec on my #100Mbps setup using 75 and 133MHz pentium classics. # #------------------------------------------ # setup a few things to=10 #sleep time trap return INT #trap ^C from the keyboard (used to exit the program) iface="$1" #NIC to listen on #------------------------------------------ #user defined variables #network has to be the network of the -srcnet IP #that is used by the copy of testlvs being run on the client #(default for testlvs is 10.0.0.0) network="10.0.0.0" netmask="255.0.0.0" #------------------------------------------- function get_packets() { cat /proc/net/dev | sed -n "s/.*${iface}:$.*$/\1/p" | \ awk '{ packets += $2} ; END { print packets }' } function call_get_packets() { while true do sleep $to p1="`get_packets "$iface"`" echo "$((($p1-$p0)/$to)) packets/sec" p0="$p1" done } #------------------------------------------- echo "Hit control-C to exit" #initialise packets at $iface p0="`get_packets "$iface"`" #reject packets from $network route add -net $network netmask $netmask reject call_get_packets #restore routing table on exit route del -net $network netmask $netmask reject #------------------------------------------- </verb> <sect2>configure director I used VS-NAT on a 2.4.2 director, with netpipe (port 5002) as the service on two real-servers. You won't be using netpipe for this test, ie you won't need a netpipe server on the real-server You just need a port that you can set up an LVS on and netpipe is in my /etc/services, so the port shows up as a name rather than a number. Here's my director <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 0.2.6 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:netpipe rr -> bashfull.mack.net:netpipe Masq 1 0 0 -> sneezy.mack.net:netpipe Masq 1 0 0 </verb> <sect2>run testlvs from client run testlvs (I used v0.1) on the client. Here testlvs is sending 256 packets from 254 addresses (the default) in the 10.0.0.0 network. (My setup handles 10,000 packets/sec. 256 packets appears to be instantaneous.) <verb> client: #./testlvs 192.168.2.110:5002 -tcp -packets 256 </verb> when the run has finished, go to the director <verb> director:/etc/lvs# ipvsadm IP Virtual Server version 0.2.6 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:netpipe rr -> bashfull.mack.net:netpipe Masq 1 0 127 -> sneezy.mack.net:netpipe Masq 1 0 127 </verb> (If you are running a 2.2.x director, you can get more information from ipchains -M -L -n, or netstat -M. For 2.4.x use cat /proc/net/ip_conntrack.) This output shows 254 connections that have closed are are waiting to timeout. A minute or so later, the InActConn will have cleared (on my machine, it's 50secs). If you send the same number of packets (256), from 1000 different addresses, (or 1000 packets to 256 addresses), you'll get the same result in the output of ipvsadm (not shown) <verb> client: #./testlvs 192.168.2.110:5002 -tcp -srcnum 1000 -packets 256 </verb> In all cases, you've made 254 connections. If you send 1000 packets from 1000 addresses, you'd expect 1000 connections. <verb> ./testlvs 192.168.2.110:5002 -tcp -srcnum 1000 -packets 1000 </verb> Here's the total number of InActConn as a function of the number of packets (connection attempts). Results are for 3 consecutive runs, allowing the connections to timeout in between. The numbers are not particularly consistent between runs (aren't computers deterministic?). Sometimes the blinking lights on the switch stopped during a test, possibly a result of the tcp race condition (see the <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance page">) <verb> packets InActConn 1000 356,368,377 2000 420,391,529 4000 639,770,547 8000 704,903,1000 16000 1000,1000,1000 </verb> You don't get 1000 InActConn with 1000 packets (connection attempts). We don't know why this is. (Julian) <quote> I'm not sure what's going on. In my tests there are dropped packets too. They are dropped before reaching the director, maybe from the input device queue or from the routing cache. We have to check it. </quote> <sect2>InActConn with drop_entry defense strategy repeating the control experiment above, but using the drop_entry strategy (see <ref id="DoS" name="the DoS strategies"> for more information). director:/etc/lvs# echo "3" >/proc/sys/net/ipv4/vs/drop_entry <verb> packets InActConn, drop_entry=3 1000 369,368,371 2000 371,380,409 4000 467,578,458 8000 988,725,790 16000 999,994,990 </verb> The drop_entry strategy drops 1/32 of the entries every second, so the number of InActConn decreases linearly during the timeout period, rather than dropping suddenly at the end of the timeout period. <sect2>InActConn with drop_packet defense strategy repeating the control experiment above, but using the drop_packet strategy (see <ref id="DoS" name="the DoS strategies"> for more information). director:/etc/lvs# echo "3" >/proc/sys/net/ipv4/vs/drop_packet <verb> packets InActConn, drop_packet=3 1000 338,339,336 2000 331,421,382 4000 554,684,629 8000 922,897,480,662 16000 978,998,996 </verb> The drop_packet=3 strategy will drop 1/10 of the packets before sending them to the real-server. The connections will all timeout at the same time (as for the control experiment, about 1min), unlike for the drop_entry strategy. With the variability of the InActConn number, it is hard to see the drop_packet defense working here. <sect2>InActConn with secure_tcp defense strategy repeating the control experiment above, but using the secure_tcp strategy (see <ref id="DoS" name="the DoS strategies"> for more information). The SYN_RECV value is the suggested value for VS-NAT. <verb> director:/etc/lvs# echo "3" >/proc/sys/net/ipv4/vs/secure_tcp director:/etc/lvs# echo "10" >/proc/sys/net/ipv4/vs/timeout_synrecv </verb> <verb> packets InActConn, drop_packet=3 1000 338,372,359 2000 405,367,362, 4000 628,507,584 8000 642,1000,886 16000 1000,1000,1000 </verb> This strategy drops the InActConn from the ipvsadm table after 10secs. <sect2>maximum number of InActConn If you want to get the maximum number of InActConn, you need to run the test for longer than the FIN timeout period (here 50secs). 2M packets is enough here. As well you want as many different addresses used as possible. Since testlvs is connecting from the 10.0.0.0/8 network, you could have 254^3=16M connections. Since only 2M packets can be passed before connections start to timeout and the director connection table reaches a steady state with new connections arriving and old connections timing out, there is no point in sending packets from more that 2M source addresses. Note: you can view the contents of the connection table with 2.2 <itemize> <item>netstat -Mn <item>cat /proc/net/ip_masquerade </itemize> 2.4 <itemize> <item>cat /proc/net/ip_vs_conn </itemize> Here's the InActConn with various defense strategies. The InActConn is the maximum number reachable, the scrnum and packets are the numbers needed to saturate the director. The time of the test must exceed the timeouts. InActConn was determined by running a command like this <verb> client: #./testlvs 192.168.2.110:5002 -tcp -srcnum 1000000 -packets 2000000 </verb> and then adding the (two) entries in the InActConn column from the output of ipvsadm. <verb> kernel DoS strategy InActConn -srcnum -packets (10k/sec) SYN cookie no secure_tcp 13,400 200,000 200,000 syn_recv=10 no none 99,400 500,000 1,000,000 yes non 70,400 1,000,000 2,000,000 </verb> <sect2>Is the number of InActConn a problem? (edited from Julian) <quote> The memory used is 128 bytes/connection and 60k connections will tie up 7M of memory. LVS does not use system sockets. LVS has its own connection table. The limit is the amount of memory you have - virtually unlimited. The masq table (by default 40960 connections per protocol). is a separate table and is used only for LVS/NAT FTP or for normal MASQ connections. </quote> However the director was quite busy during the testlvs test. Attempts to connect to other LVS'ed services (not shown in the above ipvsadm table) failed. Netpipe tests run at the same time from the client's IP (in the 192.168.1.0/24 network) stopped, but resumed at the expected rate after the testlvs run completed (i.e. but before the InActConn count dropped to 0). <sect1>Debugging LVS new way <verb> echo x > /proc/sys/net/ipv4/debug_level where 0<x<9 </verb> old way (may still work - haven't tested it) (Wensong) <verb> > Is there any way to debug/watch the path between the director and the > real-server? </verb> below the entry CONFIG_IP_MASQUERADE_VS_WLC=m in /usr/src/linux/.config add the line CONFIG_IP_VS_DEBUG=y This switch affects ip_vs.h and ip_vs.c make clean in /usr/src/linux/net/ipv4 and rebuild the kernel and modules. <verb> (other switches you will find in the code are IP_VS_ERR IP_VS_DBG IP_VS_INFO ) </verb> Look in syslog/messages for the output. The actual location of the output is determined by /etc/syslog.conf. For instance kern.* /usr/adm/kern sends kernel messages to /usr/adm/kern (re-HUP syslogd if you change /etc/syslog.conf). Here's the output when LVS is first setup with ipvsadm $ tail /var/adm/kern Nov 13 17:26:52 grumpy kernel: IP_VS: RR scheduling module loaded. <verb> ( Note CONFIG_IP_VS_DEBUG is not a debug level output, so you don't need to add *.=debug /usr/adm/debug to your syslog.conf file ) </verb> Finally check whether packets are forwarded successfully through direct routing. also you can use tcpdump to watch packets between machines. Here's some information from Ratz <tt/ratz@tac.ch/ Since some recent lvs-versions extensive debugging can be enabled to get either more information about what's exactly going on or to help you understanding the process of packet handling within the director's kernel. Be sure to have compiled in debug support for LVS (CONFIG_IP_VS_DEBUG=yes in .config) You can enable debugging by setting: echo $DEBUG_LEVEL > /proc/sys/net/ipv4/vs/debug_level where DEBUG_LEVEL is between 0 and 10. The do a tail -f /var/log/kernlog and watch the output flying by while connecting to the VIP from a CIP. If you want to disable debug messages in kernlog do: echo 0 > /proc/sys/net/ipv4/vs/debug_level If you run tcpdump on the director and see a lot of packets with the same ISN and only SYN and the RST, then either <itemize> <item>you haven't handled the <ref id="arp_problem" name="arp problem"> (most likely) <item>you're trying to connect directly to the VIP from within the cluster itself </itemize> <sect1>Security Issues The HOWTO doesn't discuss securing your LVS (we can't do everything at once). However you need to handle it someway. Roberto Nibali <tt/ratz@tac.ch/ 03 May 2001 <quote> It doesn't matter whether you're running an e-gov site or you mom's homepage. You have to secure it anyway, because the webserver is not the only machine on a net. A breach of the webserver will lead to a breach of the other systems too. </quote> from Ratz <tt/ratz@tac.ch/ The load balancer is basically on as secure as Linux itself is. ipchains settings don't affect LVS functionality (unless by mistake you use the same mark for ipchains and ipvsadm). LVS itself has some builtin security mainly to try to secure real-server in case of a DoS attack. There are several parameters you might want to set in the proc-fs. <itemize> <item>/proc/sys/net/ipv4/vs/amemthresh <item>/proc/sys/net/ipv4/vs/am_droprate <item>/proc/sys/net/ipv4/vs/drop_entry <item>/proc/sys/net/ipv4/vs/drop_packet <item>/proc/sys/net/ipv4/vs/secure_tcp <item>/proc/sys/net/ipv4/vs/debug_level With this you select the debug level (0: no debug output, >0: debug output in kernlog, the higher the number to higher the verbosity) The following are timeout settings. For more information see TCP/IP Illustrated Vol. I, R. Stevens. <item>/proc/sys/net/ipv4/vs/timeout_close - CLOSE <item>/proc/sys/net/ipv4/vs/timeout_closewait - CLOSE_WAIT <item>/proc/sys/net/ipv4/vs/timeout_established - ESTABLISHED <item>/proc/sys/net/ipv4/vs/timeout_finwait - FIN_WAIT <item>/proc/sys/net/ipv4/vs/timeout_icmp - ICMP <item>/proc/sys/net/ipv4/vs/timeout_lastack - LAST_ACK <item>/proc/sys/net/ipv4/vs/timeout_listen - LISTEN <item>/proc/sys/net/ipv4/vs/timeout_synack - SYN_ACK <item>/proc/sys/net/ipv4/vs/timeout_synrecv - SYN_RECEIVED <item>/proc/sys/net/ipv4/vs/timeout_synsent - SYN_SENT <item>/proc/sys/net/ipv4/vs/timeout_timewait - TIME_WAIT <item>/proc/sys/net/ipv4/vs/timeout_udp - UDP </itemize> You don't want your director replying to pings. <sect1>MTU discovery<label id="MTU_discovery"> (code for this was added sometime in 2000) Eric Mehlhaff wrote: <verb> > I was just updating ipchains rules and it struck me that I dont know what > LVS does with the ICMP needs-fragmentation packets required for path > MTU discovery to work. What does LVS do with such packets, when its not > immediately obvious which real server they are supposed to go to? (Wensong) Sorry that there is no LVS code to handle ICMP packets and send them to the corresponding real servers. But, I am thinking about adding some code to handle this. Date: Wed, 13 Dec 2000 22:02:43 +0000 (GMT) From: Julian Anastasov <tt/ja@ssi.bg/ > what happens with ICMP messages > specified for a Realserver. Or more exactly what happens if for example > an ICMP host unreachable messages is send to the LVS because a client > got down ? > Are the entrys from the connection table removed ? No > Are the messages forwarded to the Realservers ? </verb> Yes, the embedded TCP or UDP datagram is inspected and this information is used to forward the ICMP message to the right real server. All other messages that are not related to existing connections are accepted locally. Eric Mehlhaff <tt/mehlhaff@cryptic.com/ passed on more info Theoreticaly, path-mtu-discovery happens on every new tcp connection. In most cases the default path MTU is fine. It's weird cases (ethernet LAN conenctions with low MTU WAN connections ) that point out broken path-MTU discovery. I.e. for a while I had my home LAN (MTU 1500) hooked up via a modem connection that I had set MTU to 500 for. The minimum MTU in this case was the 500 for my home but there were many broken web sites I could not see because they had blocked out the ICMP-must-fragment packets on their servers. One can also see the effects of broken path mtu discovery on FDDI local networks. Anyway, here's some good web pages about it: <verb> http://www.freelabs.com/~whitis/isp_mistakes.html http://www.worldgate.com/~marcs/mtu/ </verb> <sect1>ICMP handling<label id="ICMP_redirects"> What happens if a real-server is connected to a client which is no longer reachable? ICMP replies go back to the VIP and will not neccessarily be forwarded to the correct real-server. Jivko Velev <tt/jiko@tremor.net/ Assume that we have TCP connections...and real server is trying to respond to the client, but it cannot reach it (the client is down, the route doesn't exist anymore, the intermadiate gateway is congested). In these cases your VIP will receive ICMP packets dest unreachable, source quench and friends. If you dont route these packets to the correct real-server you will affect performance of the LVS. For example the real-server will continue to resend packets to the client because they are not confirmed, and gateways will continue to send you ICMP packets back to VIP for every packets they droped. The TCP stack will drop these kind of connections after his timeouts expired, but if the director forwarded the ICMP packets to the appropriate real-server, this will occur a little bit earlier, and will avoid overloading the redirector with ICMP stuff. When you receive a ICMP packet it contains the full IP header of the packet that cause this ICMP to be generated + 64bytes of its data, so you can assume that you have the TCP/UDP header too. So it is possible to implement "Persitance rules" for ICMP packages. Summary: This problem was handled in kernel 2.2.12 and earlier by having the configure script turn off icmp redirects in the kernel (through the proc interface). For 2.2.13 the ipvs patch handles this. The configure script knows which kernel you are using on the director and does the Right Thing (TM). Date: Wed, 13 Dec 2000 22:02:43 +0000 (GMT) From: Julian Anastasov <tt/ja@ssi.bg/ <verb> > On Wed, 13 Dec 2000, joern maier wrote > what happens with ICMP messages specified for a Realserver. > Or more exactly what happens if for example an ICMP host > unreachable messages is send to the LVS because a client got down ? > Are the entrys from the connection table removed ? No > Are the messages forwarded to the Realservers ? </verb> Julian 13 Dec 2000 Yes, the embedded TCP or UDP datagram is inspected and this information is used to forward the ICMP message to the right real server. All other messages that are not related to existing connections are accepted locally. from a posting I picked off Dejanews by Barry Margolin <label id="icmp_redirects"> the criteria for sending a redirect are: 1. The packet is being forwarded out the same physical interface that it was received from, 2. The IP source address in the packet is on the same Logical IP (sub)network as the next-hop IP address, 3. The packet does not contain an IP source route option. Routers ignore redirects and shouldn't even be receiving them in the first place, because redirects should only be sent if the source address and the preferred router address are in the same subnet. If the traffic is going through an intermediary router, that shouldn't be the case. The only time a router should get redirects is if it's originating the connections (e.g. you do a "telnet" from the router's exec), but not when it's doing normal traffic forwarding. <verb> > Well, remember that ICMP redirects are just bandages to cover routing > problems. No one really should be routing that way. > > ICMP redirects are easily spoofed, so many systems ignore them. > Otherwise they risk having their connectivity being disconnected on whim. > Also, many systems no longer send ICMP redirects because some people > actually want to pass traffic through an intervening system! I don't know > how FreeBSD ships these days, but I suggest that it should ship with > ignore ICMP redirects as the default. </verb> <sect2>LVS code handles only needs to handle icmp redirects for VS-NAT (and not for VS-DR and VS-Tun) (Julian: 12 Jan 2001) Only for VS-NAT do the packets from the real servers hit the forward chain, i.e. the outgoing packets. VS-DR and VS-TUN receive packets only to LOCAL_IN, i.e. the FORWARD chain, where the redirect is sent, is skipped. The incoming packets for LVS/NAT use ip_route_input() for the forwarding, so they can hit the FORWARD chain too and to generate ICMP redirects after the packet is translated. So, the problem always exists for LVS/NAT, for packets in the both directions because after the packets are translated we always use ip_forward to send the packets to the both ends. I'm not sure but may be the old LVS versions used ip_route_input() to forward the DR traffic to the real servers. But this was not true for the TUN method. This call to ip_route_input() can generate ICMP redirects and may be you are right that for the old LVS versions this is a problem for DR. Looking in the Changelog it seems this change occured in LVS version 0.9.4, near Linux 2.2.13. So, in the HOWTO there is something that is true: there is no ICMP redirect problem for LVS/DR starting from Linux 2.2.13 :) But the problems remains for LVS/NAT even in the latest kernel. But this change in LVS is not created to solve the ICMP redirect problem. Yes, the problem is solved for DR but the goal was to speedup the forwarding for the DR method by skipping the forward chain. When the forward chain is skipped the ICMP redirect is not sent. ICMP redirects and LVS: (mostly from Wensong) The test setups shown in this HOWTO for VS-DR and VS-Tun have the client, director and real-servers on the same network. In production the client will connect via a router from a remote network (and for VS-Tun the real-servers could be remote and all on separate networks). The client forwards the packet for VIP to the director, the director receives the packet on the eth0 (eth0:1 is an alias of eth0), then forwards the packet to the real server through eth0. The director will think that the packet came and left through the same interface without any change, so an icmp redirect is send to the client to notify it to send the packets for VIP directly to the RIP. However, when all machines are on the same network, the client is not a router and is directly connected to the director, and ignores the icmp redirect message and the LVS works properly. If there is a router between the client and the director, and it listens to icmp redirects, the director will send an icmp redirect to the router to make it send the packet for VIP to the real server directly, the router will handle this icmp redirect message and change its routing table, then the LVS/DR won't work. The symptoms is that once the load balancer sends an ICMP redirect to the router, the router will change its routing table for VIP to the real server, then all the LVS won't work. Since you did your test in the same network, your LVS client is in the same network that the load balancer and the server are, it doesn't need to pass through a router to reach the LVS, you won't have such a symptom. :) Only when LVS/DR is used and there is only one interface to receive packets for VIP and to connect the real server, there is a need to suppress the ICMP redirects of the interface. <verb> (The ICMP redirects is turned on in the kernel 2.2 by default. The configure.pl script turns off icmp redirects on the director using sysctl echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects ) </verb> (Wensong) In the reverse direction, replies coming back from the real-server to the client <verb> |<------------------------| | real server client <--> tunlhost1=======tunlhost2 --> director ------->| </verb> After the first response packet arrives from the real-server at the tunlhost2, tunlhost2 will try to send the packet through the tunnel. If the packet is too big, then tunlhost2 will send an ICMP packet to the VIP to fragment the packet. In the previous versions of ipvs, the director won't forward the ICMP packet to (any) real server. With 2.2.13 code has been added to handle the icmp redirects and make the director forward icmp packets to the corresponding servers. <verb> > If a real-server goes down after the connection is established, will the > client get a dest_unreachable from the director? </verb> No. Here is a design issue. If the director sends an ICMP_DEST_UNREACH immediately, all tranfered data for the established connection will be lost, the client needs to establish a new connection. Instead, we would rather wait for the timeout of connection, if the real server recovers from the temporary down (such as overloaded state) before the connection expires, then the connection can continue. If the real server doesn't recover before the expire, then an ICMP_DEST_UNREACH is sent to the client. <verb> > If the client goes down after the connection is established, where do the > dest_unreachable icmp packets generated by the last router go? </verb> If the client is unreachable, some router will generate an ICMP_DEST_UNREACH packet and sent to the VIP, then the director will forward the ICMP packet to the real server. <verb> > Since icmp packets are udp, are the icmp packets routed through the > director independantly of the services that are being LVS'ed. ie if the > director is only forwarding port 80/tcp, from CIP to a particular RIP, > does the LVS code which handles the icmp forward all icmp packets from the > CIP to that RIP. What if the client has a telnet session to one real-server > and http to another real-server? </verb> It doesn't matter, because the header of the original packet is encapsulated in the icmp packet. It is easy to identify which connection is the icmp packet for. <verb> >If the client has two connections to the LVS (say telnet and http) >each to 2 different real-servers and the client goes down, the director >gets 2 ICMP_DEST_UNREACH packets. The director knows from the CIP:port >which real-server to send the icmp packet to? </verb> (Jerry Glomph Black) <verb> > The kernel debug log (dmesg) occasionally gets bursts of > messages of the following form on the LVS box: > > IP_MASQ:reverse ICMP: failed checksum from 199.108.9.188! > IP_MASQ:reverse ICMP: failed checksum from 199.108.9.188! > IP_MASQ:reverse ICMP: failed checksum from 199.108.9.188! > IP_MASQ:reverse ICMP: failed checksum from 199.108.9.188! > IP_MASQ:reverse ICMP: failed checksum from 199.108.9.188! > > What is this, is it a serious problem, and how to deal with it? > I don't think it is a serious problem. If these messages are generated, the ICMP packets must fail in checksum. Maybe the ICMP packets from 199.108.9.188 is malformed for some unknown reason. </verb> <verb> > On Fri, 21 Jan 2000, Wensong Zhang wrote: > > > No, it is not right. The director handles ICMP packets for virtual > > services long time ago, please check the ChangeLog of the code. > > from ChangeLog for 0.9.3-2.2.13 > > The incoming ICMP packets for virtual services will be forwarded > to the right real servers, and outgoing ICMP packets from virtual > services will be altered and send out correctly. This is important > for error and control notification between clients and servers, > such as the MTU discovery. > > If a realserver goes down after the connection is established, will the > client get a dest_unreachable from the director? </verb> No. Here is a design issue. If the director sends an ICMP_DEST_UNREACH immediately, all tranfered data for the established connection will be lost, the client needs to establish a new connection. Instead, we would rather wait for the timeout of connection, if the real server recovers from the temporary down (such as overloaded state) before the connection expires, then the connection can continue. If the real server doesn't recover before the expire, then an ICMP_DEST_UNREACH is sent to the client. <verb> > If the client goes down after the connection is established, where do the > dest_unreachable icmp packets generated by the last router go? </verb> If the client is unreachable, some router will generate an ICMP_DEST_UNREACH packet and sent to the VIP, then the director will forward the ICMP packet to the real server. <verb> > Since icmp packets are udp, are the icmp packets routed through the > director independantly of the services that are being LVS'ed. ie if the > director is only forwarding port 80/tcp, from CIP to a particular RIP, > does the LVS code which handles the icmp forward all icmp packets from the > CIP to that RIP. What if the client has a telnet session to one realserver > and http to another realserver? </verb> It doesn't matter, because the header of the original packet is encapsulated in the icmp packet. It is easy to identify which connection is the icmp packet for. <sect1>Filesystems for real-server content: the many reader, single writer problem The client can be assigned to any real-server. One of the assumptions of LVS is that all real-servers have the same content. This assumption is easy to fullfill for services like http, where the administrator updates the files on all real-servers when needed. For services like mail or databases, the client writes to storage on one real-server. The other real-servers do not see the updates unless something intervenes. Various tricks are described elsewhere here for mailservers and databases. These require the real-servers to write to common storage (for mail the mailspool is nfs mounted; for databases, the LVS client connects to a database client on each real-server and these database clients write to a single databased on a backend machine, or the databased's on each real-server are capable of replicating). One solution is to have a file system which can propagate changes to other real-servers. We have mentioned gfs and coda in several places in this HOWTO as holding out hope for the future. People now have these working. Wensong Zhang <tt/wensong@gnuchina.org/ 05 May 2001 <quote> It seems to me that <url url="http://www.coda.cs.cmu.edu" name="Coda"> is becoming quite stable. I have run coda-5.3.13 with the root volume replicated on two coda file servers for near two months, I haven't met problem which need manual maintance until now. BTW, I just use it for testing purposes, it is not running in production site. </quote> Mark Hlawatschek <tt/hlawatschek@atix.de/ 2001-05-04 <quote> we made some good experiences with the use of <url url="http://www.globalfilesystem.org/" name="GFS"> (now moved to <url url="http://www.sistina.com/gfs/" name="GFS">. We are using LVS in conjunction with the GFS for about one year in older versions and it worked quite stable. We successfully demonstrated the solution with a newer version of GFS (4.0) at the CEBit 2001. Several domains (i.e. http://www.atix.org) will be served by the new configuration next week. </quote> Mark's slides from his <url url="http://ace73.atix.de/downloads/santime/decus-hlawatschek-release.pdf" name="talk in German at DECUS in Berlin (2001)"> is available. <sect>Failover protection<label id="failover"> Don't even think about doing this till you've got your LVS working properly. If you want the LVS to survive a server or director failure, you can add software to do this after you have the LVS working. For production systems, failover may be useful or required. On director or real-server failure, the connection/session from the client to the real-server is lost. Preserving session information is a difficult problem. Do not expect a solution real soon. After failover, the client will be presented with a new connection. Unlike on a single server, only some of the clients will experience loss of connection. The most common problem here is loss of network access or extreme slowdown. Hardware failure or an OS crash (on unix) is less likely. However redundancy of services on the real-servers is one of the useful features of LVS. One machine/service can be removed from the functioning virtual server for upgrade or moving of the machine and can be brought back on line later without interruption of service to the client. <sect1>Director failure What happens if the director dies? The usual solution is make a duplicate director(s) and to run heartbeat between them. One director defaults to being the operational director and the other takes over when heartbeat detects that the default director has died. <verb> ________ | | | client | |________| | | (router) | | ___________ | ___________ | | | DIP | | | director1 |-----|------| director2 | |___________| | VIP |___________| | <- heartbeat-> | |---------- | ----------| | ------------------------------------ | | | | | | RIP1, VIP RIP2, VIP RIP3, VIP ______________ ______________ ______________ | | | | | | | real-server1 | | real-server2 | | real-server3 | |______________| |______________| |______________| </verb> The <url url="http://ultramonkey.sourceforge.net" name="Ultra Monkey project"> uses Heartbeat from the Linux-HA project and ldirectord to monitor the real-servers. Fake, heartbeat and mon are available at the <url url="http://linux-ha.org/download" name="Linux High Availability site">. The setup of Ultra Monkey is also covered on the LVS website. <sect2>Two box HA LVS Doug Sisk <tt/sisk@coolpagehosting.com/ 19 Apr 2001 <quote> Is it possible to create a two server LVS with fault tolerance? It looks straight forward with 4 servers ( 2 Real and 2 Directors), but can it be done with just two boxes, ie directors, each director being a real-server for the other director and a real-server running localnode for itself? </quote> Horms Take a look at ultramonkey.org, that should give you all the bits you need to make it happen. You will need to configure heartbeat on each box, and then LVS (ldirectord) on each box to have two real servers: the other box, and localhost. <sect1>seemless director failover: saving connection state info For true director failover, all connection state information from the failed director should be transferred to the new director. There is a lot of this and it's changing all the time. This is a similar problem to backing up a hot database. People have discussed ways of doing this that are likely to work and don't consume all the CPU resources on the director, but no-one seems terribly enthused about starting work on it. Director failure doesn't happen real often (and hopefully is planned) and you're only going to loose a few sessions at most. Lars Marowski-Bree <tt/lmb@suse.de/ <quote> The <ref id="scheduler" name="-sh and -dh schedulers"> in 2.4 should make this possible, as there is no state information to transfer ;-) </quote> <sect1>Real-server failure An agent external to the ipvs code on the director is used to monitor the services. LVS itself can't monitor the services as LVS is just a packet switcher. If a real-server fails, the director doesn't get the failure, the client does. For the standard VS-Tun and VS-DR setups (ie receiving packets by an ethernet device and not by TP), the reply packets from the real-server go to its default gw and don't go through the director, so the LVS can't detect failure even if it wants to. For some of the mailings concerning why the LVS does not monitor itself and why an external agent (eg mon) is used instead, see the postings on <ref id="agent" name="external agents">. In a failure protected LVS, if the real-server at the end of a connection fails, the client will loose their connection to the LVS, and the client will have to start with a new connection, as would happen on a regular single server. With a failure protected LVS, the failed real-server will be switched out of the LVS and a working new server will be made available to you transparently (the client will connect to one of the still working servers, or possibly a new server if one is brought on-line). Here the functioning and setup of "mon" is described. In the Ultra Monkey version of LVS, ldirectord fills the same role. (I haven't compared ldirectord and mon. I used mon because it was available at the time, while ldirectord was either not available or I didn't know about it.) The configure script will setup mon to monitor the services on the real-servers. Get "mon" and "fping" from http://www.kernel.org/software/mon/ (I'm using mon-0.38.20) (from ian.martins@mail.tju.edu - comment out line 222 in fping.c if compiling under glibc) Get the perl package "Period" from CPAN, ftp://ftp.cpan.org) To use the fping and telnet monitors, you'll need the tcp_scan binary which can be built from satan. The standard version of satan needs patches to compile on Linux. The patched version is at ftp://sunsite.unc.edu/pub/Linux/system/network/admin <sect1>Service/real-server failout To activate real-server failover, you can install mon on the director. Several people have indicated that they have written/are using other schemes. RedHat's piranha has monitoring code, and handles director failover and is documented there. ldirectord handles director failover and is part of the Linux High Availability project. ldirectord needs Net::SSLeay only if you are monitoring https (Emmanuel Pare <tt/emman@voxtel.com/, Ian S. McLeod <tt/ian@varesearch.com/) To get ldirectord - From: <tt/jacob.rief@tis.at/ <verb> the newest version available from cvs.linux-ha.org:/home/cvs/ user guest, passwd guest module-name is: ha-linux file: ha-linux/heartbeat/resource.d/ldirectord documentation: ha-linux/doc/ldirectord </verb> Here's a possible alternative to mon - From: Doug Bagley <tt/doug@deja.com/ 17 Feb 2000 Looking at mon and ldirectord, I wonder what kind of work is planned for future service level monitoring? mon is okay for general purposes, but it forks/execs each monitor process, if you have 100 real services and want to check every 10 seconds, you would fork 10 monitor processes per second. This is not entirely untenable, but why not make an effort to make the monitoring process as lightweight as possible (since it is running on the director, which is such an important host)? ldirectord, uses the perl LWP library, which is better than forking, but it is still slow. It also issues requests serially (because LWP doesn't really make parallel requests easy). I wrote a very simple http monitor last night in perl that uses non-blocking I/O, and processes all requests in parallel using select(). It also doesn't require any CPAN libraries, so installation should be trivial. Once it is prototyped in perl, conversion to C should be straightforward. In fact, it is pretty similar to the Apache benchmark program (ab). In order for the monitor (like ldirectord) to do management of the ipvs kernel information, it would be easier if the /proc interface to ipvs gave a more machine readable format. From: Michael Sparks <tt/zathras@epsilon3.mcc.ac.uk/ Agreed :-) <verb> It strikes me that rather than having: type serviceIP:port mechanism -> realip:port tunnel weight active inactive -> realip:port tunnel weight active inactive -> realip:port tunnel weight active inactive -> realip:port tunnel weight active inactive </verb> If the table was more like: <verb> type serviceIP:port mechanism realip:port tunnel weight active inactive </verb> Then this would make shell/awk/perl/etc scripts that do things with this table easier to cobble together. <verb> > That seems like a far reaching precedent to me. On the other hand, if > the ipvsadm command wished to have a option to represent that > information in XML, I can see how that could be useful. </verb> This reminds me I should really finish tweaking the prog I wrote to allow pretty printing of the ipvsadm table, and put it somewhere else for others to play with if they like - it allows you to specify a template file for formatting the output of of ipvsadm, making displaying the stuff as XML, HTML, plain text, etc simpler/quicker. (It's got a few hardcoded settings at the mo which I want to ditch first :-) <sect1>Mon for server/service failout Here's the prototype LVS <verb> ________ | | | client | |________| | | (router) | | | __________ | DIP | | |------| director | | VIP |__________| | | | ------------------------------------ | | | | | | RIP1, VIP RIP2, VIP RIP3, VIP ______________ ______________ ______________ | | | | | | | real-server1 | | real-server2 | | real-server3 | |______________| |______________| |______________| </verb> Mon has two parts: <itemize> <item>monitors: these are programs (usually perl scripts) which are run periodically to detect the state of a service. E.g. the telnet monitor attempts to login to the machine of interest and checks whether a program looking like telnetd responds (in this case looking for the string "login:"). The program returns success/fail. Monitors have been written for many services and new monitors are easy to write. <item> Mon demon: reads a config file, specifying the hosts/services to monitor and how often to poll them (with the monitors). The conf file lists the actions for failure/success of each host/service. When a failure (or recovery) of a service is detected by a monitor, an "alert" (another perl script) is run. There are alerts which send email, page you or write to a log. LVS supplies a "virtualserver.alert" which executes ipvsadm commands to remove or add servers/services, in response to host/services changing state (up/down). </itemize> <sect1>BIG CAVEAT *Trap for the unwary* Mon runs on the director, but... Remember that you cannot connect to any of the LVS controlled services from within the LVS (including from the director) (see <ref id="gotcha" name="gotchas">). You can only connect to the LVS'ed services from the outside (eg from the client). If you are on the director, the packets will not return to you and the connection will hang. If you are connecting from the outside (ie from a client) you cannot tell which server you have connected to. This means that mon (or any agent), running on the director (which is where is needs to be to execute ipvsadm commands), cannot tell whether an LVS controlled service is up or down. With VS-NAT an agent on the director can access services on the RIP of the real-servers (on the director you can connect to the http on the RIP of each real-server). Normal (i.e. non LVS'ed) IP communication is unaffected on the private director/real-server network of VS-NAT. If ports are not re-mapped then a monitor running on the director can watch the httpd on server-1 (at 10.1.1.2:80). If the ports are re-mapped (eg the httpd server is listening on 8080), then you will have to either modify the http.monitor (making an http_8080.monitor) or activate a duplicate http service on port 80 of the server. For VS-DR and VS-Tun the service on the real-server is listening to the VIP and you cannot connect to this from the director. The solution to monitoring services under control of the LVS for VS-DR and VS-Tun is to monitor proxy services whose accessability should closely track that of the LVS service. Thus to monitor an LVS http service on a particular server, the same webpage should also be made available on another IP (or to 0.0.0.0), not controlled by LVS on the same machine. Example: <verb> VS-Tun, VS-DR lvs IP (VIP): eth0 192.168.1.110 director: eth0 192.168.1.1/24 (normal login IP) eth1 192.168.1.110/32 (VIP) real-server: eth0 192.168.1.2/24 (normal login IP) tunl0 (or lo:0) 192.168.1.110/32 (VIP) </verb> On the real-server, the LVS service will be on the tunl (or lo:0) interface of 192.168.1.110:80 and not on 192.168.1.2:80. The IP 192.168.1.110 on the real-server 192.168.1.2 is a non-arp'ing device and cannot be accessed by mon. Mon running on the director at 192.168.1.1 can only detect services on 192.168.1.2 (this is the reason that the director cannot be a client as well). The best that can be done is to start a duplicate service on 192.168.1.2:80 and hope that its functionality goes up and down with the service on 192.168.1.110:80 (a reasonable hope). <verb> VS-NAT lvs IP (VIP): eth0 192.168.1.110 director: eth0 192.168.1.1/24 (outside IP) eth0:1 192.168.1.110/32 (VIP) eth1 10.1.1.1/24 (DIP, default gw for real-servers) real-server: eth0 10.1.1.2/24 </verb> Some services listen to 0.0.0.0:port, ie will listen on all IPs on the and you will not have to start a duplicate service. <sect1>About Mon Mon doesn't know anything about the LVS, it just detects the up/down state of services on remote machines and will execute the commands you tell it when the service changes state. We give Mon a script which runs ipvsadm commands to remove services from the ipvsadm table when a service goes down and another set of ipvsadm commands when the service comes back up. Good things about Mon: <itemize> <item>It's independant of LVS, ie you can setup and debug the LVS without Mon running. <item>You can also test mon independantly of LVS. <item>The monitors and the demon are independant. <item>Most code is in perl (one of he "run anywhere" languages) and code fixes are easy. </itemize> Bad things about Mon: <itemize> <item>I upgraded to 0.38.20 but it does't run properly. I downgraded back to v0.37l. (Mar 2001 - Well I was running 0.37l. I upgraded to perl5.6 and some of the monitors/alerts don't work anymore. Mon-0.38.21 seems to work, with minor changes in the output and mon.cf file.) <item>the author doesn't reply to e-mail. </itemize> mon-0.37l, keeps executing alerts every polling period following an up-down transition. Since you want your service polled reasonable often (eg 15secs), this means you'll be getting a pager notice/email every 15secs once a service goes down. Tony Bassette <tt/kult@april.org/ let me know that mon-0.38 has a numalert command limiting the number of notices you'll get. <sect1>Mon Install Mon is installed on the director. Most of mon is a set of perl scripts. There are few files to be compiled - it is mostly ready to go (rpc.monitor needs to be compiled, but you don't need it for LVS). You do the install by hand. <verb> $ cd /usr/lib $ tar -zxvof /your_dirpath/mon-x.xx.tar.gz </verb> this will create the directory /usr/lib/mon-x.xx/ with mon and its files already installed. LVS comes with virtualserver.alert (goes in alert.d) and ssh.monitor (goes in mon.d). Make the directory "mon-x.xx" accessable as "mon" by linking it to "mon" or by renaming it <verb> $ln -s mon-x.xx mon or $mv mon-x.xx mon </verb> Copy the man files (mon.1 etc) into /usr/local/man/man1 Check that you have the perl packages required for mon to run $perl -w mon do the same for all the perl alerts and monitors that you'll be using (telnet.monitor, dns.monitor, http_t.monitor, ssh.monitor). DNS in /etc/services is known as "domain" and "nameserver" but not "dns". To allow the use of the string "dns" in the lvs_xxx.conf files and to enable configure_lvs.pl to autoinclude the dns.monitor, add the string "dns" to the port 53 services in /etc/services with an entry like <verb> domain 53/tcp nameserver dns # name-domain server domain 53/udp nameserver dns </verb> Mon expects executables to be in /bin, /usr/bin or /usr/sbin. The location of perl in the alerts is #!/usr/bin/perl (and not /usr/local/bin/perl) - make sure this is compatible with your setup. (Make sure you don't have one version of perl in /usr/bin and another in /usr/local/bin). The configure script will generate the required mon.cf file for you (and if you like copy it to the cannonical location of /etc/mon). Add an auth.cf file to /etc/mon I use <verb> #auth.cf ---------------------------------- # authentication file # # entries look like this: # command: {user|all}[,user...] # # THE DEFAULT IS TO DENY ACCESS TO ALL IF THIS FILE # DOES NOT EXIST, OR IF A COMMAND IS NOT DEFINED HERE # list: all reset: root loadstate: root savestate: root term: root stop: root start: root set: root get: root dump: root disable: root enable: root #end auth.cf ---------------------------- </verb> <sect1>Mon Configure This involves editing /etc/mon/mon.cf, which contains information about <itemize> <item>nodes monitored <item>how to detect if a node:service is up (does the node ping, does it serve http...?) <item>what to do when the node goes down and what to do later when it comes back up. </itemize> The mon.cf generated by configure_lvs.pl <itemize> <item> assigns each node to its own group (nodes are brought down one at a time rather than in groups - I did this because it was easier to code rather than for any good technical reason). <item>detects whether a node is serving some service (eg telnet/http) selecting, if possible, a monitor for that service, otherwise defaulting to fping.monitor which detects whether the node is pingable. <item>on failure of a real-server, mon sends mail to root (using mail.alert) and removes the real-server from the ipvsadm table (using virtualserver.alert). <item>on recovery sends mail to root (using mail.alert) and adds the real-server back to the pool of working real-servers in the ipvsadm table (using virtualserver.alert). </itemize> <sect1>Testing mon without LVS The instructions here show how to get mon working in two steps. First show that mon works independantly of LVS, then second bring in LVS. The example here assumes a working VS-DR with one real-server and the following IPs. VS-DR is chosen for the example here as you can set up VS-DR with all machines on the same network. This allows you to test the state of all machines from the client (ie using one kbd/monitor). (Presumably you could do it from the director too, but I didn't try it.) <verb> lvs IP (VIP): eth0 192.168.1.110 director: eth0 192.168.1.1/24 (admin IP) eth0:1 192.168.1.110/32 (VIP) real-server: eth0 192.168.1.8/24 </verb> On the director, test ping.monitor (in /usr/lib/mon/mon.d) with <verb> $ ./fping.monitor 192.168.1.8 </verb> You should get the command prompt back quickly with no other output. As a control test for a machine that you know is not on the net <verb> $ ./fping.monitor 192.168.1.250 192.168.1.250 </verb> ping.monitor will wait for a timeout (about 5secs) and then return the IP of the unpingable machine on exit. Check test.alert (in /usr/lib/mon/alert.d) - it writes a file in /tmp $ ./test.alert foo you will get the date and "foo" in /tmp/test.alert.log As part of generating the rc.lvs_dr script, you will also have produced the file mon_lvsdr.cf. To test mon, place this in /etc/mon/mon.cf <verb> #------------------------------------------------------ #mon.cf # #mon config info, you probably don't need to change this very much # alertdir = /usr/lib/mon/alert.d mondir = /usr/lib/mon/mon.d #maxprocs = 20 histlength = 100 #delay before starting #randstart = 60s #------ hostgroup LVS1 192.168.1.8 watch LVS1 #the string/text following service (to OEL) is put in header of mail messages #service "http on LVS1 192.168.1.8" service fping interval 15s #monitor http.monitor #monitor telnet.monitor monitor fping.monitor allow_empty_group period wd {Sun-Sat} #alertevery 1h #alert mail.alert root #upalert mail.alert root alert test.alert upalert test.alert #-V is virtual service, -R is remote service, -P is protocol, -A is add/delete (t|u) #alert virtualserver.alert -A -d -P -t -V 192.168.1.9:21 -R 192.168.1.8 #upalert virtualserver.alert -A -a -P -t -V 192.168.1.9:21 -R 192.168.1.8 -T -m -w 1 #the line above must be blank #mon.cf--------------------------- </verb> Now we will test mon on the real-server 192.168.1.8 independantly of LVS. Edit /etc/mon/mon.cf and make sure that all the monitors/alerts except for fping.monitor and test.alert are commented out (there is an alert/upalert pair for each alert, leave both uncommented for test.alert). Start mon with rc.mon (or S99mon) Here is my rc.mon (copied from the mon package) <verb> # rc.mon ------------------------------- # You probably want to set the path to include # nothing but local filesystems. # echo -n "rc.mon " PATH=/bin:/usr/bin:/sbin:/usr/sbin export PATH M=/usr/lib/mon PID=/var/run/mon.pid if [ -x $M/mon ] then $M/mon -d -c /etc/mon/mon.cf -a $M/alert.d -s $M/mon.d -f 2>/dev/null #$M/mon -c /etc/mon/mon.cf -a $M/alert.d -s $M/mon.d -f fi #-end-rc.mon---------------------------- </verb> After starting mon, check that mon is in the ps table (ps -auxw | grep perl). When mon comes up it will read mon.cf and then check 192.168.1.8 with the fping.monitor. On finding that 192.168.1.8 is pingable, mon will run test.alert and will enter a string like Sun Jun 13 15:08:30 GMT 1999 -s fping -g LVS3 -h 192.168.1.8 -t 929286507 -u -l 0 into /tmp/test.alert.log. This is the date, the service (fping), the hostgroup (LVS), the host monitored (192.168.1.8), unix time in secs, up (-u) and some other stuff I didn't need to figure out to get everything to work. Check for the "-u" in this line, indicating that 192.168.1.8 is up. If you don't see this file within 15-30secs of starting mon, then look in /var/adm/messages and syslog for hints as to what failed (both contain extensive logging of what's happening with mon). (Note: syslog appears to be buffered, it may take a few more secs for output to appear here). If neccessary kill and restart mon <verb> $ kill -HUP `cat /var/run/mon.pid` </verb> Then pull the network cable on machine 192.168.1.8. In 15secs or so you should hear the whirring of disks and the following entry will appear in /tmp/test.alert.log Sun Jun 13 15:11:47 GMT 1999 -s fping -g LVS3 -h 192.168.1.8 -t 929286703 -l 0 Note there is no "-u" near the end of the entry indicating that the node is down. Watch for a few more entries to appear in the logfile, then connect the network cable again. A line with -u should appear in the log and no further entries should appear in the log. If you've got this far, mon is working. Kill mon and make sure root can send himself mail on the director. Make sure sendmail can be found in /usr/lib/sendmail (put in a link if neccessary). Next activate mail.alert and telnet.monitor in /etc/mon/mon.cf and comment out test.alert. (Do not restart mon yet) Test mail.alert by doing <verb> $ ./mail.alert root hello ^D </verb> root is the address for the mail, hello is some arbitrary STDIN and controlD exits the mail.alert. Root should get some mail with the string "ALERT" in the subject (indicating that a machine is down). Repeat, this time you are sending mail saying the machine is up (the "-u") <verb> $ ./mail.alert -u root hello ^D </verb> Check that root gets mail with the string "UPALERT" in the subject (indicating that a machine has come up). Check the telnet.monitor on a machine on the net. You will need tcp_scan in a place that perl sees it. I moved it to /usr/bin. Putting it in /usr/local/bin (in my path) did not work. <verb> $ ./telnet.monitor 192.168.1.8 </verb> the program should exit with no output. Test again on a machine not on the net <verb> $ ./telnet.monitor 192.168.1.250 192.168.1.250 </verb> the program should exit outputting the IP of the machine not on the net. Start up mon again (eg with rc.mon or S99mon), watch for one round of mail sending notification that telnet is up (an "UPALERT) (note: for mon-0.38.21 there is no initial UPALERT). There should be no further mail while the machine remains telnet-able. Then pull the network cable and watch for the first ALERT mail. Mail should continue arriving every mon time interval (set to 15secs in mon_lvs_test.cf). Then plug the network cable back in and watch for one UPALERT mail. If you don't get mail, check that you re-edited mon.cf properly and that you did kill and restart mon (or you will still be getting test.alerts in /tmp). Sometimes it takes a few seconds for mail to arrive. If this happens you'll get an avalanche when it does start. If you've got here you are really in good shape. Kill mon (kill `cat /var/run/mon.pid`) <sect1>Can virtualserver.alert send commands to LVS? (virtualserver.alert is a modified version of Wensong's orginal file, for use with 2.2 kernels. I haven't tested it back with a 2.0 kernel. If it doesn't work and the original file does, let me know) run virtualserver.alert (in /usr/lib/mon/alert.d) from the command line and check that it detects your kernel correctly. $ ./virtualserver.alert you will get complaints about bad ports (which you can ignore, since you didn't give the correct arguments). If you have kernel 2.0.x or 2.2.x you will get no other output. If you get unknown kernel errors, send me the output of `uname -r`. Debugging print statements can be uncommented if you need to look for clues here. Make sure you have a working VS-DR LVS serving telnet on a real-server. If you don't have the telnet service on real-server 192.168.1.8 then run $ipvsadm -a -t 192.168.1.110:23 -r 192.168.1.8 then run ipvsadm in one window. $ipvsadm and leave the output on the screen. In another window run $ ./virtualserver.alert -V 192.168.1.110:23 -R 192.168.1.8 this will send the down command to ipvsadm. The entry for telnet on real-server 192.168.1.8 will be removed (run ipvsadm again). Then run $ ./virtualserver.alert -u -V 192.168.1.110:23 -R 192.168.1.8 and the telnet service to 192.168.1.8 will be restored in the ipvsadm table. <sect1>Running mon with LVS Connect all network connections for the LVS and install a VS-DR LVS with INITIAL_STATE="off" to a single telnet real-server. Start with a file like lvs_dr.conf.single_telnet_off adapting the IP's for your situation and produce the mon_xxx.cf and rc.lvs_xxx file. Run rc.lvs_xxx on the director and then the real-server. The output of ipvsadm (on the director) should be <verb> grumpy:/etc/mon# ipvsadm IP Virtual Server (Version 0.9) Protocol Local Address:Port Scheduler -> Remote Address:Port Forward Weight ActiveConn FinConn TCP 192.168.1.110:23 rr </verb> showing that the scheduling (rr) is enabled, but with no entries in the ipvsadm routing table. You should NOT be able to telnet to the VIP (192.168.1.110) from a client. Start mon (it's on the director). Since the real-server is already online, mon will detect a functional telnet on it and trigger an upalert for mail.alert and for virtualserver.alert. At the same time as the upalert mail arrives run ipvsadm again. You should get <verb> grumpy:/etc/mon# ipvsadm IP Virtual Server (Version 0.9) Protocol Local Address:Port Scheduler -> Remote Address:Port Forward Weight ActiveConn FinConn TCP 192.168.1.110:23 rr -> 192.168.1.8:23 Route 1 0 0 </verb> which shows that mon has run ipvsadm and added direct routing of telnet to real-server 192.168.1.8. You should now be able to telnet to 192.168.1.110 from a client and get the login prompt for machine 192.168.1.8. Logout of this telnet session, and pull the network cable to the real-server. You will get a mail alert and the entry for 192.168.1.8 will be removed from the ipvsadm output. Plug the network cable back in and watch for the upalert mail and the restoration of LVS to the real-server (run ipvsadm again). If you want to, confirm that you can do this for http instead of telnet. You're done. Congratulations. You can use the mon_xxx.cf files generated by configure.pl from here. <sect1>Why is the LVS monitored for failures by an external agent rather than by the kernel?<label id="agent"> (Patrick Kormann <tt/pkormann@datacomm.ch/) <verb> > Wouldn't it be nice to have a > switch that would tell ipvsadm 'If one of the real-servers is > unreachable/connection refused, take it out of the list of real servers for > x seconds' or even 'check the availability of that server every x seconds, > if it's not available, take it out of the list, if it's available again, put > it in the list'. </verb> (Lars) That does not belong in the kernel. This is definetely the job of a userlevel monitoring tool. I admit it would be nice if the LVS patch could check if connections directed to the real-server were refused and would log that to userlevel though, so we could have even more input available for the monitoring process. <verb> > and quirks to make lvs a real high-availability system. The problem is that > all those external checks are never as effective as a decition be the > 'virtual server' could be. </verb> That's wrong. A userlevel tool can check reply times, request specific URLs from the servers to check if they reply with the expected data, gather load data from the real servers etc. This functionality is way beyond kernel level code. (Michael Sparks <tt/zathras@epsilon3.mcc.ac.uk/) Personally I think monitoring of systems is probably one of the things the lvs system shouldn't really get into in it's current form. My rationale for this is that LVS is a fancy packet forwarder, and in that job it excels. For the LVS code to do more than this, it would require for TCP services the ability to attempt to connect to the *service* the kernel is load balancing - which would be a horrible thing for a kernel module to do. For UDP services it would need to do more than pinging... However, in neither case would you have a convincing method for determining if the *services* on those machines was still running effectively, unless you put a large amouunt of protocol knowledge into the kernel. As a result, you would still need to have external monitoring systems to find out whether the services really are working or not. For example, in the pathological case (of many that we've seen :-) of a SCSI subsystem failure resulting in indestructable inodes on a cache box, a cache box can reach total saturation in terms of CPU usage, but still respond correctly to pings and TCP connections. However nothing else (or nothing much) happens due to the effective system lockup. The only way round such a problem is to have a monitoring system that knows about this sort of failure, and can then take the service out. There's no way this sort of failure could be anticipated by anyone, so putting this sort of monitoring into the kernel would create a false illusion of security - you'd still need an auxillary monitoring system. Eg - it's not just enough for the kernel to mark the machine out of service - you need some useful way of telling people what's gone wrong (eg setting off people's pager's etc), and again, that's not a kernel thing. (Lars) Michael, I agree with you. However, it would be good if LVS would log the failures it detects. ie, I _think_ it can notice if a client receives a port unreachable in response to a forwarded request if running masquerading, however it cannot know if it is running DR or tunneling because in that case it doesn't see the reply to the client. (Wensong) > _think_ it can notice if a client receives a port unreachable in response to a Currently, the LVS can handle ICMP packets for virtual services, and forward them to the right place. It is easy to set the weight of the destination zero or temperarily remove the dest entry directly, if an PORT_UNREACH icmp from the server to the client passes through the LVS box. If we want the kernel to notify monitoring software that a real server is down in order to let monitoring software keep consistent state of virtual service table, we need design efficient way to notify monitoring, more code is required. Anyway, there is a need to develop efficient communication between the LVS kernel and monitoring software, for example, monitoring software get the connection number efficiently, it is time-consuming to parse the big IPVS table to get the connection numbers; how to efficiently support ipvsadm -L <protocol, ip, port>? it is good for applications like Ted's 1024 virtual services. I checked the procfs code, it still requires one write_proc and one read_proc to get per virtual service print, it is a little expensive. Any idea? <verb> (Julian Anastasov <tt/uli@linux.tu-varna.acad.bg/) > Currently, the LVS can handle ICMP packets for virtual services, and > forward them to the right place. It is easy to set the weight of the > destination zero or temperarily remove the dest entry directly, if an > PORT_UNREACH icmp from the server to the client passes through the LVS > box. </verb> PORT_UNREACH can be returned when the packet is rejected from the real server's firewall. In fact, only UDP returns PORT_UNREACH when the service is not running. TCP returns RST packet. We must carefully handle this (I don't know how) and not to stop the real server for all clients if we see that one client is rejected. And this works only if the LVS box is default gw for the real servers, i.e. for any mode: MASQ(it's always def gw), DROUTE and TUNNEL (PROT_UNREACH can be one of the reasons not to select other router for the outgoing traffic for these two modes). But LVS cn't detect the outgoing traffic for DROUTE/TUNNEL mode. For TUNNEL it can be impossible if the real servers are not on the LAN. So, the monitoring software can solve more problems. The TCP stack can return PORT_UNREACH but if the problem with the service in the real server is more complex (real server died, daemon blocked) we can't expect PORT_UNREACH. It is send only when the host is working but the daemon is stooped. Please restart this daemon. So, don't rely on the real server, in most of the cases he can't tell "Please remove me from the VS configuration, I'm down" :) This is job for the monitoring software to exclude the destinations and even to delete the service (if we switch to local delivery only, i.e. when we switch from LVS to WEB only mode for example). So, I vote for the monitoring software to handle this :) (Wensong) Yeah, I prefer that monitoring software handles this too, because it is a unified approach for VS-NAT, VS-Tun and VS-DR, and monitoring software can detect more failures and handle more things according to the failures. What we discuss last time is that the LVS kernel sets the destination entry unavailable in virtual server table if the LVS detect some icmp packets (only for VS-NAT) or RST packet etc. This approach might detect this kinds of problems just a few seconds earlier than the monitoring software, however we need more code in kernel to notify the monitoring software that kernel changes the virtual server table, in order to let the monitoring software keep the consistent view of the virtual server table as soon as possible. Here is a tradeoff. Personally, I prefer to keeping the system simple (and effective), only one (monitoring software) makes decision and keeps the consistent view of VS table. <sect>Misc/FAQ/Wisdom from the mailing list<label id="Wisdom"> These topics were too short or not central enough to LVS operation to have their own section. <sect1>Setting up an LVS with inetd from Ratz<tt/ratz@tac.ch/ We're going to set up a LVS cluster from scratch. you need <itemize> <item>4 machines (2 real-server, 1 load balancer, 1 client) wired like described in various sketches throughout this howto. <item>fun and some spare time (actually quite some if it doesn't work out the first time like described) </itemize> The goal is to set up an own loadbalanced tcp application. The application will consist of a own written shell script being invoked by inetd. As you might have guessed, security is very low priority, you should get the idea behind this. Of course I should take xinetd and of course I should use a tcpwrapper and maybe even SecurID authentication but here the goal is to understand the fundamental design principals of a LVS cluster and its deploy. All instructions will be done as root. Setting up the real-server <verb> Edit /etc/inetd.conf and add following line: lvs-test stream tcp nowait root /usr/bin/lvs-info lvs-info Edit /etc/services and add following line: lvs-test 31337/tcp # supersecure lvs-test port </verb> Now you need to get inetd running. This is different for every Unix. So please have a look at it yourself. You verify if it's running with 'ps ax|grep [i]netd' And to verify if it really runs this port you do a 'netstat -an|grep LISTEN' and if there is a line: <verb> tcp 0 0 0.0.0.0:31337 0.0.0.0:* LISTEN </verb> you're one step closer to the truth. Now we have to supply the script that will be called if you connect to real-server# port 31337. So simply do this on your command line (copy 'n' paste): <verb> cat > /usr/bin/lvs-info << EOF && chmod 755 /usr/bin/lvs-info #!/bin/sh echo "This is a test of machine `ifconfig -a | grep HWaddr | awk '{print $1}'`" echo EOF </verb> Now you can test if it really works with telnet or netcat: <verb> telnet localhost 31337 netcat localhost 31337 </verb> This should spill out something like: <verb> hog:/ # netcat localhost 31337 This is a test of machine 192.168.1.11 hog:/ # </verb> If it worked, do the same procedure to set up the second real-server. Now we're ready to set up the load balancer. These are the required commands to set it up for our example: <verb> ipvsadm -A -t 192.168.1.100:31337 -s wrr ipvsadm -a -t 192.168.1.100:31337 -r 192.168.1.11 -g -w 1 ipvsadm -a -t 192.168.1.100:31337 -r 192.168.1.12 -g -w 1 </verb> Check it with ipvsadm -L -n: <verb> hog:~ # ipvsadm -L -n IP Virtual Server version 0.9.14 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.1.100:31337 wrr -> 192.168.1.12:31337 Route 1 0 0 -> 192.168.1.11:31337 Route 1 0 0 hog:~ # </verb> Now if you connect from outside with the client node to the VIP=192.168.1.100 you should get to one of the two real-server (presumably to ~.12) Reconnect to the VIP again an you should get to the other real-server. If so, be happy, if not go back, check netstat -an, ifconfig -a, arp-problem, routing tables and so on ... <sect1>How to bring down a real-server for maintenance (eg swap disks) <verb> >From Michael Sparks <tt/sparks@mcc.ac.uk/ > I want to use virtual server functionality to allow switching over > from one pool of server processes to another without an interruption > in service to clients. current real-servers : A,B,C servers to swap into the system instead D,E,F * Add servers D,E,F into the system all with fairly high weights (perhaps ramping the weights up slowly so as not to hit them too hard:-) * Change the weights of servers A,B,C to 0. * All new traffic should now go to D,E,F * When the number of connections through A,B,C reaches 0, remove them from the service. This can take time I know but... </verb> from Joe A planned feature for ipvsadm will be to give a real-server a weight of 0 (now implemented). This real-server will not be sent any new connections and will continue serving its current connections till they close. You may have to wait a while if a user is downloading a 40M file from the real-server. <sect1>Howto turn your single node ftp/http server into an LVS without taking it off-line eg if you want to test LVS on your BIG Sunserver and how to restore an LVS to a single node server again. <verb> current ftp server: standalone A planned LVS (using VS-DR): real-server A director Z </verb> Setup the LVS in the normal way with the director's VIP being a new IP for the network. The IP of the standalone server will now also be the IP for the real-server. You can access the real-server via the VIP while the outside users continue to connect to the original IP of A. When you are happy that the VIP gives the right service, change the DNS IP of your ftp site to the VIP. Over the next 24hrs as the new DNS information is propagated to the outside world, users will change over to the VIP to access the server. To expand the number of servers (to A, B,...), add another server with duplicated files, add an extra entry into the director's tables with ipvsadm. To restore - in your DNS, change the IP for the service to the real-server IP. When no-one is accessing the VIP anymore, unplug the director. <sect1>Other projects like LVS - Beowulf The difference between a beowulf and an LVS: The Beowulf project has to do with processor clustering over a network -- parallel computing... Basically putting 64 nodes up and running that all are a part of a collective of resources. Like SMP -- but between a whole bunch of machines with a fast ethernet as a backplane. LVS, however, is about load-balancing on a network. Someone puts up a load balancer in front of a cluster of servers. Each one of those servers is independent and knows nothing about the rest of the servers in the farm. All requests for services go to the load balancer first. That load balancer then distributes requests to each server. Those servers respond as if the request came straight to them in the first place. So -- with the more servers one adds -- the less load goes to each server. A person might go to a web site that is load balanced, and their requests would be balanced between four different machines. (Or perhaps all of their requests would go to one machine, and the next person's request would go to another machine) However, a person who used a Beowulf system would actually be using one processing collaborative that was made up of multiple computers... I know that's not the best explanation of each, and I apologize for that, but I hope it at least starts to make things a little clearer. Both projects could be expanded on to a great extent, but that might just confuse things farther. (Joe) - both use several (or a lot of) nodes. A beowulf is a collection of nodes working on a single computation. The computation is broken into small pieces and passed to a node, which replies with the result. Eventually the whole computation is done. THe beowulf usually has a single user and the computations can run for weeks. An LVS is a group of machines offering a service to a client. A dispatcher connects the client to a particular server for the request. When the request is completed, the dispatcher removes the connection between the client and server. The next request from the same client may go to a different server but the client cannot tell which server it has connected to. The connection between client and server may only be seconds long from a posting to the beowulf mailing list by Alan Heirich - Thomas Sterling and Donald Becker made "Beowulf" a registered service mark with specific requirements for use: <verb> -- Beowulf is a cluster -- the cluster runs Linux -- the O/S and driver software are open source -- the CPU is multiple sourced (currently, Intel and Alpha) </verb> I assume they did this to prevent profit-hungry vendors from abusing this term; can't you just imagine Micro$oft pushing a "Beowulf" NT-cluster? (Joe - I looked up the Registered Service Marks on the internet and Beowulf is not one of them.) (Wensong) Beowulf is for parallel computing, Linux Virtual Server is for scalable network services. They are quite different now. However, I think they may be unified under "single system image" some day. In the "single system image", every node can see a single system image (the same memory space, the same process space, the same external storage), and the processes/threads can be transparently migrated to other nodes in order to achieve load balance in the cluster. All the processes are checkpointed, they can be restarted in the node or the others if they fails, full fault tolerant can be made here. It will be easy for programmers to code because of single space, they don't need to statically partition jobs to different sites and let them communicate through PVM or MPI. They just need identify the parallelism of his scientific application, and fork the processes or generate threads, because processes/threads will be automatically load balanced on different nodes. For network services, the service daemons just need to fork the processes or generates threads, it is quite simple. I think it needs lots of investigation in how to implement these mechanisms and make the overhead as low as possible. What Linux Virtual Server has done is very simple, Single IP Address, in which parallel services on different nodes is appeared as a virtual service on a single IP address. The different nodes have their own space, it is far from "single system image". It means that we have a long way to run. :) <sect1>Projects like LVS - Eddie Eddie http://www.eddieware.org (Jacek Kujawa <tt/blady@cnt.pl/) Eddie is a load balancing software, using NAT (only NAT), for webservers, written in language erlang. Eddie include intelligent HTTP gateway and Enhanced DNS. (Joe) Erlang is a language for writing distrubuted applications. <sect1>Troubles with tulip cards (Joe - I don't know if this is still a problem) <verb> Date: Wed, 05 May 1999 16:45:34 +0100 From: John Connett <tt/jrc@art-render.com/ Subject: Re: Configuration help John Connett wrote: > John Connett wrote: > > Any suggestions as to how to narrow it down? I have an Intel > > EtherExpress PRO 100+ and a 3COM 3c905B which I could try instead of the > > KNE 100TX to see if that makes a difference. > > A tiny light at the end of the tunnel! Just tried an Intel EtherExpress > PRO 100+ and it works! Unfortunately, the hardware is fixed for the > application I am working on and has to use a single Kingston KNE 100TX > NIC ... </verb> Some more information. The LocalNode problem has been observed with both the old style (21140-AF) and the new style (21143-PD) of Kingston KNE 100TX NIC. This suggests that there is a good chance that it will be seen with other "tulip" based NICs. It has been observed with both the "v0.90 10/20/98" and the "v0.91 4/14/99" versions of tulip.c. I have upgraded to vs-0.9 and the behaviour remains the same: the EtherExpress PRO 100+ works; the Kingston KNE 100TX doesn't work. It is somewhat surprising that the choice of NIC should have this impact on the LocalNode behaviour but work successfully on connections to slave servers. Any suggestions as to how I can identify the feature (or bug) in the tulip driver would be gratefully received. If it is a bug I will raise it on the tulip mailing list. <sect1>About PPC (persistent port connection) (for 2.2.12 kernels) - was PCC (persistent client connection) for previous kernels PPC is used for clients who must maintain a session with the same real-server throughout a session (eg for various SSL protocols, to an http server sending <ref id="cookie" name="cookies">...). The default session timeout is 5mins (ip_vs_pcc.h). PCC (kernel < 2.2.12) was removed in 2.2.12 and has resurfaced as a more general persistance feature called persistant port. PCC connects (some or all) ports from a client IP through to the same ports on a single real-server (the real-server is selected on the first connection, after than all subsequent port requests from the same IP go to the same real-server). With persistant port, the persistant connection is on a port by port basis and not by IP. If persistant port is called with a port of "0" then the connection will be the same as PCC. here's the syntax for 2.2.12 kernels (Wensong) To use persistent port, the commands are as follows: <verb> ipvsadm -A -t <VIP>:<port> [-s <scheduler>] -p ipvsadm -a -t <VIP>:<port> -R <real server> ... ... </verb> if port=0 then all ports from the CIP will be mapped through to the real-server (the old PCC behaviour). If port=443, then only port 443 from the CIP will be mapped through to the real-server as a persistant connection. If the virtual service port is set persistent, connections from the same clients are gauranteed to direct to the same server. When a client sends request for the service at the first time, the load balancer (director) selects a server by the scheduling method and creates a connection and the template. Then, the following connections from the same client will be forwarded to the same server according to the template in the specified time. The source address of an incoming packet is used to lookup connection template. from Peter Kese (who implemented pcc) PCC (persistent client connection) scheduling algorithm needs some more explanation. When PCC scheduling is used, the connections are scheduled on a per client base instead of per connection. That means, the scheduling is performed only the first time a certain client connects to the virtual IP. Once the real server is chosen, all further connections from the same client will be forwarded to the same real server. PCC scheduling algorithm can either be attached to a certain port or to the server as whole. By setting the service port to 0 (example: ipvscfg -A -t 192.168.1.10:0 -s pcc) the scheduler will accept all incoming connections and will schedule them to the same real server no matter what the port number is. As Wensong had noted before, the PCC scheduling algorithm might produce some imbalance of load on real servers. This happens because the number of connections established by clients might vary a lot. (There are some large companies for example, that use only one IP address for accessing the internet. Or think about what happens when a search engine comes to scan the web site in order to index the pages.) On the other hand, the PCC scheduler resolves some problems with certain protocols (e.g. FTP) so I think it is good to have it. and a comment about load balancing using pcc/ssl. (the problem: once someone comes in from aol.com to one of the real-servers, all subsequent connections from aol.com will also go to the same server) - (Lars) Lets examine what happens now with SSL session comeing in from a big proxy, like AOL. Since they are all from the same host, they get forwarded to the same server - *thud*. Now, SSL carries a "session id" which identifies all requests from a browser. This can be used to separate the multiple SSL sessions, even if comeing in from one big proxy and load balance them. <verb> > > SSL connections will not come from the same port, since the clients open > > many of them at once, just like with normal http. > so would we be able to differentiate all the people coming from aol by > the port number? </verb> No. A client may open multiple SSL connections at once, which obviously will not come from the same port - but I think they will come in with the same SSL id. Lars Marowsky-Brie and from Wensong <verb> > But like I said: really hard to get working, and even harder to get right ;-) > > (At least I think so) </verb> No, not really! As I know, the PCC (Persistent Client Connection) scheduling in the VS patch for kernel 2.2 can solve connection affinity problem in SSL. When a SSL connection is made (crypted with server's public key), port 443 for secure Web servers and port 465 for secure mail server, a key (session id) must be generated and exchanged between the server and the client. The later connections from the same client are granted by the server in the life span of the SSL key. So, the PCC scheduling can make sure that once SSL "session id" is exchanged between the server and the client, the later connections from the same client will be directed to the same server in the life span of the SSL key. However, I haven't tested it myself. I will download ApacheSSL and test it sometime. Anyone who have tested or are going to test it, please let me know the result, no matter it is good or bad. :-) (a bit later) <verb> > I tested LVS with servers running Apache-SSL. > LVS uses the VS patch for kernel 2.2.9, and uses > the PCC scheduling. It worked without any problem. </verb> (some more) SSL is a little bit different. In use, the client will send a connection request to the server. The server will return a signed digital certificate. The client then authenticates the certificate using the digital signature and the public key of the CA. If the certificate is not authentic the connection is dropped. If it is authentic then the client sends a session key (such as a) and encrypts the data using the servers public key. This ensures only the server can read it since decrypting requires knowing the server private key. The server sends its session key (such as b) and encrypts with its private key, the client decrypt it with server's public key and get b. Since both the client and the server get a and b, they can generate the same session key based on a and b. Once they have the session key, they can use this to encrypt and decrypt data in communication. Since the data sent between the client and server is encrypted, it can't be read by anyone else. Since the key exchange and generating is very time-consuming, for performance reasons, once the SSL session key is exchanged and generated in a TCP connection, other TCP connections can also use this session key between the client and the server in the life-span of the key. So, we have make the connections from the same client is sent to the same server in the life-span of the key. That's why the PCC scheduling is used here. About longer timeouts From: felix k sheng <tt/felix@deasil.com/ and Ted Pavlic <verb> >> 2. The PCC feature....can I set the permanent connection for something >else >> than the default value ( I need to maintain the client on the same server >> for 30 minutes at maximum) ? > >If people connecting to your application will contact your web server at >least once every five minutes, setting that value to five minutes is fine. >If you expect people to be idle for up to thirty minutes before contacting >the server again, then feel free to change it to thirty minutes. Basically >remember that the clock is reset every time they contact the server again. >Persistence lasts for as long as it's needed. It only dies after the amount >of seconds in that value passes without a connection from that address. > >So if you really want to change it to thirty minutes, check out >ip_vs_pcc.h -- there should be a constant that defines how many seconds to >keep the entry in the table. (I don't have access to a machine with IPVS on >it at this location for me to give you anything more precise) </verb> I think this 30 minute idea is a web specific time out period. That is, default timeout's for <ref id="cookie" name="cookies"> are 30 minutes, so many web sites use that value as the length of a given web "session". So if a user hits your site, stops and does nothing for 29 minutes, and then hits your site again, most places will consider that the same session - the same session cookies will still be in place. So it would probably be a nice to have them going to the same server. <sect1>Related to PPC - Sticky connections (Wensong) Since there are many messages about passive ftp problem and sticky connection problem, I'd better send a separate message to make it clear. In LinuxDirector (by default), we have assumed that each network connection is independent of every other connection, so that each connection can be assigned to a server independently of any past, present or future assignments. However, there are times that two connections from the same client must be assigned to the same server either for functional or for performance reasons. FTP is an example for a functional requirement for connection affinity. The client establishs two connections to the server, one is a control connection (port 21) to exchange command information, the other is a data connection (usually port 20) that transfer bulk data. For active FTP, the client informs the server the port that it listens to, the data connection is initiated by the server from the server's port 20 and the client's port. LinuxDirector could examine the packet coming from clients for the port that client listens to, and create any entry in the hash table for the coming data connection. But for passive FTP, the server tells the clients the port that it listens to, the client initiates the data connection connectint to that port. For the VS-Tunneling and the VS-DRouting, LinuxDirector is only on the client-to-server half connection, so it is imposssible for LinuxDirector to get the port from the packet that goes to the client directly. SSL (Secure Socket Layer) is an example of a protocol that has connection affinity between a given client and a particular server. When a SSL connection is made, port 443 for secure Web servers and port 465 for secure mail server, a key for the connection must be chosen and exchanged. The later connections from the same client are granted by the server in the life span of the SSL key. Our current solution to client affinity is to add persistent client connection scheduling in LinuxDirector. In the PCC scheduling, when a client first access the service, LinuxDirector will create a connection template between the give client and the selected server, then create an entry for the connection in the hash table. The template expires in a configurable time, and the template won't expire if it has its connections. The connections for any port from the client will send to the server before the template expires. Although the PCC scheduling may cause slight load imbalance among servers, it is a good solution to connection affinity. The configuration example of PCC scheduling is as follows: <verb> ipvsadm -A -t <VIP>:0 -s pcc ipvsadm -a -t <VIP>:0 -R <your server> </verb> BTW, PCC should not be considered as a scheduling algorithm in concept. It should be a feature of virtual service port, the port is persistent or not. I will write some codes later to let user to specify whether port is persistent or not. (and what if a real-server holding a sticky connection crashes?) From: Ted Pavlic <tt/tpavlic_list@netwalk.com/ Is this a bug or a feature of the PCC scheduling... A person connects to the virtual server, gets direct routed to a machine. Before the time set to expire persistent connections, that real machine dies. mon sees that the machine died, and deletes the real server entries until it comes back up. But now that same person tries to connect to the virtual server again, and PCC *STILL* schedules them for the non-existent real server that is currently down. Is that a feature? I mean -- I can see how it would be good for small outages... so that a machine could come back up really quick and keep serving its old requests... YET... For long outages those particular people will have no luck. (Wensong) > Is this a bug or a feature of the PCC scheduling... I don't know how to answer this. :-) You can set the timeout of template masq entry into a small number now. It will be expired soon. Or, I will add some codes to let each real server entry keep a list of its template masq entries, remove those template masq entries if the real server entry is deleted. <verb> > To me, this seems most sensible. Lowering the timeouts has other effects, > affecting general session persistence... (Ted) </verb> I agree with this. This was what I was hoping for when I sent the original message. I figure, if the server the person was connecting to went down, any persistence wouldn't be that useful when the server came back up. There might be temporary files in existence on that server that don't exist on another server, but otherwise... FTP or SSL or anything like that -- it might as well be brought up anew on another server. Plus, any protocol that requires a persistent connection is probably one that the user will access frequently during one session. It makes more sense to bring that protocol up on another server than waiting for the old server to come back up -- will be more transparent to the user. (Even though they may have to completely re-connect once) So, yes, deleting the entry when a real server goes down sounds like the best choice. I think you'll find most other load balancers do something similar to this. a similar question by Andres Reiner <tt/areiner@nextron.ch/ <verb> > I found some strange behaviour using 'mon' for the > high-availability. If a server goes down it is correctly removed from > the routing table. BUT if a client did a request prior to the server's > failure, it will still be directed to the failed server afterwards. I > guess this got something to do with the persistent connection setting > (which is used for the cold fusion applications/session variables). > > In my understanding the LVS should, if a routing entry is deleted, no > longer direct clients to the failed server even if the persistent > connection setting is used. > > Is there some option I missed or is it a bug ? </verb> No, you didn't miss anything and it is not a bug either. :) In the current design of LVS, the connection won't be drastically removed but silently drop the packet once the destination of the connection is down, because monitering software may marks the server temporary down when the server is too busy or the monitering software makes some errors. When the server is up, then the connection continues. If server is not up for a while, then the client will timeout. One thing is gauranteed that no new connections will be assigned to a server when it is down. When the client reestablishs the connection (e.g. press reload/refresh in the browser), a new server will be assigned. Wensong <sect1>Thundering herd problem, when down machine(s) come on line<label id="thundering_herd"> (now handled by code added to the scheduler) From: Christopher Seawood <tt/cls@aureate.com/ LVS seems to work great until a server goes down (this is where mon comes in). Here's a couple of things to keep in mind. If you're using the Weighted Round-Robin scheduler, then LVS will still attempt to hit the server once it goes down. If you're using the Least Connections scheduler, then all new connections will be directed to the down server because it has 0 connections. You'd think using mon would fix these problem but not in all cases. Adding mon to the LC setup didn't help matters much. I took one of three servers out of the loop and waited for mon to drop the entry. That worked great. When I started the server back up, mon added the entry. During that time, the 2 running servers had gathered about 1000 connections apiece. When the third server came back up, it immediately received all of the new connections. It kept receiving all of the connections until it had an equal number of connections with the other servers (which by this time...a minute or so later...had fallen to ~700). By this time, the 3rd server had been restarted after due to triggering a high load sensor also monitoring the machine (a necessary evil or so I'm told). At this point, I dropped back to using WRR as I could envision the cycle repeating itself indefinitely. <sect1>on the need for extended testing (this must have been solved, no-one is complaining about memory leaks now :-) From: Jerry Glomph Black <tt/black@real.com/ We have successfully used 2.0.36-vs (direct routing method), but it does fail at extremely high loads. Seems like a cumulative effect, after about a billion or so packets forwarded. Some kind of kernel memory leak, I'd guess. <sect1>loopback on Solaris The thing I have found out is that on Solaris 2.6, and probably other versions of Solaris, you have to to some magic to get the loopback alias setup. You must run the following commands one at a time: <verb> ifconfig lo0:1 <VIP> ifconfig lo0:1 <VIP> <VIP> ifconfig lo0:1 netmask 255.255.255.255 ifconfig lo0:1 up </verb> Which works well and is actually a pointopoint link like ppp which must be the way Solaris defines aliases to the lo interface. It will not let you do this all at once, just each step at a time or you have to start over from scratch on the interface. <verb> Chris Kennedy I-Land Internet Services <tt/ckennedy@iland.net/ </verb> <sect1>Having one director handling multiple LVS sites <verb> From: James CE Johnson <tt/jjohnson@mobsec.com/ Subject: Re: Can VS handle more than one site per director? Keith Rowland wrote: > Can I use Virtual Server to host multiple domains on the > cluster? Can VS be setup to respond to multiple 10-20 different > IP addresses and use the clusters to reposnd to any one of them > with the proper web directory. If I understand the question correctly, then the answer is yes :-) I have one system that has two IP addresses and responds to two names: foo.mydomain.com A.B.C.foo eth1 bar.mydomain.com A.B.C.bar eth1:0 On that system (kernel 2.0.36 BTW) I have LVS setup as: ippfvsadm -A -t A.B.C.foo:80 -R 192.168.42.50:80 ippfvsadm -A -t A.B.C.bar:80 -R 192.168.42.100:80 </verb> To make matters even more confusing, 192.168.42.(50|100) are actually one system where eth0 is 192.168.42.100 and eth0:0 is 192.168.42.50. We'll call that 'node'. Apache on 'node' is setup to serve foo.mydomain.com on ...100 and bar.mydomain.com on ...50. It took me a while to sort it out but it all works quite nicely. I can easily move bar.mydomain.com to another node within the cluster by simply changing the ippfvsadm setup on the externally addressable node. <sect1>Running multiple directors (each with their own IP) On a normal LVS (one director, multiple real-servers being failed-over with mon), the single director is a SPOF (single point of failure). Director failure can be handled (in principle) with heartbeat, but no-one is doing this yet. In the meantime, you can have two directors each with their own VIP known to the users and set them up to talk to the same set of real-servers. (You can have two VIP's on one director box too). (The configure.pl script doesn't handle this yet.) (Michael Sparks <tt/michael.sparks@mcc.ac.uk/ <verb> > Also has anyone tried this using 2 or more masters - each master with it's > own IP? (*) From what I can see theoretically all you should have to do is > have one master on IP X, tunneled to clients who recieve stuff via tunl0, > and another master on IP Y, tunneled to clients on tunl1 - except when I > just tried doing that I can't get the kernel to accept the concept of a > tunl1... Is this a limitation of the IPIP module ??? </verb> (Stephen D. WIlliams <tt/sdw@lig.net/) Do aliasing. I don't see a need for tunl1. In fact, I just throw a dummy address on tunl0 and do everything with tunl0:0, etc. We plan to run at least two LinuxDirector/DR systems with failover for moving the two (or more) public IP's between the systems. We also use aliased, movable IP's for the real server addresses so that they can failover also. <sect1>Running clients (eg telnet) on real-servers<label id="clients_on_real-servers"> There are two types of clients on real-servers from the point of view of LVS. <itemize> <item>Clients which have src_addr=RIP (eg telnet run from the command line). These are simpler to handle. <item>Clients which need to have src_addr=VIP (but call from RIP). These are usually call-backs from the LVS'ed service to a demon on the LVS client. Handling these is somewhat problematic. The instances that we know about of this. <itemize> <item> <ref id="authd" name="authd"> <item> <ref id="rshd" name="rshd"> <item> <ref id="passive_ftp" name="passive ftp"> </itemize> </itemize> Both types of clients require the same understanding of LVS, but because the first case is simple, it is discussed here. The second case has all sorts of ramifications for LVS and for that reason is discussed in the <ref id="authd" name="section on authd">. You might have valid reasons for running clients on real-servers, e.g. so that the sysadmin could telnet to a remote site. The way to allow clients on the real-servers to connect to outside servers is to configure these requests so that they are independant of the LVS setup (you do have to use the network and default gw set by the LVS). The solution is to NAT the client requests. <sect2>client requests from real-servers in a VS-NAT LVS This is simple <itemize> <item>the director is already the default gw for the real-server (a requirement for NAT). <item>each real-server is replying to LVS packets with its RIP, which is unique (there is no VIP on the real-servers with VS-NAT). NAT'ed client requests will return to the correct real-server. </itemize> Here's the command to run on a 2.2.x director to allow realserver1 to telnet to the outside world. <verb> director:# ipchains -A forward -p tcp -j MASQ -s realserver1 telnet -d 0.0.0.0/0 </verb> You may have to turn off icmp redirects, if you have a <ref id="one_network" name="one network VS-NAT">. <verb> director: #echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects director: #echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects </verb> After running this command you can telnet from the real-servers. You can do this even if telnet is an LVS'ed service, since the telnet client and demon operate independantly of each other. You can use NAT the rshd and identd clients in the same way (replace telnet with rsh/identd and clients on the real-server can connect to their demons on outside machines). <sect2>client requests from real-servers in VS-DR or VS-Tun LVS's In general this has not been solved. Calls initiated by the identd client on a real-server will come from the VIP, not the RIP. Some hare-brained schemes have been tried but did not work (NAT'ing out the request from the VIP, so that it emerges from the real-server with src_addr=RIP and then NAT'ing the packet again on the director, so it emerges with src_addr=VIP). There are specific solutions In VS-DR/VS-Tun, if the client and RIP are on the same network. Usually the RIP's on VS-DR real-servers are private addresses. However if the LVS clients and the LVS are all local and on the same network, this will work. Clients not associated with the LVS'ed services (ie telnet even if telnetd is LVSed, but not authd or rshd) can still be NAT'ed out, since the connect request will come from the RIP and not the VIP. Since the default gw for the real-server in VS-DR is not the director, you can handle this 2 ways <itemize> <item>do the NAT'ing on the default gw box (you may not have access to this machine) <item>make the director the default gw for packets from the RIP (see setting up <ref id="NAT_with_VS-DR" name="NAT for clients on VS-DR">). </itemize> <sect1>Timeouts<label id="timeout"> <sect2>TCP/UDP (from Julian, 28 Jun 00) LVS uses the default timeouts for idle connections set by MASQ for (EST/FIN/UDP) of 15,2,5 mins. These values are fine for ftp or http, but if you have people sitting on a LVS telnet connection they won't like the 15min timeout. You can't read these timeout values, but you can set them with ipchains. $ipchains -M -S 36000 0 0 will set the timeout for established connections to 10hrs, while the other two 0 values will leave the FIN and UDP timeouts unchanged. The timeouts are set in /usr/src/linux/include/net/ip_masq.h <sect2>director with many entries in FIN state With the FIN timeout being about 1 min (2.2.x kernels), if most of your connections are non-persistent http (only taking 1 sec or so), most of your connections will be in the InActConn state. Date: Tue, 20 Mar 2001, Hendrik Thiel wrote: > we are using a lvs in NAT Mode and everything works fine ... > Probably, the only Problem seems to be the huge number of (idle) > Connection Entries. > > ipvsadm shows a lot of inActConn (more than 10000 entries per > Realserver) entries. > ipchains -M -L -n shows that these connections last 2 minutes. > Is it possible to reduce the time to keep the Masquerading Table > small? e.g. 10 seconds ... (Joe - you can use netstat -M instead of ipchains -M -L, for 2.4.x kernels use cat /proc/net/ip_conntrack.) (Julian) One entry occupies 128 bytes. 10k entries mean 1.28MB memory. This is not a lot of memory and may not be a problem. To reduce the number of entries in the ipchains table, you can reduce the timeout values. You can edit the TIME_WAIT, FIN_WAIT values in ip_masq.c, or enable the <url url="http://www.linux-vs.org/defense.html" name="secure_tcp strategy"> and alter the proc values there. FIN_WAIT can also be changed with ipchains. <sect2>ICMP On Wed, 14 Feb 2001 From: Laurent Lefoll <tt/Laurent.Lefoll@mobileway.com/ <verb> > what is the usefulness of the ICMP packets that are sent when > new packets arrives for a TCP connection that timed out for in LVS box ? I > understand obviously for UDP but I don't see their role for a TCP connection... </verb> (Julian) I assume your question is about the reply after ip_vs_lookup_real_service. It is used to remove the open request in SYN_RECV state in the real server. LVS replies for more states and may be some OSes report them as soft errors (Linux), others can report them as hard errors, who knows. <verb> > it's about ICMP packets from a VS-NAT director to the client. > For example, a client accesses a TCP virtual service and then stops sending data > for a long time, enough for the LVS entry to expire. When the client try to send > new data over this same TCP connection the LVS box sends ICMP (port unreachable) > packets to the client. For a TCP connection how do these ICMP packets > "influence" the client ? It will stop sending packets to this expired (for the > LVS box...) TCP connection only after its own timeouts, doesn't it ? </verb> By default TCP replies RST to the client when there is no existing socket. LVS does not keep info for already expired connections and so we can only reply with an ICMP rather than sending a TCP RST. (If we implement TCP RST replies, we could reply TCP RST instead of ICMP). What does the client do with this ICMP packet? By default, the application does not listen for ICMP errors and they are reported as soft errors after a TCP timeout and according to the TCP state. Linux at least allows the application to listen for such ICMP replies. The application can register for these ICMP errors and detect them immediately as they are received by the socket. It is not clear whether it is a good idea to accept such information from untrusted sources. ICMP errors are reported immediately for some TCP (SYN) states. <sect1>tcpdump On Fri, 16 Mar 2001, Joseph Mack wrote: <verb> > I'm looking at packets after they've been accepted by TP > and I'm using (among other things) tcpdump. > > Where in the netfilter chain does tcpdump look at incoming > and outgoing packets? When they are put on/received from > the wire? After the INPUT, before the OUTPUT chain...? </verb> (Julian) Before/after any netfilter chains. Such programs hook at packet level before/after the IP stack just before/after the packet is received/must be sent from/by the device. They work for other protocols. tcpdump is a packet receiver just like the IP stack is in the network stack. <sect1>Bringing down aliased devices (without bringing them all down) Problem: if down/delete an aliased device (eg eth0:1) you also bring down the other eth0 devices. This means that you can't bring down an alias remotely as you loose your connection (eth0) to that machine. You then have to go the console of the remote machine to fix it by rmmod'ing the device driver for the device and bring it up again. The configure script handles this for you and will exit (with instructions on what to do next) if it finds that an aliased device needs to be removed by rmmod'ing the module for the NIC. (I'm not sure that all of the following is accurate, please test yourself first). (Stephen D. WIlliams <tt/sdw@lig.net/) whenever you want to down/delete an alias, first set it's netmask to 255.255.255.255. This avoids also automatically downing aliases that are on the same netmask and are considered 'secondaries' by the kernel. (Joe) To bring up an aliased device $ifconfig eth0:1 192.168.1.10 netmask 255.255.255.0 to bring eth0:1 down without taking out eth0, you do it in 2 steps, first change the netmask $ifconfig eth0:1 192.168.1.10 netmask 255.255.255.255 then down it $ifconfig eth0:1 192.168.1.10 netmask 255.255.255.255 down then eth0 device should be unaffected, but the eth0:1 device will be gone. This works on one of my machines but not on another (both with 2.2.13 kernels). I will have to look into this. Here's the output from the machine for which this procedure doesn't work. Examples: Starting setup. The real-server's regular IP/24 on eth0, the VIP/32 on eth0:1 and another IP/24 for illustration on eth0:2. Machine is SMP 2.2.13 net-tools 1.49 <verb> chuck:~# ifconfig -a eth0 Link encap:Ethernet HWaddr 00:90:27:71:46:B1 inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1 RX packets:6071219 errors:0 dropped:0 overruns:0 frame:0 TX packets:6317319 errors:0 dropped:0 overruns:4 carrier:0 collisions:757453 txqueuelen:100 Interrupt:18 Base address:0x6000 eth0:1 Link encap:Ethernet HWaddr 00:90:27:71:46:B1 inet addr:192.168.1.110 Bcast:192.168.1.110 Mask:255.255.255.255 UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1 Interrupt:18 Base address:0x6000 eth0:2 Link encap:Ethernet HWaddr 00:90:27:71:46:B1 inet addr:192.168.1.240 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1 Interrupt:18 Base address:0x6000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:3924 Metric:1 RX packets:299 errors:0 dropped:0 overruns:0 frame:0 TX packets:299 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 chuck:~# netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 192.168.1.110 0.0.0.0 255.255.255.255 UH 0 0 0 eth0 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0 Deleting eth0:1 with netmask /32 chuck:~# ifconfig eth0:1 192.168.1.110 netmask 255.255.255.255 down chuck:~# ifconfig -a eth0 Link encap:Ethernet HWaddr 00:90:27:71:46:B1 inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1 RX packets:6071230 errors:0 dropped:0 overruns:0 frame:0 TX packets:6317335 errors:0 dropped:0 overruns:4 carrier:0 collisions:757453 txqueuelen:100 Interrupt:18 Base address:0x6000 eth0:2 Link encap:Ethernet HWaddr 00:90:27:71:46:B1 inet addr:192.168.1.240 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1 Interrupt:18 Base address:0x6000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:3924 Metric:1 RX packets:299 errors:0 dropped:0 overruns:0 frame:0 TX packets:299 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 If you do the same thing with eth0:2 with the /24 netmask chuck:~# ifconfig eth0:2 192.168.1.240 netmask 255.255.255.0 down chuck:~# ifconfig -a eth0 Link encap:Ethernet HWaddr 00:90:27:71:46:B1 inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1 RX packets:6071237 errors:0 dropped:0 overruns:0 frame:0 TX packets:6317343 errors:0 dropped:0 overruns:4 carrier:0 collisions:757453 txqueuelen:100 Interrupt:18 Base address:0x6000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:3924 Metric:1 RX packets:299 errors:0 dropped:0 overruns:0 frame:0 TX packets:299 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 tunl0 Link encap:IPIP Tunnel HWaddr unspec addr:[NONE SET] Mask:[NONE SET] NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 </verb> <sect1>Backing up real-servers/keeping in sync Real-servers should have indentical files for any particular service (since the client can connect to any of them). To keep servers in sync (after adding content/files) you can use rsync. (Wensong) If you just have two servers, it might be easy to use rsync to synchronize the backup server, and put the rsync job in the crontab in the primary. See http://rsync.samba.org/ for rsync. If you have a big cluster, you might be interested in Coda, a fault-tolerant distributed filesystem. See http://www.coda.cs.cmu.edu/ for more information. (note from Joe, coda doesn't seem ready for prime time yet, we do have hopes for Intermezzo http://www.inter-mezzo.org) On Mon, 27 Sep 1999, J Saunders wrote: <verb> > I plan to start a frequently updated web site (potentially every minute > or so). </verb> ("Baeseman, Cliff" <tt/Cliff.Baeseman@greenheck.com/) I use mirror to do this!. I created a ftp point on the director. All nodes run against the director ftp directory and update the local webs. It runs very fast and very solid. upload to a single point and the web propagates across the nodes. <sect1>Malicious attacks (SYN floods) LVS has been tested with a 100Mbit/sec syn-flooding attack by Alan Cox and Wensong. Each connection requires 128 bytes. A machine with 128M of free memory could hold 1M concurrent connections. An average connection lasts 300secs. Connections which just receive the syn packet are expired in 30secs (starting ipvs 0.8 ). An attacker would have to initiate 3k connections/sec (600Mbps) to maintain the memory at the 128M mark and would require several T3 lines to keep up the attack. <sect1>Does SMP help on the director? only one CPU can be in the kernel with 2.2. Since LVS is all kernel code, there is no benefit to LVS by using SMP with 2.2.x. Kernel 2.[3-4] can use multiple CPUs. While standard (300MHz pentium) directors can easily handle 100Mbps networks, they cannot handle an LVS at Gbps speeds. Either SMP directors with 2.4.x kernels or multiple directors (each with a separate VIP all pointing to the same real-servers) are needed. Since VS-NAT requires computation on the director (to rewrite the packets) not needed for VS-DR and VS-Tun, SMP would help throughput. <verb> (Julian Anastasov <tt/uli@linux.tu-varna.acad.bg/) > > Does somebody have any idea or data ? > > it depends :-) > > If you're using VS-NAT then you'll need a machine that can handle the full > bandwidth of the expected connections. If this is T1, you won't need much > of a machine. If it's 100Mbps you'll need more (I can saturate 100Mbps > with a 75MHz machine). If you're running VS-DR or VS-Tun you'll need > less horse power. Since most LVS is I/O I would suspect that SMP won't > get you much. However if the director is doing other things too, > then SMP might be useful </verb> Yep, LVS in 2.2 can't use both CPUs. This is not a LVS limitation. It is already solved in the latest 2.3 kernels: softnet. If you are using the director as real server too, SMP is recommended. <verb> > Date: Wed, 03 Jan 2001 11:08:41 -0500 > From: Pat O'Rourke <tt/orourke@mclinux.com/ > > In our experiments we've been seeing > an SMP director perform significantly worse than a uni-processor one > (using the same hardware - only difference was booting an SMP kernel > or uni-processor). > > We've been using a 2.2.17 kernel with the 1.0.2 LVS patch and bumped > the send / recv socket buffer memory to 1mb for both the uni-processor > and SMP scenarios. The director is an Intel based system with 550 > mhz Pentium 3's. </verb> Date: Tue, 26 Dec 2000 14:25:04 -0600 (CST) From: Michael E Brown <tt/michael_e_brown@dell.com/ <verb> > In some tests I've done with FTP, I have seen > *significant* improvements using dual and quad processors using 2.4. Under > 2.2, there are improvements, but not astonishing ones. > > Things like 90% saturation of a Gig link using quad processors, 70% using > dual processors and 55% using a single processor under 2.4.0test. Really > amazing improvements. > > > What are the percentage differences on each processor configuration > > between 2.2 and 2.4? How does a 2.2 system compare to a 2.4 system on the > > same hardware? > > I haven't had much of a chance to do a full comparison of 2.2 vs 2.4, but > most of the evidence on tests that I have run points to a > 100% > improvement for *network intensive* tasks. </verb> <verb> Date: 03 Jan 2001 > Pat O'Rourke wrote: > > In our experiments we've been seeing > an SMP director perform significantly worse than a uni-processor one > (using the same hardware - only difference was booting an SMP kernel > or uni-processor). </verb> <sect1>Multiple IPs on the Director (Michael Sparks) It's useful for the director to have 3 IP addresses. One which is the real machines base IP address, one which is the virtual service IP address, and then another virtual IP address for servicing the director. The reason for this is associated with director failover. Suppose: <itemize> <item> X real-servers pinging director on real IP A (assume a heartbeat style monitor) serving pages off virtual IP V. (IP A would be in place of hostip above) <item>Director on IP A fails, backup director (*) on IP B comes online taking over the virtual IP V. By not taking over IP A, IP B can watch for IP A to come back online via the network, rather than via a serial link (etc). <item> Problem is the real-servers are still sending to IP A for the heartbeat code to be valid on IP B, the real-servers need to send their pings to IP B instead. IMO the easiest solution is to allocate a we need a "heartbeat"/monitor virtual IP. (this is the vhostip) </itemize> <sect1>Expanding port number range <verb> Wang Haiguang wrote: > My client machine it uses port numbers between 1024 - 4096. > After reaching 4096, it will loop back to 1024 > and reuse the ports. I want to use more port nubmers <tt/michael_e_brown@dell.com/ 06 Feb 2001 echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range /usr/src/linux/Documentation/networking/ip-sysctl.txt </verb> <verb> From: Julian Anastasov <ja@ssi.bg> On Tue, 27 Feb 2001, LVS Account wrote: > I'm trying to do some load testing of LVS using a reverse proxy cache server > as the load balanced app. The error I get is from a load generating app.. > Here is the error: > > byte count wrong 166/151 Broken app. > this goes on for a few hundred requests then I start getting: > Address already in use App uses too many local ports. > This is when I can't telnet to port 80 any more... If I try to telnet to > 10.0.0.80 80 I get this: > > $ telnet 10.0.0.80 80 > Trying 10.0.0.80... > telnet: Unable to connect to remote host: Resource temporarily unavailable No more free local ports. > If I go directly to the web server OR if I go directly to the IP of the > reverse proxy cache server, I don't get these errors. Hm, there are free local ports now. > I'm using a load balancing app that I call this way: > > /home/httpload/load -sequential -proxyaddr 10.0.0.80 -proxyport > 0 -parallel 120 -seconds 6000000 /home/httpload/url Thanks, upping the local port range has helped tremendously </verb> <sect1>Performance Hints from the Squid people There is information on the Squid site about tuning a squid box for performance. This should apply to an LVS director. For a 100Mbps network, current PC hardware on a director can saturate a network without these optimizations. However current single processor hardware cannot saturate 1Gpbs network, and optimizations are helpful. Thei squid information is as good a place to start as any. Here's some more info <verb> > Date: Fri, 29 Dec 2000 15:50:04 -0600 (CST) > From: Michael E Brown <tt/michael_e_brown@dell.com/ > > How much memory do you have? How fast of network links? There are some > kernel parameters you can tune in 2.2 that help out, and there are even > more in 2.4. From the top of my head, > > 1) /proc/sys/net/core/*mem* <-- tune to your memory spec. The defaults are > not optimized for network throughput on large memory machines. > > 2.) 2.4 only /proc/sys/net/ipv4/*mem* > > 3.) For fast links, with multiple adapters (Two gig links, dual CPU) 2.4 > has NIC-->CPU IRQ binding. That can really help also on heavily loaded > links. > > 4.) For 2.2 I think I would go into your BIOS or RCU (if you have one) and > hardcode all NIC adapters (Assuming identical/multiple NICS) to the same > IRQ. You get some gain due to cache affinity, and one interrupt may > service IRQs from multiple adapters in one go, on heavily loaded links. > > 5.) Think "Interrupt coalescing". Figure out how your adapter driver turns > this on and do it. If you are using Intel Gig links, I can send you some > info on how to tune it. Acenic Gig adapters are pretty well documented. > > For a really good tuning guide, go to spec.org, and look up the latest TUX > benchmark results posted by Dell. Each benchmark posting has a full list > of kernel parameters that were tuned. This will give you a good starting > point from which to examine your configuration. > > The other obvious tuning recommendation: Pick a stable 2.4 kernel and use > that. Any (untuned) 2.4 kernel will blow away 2.2 in a multiprocessor > configuration. If I remember correctly 2.4.0test 10-11 are pretty stable. </verb> Some information is on http://www.LinuxVirtualServer.org/lmb/LVS-Announce.html <sect1>Problems with large uptimes (linux with an eepro100 can't pass more than 2^31-1 packets) From: Jerry Glomph Black <tt/black@real.com/ Subject: 2-billion-packet bug? I've seen several 2.2.12/2.2.13 machines lose their network connections after a long period of fine operation. Tonight our main LVS box fell off the net. I visited the box, it had not crashed at all. However, it was not communicating via its (Intel eepro100) ethernet port. The evil evidence: <verb> eth0 Link encap:Ethernet HWaddr 00:90:27:50:A8:DE inet addr:172.16.0.20 Bcast:172.16.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:15 errors:288850 dropped:0 overruns:0 frame:0 TX packets:2147483647 errors:1 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 Interrupt:10 Base address:0xd000 </verb> Check out the TX packets number! That's 2^31-1. Prior to the rollover, In-and-out packets were roughly equal. I think this has happened to non-LVS systems as well, on 2.2 kernels. ifconfigging eth0 down-and-up did nothing. A reboot (ugh) was necessary. <sect1>Testimonials This isn't particularly inclusive. We don't pester people for testimonials as we don't want to scare people from posting to the mailing list and we don't want inflated praise. People seem to understand this and don't pester us with their performance data either. Much of it isn't scientific, but it is nice to hear. The people who don't like LVS presumably go somewhere else, and we don't hear any complaints from them. From: Michael Sparks <tt/zathras@epsilon3.mcc.ac.uk/ On Wed, 2 Feb 2000, [iso-8859-1] "Daniel Erd�s" wrote: <verb> > How many connections did you really handled? What are your impressions and > experiences in "real life"? What are the problems? </verb> Problems - LVS provides a load balancing mechanism, nothing more, nothing less, and does it *extremely* well. If your back end real servers are flakey in anyway, then unless you have monitoring systems in place to take those machines out of service as soon as there are problems with those servers, then users will experience glitches in service. NB, this is essentially a real server stability issue, not an LVS issue - you'd need good monitoring in place anyway if you weren't using LVS! Another plus in LVS's favour in something like this over the commercial boxes, is the fact that the load balancer is a Unix type box - meaning your monitoring can be as complex or simple as you like. For example load balancing based on wlc could be supplemented by server info sent to the director. <verb> Date: Thu, 23 Mar 2000 16:15:14 -0800 (PST) From: Drew Streib <tt/ds@varesearch.com/ Subject: Re: question about lvs and 2.3.x, 2.4 </verb> I can vouch for all sorts of good performance from lvs. I've had single processor boxes handle thousands of simultaneous connections without problems, and yes, the 50,000 connections per second number from the VA cluster is true. lvs powers SourceForge.net, Linux.com, Themes.org, and VALinux.com. SourceForge uses a single lvs server to support 22 machines, multiple types of load balancing, and an average 25Mbit/sec traffic. With 60Mbit/sec of traffic flowing through the director (and more than 1000 concurrent connections), the box was having no problems whatsoever, and in fact was using very little cpu. Using DR mode, I've sent request traffic to an director box resulting in near gigabit traffic from the real servers. (Request traffic was on the order of 40Mbit.) I can say without a doubt that lvs toasts F5/BigIP solutions, at least in our real world implementations. I wouldn't trade a good lvs box for a Cisco Local Director either. <verb> > The 50,000 figure is unsubstantiated and was _not_ claimed by anyone at VA > Linux Systems. A cluster with 16 apache servers and 2 LVS servers in a was > configured for Linux World New York but due to interconnect problems the > performance was never measured - we weren't happy with the throughput of the > NICs so there didn't seem to be a lot of point. This problem has been > resolved and there should be an opportunity to test this again soon. </verb> In recent tests, I've taken multinode clusters to tens of thousands of connections per second. Sorry for any confusion here. The exact 50,000 number from LWCE NY is unsubstantiated. <verb> Date: Thu, 23 Mar 2000 16:34:21 -0800 (PST) From: Jerry Glomph Black <tt/black@real.com/ Subject: LVS testimonials </verb> We ran a very simple LVS-DR arrangement with one PII-400 (2.2.14 kernel)directing about 20,000 HTTP requests/second to a bank of about 20 Web servers answering with tiny identical dummy responses for a few minutes. Worked just fine. Now, at more terrestrial, but quite high real-world loads, the systems run just fine, for months on end. (using the weighted-least-connection algorithm, usually). We tried virtually all of the commercial load balancers, LVS beats them all for reliability, cost, manageability, you-name-it. <verb> Jerry Glomph Black Director, Internet & Technical Operations RealNetworks Seattle Washington USA </verb> Well I guess 2 testimonials isn't too bad. :-) <sect1>Transport Layer Security(TLS) From: Ted Pavlic <tt/tpavlic@netwalk.com/, Nov 2000 <quote> I don't see any reason why LVS would have any bearing on TLS. As far as LVS was concerned, TLS connections would just be like any other connections. Perhaps you are referring to HTTPS over TLS? Such a protocol has not been completed yet in general, and when it does it still will not need any extra work to be done in the LVS code. The whole point of TLS is that one connects to the same port as usual and then "upgrades" to a higher level of security on that port. All the secure logic happens at a level so high that LVS wouldn't even notice a change. Things would still work as usual. </quote> Julian Anastasov <tt/ja@ssi.bg/ <quote> This is an end-to-end protocol layered on another transport protocol. I'm not a TLS expert but as I understand TLS 1.0 is handled just like the SSL 3.0 and 2.0 are handled, i.e. they require only a support for persistent connections. </quote> <sect1>rcp and friends on LVS (better to use ssh) David Lambe <tt/david.lambe@netunlimited.com/ Mon, 13 Nov 2000 <verb> > I've recently completed "construction" of a LVS cluster consisting of 1 LVS > and 3 real servers. Everything seems to work OK with the setup except for rcp. . > All it ever gives is "Permission Denied" when running rcp blahfile node2:/tmp/blahfile > from a console on node1. > > Both rsh and rlogin function, BUT require the password to be entered twice. </verb> Joe sounds like you are running RedHat. You have to fix the pam files. The beowulf people have been through all of this. You can either recompile the r* executables without pam (my solution), or you can fiddle with the pam files. For suggestions, go to the beowulf mailing list search engine at <url url="http://www.scyld.com/search.html" name="scyld beowulf"> and look for "rsh", "root", "rlogin". (hmm it seems to have gone. Looks like you have to download the whole archive at <url url="http://www.beowulf.org/pipermail/beowulf/" name="whole archive"> and grep through it.) If you go to the beowulf site, you'll find people are moving to replace rsh etc with ssh etc on sites which could be attacked from outside (and turning off telnet, r* etc) My machines aren't connected to the outside world so I have root with no passwd. To compile ssh do ./configure --with-none and use the config file I've attached (the docs on passwordless root logins were not helpfull) <verb> # This is ssh server systemwide configuration file. Port 22 #Protocol 2,1 ListenAddress 0.0.0.0 #ListenAddress :: HostKey /usr/local/etc/ssh_host_key ServerKeyBits 768 LoginGraceTime 600 KeyRegenerationInterval 3600 PermitRootLogin yes #PermitRootLogin without-password # # Don't read ~/.rhosts and ~/.shosts files IgnoreRhosts yes # Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication #IgnoreUserKnownHosts yes StrictModes yes X11Forwarding no X11DisplayOffset 10 PrintMotd yes KeepAlive yes # Logging SyslogFacility AUTH LogLevel INFO #obsoletes QuietMode and FascistLogging RhostsAuthentication no # # For this to work you will also need host keys in /usr/local/etc/ssh_known_hosts #RhostsRSAAuthentication no RhostsRSAAuthentication yes # RSAAuthentication yes # To disable tunneled clear text passwords, change to no here! PasswordAuthentication yes #PermitEmptyPasswords no PermitEmptyPasswords yes # Uncomment to disable S/key passwords #SkeyAuthentication no #KbdInteractiveAuthentication yes # To change Kerberos options #KerberosAuthentication no #KerberosOrLocalPasswd yes #AFSTokenPassing no #KerberosTicketCleanup no # Kerberos TGT Passing does only work with the AFS kaserver #KerberosTgtPassing yes CheckMail no #UseLogin no # Uncomment if you want to enable sftp #Subsystem sftp /usr/local/libexec/sftp-server #MaxStartups 10:30:60 </verb> <sect1>Forwarding an httpd request based on file name not load (mod_proxy) On Mon, 25 Dec 2000, Sean wrote: > I need to forward request using the Direct Routing method to a server. > However I determine which server to send the request to depending on the > file it has requested in the HTTP GET not based on it's load. For this I am From: Michael E Brown <tt/michael_e_brown@dell.com/ Mon, 25 Dec 2000 Use LVS to balance the load among several servers set up to reverse-proxy your real-servers, set up the proxy servers to load-balance to real-servers based upon content. <tt/atif.ghaffar@4unet.net/ On the LVS servers you can run apache with mod_proxy comiled in, then redirect traffic with it. Example <verb> ProxyPass /files/downloads/ http://internaldownloadserver/ftp/ ProxyPass /images/ http://internalimagesserver/images/ </verb> See <url url="http://www.linuxfocus.org/English/March2000/article147.html" name="more on Proxy pass"> and <url url="http://www.stevek.com/projects/mod_tproxy/" name="transparent proxy module for apache">. You can use mod_rewrite if your real-servers are reachable from the net. <sect1>URL parsing <verb> > Is there any way to do URL parsing for http requests (ie send cgi-bin > requests to one server group, static to another group?) </verb> John Cronin <tt/jsc3@havoc.gtf.org/ 13 Dec 2000 Probably the best way to do this is to do it in the html code itself; make all the cgis hrefs to cgi.your-domain.com. Similarly, you can make images hrefs to image.your-domain.com. You then set these up as additional virtual servers, in addition to your www virtual server. That is going to be a lot easier than parsing URLs; this is how they have done it at some of the places I have done consulting for; some of those places were using Extreme Networks load balancers, or Resonate, or something like that, using dozens of Sun and Linux servers, in multiple hosting facilities. <sect1>can I run my ipchains firewall and LVS on the same box? "K.W." <tt/kathiw@erols.com/ <verb> > can I run my ipchains firewall and LVS (piranha in this case) on the > same box? It would seem that I cannot, since ipchains can't understand > virtual interfaces such as eth0:1, etc. </verb> Brian Edmonds <tt/bedmonds@antarcti.ca/ 21 Feb 2001 I've not tried to use ipchains with alias interfaces, but I do use aliased IP addresses in my incoming rulesets, and it works exactly as I would expect it to. Julian I'm not sure whether piranha already supports kernel 2.4, I have to check it. ipchains does not understand interfaces aliase even in Linux 2.2. Any setup that uses such aliases can be implemented without using them. I don't know for routing restrictions that require using aliases. <verb> > I have a full ipchains firewall script, which works (includes port > forwarding), and a stripped-down ipchains script just for LVS, and they > each work fine separately. When I merge them, I can't reach even just > the firewall box. As I mentioned, I suspect this is because of the > virtual interfaces required by LVS. </verb> LVS does not require any (virtual) interfaces. LVS never checks the devices nor any aliases. I'm not sure what is the port forwarding support in ipchains too. Is that the support provided from ipmasqadm: the portfw and mfw modules? If yes, they are not implemented (yet). And this support is not related to ipchains at all. Some good features are still not ported from Linux 2.2 to 2.4 including all these autofw useful things. But you can use LVS in the places where use ipmasqadm portfw/mfw but not for the autofw tricks. LVS can perfectly do the portfw job and even to extend it after the NAT support: there are DR and TUN methods too. <verb> >I have a full ipchains firewall script, which works (includes port >forwarding), and a stripped-down ipchains script just for LVS, and they >each work fine separately. When I merge them, I can't reach even just >the firewall box. As I mentioned, I suspect this is because of the >virtual interfaces required by LVS. </verb> Lorn Kay <tt/lorn_kay@hotmail.com/ I ran into a problem like this when adding firewall rules to my LVS ipchains script. The problem I had was due to the order of the rules. Remember that once a packet matches a rule in a chain it is kicked out of the chain--it doesn't matter if it is an ACCEPT or REJECT rule(packets may never get to your FWMARK rules, for example, if they do not come before your ACCEPT and REJECT tests). I am using virtual interfaces as well (eg, eth1:1) but, as Julian points out, I had no reason to apply ipchains rules to a specific virtual interface (even with an ipchains script that is several hundred lines long!) <verb> > Remember that once a packet matches a rule in a chain it is kicked out > of the chain--it doesn't matter if it is an ACCEPT or REJECT > rule(packets may never get to your FWMARK rules, for example, if they > do not come before your ACCEPT and REJECT tests). </verb> FWMARKing does not have to be a part of an ACCEPT rule. If you have a default DENY policy and then say: <verb> /sbin/ipchains -A input -d $VIP -j ACCEPT /sbin/ipchains -A input -d $VIP 80 -p tcp -m 3 /sbin/ipchains -A input -d $VIP 443 -p tcp -m 3 </verb> To maintain persistence between port 80 and 443 for https, for example, the packets will match on the ACCEPT rule, get kicked out of the input chain tests, and never get marked. <sect1>Setting up a hot spare server Mark Miller <tt/markm@cravetechnology.com/ 09 May 2001 Subject: Hot Spare config with LVS? <quote> We want a configuration where two Solaris based web servers will be setup in a primary and secondary configuration. Rather than load balancing between the two we really want the secondary to act as a hot spare for the primary. Here is a quick diagram to help illustrate this question: <verb> Internet LD1,LD2 - Linux 2.4 kernel | RS1,RS2 - Solaris Router | -------+------- | | ----- ----- |LD1| |LD2| ----- ----- | | -------+------- | Switch | --------------- | | ----- ----- |RS1| |RS1| ----- ----- </verb> </quote> Paul Baker <tt/pbaker@where2getit.com/ 09 May 2001 <quote> Just use heartbeat on the two firewall machines and heartbeat on the two solaris machines. </quote> Horms <tt/horms@vergenet.net/ 09 May 2001 <quote> You can either add and remove servers from the virtual service (using ipvsadm) or toggle the weights of the servers from zero to non-zero values. </quote> Alexandre Cassen <tt/alexandre.cassen@canal-plus.com/ 10 May 2001 <quote> For your 2 LDs you need to run a Hot standby protocol. Hearthbeat can be used, you can also use vrrp or hsrp. I am actually working on the IPSEC AH implementation for vrrp. That kind of protocol can be usefull because your LD backup server can be used even if it is in backup state (you simply create 2 LDs VIP and set default gateway of your serveur pool half on LD1 and half on LD2). For your webserver hot-spare needs, you can use the next keepalived (http://keepalived.sourceforge.net) in wich there will be "sorry server" facility. This mean exactly what you need => You have a RS server pool, if all the server of this RS server pool are down then the sorry server is placed into the ipvsadm table automaticaly. If you use keepalived keep in mind that you will use NAT topology. </quote> Joe 11 May 2001 <quote> Unless there's something else going on that I don't know about, I expect this isn't a great idea. The hot spare is going to degrade (depreciate, disk wear out - although not quite as fast, software need upgrading) just as fast idle as doing work. You may as well have both working all the time and for the few hours of down time a year that you'll need for planned maintenance, you can make do with one machine. If you only need the capacity of 1 machine, then you can use two smaller machines instead. </quote> <sect>Useful things that have other place (yet) <sect1>Files which are kernel version dependant eg System.map and ipvsadm I'm always booting into versions of the kernel which don't match the version of the kernel in /usr/src/linux. This gives me the warning (eg when running `ps`) <verb> Warning: /usr/src/linux/System.map has an incorrect kernel version. </verb> As well, my version of ipvsadm doesn't work anymore (and will probably tell me that I don't have ipvsadm installed when I do). My /usr/src directory has entries like <verb> linux-1.0.3-2.2.18-TP </verb> for the 2.2.18 kernel with the 1.0.3 ipvs patch applied and built with transparent proxy. I link this to linux-2.2.18 My /sbin director has ipvsadm-2.2.18 and ipvsadm-2.4.1 (ipvsadm versions match ipvs versions and not kernel versions, but it's close enough to kernel versions that labelling them by kernel versions works). At bootup I run this script to get the right versions of these files. <verb> #always have the right System.map, ipvsadm... #no matter which kernel version you're running #Joseph Mack (C) 2001 released under GPL, jmack@wm7d.net, #To use: #for files/directories which are kernel version specific. # #eg you'll get error messages about System.map being the wrong version unless #you have /usr/src/linux linked to the same version of linux as you booted from. #eg in /sbin: ipvsadm-2.4.1, ipvsadm-2.2.18 which is called as /sbin/ipvsadm # #in main you send a list of directories/files like /usr/src/linux/, /sbin/ipvsadm #for each item in this list, this program will #a. delete /usr/src/linux if linux is a link, otherwise do nothing if linux is a file #b. look for linux-x.x.x where x.x.x is the kernel version #c. ln -s linux-x.x.x linux, else issue notice if there is no linux-x.x.x # #--------------------- ip_vs(){ #IPVS_VERSION is assigned the version number of IPVS currently loaded if [ "$LINUX_KERNEL_SERIES" = "2.2" ] then IPVS_VERSION=`grep "IP Virtual Server version" /proc/net/ip_vs | awk '{print \$5}'` echo $IPVS_VERSION elif [ "$LINUX_KERNEL_SERIES" = "2.4" ] then IPVS_VERSION=`grep "IP Virtual Server version" /proc/net/ip_vs | awk '{print \$5}'` echo $IPVS_VERSION else echo "Error: kernel $UNAME_R not from 2.2 or 2.4 series" exit -1 fi } make_link(){ BASENAME=`/usr/bin/basename $1` #echo "BASENAME $BASENAME" cd `/usr/bin/dirname $1` #if $BASENAME a link or doesn't exist if [ -L $BASENAME -o ! $ -e $BASENAME $ ] then #only do something if we have a target if [ -d $BASENAME-${UNAME_R}-${IPVS_VERSION} -o -f $BASENAME-${UNAME_R}-${IPVS_VERSION} ] then if [ -L $BASENAME ] #if the filename is a link, delete it then /bin/rm $BASENAME fi #there is no $BASENAME now. /bin/ln -s $BASENAME-${UNAME_R}-${IPVS_VERSION} $BASENAME elif [ -d $BASENAME-${IPVS_VERSION}-${UNAME_R} -o -f $BASENAME-${IPVS_VERSION}-${UNAME_R} ] then if [ -L $BASENAME ] #if the filename is a link, delete it then /bin/rm $BASENAME fi #there is no $BASENAME now. /bin/ln -s $BASENAME-${IPVS_VERSION}-${UNAME_R} $BASENAME elif [ -d $BASENAME-${IPVS_VERSION} -o -f $BASENAME-${IPVS_VERSION} ] then if [ -L $BASENAME ] #if the filename is a link, delete it then /bin/rm $BASENAME fi #there is no $BASENAME now. /bin/ln -s $BASENAME-${IPVS_VERSION} $BASENAME elif [ -d $BASENAME-${UNAME_R} -o -f $BASENAME-${UNAME_R} ] then if [ -L $BASENAME ] #if the filename is a link, delete it then /bin/rm $BASENAME fi #there is no $BASENAME now. /bin/ln -s $BASENAME-${UNAME_R} $BASENAME else echo "no $BASENAME-${UNAME_R} or $BASENAME-${IPVS_VERSION} to link to" fi else echo "cannot delete $BASENAME, doesn't exist or is a regular file" fi cd - } #----------------- #main: echo "rc.system_map " UNAME_R=`/bin/uname -r` LINUX_KERNEL_SERIES=${UNAME_R%.*} ip_vs if [ $? != "0" ] then echo "Error: unable to determine IPVS version" exit -1 fi make_link /usr/src/linux make_link /sbin/ipvsadm make_link /sbin/ipvsadm-save make_link /sbin/ipvsadm-restore make_link /usr/src/ipvs #make_link /bin/foo #a test, foo-x.x.x doesn't exist #----------------- </verb> Note:klogd is supposed to read files like /boot/System.map-<kernel_version> allowing you to have several kernels in /. However this doesn't solve the problem for general executables. <sect1>Ramdisk I needed a ramdisk for testing once and couldn't find the instructions for setting it up. Here they are From: Jerry Glomph Black <tt/black@real.com/ You specify the ramdisk size when you load the rd module, as an option. /sbin/insmod rd rd_size=32768 /sbin/mke2fs -m0 /dev/ram0 mount -t ext2 /dev/ram0 /mnt <sect1>cscope cscope is a code symbol navigating tool. from Patrick O'Rourke, <tt/orourke@missioncriticallinux.com/ A cscope for Linux is at http://sourceforge.net/projects/cscope/ <sect>FAQ <sect1>Help! My LVS doesn't work We need more information than this. For suggestions look at the <ref id="problem_report" name="information needed to solve problems">. <sect1>My LVS doesn't work: ipvsadm shows entries in InActConn, but none in ActiveConn The usual mistake is to have the default gw for the real-servers set incorrectly. For VS-NAT its the director and for VS-DR and VS-Tun, it's _not_ the director. Setting up an LVS by hand is tedious. You can use the <ref id="configure" name="configure script"> which will trap most errors in setup. <sect1>initial connection is delayed, but once connected everything is fine Usually you have problems with <ref id="authd" name="authd/identd">. Simplest thing is to stop your service from calling the identd server on the client (i.e.disconnect your service from identd). <sect1>How fast/big should my director be There isn't a simple answer. The speed of the director is determined by the packet throughput from/to the clients and not by the number of real-servers. From the mailing list, 3-500MHz UMP directors running 2.2.x kernels with the ipvs patch can handle 100Mbps throughput. We don't know what is needed for 1Gpbs throughput, but postings on the mailing list show that top end UMP machines (eg 800MHz) can't handle it. For the complicated answer, see the <ref id="performance" name="section on estimating director performance">. <sect1>Does the director handle ICMP? yes. For LVS'ed services, the director handles <ref id="ICMP_redirects" name="ICMP redirects"> and <ref id="MTU_discovery" name="MTU discovery"> delivering them to the correct real-server. ICMP packets for non-LVS'ed services are delivered locally. <sect1>I get "connection refused" from the client This means that no service is listening for your client's requests and that some machine at the end is replying that it is not listening for that service. If the LVS is working, then the director is forwarding packets to a real-server which doesn't have the service set up. <sect1>Any recommendations for a NIC? No they all seem to work OK enough. If you are going into production, you should test that yours work with a netpipe test (see the <url url="http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html" name="performance page">). <sect1>Does SMP help? LVS is kernel code. In particular the network code is kernel code. Kernel code is only SMP in 2.4.x kernels. To take advantage of SMP for LVS then you must be running a 2.4.x kernel. Michael Brown <tt/michael_e_brown@dell.com/ wrote on 26 Dec 2000 <quote> I've seen significant improvements using dual and quad processors with 2.4. Under 2.2 there are improvements but not astonishing ones. Things like 90% saturation of a Gig link using quad processors. 70% using dual processors and 55% using a single processor under 2.4.0test. I haven't had much of a chance to do a full comparison of 2.2 vs 2.4, but most evidence points to >100% improvement for network intensive tasks. </quote> </article>