LVS-HOWTO Joseph Mack mack (at) wm7d (dot) net v2006.09 Sep 2006, released under GPL. 1999 2000 2001 2002 2003 2004 2005 2006 Joseph Mack Install, testing and running of a Linux Virtual Server with 2.2.x, 2.4.x, 2.6.x kernels search the LVS documentation search the LVS documenation with htdig. search the two mailing list archives
LVS: Introduction This LVS-HOWTO is posted to the LVS-HOWTO homepage, http://www.austintek.com/LVS/LVS-HOWTO/ about once a month (although I do miss occasional months). Some of the material is from my own testing and I've tried to make it into a coherent story. Much of the material is from the lvs-users mailing list and is listed chronologically (sometimes forward and sometimes backwards in time) and will thus look like a blog.
Thanks Contributions to this HOWTO came from the mailing list and are attibuted to the poster (with e-mail address). Postings may have been edited to fit into the flow of the HOWTO. The LVS logo (Tux with 3 lighter shaded penguins behind him representing a director and 3 realservers) is by Mike Douglas spike (at) bayside (dot) net LVS homepage is running on a machine donated by Horms. (Until Jul 2002, we used a machine donated by Voxel). LVS mailing list is hosted by Lars in Germany lmb (at) suse (dot) de
About the HOWTO
Purpose To enable you to understand how a Linux Virtual Server (LVS) works. The LVS-mini-HOWTO (http://www.austintek.com/LVS/LVS-HOWTO/mini-HOWTO/LVS-mini-HOWTO.html) tells you how to setup and install an LVS without understanding how the LVS works. The material here covers directors and realservers with 2.2, 2.4 and 2.6 kernels. The original material was written for 2.2.x kernels and ipchains. Not all material has been updated for 2.4.x kernels and iptables. The layout of this HOWTO is almost flat - you go to the section you want information on. You aren't supposed to read it from start to finish. Within any section, newer information may be combined with older information that says different things. I just don't have time to edit everything - I'll be glad if you straighten me out. The only information one level up is how LVS works (from this HOWTO or from documentation on the website, e.g. Wensong's early documents) setting up an LVS in the LVS-mini-HOWTO.html. The code for 2.0.x kernels still works fine and was used on production systems when 2.0.x kernels were current, but is not being developed further. For 2.2 kernels, the Linux kernel networking code was rewritten, producing for us . This changes the installation of LVS from a simple process that can be done by almost anyone, to a thought provoking, head scratching exercise, which requires detailed understanding of the workings of LVS. For 2.0 and 2.2, LVS is stand-alone code, based on ip_masquerading and doesn't integrate well with other uses of ip_masquerading. For 2.4 kernels, LVS was rewritten as a netfilter module to allow it to fit into and be visible to other netfilter modules. Unfortunately the fit isn't perfect, but cooperation with netfilter does work in most cases. (You will have trouble using your director as a firewall; see the .) Being a netfilter module, the latency and throughput are slightly worse for 2.4 LVS than for the 2.2 code. However with modern CPUs being running at 800MHz, the bottleneck now is network throughput rather than LVS throughput (you only need a small number of realservers to saturate 100Mbps ethernet). In general ipvsadm commands and services have not changed between kernels.
HOWTO source is xml The HOWTO was originally written in sgml. It is now xml. The char '&' found in C source code has to be written as & in sgml. If you swiped patches from the sgml rather than the html rendering, you would get code that needed to be edited to fix the &. Now that the HOWTO is in xml, this munging is not needed. Although I've tried to remove all munged ampersands, I expect some will persist for a while. Ampersands in URLs still have to be munged.
e-mail addresses in the HOWTO are spam protected Well we hope so anyhow. An article on spambots describes robots which ignore the robots.txt file and scan for e-mail addresses in readable files on websites. The author suggests removing any 'mailto:' strings and spam protecting e-mail addresses, by changing them from machine-readable to human-readable format. If you have a better scheme than implemented here, (and I can do it with vi) let me know. (May 2002): BTW, 160 people have contributed to the HOWTO (as judged by unique e-mail addresses).
Nomenclature/Abbreviations If you use these terms when you mail us, we'll know what you're talking about.
Preferred names IPVS,ipvs,ip_vs the code that patches the linux kernel on the director. LVS, linux virtual server This is the director + realservers. Together these machines are the virtual server, which appears as one machine to the client(s). director: the node that runs the ipvs code. Clients connect to the director. The director forwards packets to the realservers. The director is nothing but an IP router with special rules that make the LVS work. realservers: the hosts that have the services. The realservers handle the requests from the clients. client the host or user level process that connects to the VIP on the director forwarding method (currently , , ). The director is a router with somewhat different rules for forwarding packets than a normal router. The forwarding method determines how the director sends packets from the client to the realservers. scheduling () - the algorithm the director uses to select a realserver to service a new connection request from a client.
synonyms Please use the first term in these lines. The other words are valid but less precise (or are redundant). director: load balancer, dispatcher, redirector. realserver: servers, realservers, real-servers. LVS: the whole cluster, the (linux) virtual server (LVS)
virtual services, scheduling groups Here's the ipvsadm output of an LVS serving telnet and squid. RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs.mack.net:squid rr -> rs1.mack.net:squid Route 1 0 0 -> rs2.mack.net:squid Route 1 0 0 -> rs3.mack.net:squid Route 1 0 0 TCP lvs.mack.net:telnet rr -> rs1.mack.net:telnet Route 1 0 0 -> rs2.mack.net:telnet Route 1 0 0 ]]> In the above LVS, there are two virtual services, telnet and squid. There are also two virtual servers; a virtual server for telnet (which has 2 realservers) and a virtual server for squid (which has 3 realservers). This is what the client sees; two services (and two servers). Connections to each virtual server are scheduled (here by "rr", round robin), to the realservers which belong to the scheduling group. Here the scheduling group for telnet is rs1,rs2. The scheduling group for quid is rs1,rs2,rs3. Connections to the telnet virtual server are scheduled independantly of connections to the squid virtual server. The above nomenclature can be extended for .
scheduling instance, scheduled unit, virtual connection We don't have a good name for this. Suggestions welcome. (We also don't talk much about this concept on the mailing list, so we've done without a name). The director needs to know how to schedule packets from the client to the realservers. The smallest unit for LVS is a tcpip connection, i.e. all packets that are part of a single tcpip session from a client will be sent to the same realserver. For a tcp virtual service, each tcp connection is scheduled separately, with the first tcp connection going to one realserver, and the next tcp connection going to the next realserver assigned a connection from the scheduler. The virtual connection is the same as the tcp connection. For a all tcp connections that are separated by less than the timeout period are regarded as belonging to the same virtual connection and are scheduled to the same realserver. For udp, there is no such thing as a connection or session and all packets from the client within a timeout period are scheduled to the same realserver. (People aren't using LVS for udp services a whole lot). The virtual connection then is all udp packets from a client within a certain time period.
backend (multi-tier) servers The realservers sometimes are frontends to other backend servers. The client does not connect to these backend servers and they are not in the ipvsadm table. e.g. a realserver may run a web application. The web application in turn connects to a database on another backend server. a webcaching realserver (e.g. a squid). The squid connects to backend webserver(s). These backend servers are setup separately from the LVS.
the term "the server" is ambiguous People sometimes call the director or the realservers, "the server". Since the whole LVS appears as a server to the client and since the realservers are also serving services, the term "server" is ambiguous. Do not use the term "the server" or "the lvs server" when talking about LVS. Most often you are referring to the "director" or the "realservers". Sometimes (e.g. when talking about throughput) you are talking about the (whole) virtual server. I use "realserver" as I despair of finding a reference to a "realserver" in a webpage using the search keys "real" and "server". Horms and I (for reasons that neither of us can remember) have been pushing the term "real-server" for about a year, on the mailing list, and no-one has adopted it. We're going back to "realserver".
names of IPs/networks in an LVS The router has traditionally not been considered part of the LVS, because often you do not have control over the router. However if you're a paying customer, then the ISP will be glad to set up the router according to your specifications. If you have access to the router, it can solve and can install filter rules. Here are the names we use for the various IPs. If you use them when asking questions on the mailing list, we'll be able to answer your questions more easily. The VIP and DIP are setup as secondary IPs, (i.e. there is another primary IP on that NIC), so they can be moved to another duplicate director following director failover. For initial setup with a single director, setting up the VIP and DIP as secondary IPs will make the transition to a failover setup easier. For a two director LVS (where directors failover), the IPs on the DRIP network are The DIP will be on the same NIC as PIP on bootup and will move to the same NIC as SIP on director failover. We don't seem to need a name for the primary IP on the outside of the director - no-one ever talks about it. If the clients are on the internet then the VIP will be a public (routable) IP. The RIPs will usually not be routable (i.e. they will be private IPs like 192.168.x.x/24). If the realservers need to contact machines on the internet for other reasons (e.g. ) then they will need routable IPs (which can be the RIPs, or can be another IP on another NIC on another network). We don't often need to explicitely name the networks in an LVS, but here's some suggestsions DRIP network: the network containing the DIP and RIPs. (OK you come up with a better name.) network facing the internet or the outside network: the network on the director which receives packets from the outside world. This shouldn't be called the VIP network, as the VIP is also in the DRIP network (but not replying to arp calls) on the realservers in LVS-DR and LVS-Tun.
What is an LVS? Can I use an LVS? A Linux Virtual Server (LVS) is a cluster of servers which appears to be one server to an outside client. This apparent single server is called here a "virtual server". The individual servers (realservers) are under the control of a director (or load balancer), which runs a Linux kernel patched to include the ipvs code. The ipvs code running on the director is the essential feature of LVS. Other user level code is used to manage the LVS (set rules for services handled, handle failover). The director is basically a layer 4 router with a modified set of routing rules. When a new connection is requested from a client to a service provided by the LVS (eg httpd), the director will choose a realserver for the client. From then, all packets from the client will go through the director to that particular realserver. The association between the client and the realserver will last for only the life of the tcp connection (or udp exchange). For the next tcp connection, the director will choose a new realserver (which may or may not be the same as the first realserver). Thus a web browser connecting to an LVS serving a webpage consisting of several hits (images, html page), may get each hit from a separate realserver. Since the director will send the client to an arbitary realserver, the services must be either read only (e.g. web services) or if read/write (e.g. an on-line shopping cart) some mechanism external to LVS must be provided for propagating the writes to the other realservers on a timescale appropriate for the service (i.e. purchase of an item must decrement the stock on all other nodes before the next client attempts to purchase the same item). At best LVS is read mostly. If you just want one of several nodes to be up at any one time, and the other node(s) to become active on failure of the primary node, then you don't need LVS: you need a high availability setup e.g. Linux-HA (heartbeat), vrrp or carp. If you want independant servers at different locations, then you want a geographically distributed server like Supersparrow. Here are some
What is a VIP? The director presents an IP called the Virtual IP (VIP) to clients. (When using , VIPs are agregated into groups of IPs, but the same principles apply as for a single IP). When a client connects to the VIP, the director forwards the client's packets to one particular realserver for the duration of the client's connection to the LVS. This connection is chosen and managed by the director. The realservers serve services (eg ftp, http, dns, telnet, nntp, smtp) such as are found in /etc/services or inetd.conf. The LVS presents one IP on the director (the virtual IP, VIP) to clients. Peter Martin p (dot) martin (at) ies (dot) uk (dot) com and John Cronin jsc3 (at) havoc (dot) gtf (dot) org 05 Jul 2001
The VIP is the address which you want to load balance i.e. the address of your website. The VIP is usually an alias (e.g. eth0:1) so that the VIP can be swapped between two directors if a fault is detected on one. The VIP is the IP address of the "service", not the IP address of any of the particular systems used in providing the service (ie the director and the realservers). The VIP be moved from one director to another backup director if a fault is directed (typically this is done by using mon and heartbeat, or something similar). The director can have multiple VIPs. Each VIP can have one or more services associated with it e.g. you could have HTTP/HTTPS balanced using one VIP, and FTP service (or whatever) balanced using another VIP, and calls to these VIPs can be answered by the same or different realservers. Groups of VIPs and/or ports can be setup with . The realservers have to be configured to work with the VIPs on the director (this includes handling the ). There can be issues, if you are using cookies or https, or anything else that expects the realserver fulfilling the requests to have some connection state information. This is also addressed on the LVS persistence page
Where do you use an LVS? For higher throughput. The cost of increasing throughput by adding realservers in an LVS increases linearly, whereas the cost of increased throughput by buying a larger single machine increases faster than linearly for redundancy. Individual machines can be switched out of the LVS, upgraded and brought back on line without interuption of service to the clients. Machines can move to a new site and brought on line one at a time while machines are removed from the old site, without interruption of service to the clients. for adaptability. If the throughput is expected to change gradually (as a business builds up), or quickly (for an event), the number of servers can be increased (and then decreased) transparently to the clients.
Client/Server relationship is preserved in an LVS Client sees only one IP address and believes it is connecting to a single machine. IPs of all servers is mapped to one IP (the VIP). While the client is connected to only one machine at a time, however subsequent connections will be assigned to a new and likely different machine. servers at different IP addresses believe they are contacted directly by the client.
LVS director is an L4 switch In the computer beastiary, the director is a layer 4 (L4) switch. The director makes decisions at the IP layer and just sees a stream of packets going between the client and the realservers. In particular an L4 switch makes decisions based on the IP information in the headers of the packets. Here's a description of an L4 switch from Super Sparrow Global Load Balancer documentation
Layer 4 Switching: Determining the path of packets based on information available at layer 4 of the OSI 7 layer protocol stack. In the context of the Internet, this implies that the IP address and port are available as is the underlying protocol, TCP/IP or UCP/IP. This is used to effect load balancing by keeping an affinity for a client to a particular server for the duration of a connection.
This is all fine except Nevo Hed nevo (at) aviancommunications (dot) com 13 Jun 2001
The IP layer is L3.
Alright, I lied. TCPIP is a 4 layer protocol and these layers do not map well onto the 7 layers of the OSI model. (As far as I can tell the 7 layer OSI model is only used to torture students in classes.) It seems that everyone has agreed to pretend that tcpip uses the OSI model and that tcpip devices like the LVS director should therefore be named according to the OSI model. Because of this, the name "L4 switch" really isn't correct, but we all use it anyhow. The director does not inspect the content of the packets and cannot make decisions based on the content of the packets (e.g. if the packet contains a cookie, the director doesn't know about it and doesn't care). The director doesn't know anything about the application generating the packets or what the application is doing. Because the director does not inspect the content of the packets (layer 7, L7) it is not capable of session management or providing service based on packet content. L7 capability would be a useful feature for LVS and perhaps this will be developed in the future (preliminary ktcpvs code is out - May 2001 - ). The director is basically a router, with routing tables set up for the LVS function. These tables allow the director to forward packets to realservers for services that are being LVS'ed. If http (port 80) is a service that is being LVS'ed then the director will forward those packets. The director does not have a socket listener on VIP:80 (i.e. netstat won't see a listener). John Cronin jsc3 (at) havoc (dot) gtf (dot) org (19 Oct 2000) calls these types of servers (i.e. lots of little boxes appearing to be one machine) "RAILS" (Redundant Arrays of Inexpensive Linux|Little|Lightweight|L* Servers). Lorn Kay lorn_kay (at) hotmail (dot) com calls them RAICs (C=computer), pronounced "rake".
LVS forwards packets to realservers The director uses 3 different methods of forwarding. LVS-NAT based on network address translation (NAT) LVS-DR (direct routing) where the MAC addresses on the packet are changed and the packet forwarded to the realserver LVS-Tun (tunnelling) where the packet is IPIP encapsulated and forwarded to the realserver. Some modification of the realserver's ifconfig and routing tables will be needed for LVS-DR and LVS-Tun forwarding. For LVS-NAT the realservers only need a functioning tcpip stack (i.e. the realserver can be a networked printer). LVS works with all services tested so far (single and 2 port services) except that LVS-DR and LVS-Tun cannot work with services that initiate connects from the realservers (so far; identd and rsh). The realservers can be indentical, presenting the same service (eg http, ftp) working off file systems which are kept in sync for content. This type of LVS increases the number of clients able to be served. Or the realservers can be different, presenting a range of services from machines with different services or operating systems, enabling the virtual server to present a total set of services not available on any one server. The realservers can be local/remote, running Linux (any kernel) or other OS's. Some methods for setting up an LVS have fast packet handling (eg LVS-DR which is good for http and ftp) while others are easier to setup (eg transparent proxy) but have slower packet throughput. In the latter case, if the service is CPU or I/O bound, the slower packet throughput may not be a problem. For any one service (eg httpd at port 80) all the realservers must present identical content since the client could be connected to any one of them and over many connections/reconnections, will cycle through the realservers. Thus if the LVS is providing access to a farm of web, database, file or mail servers, all realservers must have identical files/content. You cannot split up a database amongst the realservers and access pieces of it with LVS. The simplest LVS to setup involved clients doing read-only fetches (e.g. a webfarm). If the client is allowed to write to the LVS (e.g. database, mail farm), then some method is required so that data written on one realserver is transferred to other realservers before the client disconnects and reconnects again. This need not be all that fast (you can tell them that their mail won't be updated for 10mins), but the simplest (and most expensive) is for the mail farm to have a common file system for all servers. For a database, the realservers can be running database clients which connect to a single backend database, or else the realservers can be running independant database daemons which replicate their data.
LVS runs on Linux and FreeBSD directors LVS was developed on Linux and historically uses a Linux director. The Intel and Dec Alpha versions of LVS are known to work. The LVS code doesn't have any Intel specific instructions and is expected to work on any machine that runs Linux. In Apr 2005, LVS was ported to FreeBSD by Li Wang Li Wang dragonfly (at) linux-vs (dot) org 2005/04/16 The URL is: FreeBSD port of LVS (http://dragon.linux-vs.org/~dragonfly/htm/lvs_freebsd.htm). Here's a performance test on FreeBSD(version 0.4.0) (http://dragon.linux-vs.org/~dragonfly/software/doc/ipvs_freebsd/performance.html).
Code for LVS is different for each kernel series There are differences in the coding for LVS for the 2.0.x, 2.2.x, 2.4.x and 2.6.x kernels. Development of LVS on 2.0.36 kernels has stopped (May 99). Code for 2.6.x kernels is relatively new. The 2.0.x and 2.2.x code is based on the masquerading code. Even if you don't explicitely use ipchains (eg with LVS-DR or LVS-Tun), you will see masquerading entries with `ipchains -M -L` (or `netstat -M`). Code for 2.4.x kernels was rewritten to be compatible with the netfilter code (i.e. its entries will show up in netfilter tables). It is now production level code. Because of incompatibilities with LVS-NAT for 2.4.x LVS was in development mode (till about Jan 2001) for LVS-NAT.
kernels from 2.4.x series are SMP for kernel code 2.4.x kernels are SMP for kernel code as well as user space code, while 2.2.x kernels are only SMP for user space code. LVS is all kernel code. A dual CPU director running a 2.4.x kernel should be able to push packets at twice the rate of the same machine running a 2.2 kernel (if other resources on the director don't become limiting). (Also see the section on .)
OS for realservers You can have almost any OS on the realservers (all are expected to work, but we haven't tried them all yet). The realservers only need a tcpip stack - a networked printer can be a realserver.
LVS works on ethernet LVS works on ethernet. There are some limitations on using ATM. Firewire: (from the Beowulf mailing list - Donald Becket 5 Dec 2002): The firewire transport layer (IEEE1394) does run IP over FireWire. However firewire is designed for fixed size repeated frames (video or continuous disk block reads), but has overhead for other communication. Throughput is 400Mbps but worst case latency is high (msec range). Oracle has released GPL libraries for clustering Linux boxes over FireWire (http://www.ultraviolet.org/mail-archives/beowulf.2002/2977.html, link dead Dec 2003).
LVS works on IPV6 Seiji Tsuchiike tsuchiike (at) yggr-drasill (dot) com 02 Jun 2002
We just implemented IPv6 to lvs. We think that Basic Mechanism is same. (http://www.yggr-drasill.com/LVS6/documents.html. link dead Dec 2003, but Sep 2004 Horms says its alive).
LVS is continually being developed LVS is continually being developed and usually only the more recent kernel and kernel patches are supported. Usually development is incremental, but with the 2.2.14 kernels the entries in the /proc file system changed and all subsequent 2.2.x versions were incompatible with previous versions.
LVS is 64 bit Kenny Chamber
Has anybody here successfully setup lvs-director on sparc64 machine? I need to know which distro is OK for this.
Ratz 16 Dec 2004 Yes. Just recently. Debian is fine, I reckon Gentoo would do as well. INFO: It could be that your ipvsadm binary that comes to instrument the kernel tables for LVS is broken with regard to 64bit'ness. You then need to download the latest sources and recompile adding '-m64' to the CFLAGS. That's all, other than that it seems to work nicely. Btw: I took Debian testing, probably not too wise but on the other hand I needed more up to date tools. I wouldn't know of too many other Distros that have up to date Sparc64 support. Suse used to have, but they dropped support a while ago unfortunately. Justin Ossevoort justin (at) snt (dot) utwente (dot) nl 16 Dec 2004 Well our plain debian-sarge here did it just as painlessly as our x86 based machines. So as long as your distro has ipvs (and of course a sparc tree ;)) support you're in the green. liuah
I want to know whether LVS can work with 64-bit boxes. If I use LVS-DR, how can I apply the hidden patch to 64-bit linux, using kernel is 2.4.18?
ratz 29 Nov 2003 Yes. The only problem I see is if either the counters or the hashtable handling has some bug with 32/64-bit signedness and wrong shift operators. Just let us know if you experience flakyness on your director ;). The hidden patch for your kernel is: http://www.ssi.bg/~ja/hidden-2.4.19pre5-1.diff I hope you are aware of the fact that 2.4.18 is really buggy in many ways. I know that some 64-bit archs have been lagging behind in the 2.4.x tree but if I was you I would upgrade to a newer kernel. Peter Mueller pmueller (at) sidestep (dot) com 29 Nov 2003 The one for straight 2.4.18 is http://www.ssi.bg/~ja/hidden-2.4.5-1.diff. Since he said 2.4.18 I would suspect he's running Debian. If you want a Debian kernel with LVS+hidden use the Ultramonkey kernel (http://www.ultramonkey.org/"). liuah liuah (at) langchaobj (dot) com (dot) cn 02 Dec 2003 The hidden patch compiles and runs on our 64-bit servers successfully.
Other documentation For more documentation, look at the LVS web site (eg a talk I gave on how LVS works on 2.0.36 kernel directors) Julian has written Netparse for which we don't have a lot of documentation yet. For those who want more understanding of netfilter/iptables etc, here are some starting places. These topics are also covered in many other places. Harald Welte (of the netfilter team) description of what happens to a packet under 2.4 Harald Welte (of the netfilter team) conntrack HOWTO. Conntrack is used in filter rules as a way of accepting "related" packets, e.g. the data packets associated with an established ftp connection. Regular filter rules written for these data packets would accept ftp data packets (port 20) even if there were not in response to a PORT call from an already established ftp connection on port 21. In this case the filter rules would accept packets that are part of a DoS attack. Conntrack is CPU intensive and lowers throughput (see effect of conntrack on throughput). the docs/FAQs/HOWTOs on the netfilter site Linux Network Administrators Guide
LVS is not simple to install, get going or keep running This is not a utility where you run ../configure && make && make check && make install, put a few values in a *.conf file and you're done. LVS rearranges the way IP works so that a router and server (here called director and realserver), reply to a client's IP packets as if they were one machine. You will spend many days, weeks, months figuring out how it works. LVS is a lifestyle, not a utility. That said, you should be able to get a simple LVS-NAT setup working in a few hours without really understanding a whole lot about what's going one (see the LVS-mini-HOWTO).
LVS Failure LVS supplies high throughput using multiple identically configured machines. You would like to be able to swap out machines for planned maintenance and to automatically handle node failure (high availability). The LVS itself does not provide high availability. As you will read here, the software layer that provides high availability should be logically separate to the layer that it monitors. The writing of software that attempts to determine whether a machine is working, is somewhat of a black art. There are several packages used to help provide high availability for LVS and these are discussed in the High Availability LVS section. While it is relatively easy to monitor the functionality of the realservers, fail-out of directors is more difficult. An even greater problem is handling failure of nodes which are holding state information.
Minimal knowledge required The mailing list and HOWTOs cover information specific to LVS. The rest you have to handle yourself. All of us knew nothing about computers when we first started, we learnt it, and you can too (we're not saying it's easy). If you can't setup a simple LVS from the LVS-mini-HOWTO, without breaking into a major sweat (or being able to tell us what's wrong with the instructions), then you need to do some more homework. (Also see Help! My LVS doesn't work.)
Ratz ratz (at) tac (dot) ch To be able to setup/maintain an LVS, you need to be able to know how to patch and compile a kernel the basics of shell-scripting have intermediate knowledge of TCP/IP have read the man-page, the online-documentation and LVS-HOWTO (this document) (and the LVS-mini-HOWTO) know basic system administration (e.g. iptables; syslog; find, compile, install code from source files; use cpan to find perl modules).
Getting Technical Help All of the people on this list are replying for free in their spare time. The best we can do on this list is to give solutions to technical problems on setting up and running LVS. I give about 15secs to a posting to decide if I've got something useful to say. The posting has to indicate that the person has analysed the problem to a stage where an answer exists. If _they_ can't describe the problem, there's no point in replying - they won't understand the answer. Please don't e-mail me privately with general questions (feel free to cc: me if you want). The mailing list will archive your question and the answer(s) which can be retrieved later. Other people may have more interesting, relevant or useful comments than I will. If you are writing to me in the hopes of avoiding the humiliation of publically showing your ignorance on the mailing list, it's not going to happen. We've had too many good ideas from "ignorant" people to let this happen. If your question has been answered many times before and it's in the HOWTO and the archives, you'll be told to read the HOWTO, that's all. To get technical help: Read the docs on the website, the HOWTOs, and search the mailing list archives. The HOWTO (at the top) has a link to a search engine of all known LVS documentation. It will probably return several webpages. You'll have to find the entry from there. The LVS-mini-HOWTO shows you how to setup a simple 3 node (client, director, realserver) LVS without you needing to understand a whole lot about how an LVS works. after you've done a search of the docs, then post to the mailing list. updates/problems/bugs - post to the mailing list Jakub Suchy jakub (at) rtfm (dot) cz 13 Jan 2005 Please read: smart questions (http://www.catb.org/~esr/faqs/smart-questions.html) before asking questions. Please only post relevant lines of a debug dump. If you post the whole dump, because you don't understand it, then it will fill up the archive machine and everyone's mail box. If we need the whole debug, we'll ask for it and you can send it to us off-list.
Problem people 1 It's hard to believe, but we get postings like
recompiling the kernel is hard (or I don't read HOWTOs), can't you guys cut me some slack and just tell me what to do?
I expect the people who post these statements don't read this HOWTO, so I may be wasting my time, but - No. The people on the mailing list answer questions for free, and have other important things to do, like keeping up with /. and checking our e-mail. When we're at home, we drink beer and watch Gilligan's Island re-runs.
Problem people 2
can anybody tell me how to setup a windows realserver? thank you very much! I'm in a hurry.
robert (dot) gehr (at) web2cad (dot) de I can't think of anyone who has set up lvs in a hurry :-)
Problem People 3: People using RedHat LVS RedHat have LVS in their standard distribution kernel. This gives people the idea that they can setup LVS from their standard RedHat distribution just by clicking on a few buttons or running some scripts. From reading the postings to the mailing list, it's more difficult than doing it our way. You still have to understand LVS and then afterwards, you have to figure out what RedHat did to it. One of the major wastes of time and source of aggravation for me personally on the LVS mailing list, is postings from people using RedHat LVS who assume that it's the same as LVS, and who post as if they're using our setup methods. Just saying that you're using a RedHat distribution doesn't tell us anything, since you can setup LVS our way in RedHat. Things you need to know before you post - There are reasons for wanting to setup LVS in a standard RedHat distribution (e.g. RedHat is "approved" in your location whereas "Linux" isn't). There is information in this HOWTO () and in the various links from here which show you how to setup RedHat LVS. We have a method of setting up LVS which works for all distributions (including RedHat). We are not interested in learning, understanding, debugging, supporting or fixing a setup method that only works for one distibution. RedHat don't talk to us about what they do and while they may monitor the LVS mailing list, rarely (only about once a year, that I can tell) do they reply to people having problems with RedHat LVS. It appears that RedHat does not think their version of LVS worthy of much support and I agree with them. If you setup LVS the RedHat way, you still need an understanding of how an LVS works and is setup (just like everyone else), before posting to the mailing list. If you are setting up with RedHat and want help with it, make sure that you describe what you've done, that you're using the RedHat files and how you've set it up, otherwise we'll assume that you're setting up using our methods.
Why you may not get an answer no-one knows. The took a long time to figure out. Since no-one else had seen the problem, we didn't know at first if it was a problem with LVS. It wasn't till 6 months later when someone else had the same symptom, and found that it only occured when the ftp helper module was loaded, that we could do something. I once needed to do something with iproute2 that I spent about 3 weeks trying to figure out. No-one on the list knew the answer. I had to post off-line to someone who could figure it out for me. We may not have a useful answer. If you post saying "I want to build an LVS with (list of hardware); do you think it will work?", all we can say is "probably". Often when questions like this come up, there are people who are happy to share their experiences, so there's no harm in posting such a question. In general the people who've been working with LVS for years will expect you to have read the docs and know what LVS does before you post. In the time I alot for a reply, I don't have time to figure out whether in your case LVS is best for you - you should pay a consultant to do this if you can't do it yourself. Your question may not be well posed. We are reading the postings in our spare time. You will get at most 30secs of attention before we figure out whether we can help you, this will take a bit of thinking, or we can't help you. If you have a long posting in which you haven't figured out which parts are causing the problem and which parts are working, then we aren't going to try to figure it out either. Post the minimum setup that will produce the problem. You haven't read the HOWTO.
Edit your posts! (top, bottom and in-line posting) Please edit the posting you're replying to, leaving only the parts relevant to your reply. We don't need to see material from previous posts irrelevant to the current posting, and the disk archive doesn't either. Reply in-line, i.e. following each statement by the poster. Here's a posting on the subject from one of the kernel mailing lists. Greg KH greg (at) kroah (dot) com 16 Nov 2005
After you've Got Technical Help In most cases when a problem is solved, there's enough info on the mailing list to see how it worked and we can write it up here for the next people. Occasionally, we get a posting "I've worked it out. Thanks for the help." When this happens we have no idea what the solution was and will have to reinvent it for the next person. If you've got help from all the unpaid people on the mailing list who've given their spare time to help you, when they could instead have been watching Gilligan's Island reruns, please write it up for the HOWTO. When I write to people asking for their solution I don't want to hear that you're busy and have a job. We're busy, have jobs, kids, homework to do and tax forms to fill in and we stopped what we were doing to help you. Here's a template. what you wanted to do why/how it didn't work what you needed to do to get it to work how the solution works
Mailing list: subscribing, unsubscribing, searching Thanks to Hank Leininger for the mailing list archive which is searchable not only by subject and author, but also by strings in the body. Hank's resource has been of great help in assembling this HOWTO. The mailing list is available for further questions. A single mailing list handles developers, new users and old users and has about 0-20 postings a day. You don't have to join the mailing list to read the archives. If you want to post questions, then you have to join. If you aren't subscribed and you post (or you post from an unsubscribed address), you'll get a reply saying that your posting is "awaiting moderator approval". It isn't; because of the volume of spam, we no longer review these messages - they're deleted.
Mailing list: posting to Please send e-mail with straight ascii (not html) and turn line-wrap on (some mails come with each paragraph on a single long line). If you're stuck with posting from a Windows machine or Lotus notes, or using Lookout, where each paragraph is sent as one line:
Francois JEANMOUGIN Francois (dot) JEANMOUGIN (at) 123multimedia (dot) com 09 Jul 2004 Global Settings -> Internet Message Format -> Default (or the one used) -> Advanced -> word wrap ]]> like shown in fixing outlook (http://www.lemis.com/email/fixing-outlook.html) especially in word wrap (http://www.lemis.com/email/exchsrvr-wordwrap.gif), but this is a very old version of exchange.
Please don't turn on your vacation message, intended only for your work mates, for messages from a list. e.g. The LVS mailing list doesn't want to know. Dan Moljar Aug 2004 For Lotus Notes: The client is not configured correctly. In the 'Out of Office' enable dialog under the 'Exceptions' tab, there is a check box for 'Do not reply to Internet Addresses'. Check it. The server shouldn't do it to begin with, but you can make the client stop. There's always new ideas and questions being posted on the mailing list. We don't expect this to stop. There are many complexities to LVS and we don't expect new people to understand any more about LVS that we did when we started. No-one is expected to know/understand everything in the docs but your questions will be better received, if you've done your homework, if you have setup the test configurations here, have at least perused this HOWTO (yes we know it's big), and have looked at the mail archives. We can't help you if you just tell us that you've read the documents and your LVS doesn't work. To you, all problems look the same ("it doesn't work"). To help you, we need more information. We at least need the forwarding method, the service(s) being forwarded, the number of networks and the output of ipvsadm in the problem state. Before you come up on the mailing list - Read the LVS-HOWTO (this document) and the LVS-mini-HOWTO Set up a simple LVS (3 nodes: client, director, realserver) with LVS-DR or LVS-NAT forwarding, with the service telnet using the instructions in the LVS-mini-HOWTO. You should be able to do this starting from a freshly downloaded kernel from ftp.kernel.org and the LVS patches (ipvs and the hidden patch if you have 2.4.x realservers). Don't setup first with http, with filter rules, with firewalls, with complicated file systems (e.g. coda, nfs) or network accelators - debug all these nifty things after you have LVS working with telnet and with no filter rules. Do use standard compilers (gcc-2.95.3), tools and utilities (ifconfig or iproute2). Do not use non-standard tools particular to a distribution designed to capture market share (e.g. ifup). If you are using one of the packages that can be used with LVS (e.g. heartbeat from the Linux HA project http://www.henge.com/~alanr/ha, or piranha from Redhat), again we may know what the problem is, but they need the feedback that you can't get it to work, not us. Many of us are on each others' mailing lists and we try to help when we can, but the best people to handle the problem are the developers for each package. Consult the LVS mailing list archives. Use our jargon as best you can. The machine names will be client, director, realserver1, realserver2... IPs are CIP, VIP, RIP, DIP. If you do this, we won't have to translate "susanne" and "annie" to their functional names as we scan your posting. we need to know your kernel (e.g. 2.2.14) and the ip_vs patch that was applied to it (eg 0.9.11), whether you are using LVS-DR, LVS-NAT or LVS-Tun. Tell us what you did what you expected what you got and why that's a problem If you don't understand your problem well, here's a suggested submission format from Roberto Nibali ratz (at) tac (dot) ch System information, such as kernel, tools and their versions. Example: ipvsadm -L -n | head -1 IP Virtual Server version 1.0.2 (size=4096) hog:~ # ipvsadm -h | head -1 ipvsadm v1.13 2000/12/17 (compiled with popt and IPVS v1.0.2) ]]> Short description and maybe sketch of what you intended to setup. Example for LVS-DR: RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.1.10:80 wlc -> 192.168.1.13:80 Route 0 0 0 -> 192.168.1.12:80 Route 0 0 0 -> 192.168.1.11:80 Route 0 0 0 ]]> The output from ifconfig from all machines (abbreviated, just need the IP, netmask etc), and the output from netstat -rn. What doesn't work. Show some output from tcpdump, ipchains/ip_tables, ipvsadm and kernlog. Later we may ask you for a more detailed configuration like routing table, OS-version or interface setup on some machines used in your setup. Tell us what you expected. Example: /proc/sys/net/ipv4/vs/debug_level && tail -f /var/log/kernlog tcpdump -n -e -i eth0 tcp port 80 route -n netstat -an ifconfig -a ]]> tcpdump listings are difficult to read. If you post one, please change the IPs to VIP, CIP, RIP1..n, DIP etc. Since you'll likely be on a switched network, tcpdump will only see packets to that NIC. Tell us which machine (director, realserver...) and the NIC (if there are two NICs on the machine) that it was run on.
Bug Fixes It's wonderful to get an unsolicited bug fix. Please let us know what it does and why it's better than the current file. A new version of a file without any information about what it does, or what it fixes isn't much use to us.
Other load balancing solutions, GPL, opensource and commercial
Open Source and GPL solutions from lvs (at) spiderhosting (dot) com a list of load balancers Brent Cook busterb (at) mail (dot) utexas (dot) edu 28 Mar 2002
There's the http://www.bsdshell.net/ HighUpTime (HUT) projec (link dead Apr 2003). It's FreeBSD.
The HUT author, Sebastian Petit spe (at) selectbourse (dot) net has joined the LVS mailing list. For L7 Switching see the DRWS project. BSD load balancing: Roberto Nibali ratz (at) tac (dot) ch 05 Nov 2003 As already mentioned by others, LVS will not work on FreeBSD as director due to the kernel part. Using FreeBSD on the RS is of course ok. The BSD folks have not shown bigger interest in adopting the LVS idea or parts of the code yet. If you're interested in load balancing and HA Solutions under FreeBSD, you could check out following links: Gavin Henry ghenry (at) suretecsystems (dot) com 06/13/2005
ClusterIP by Harald Welte. What is the list's view on it?
Gavin Henry ghenry (at) suretecsystems (dot) com 13 Jun 2005 The man page for more recent versions of iptables says:
CLUSTERIP: This module allows you to configure a simple cluster of nodes that share a certain IP and MAC address without an explicit load balancer in front of them
Horms Been there, done that. Works, but is it neccessary? LVS with upto 16 directors active (http://www.ultramonkey.org/papers/active_active/)
List of Commercial Solutions Cahya Wirawan cwirawan (at) email (dot) archlab (dot) tuwien (dot) ac (dot) at 19 Feb 2004
I'm implementing proxy, smtp and webserver with LVS as local node, and I have tested it and it's running fine, but because someone from management section thinks that such an implementation is easy (just run setup.exe and everything is installed and ready to use), he pushed me to move the setup into production, and create another one as soon as possible. I want to tell him that such an implementation is not a trivial thing and needs time to setup and to test before we go into production. I want to show him a list of companies who have such complete solutions, so he can see the cost. Then he can understand that high availability and load balancing is not easy to setup, and will cost alot of money if we buy a complete solution.
Vendors just rub their hands with glee on finding management like this - see my review of the book "The IBM Way" (http://www.austintek.com/book_reviews/the_ibm_way.html) for how IBM handles the situation. Peter Mueller Prices at this level are negotiable. Who knows what you could pay? http://www.cisco.com/ - the old man on the LB-gig. http://www.f5.com/f5products/bigip/LB520/ - the second old man in the LB gig. http://www.suse.com/us/business/products/server/ - Suse has always been a big player in the Linux-HA world. http://www.redhat.com/software/rhel/purchase/ - they have clustering based on LVS, not sure about price. At this point you have to buy enterprise edition (or http://www.whiteboxlinux.com) to use the clustering software. http://www.ibm.com/ - always an option... http://www.dell.com/ - moving up in the datacenter world. I see lots of Dells now.. http://www.ebay.com/ - see how much the gear is worth on the open market. http://www.linuxvirtualserver.org/ - $0. There's plenty of people on list who can help you and your boss feel more comfortable with your setup. I'm sure if you posted something some people would be willing to help make you sleep better at night. BTW, you know about the http://www.ultramonkey.org/ and http://www.keepalived.org/ projects, right?
Radware Joe Oct 2005: From a presentation by Radware) (http://www.radware.com/) given to North Carolina Systems Administrators (NCSA) (http://www.ncsysadmin.org/) on 10 Oct 2005. Unfortunately I was the guy getting the pizzas for the meeting, so I missed most of the talk (which I wanted to hear). Radware is used by Ebay and Accuweather. Radware has a NAT loadbalancing director that appears to function similarly to an LVS-NAT director. The servers can have private IPs. Radware's loadbalancing director is only a small part of their offering. Radware have boxes that filter based on packet content (looking for viruses) that sit in the flow of packets (possibly before the director, possibly after - didn't find this out). They have boxes which just handle SYN floods. They use SYN cookies and do a statistical analysis of the packets, letting some through to see which machines reply to the SYN-ACKs. Radware has a gui to control the loadbalancer, which can do things like shutting down some of the backend servers at sometime in the future (e.g. at 10pm later that night) for new connections, so that by 8am next morning these machine have few or no connections and can be taken offline for servicing. Much of their hardware is ASIC based. Health checking seems to be done from the director, and checks are made through to 3rd-Tier components of the backend servers (e.g. database machines behind the webservers that the client doesn't directly connect to). Each local NAT'ed load balancing setup is itself a member of a distributed DNS-based load balancer. So www.foo.net might have a loadbalanced set of servers in different sites eg London, New York, San Francisco and Tokyo. Each local setup has an authoritative nameserver for www.foo.net The way is works is client in Scotland asks for the IP of www.foo.net the client's nameserver doesn't know the IP and asks a rootserver for the machine authoritative for foo.net. The rootserver has a list of 4 authoritative nameservers for foo.net and selects the next nameserver by round robin. If the next one in its list is in New York, it tells the client's nameserver to go query the nameserver in New York. The New York nameserver for foo.net measures the packet latency to the client's nameserver and then returns the VIP for www.foo.net associated with the New York installation of www.foo.net. The latency is propagated to the other foo.net nameservers (in Tokyo, London and San Francisco). Sometime later after the client's nameserver has flushed the IP entry for www.foo.net from its cache, another (or the same) client using the same nameserver asks for the IP of www.foo.net again and this time the rootserver will possibly send the request to another of the sites (say London). The London machine already knows the latency from New York to the client (without knowing where the client is), and sees that its latency to the client is lower than the latency from New York to the client, and returns the IP of its copy of www.foo.net. The London nameserver also updates the latency tables at the other sites (New York, San Francisco and Tokyo). If the next nameserver request from the client site is sent to Tokyo, then the Tokyo machine updates the latency tables in all the other nameservers, and knowing that the latency is lowest to the London nameserver, returns the IP of www.foo.net in London. In this way the four nameserver accumulate the latencies to all nameservers in the world. This works provided that the latencies don't change a lot with time of day (or throughput). Presumably you could store successive latencies and pick the shortest as reflecting the true network distance. The amount of memory required to do this must be small - there can't be more than a million nameservers, can there? 1 million 8 bit latencies is not much to store in memory. Although I didn't get to ask how it works, if a client winds up at a more distant site (network wise), then http redirects will send the client to a closer site. Radware SSL accelarators: When I commented to the speaker that the main reason to use SSL accelarators is financial, i.e. to only have one copy of the certificate, rather than one on each realserver, they said "it's also for certificate management". Presumably some sites have large numbers of certificates. (They didn't disagree with my statement.) The SSL accelarators in the Radware design don't sit between the director and the realservers (or in front of the director i.e. between the client and the director), but sit at the same level as the other realservers. The https request is balanced by the director to an accelarator, which decrypts the packets and sends the decrypted packet back to the director for loadbalancing as http traffic. Since the director is a NAT balancer, the return http traffic from the http servers, goes back through the director, and then recursively back to the SSL accelarator then back to the director at https traffic and then back to the client. Being able to have the SSL accelarator as a realserver in LVS would require the realservers to be a client of the director, something that we can do for LVS-NAT, but not for LVS-DR. This is not a capability that we've paid much attention to for LVS. If you need a realserver to be in the path in both inward and outward directions (like an SSL accelarator) then you will have to use LVS-NAT. Francois JEANMOUGIN Francois (dot) JEANMOUGIN (at) 123multimedia (dot) com 12 Oct 2005 Note that we removed our Radware appliance to use LVS instead. Load Balancing using DNS is _evil_, especially with mobile internet and all those misconfigured operator gateways. Most mobile gateway are written in Java, and I'm probably the only one who read the java.security file. Just have a look on this ugly stuff you can find in it and the unbelievable silly explanation given: For security reasons! Guys! Well. So we removed radware. Note that we had other problem with radware. The DNS cache of the clients is one, the response time of the DNS was another. Several technical issues when you reach some trafic limits was the last. Henrik Holst
still, geographic load balancing would be very nice to have and I cannot figure out another way to do it than involve DNS round-robin.
Francois Round-Robin DNS could work if You have enough clients Clients are using DNS as expected Clients are dealing with TTL Client DNS caches or provider DNS are honouring DNS TTL All your sites are always up and working (you can't use a DNS solution for failover) My clients are mobile phones, basically points 1 to 4 are not OK :). And I have to deal with multiple sources for the same client (the transaction begin in the gallery gateway and continues in the standard surf gateway, and I have to use fwmarks to keep the session)... We used RadWare to try to load-balance between our two peers. It clearly was not working. Unfortunately, I don't have all the details. Horms If you want to distribute traffic between hosts that have fast, reliable links, like a LAN, then LVS is a good option. No, an excellent option. If you want to distribute traffic between geographically separated hosts, then you don't want something like LVS that channles packets through a single location then to another. Something DNS based is probably the way to go - though round robin is not nearly smart enough for my liking. In practice, if you do have geographically distributed sites, then each site should probably be an LVS cluster. So essentially you end up using two techniques to solve different parts of the same problem. I wrote quite a lot of this on supersparrow.org once upon a time, its still there if people want to read/play/enhance/. (links through ).
Books on LVS Karl Kopper has tackled this. Writing a book on a moving target like LVS is a difficult proposition certainly more than I was prepared to tackle. The book is available at your usual suppliers. I'm loath to mention the names of internet booksellers who require your e-mail address as part of your purchase, so that they can spam you later. I've been buying my books by phone at a marginally higher price since realising their business practices. However recently (Jul 2004) I've discovered disposable e-mail addresses e.g. the free service from Jetable.org (http://www.jetable.org/). They have a google-like (i.e. simple) interface. You give them your e-mail address, the required lifetime of the address (1-8days), and click. Up comes an e-mail address (test by sending a message to it) that you can give to your internet vendor, and mail will be forwarded to you for the period selected. After that time, no more mail will get to you. I've been using jetable since Jul 2004 (now Sep 2004) and have not got any spam from Jetable or from internet vendors.
LVS in the news "Wired" Magazine in Jun 2004 has a small article about LVS, illustrating the multinational cooperative nature of GPL software development. The page is here (http://www.wired.com/wired/archive/12.06/images/atlas_software.pdf), or a local copy of the article on this server.
LVS: Install, Configure, Setup
Installing from Source Code Doing this from source code is now described in the LVS-mini-HOWTO. Two methods of setup are described Setup from the command line. This is fine to understand what's going on, and if you only want to have a single type of setup. For LVSs which you're reconfiguring a lot, it's tedious and mistake prone. If it doesn't work, you will spend some time figuring out why. From a configure script which sets up an LVS with a single director. This script is fine for initial setups: it's mistake proof (will give you enough information about failures to figure out what might be wrong) and I used it for all my testing of LVS. Since it's not easily expandable to handle director failover and other configuration tools handle this now, the configure script is not being developed anymore. For production, where you need failover directors, you should use other setup tools or save your hand-built setup as a script (e.g. with ipvsadm-sav).
Ultra Monkey Ultra Monkey is a packaged set of binaries for LVS, including Linux-HA for director failover and ldirectord for realserver failover. It's written by Horms, one of the LVS developers. Ultra Monkey was used on many of the server setups sold by VA Linux and presumably made lots of money for them. Ultra Monkey has been around since 2000 and is mature and stable. Questions about Ultra Monkey are answered on the LVS mailing list. Ultra Monkey is mentioned in many places in the LVS-HOWTO.
Keepalived Keepalived is written by Alexandre Cassen, and is based on vrrpd for director failover. Health checking for realservers is included. It has a lengthy but logical conf file and sets up an LVS for you. Alexandre released code for this in late 2001. There is a keepalived mailing list and Alexandre also monitors the LVS mailing list (May 2004, most of the postings have moved to the keepalived mailing list). The LVS-HOWTO has some information about Keepalived.
Alternate hardware: Soekris (and embedded hardware) Clint Byrum cbyrum (at) spamaps (dot) org 27 Sep 2004
I'd like to setup a two node Heartbeat/LVS load balancer using Soekris Net4801 machines. These have a 266Mhz Geode CPU, 3 Ethernet, and 128MB of RAM. The OS (probably LEAF) would live on a CF disk. If these are overkill, I'd also consider a Net4501, which has a 133Mhz CPU, 64MB RAM, and 3 ethernet. I'd need to balance about 300 HTTP requests per second, totaling about 150kB/sec, between two servers. I'm doing this now with the servers themselves (big dual P4 3.02 Ghz servers with lots and lots of RAM). This is proving problematic as failover and ARP hiding are just a major pain. I'd rather have a dedicated LVS setup. 1) anybody else doing this? 2) IIRC, using the DR method, CPU usage is not a real problem because reply traffic doesn't go through the LVS boxes, but there is some RAM overhead per connection. How much traffic do you guys think these should be able to handle?
Ratz 28 Sep 2004 The Net4801 machines are horribly slow but for your purpose enough. The limiting factor on those boxes are almost always the cache sizes. I've waded through too many processor sheets of those Geode derivates to give your specific details on your processor but I would be surprised if it had more than 16kb i/d-cache each.
16k unified cache. :-/
Make sure that your I/O rate is as low as possible or the first thing to blow is your CF disk. I've worked with hundreds of those little boxes in all shapes, sizes and configurations. The biggest common mode failures were CF disk due to temperature problems and I/O pressure (MTTF was 23 days); other problems only showed up in really bad NICs locking up half of the time.
I haven't ever had an actual CF card blow on me. LEAF is made to live on readonly media.. so its not like it will be written to a lot.
Sorry, blow is exaggerated, I mean they simply fail because they only have limited write capacity on the cells. RO doesn't mean that there's no I/O going to your disk as you correctly noted. The problem is that if you plan on using them 24/7 I suggest you monitor your block I/O on your RO partitions using the values from /proc/partitions or the wonderful iostat tool. Then extrapolate about 4 hours worth of samples, check your CF vendor specification on how many writes it can endure and see how long you can expect the thing to run. I have to add that thermal issues were adding to our high failure rates. We wanted to ship those little nifty boxes to every branch of a big customer to do a big VPN network. Unfortunately the customer is in the automobile industry and this means that those boxes were put in the stranges places imaginable in garages sometimes causing major heat congestion. Also as it is usual in this sector of industry people are used to reliable hardware and so they don't care if at the end of a working day they simply shut down the power of the whole garage. Needless to say that this adds up to the reduced lifetime of a CF. I then did a reliability analysis using the MGL (multiple greek letter, derived from the beta-factor model) model to calculate the average risk in terms of failure*consequence and we had to refrain from using those little nifty things. The costs of repair (detection of failure -> replacement of product) at a customer would exceed the income our service provided through a mesh of those boxes.
If these are overkill, I'd also consider a Net4501, which has a 133Mhz CPU, 64MB RAM, and 3 ethernet.
I'd go with the former ones, just to be sure ;).
Forgive me for being frank, but it sounds like you wouldn't go with either of them.
I don't know your business case so it's very difficult to give you a definite answer. I only give you an (somewhat intimidating) experience report, someone might just as well give you a much better report.
I'd need to balance about 300 HTTP requests per second, totaling about 150kB/sec, between two servers.
So one can assume a typical request to your website is 512 Bytes, which is rather quite high. But not really an issue for LVS-DR.
I didn't clarify that. The 150kB/sec is outgoing. This isn't for all of the website, just the static images/html/css.
I'm doing this now with the servers themselves (big dual P4 3.02 Ghz servers with lots and lots of RAM). This is proving problematic as failover and ARP hiding are just a major pain. I'd rather have a dedicated LVS setup.
I'd have to agree to this.
1) anybody else doing this?
Maybe. Stupid questions: How often did you have to failover and how often did it work out of the box?
Maybe once every 2 or 3 months I'd need to do some maintenance and switch to the backup. Every time there was some problem with noarp not coming up or some weird routing issue with the IPs. Complexity bad. :)
So frankly speaking: your HA solution didn't work as expected ;).
2) IIRC, using the DR method, CPU usage is not a real problem because reply traffic doesn't go through the LVS boxes, but there is some RAM overhead per connection. How much traffic do you guys think these should be able to handle?
This is very difficult to say since these boxes impose limits also through their inefficiant PCI busses, their rather broken NICs and the dramatically reduced cache. Also it would be interesting to know if you're planning on using persistency on your setup.
Persistency is not a requirement. Note that most of the time a client opens a connection once, and keeps it up as long as they're browsing with keepalives.
Yes, provided most clients use HTTP/1.1. But since on an application level you don't need persistency. But to give you a number to start with, I would say those boxes should be able (given your constraints) to sustain 5Mbit/s of traffic with about 2000pps (~350 Bytes/packet) and only consume 30 Mbyte of your precious RAM when running without persistency. This is if every packet of your 2000pps is a new client requesting a new connection to the LVS and will be inserted by the template at an average of 1 Minute. As mentioned previously, you HW configuration is very hard to compare to actual benchmarks, thus take those numbers with a grain of salt, please.
Thats not encouraging. I need something fairly cheap.. otherwise I might as well go down the commercial load balancer route.
Well, I have given you number which are (at a second look) rather low estimates ;). Technically, your system should be able to deliver 25000pps (yes, 25k) at a 50Mbit/s rate. You would then, if every packet was a new client, consume about all the memory of your system :). So somewhere in between those two numbers I would place the performance of your machine. Bubba Parker sysadmin (at) citynetwireless (dot) net 27 Sep 2004 In my tests, the Soekris net4501, 4511, and 4521 all were able to route almost 20Mbps at wire-speed. I would suspect the 4801 to be in excess of 50Mbps, but remember, your Soekris board has 3 nics, but what they don't tell you is that they all share the same interrupt, so performance degredation is exponential with many packets per second. Ratz 28 Sep 2004 For all Geode based boards I've received more technical documentation than I was ever prepared to dive in. Most of the time you get a very accurate depiction of your hardware including south and north bridges and there you can see that the interrupt lines are hardwired and require a interrupt sharing. However this is not a problem since there's not a lot of devices on the bus anyway that would occupy it and if you're really unhappy about the bus speed, use setpci to reduce latency for the NIC's IRQs. Newer kernels have excellent handling for shared IRQs btw. Did you measure exponential degradation? I know you get a pretty steep performance reduction once you push the pps too high but I newer saw exponential behaviour. Peter Mueller 2004-09-27
What about not using these Soekris's and just using those two beefy servers? e.g., http://www.ultramonkey.org/2.0.1/topologies/ha-overview.html or http://www.ultramonkey.org/2.0.1/topologies/sl-ha-lb-overview.html
Clint Byrum 27 Sep 2004 Thats what I'm doing now. The setup works, but its complexity causes issues. Bringing up IPs over here, moving them from eth0 to lo over there, running noarpctl on that box. Its all very hard to keep track of. Its much simpler to just have two boxes running LVS, and not worry about whats on the servers. Simple things are generally easier to fix if they break. It took me quite a while to find a simple typo in a script on my current setup, because it was very non-obvious at what layer things were failing.
LVS on a CD: Malcolm Turnbull's ISO files Malcolm Turnbull Malcolm (dot) Turnbull (at) crocus (dot) co (dot) uk 03 Jun 2003, has released a Bootable ISO image of his Loadbalancer.org appliance software. The link was at http://www.loadbalancer.org/modules.php?name=Downloads&d_op=viewdownload&cid=2 but is now dead (Dec 2003). Checking the website (Apr 2004) I find that the code is available as a 30 day demo (http://www.loadbalancer.org/download.html, link dead Feb 2005). Here's the original blurb from Malcolm
The basic idea is creating an easy to use layer 4 switch appliance to compete with Coyote Point Equalizer/ CISCO local director... All my source code is GPL, but the ISO distribution contains files that are non-GPL to protect the work and allow vendors to licence the software. The ISO requires a license before you can legally use it in production. Burn it to CD and then use it to boot a spare server with pentium/celeron + ATAPI CD + 64MB RAM + 1 or 2 NICs+20GB HD Default setup is DR so just plug it straight into the same hub as your web servers and have a play.. Download the manuals for configuration info...
LVS: Ipvsadm and Schedulers ipvsadm is the user code interface to LVS. The scheduler is the part of the ipvs kernel code which decides which realserver will get the next new connection. There are patches for ipvsadm machine readable error codes for ipvsadm stateless entry of ipvsadm commands Mar 2004. There appears to have been introduced a bug in the wrr code. Presumably this will be fixed sometime in the main code, and presumably older versions of ipvs still work (but I don't know how far back you need to go, presumably to the 2.4 kernels). Here are some postings on the matter and links to a patch. Jan Kasprzak kas (at) fi (dot) muni (dot) cz 2005/03/25
port unreachable after RS removal: I use IPVS with direct routing and wrr scheduler. The problem is that for some configurations I get "icmp port unreachable" when one of the real servers fails and is removed from the ip_vs tables. The smallest case where I can replicate the problem is the following: `-' Resolving virtual.service... 1.2.3.4 Connecting to virtual.service[1.2.3.4]:80... failed: Connection refused. ]]> I have verified by tcpdump that no traffic is sent to realserver2 after it is removed from the virtual.service pool. The ICMP "tcp port unreachable" is sent by the ipvs director. This appears to be a problem in the wrr scheduler. With wlc or rr it works as expected. The director is Fedora Core 3 with vanilla 2.6.11.3 kernel, but I have been experiencing this for a longer time.
Sent by: lvs-users-bounces@LinuxVirtualServer.org 2005/03/26 This is exactly the problem I described in my previous mails, and for which a patch is available from Wensong and/or Horms. Search the mailinglist archive for 'overload flag not resetting' which was my initial (wrong) diagnosis. See
Using ipvsadm You use ipvsadm from the command line (or in rc files) to setup: - services/servers that the director directs (e.g. http goes to all realservers, while ftp goes only to one of the realservers). weighting given to each realserver - useful if some servers are faster than others. Horms 30 Nov 2004 The weights are integers, but sometimes they are assigned to an atomic_t, so they can only be 24bits i.e. values so 0 to (2^24-1) should work. scheduling algorithm You use can also use ipvsadm to add services: add a service with weight >0 shutdown (or quiesce) services: set the weight to 0. This allows current connections to continue, untill they disconnect or expire, but will not allow new connections. When there are no connections remaining, you can bring down the service/realserver. delete services: this stops traffic for the service (the connection will hang), but the entry in the connection table is not deleted till it times out. This allows deletion, followed shortly thereafter by adding back the service, to not affect established (but quiescent) connections. Once you have a working LVS, save the ipsvadm settings with ipvsadm-sav ipvsadm.sav ]]> and then after reboot, restore the ipvsadm settings, with ipvsadm-restore Both of these commands can be part of an ipvsadm init script. list version of ip_vs (here 0.9.4, with a hash table size of 4096) list version of ipvsadm (here 1.20)
sysctl documentation the sysctl for ipvs will be in Documentation/networking/ipvs-sysctl.txt for 2.6.18 (hopefully). It is derived from http://www.linuxvirtualserver.org/docs/sysctl.html v1.4.
Compile a version of ipvsadm that matches your ipvs Compile and install ipvsadm on the director using the supplied Makefile. You can optionally compile ipvsadm with popt libraries, which allows ipvsadm to handle more complicated arguments on the command line. If your libpopt.a is too old, your ipvsadm will segv. (I'm using the dynamic libpopt and don't have this problem). Since you compile ipvs and ipvsadm independantly and you cannot compile ipvsadm until you have patched the kernel headers, a common mistake is to compile the kernel and reboot, forgetting to compile/install ipvsadm. Unfortunately there is only rudimentary version detection code into ipvs/ipvsadm. If you have a mismatched ipvs/ipvsadm pair, many times there won't be problems, as any particular version of ipvsadm will work with a wide range of patched kernels. Usually with 2.2.x kernels, if the ipvs/ipvsadm versions mismatch, you'll get weird but non-obvious errors about not being able to install your LVS. Other possibilities are that the output of ipvsadm -L will have IP's that are clearly not IPs (or not the IP's you put in) and ports that are all wrong. It will look something like this RemoteAddress:Port Forward Weight ActiveConn InActConn TCP C0A864D8:0050 rr -> 01000000:0000 Masq 0 0 0 ]]> rather than RemoteAddress:Port Forward Weight ActiveConn InActConn TCP lvs2.mack.net:ssh rr -> RS2.mack.net:ssh Route 1 0 0 ]]> There was a change in the /proc file system for ipvs about 2.2.14 which caused problems for anyone with a mismatched ipvsadm/ipvs. The ipvsadm from different kernel series (2.2/2.4) do not recognise the ipvs kernel patches from the other series (they appear to not be patched for ipvs). The later 2.2.x ipvsadms know the minimum version of ipvs that they'll run on, and will complain about a mismatch. They don't know the maximum version (which will be produced presumably some time in the future) that they will run on. This protects you against the unlikely event of installing a new 2.2.x version of director:/etc/lvs# ipvsadm on an older version of ipvs, but will not protect you against the more likely scenerio where you forget to compile ipvsadm after building your kernel. The ipvsadm maintainers are aware of the problem. Fixing it will break the current code and they're waiting for the next code revision which breaks backward compatibility. If you didn't even apply the kernel patches for ipvs, then ipvsadm will complain about missing modules and exit (i.e. you can't even do `ipvsadm -h`).
Other compile problems Ty Beede tybeede (at) metrolist (dot) net
Ty Beede tybeede (at) metrolist (dot) net on a slackware 4.0 machine I went to compile ipvsadm and it gave me an error indicating that the iphdr type was undefined and it didn't like that when it saw Ty Beede tybeede (at) metrolist (dot) net to ip_fw.h I added ]]> Ty Beede tybeede (at) metrolist (dot) net in ipvsadm.c, which is where the iphdr #structure is defined and everything went ok
Doug Bagley doug (at) deja (dot) com The reason that it fails "out of the box" is because fwp_iph's type definition (struct iphdr) was ]]> (and not included anywhere else) since the symbol __KERNEL_ was undefined. before ]]> in the .c file did the trick.
put realservers in /etc/hosts (from a note by Horms 26 Jul 2002) ipvsadm by default outputs the names of the realservers rather than the IPs. The director then needs name resolution. If you don't have it, ipvsadm will take a long time (upto a minute) to return, as it waits for name resolution to timeout. The only IPs that the director needs to resolve are of the realservers. DNS is slow. To prevent the director from needing DNS, put the names of the realservers in /etc/hosts. This lookup is quicker than DNS and you won't need to open a route from the director to a nameserver. Or you could use `ipvsadm -n` which outputs the IPs of the realservers instead.
RR and LC schedulers On receiving a connect request from a client, the director assigns a realserver to the client based on a "schedule". The scheduler type is set with ipvsadm. The schedulers available are round robin (rr), weighted round robin (wrr) - new connections are assigned to each realserver in turn least connected (lc), weighted least connection (wlc) - new connections go to realserver with the least number of connections. This is not neccessarily the least busy realserver but is a step in that direction. Doug Bagley doug (at) deja (dot) com points out that *lc schedulers will not work properly if a particular realserver is used in two different LVSs. LBLC: a persistent memory algorythm DH: destination hash SH: source hash The original schedulers are rr, and lc (and their weighted versions). Any of these will do for a test setup. In particular, round robin will cycle connections to each realserver in turn, allowing you to check that all realservers are functioning in the LVS. The rr,wrr,lc,wlc schedulers should all work similarly when the director is directing identical realservers with identical services. The lc scheduler will better handle situations where machines are brought down and up again (see thundering herd problem). If the realservers are offering different services and some have clients connected for a long time while others are connected for a short time, or some are compute bound, while others are network bound, then none of the schedulers will do a good job of distributing the load between the realservers. LVS doesn't have any load monitoring of the realservers. Figuring out a way of doing this that will work for a range of different types of services isn't simple (see load and failure monitoring).
Netmask for VIP You setup the RIPs, DIP and other networking with whatever netmask you choose. For the VIP For LVS-DR, LVS-Tun: netmask for VIP on director, realservers must be /32. For LVS-NAT: the netmask can be /32 or the netmask of the RIPs, DIP. You will need to setup the routing for the VIP to match the netmask. For more details, see the chapters for each forwarding method. Horms 12 Aug 2004 The real story is that the netmask works a little differently on lo to other interfaces. On lo the interface will answer to _all_ addresses covered by the netmask. This is how 127.0.0.1/8 on lo ends up answering 127.0.0.0-127.255.255.255. So if you add 172.16.4.222/16 to eth0 then it will answer 172.16.4.222 and only 172.16.4.222. But if you add the same thing to lo then it will answer 172.16.0.0-172.16.255.255. So you need to use 172.16.4.222/32 instead. To clarify - Add 192.168.10.10 to eth0 ifconfig lo:0 192.168.10.10 netmask 255.255.255.0 broadcast 192.168.10.255 up -> Add 192.168.10.0 - 192.168.10.255 to lo ifconfig lo:0 192.168.10.0 netmask 255.255.255.0 broadcast 192.168.10.255 up -> Same as above, add 192.168.10.0 - 192.168.10.255 to lo ifconfig lo:0 192.168.10.10 netmask 255.255.255.255 broadcast 192.168.10.10 up -> Add 192.168.10.10 to lo ]]> Malcolm Turnbull malcolm (at) loadbalancer (dot) org 2005/04/21 On all platforms apart from windows you want 255.255.255.255 for the loopback. On windows you can get away with 255.255.255.0 IF you use a priority 254 80% of the time. 255.255.255.255 can be used if you mod the registry... But we've found that 255.0.0.0 will work better 99% of the time because windows by default uses the smallest subnet first for routing and a class A will never be used instead of a class C.
LBLC, DH schedulers The LBLC code (by Wensong) and the DH scheduler (by Wensong, inspired by code submitted by Thomas Proell proellt (at) gmx (dot) de) are designed for web caching realservers (e.g. squids). For normal LVS services (eg ftp, http), the content offered by each realserver is the same and it doesn't matter which realserver the client is connected to. For a web cache, after the first fetch has been made, the web caches have different content. As more pages are fetched, the contents of the web caches will diverge. Since the web caches will be setup as peers, they can communicate by ICP (internet caching protocol) to find the cache(s) with the required page. This is faster than fetching the page from the original webserver. However, it would be better after the first fetch of a page from http://www.foo.com/*, for all subsequent clients wanting a page from http://www.foo.com/ to be connected to that realserver. The original method for handling this was to make connections to the realservers persistent, so that all fetches from a client went to the same realserver. The -dh (destination hash) algorythm makes a hash from the target IP and all requests to that IP will be sent to the same realserver. This means that content from a URL will not be retrieved multiple times from the remote server. The realservers (eg squids in this case) will each be retreiving content from different URLs. Wensong Zhang wensong (at) gnuchina (dot) org 16 Feb 2001 Please see "man ipvsadm" for short description of DH and SH schedulers. I think some examples to use those two schedulers. Example: cache cluster shared by several load balancers. The DH scheduler can keep the two load balancer redirect requests destined for the same IP address to the same cache server. If the server is dead or overloaded, the load balancer can use cache_bypass feature to send requests to the original server directly. (Make sure that the cache servers are added in the two load balancers in the same order) Diego Woitasen 12 Aug 2003
The scheduling algorithms that use dest IP for selecting the realserver to use (like DH, LBLC, LBLCR) is only aplicable to transparent proxy, this being the only aplication where the dest ip could be variable.
Wensong Zhang wensong (at) linux-vs (dot) org 12 Aug 2003 Yes, you are almost right. LBLC and LBLCR are written for transparent proxy clusters only. DH can be used for transparent proxy cluster and can be used in other clusters needing static mapping. Here's follows a set of exchanges between a Chinese person and Wensong, that were in English, that I didn't follow at all. Apparently it was clear to Wensong.
If lblc uses dh, then is lblc = dh + lc?
Wensong Zhang 09 Mar 2004 Maybe lblc = dh + wlc. n.weight AND * there is a node m with m.conns The difference between lblc and lblcr is that cachenode[dest_ip] in lblc is a server, and cachenode[dest_ip] in lblcr is a server set.
In lblc the server has overloaded and lvs use wlc and allocate a server in half load of the server, Allocate the weighted least-connection server to IP address. Is this means after allocation for ip address, it will not return to past server ?
No, it will not in most cases. There is only one possible situation that the current map expires after it is not used for six minutes, and the past server is the one with least connections when next access to the ip address comes.
scheduling <link linkend="squids">squids</link> The usual problem with squids not using a cache friendly scheduler is that fetches are slow. In this case the website is sending hits to several different RIPs. Some websites detect this and won't even serve you the pages. Palmer J.D.F. J (dot) D (dot) F (dot) Palmer (at) Swansea (dot) ac (dot) uk 18 Mar 2002/
I tried https and online banking sites (e.g. www.hsbc.co.uk). It seems that this site and undoubtedly many other secure sites don't like to see connections split across several IP addresses as happens with my cluster. Different parts of the pages are requested by different realservers, and hence different IP addresses. It gives an error saying... "...For your security, we have disconnected you from internet banking due to a period of inactivity..." I have had caching issues with HSBC before, they seem to be a bit more stringent than other sites. If I send the requests through one of the squids on it's own it works fine, so I can only assume it's because it is seeing fragmented requests, maybe there is a keepalive component that is requested. How do I combat this? Is this what persistence does or is there a way of making the realservers appear to all have the same IP address?
Joe change -rr (or whatever you're running) to -dh. Lars Use a different scheduler, like lblc or lblcr. Harry Yen hyen1 (at) yahoo (dot) com 16 April 2002 What is the purpose of using LVS with Squid to a https site? HTTPs based material typically is not cachable. I don't understand why you need Squid at all. Once a request reaches a Squid and incurs a cache miss, the forwarded request will have Squid IP as the source address. So you need to find a way to make sure all connections from the same client IP to go to the same Squid farm. Then when they incur cache misses, they will wind up via LVS persistency to the same real sever.
The reason https is sent to the squids is because it's much easier to send all browser traffic to the squids and then let them handle it. The only way I seemed to be able to get this to work (IE access the bank site) is to set a persistence (360 seconds), and using lblc scheduling. The current output of ipvsadm is this... I am a tad concerned at the apparent lack of load balancing. squidfarm1.swan.ac.uk:squid Route 1 202 1045 -> squidfarm2.swan.ac.uk:squid Route 1 14 8 ]]> HSBC seems to be a bit more stringent than other sites. If I send the requests through one of the squids on it's own it works fine, so I can only assume it's because it is seeing fragmented requests, maybe there is a keepalive component that is requested. How do I combat this? Is this what persistence does or is there a way of making the realservers appear to all have the same IP address? I have sorted it by using persistence, couldn't get any of the dedicated squid schedulers to work properly. I'm currently running wlc, and 360s persistance. Seems to be holding up really well. Still watching it with eagle eyes though.
The -dh scheduler was written expressly to handle squids. Jezz tried it and didn't get it to work satisfactorily but found that persistence worked. We don't understand this yet. Jakub Suchy jakub (at) rtfm (dot) cz 2005/02/23 round-robin algorithm is not usable for squid. Some servers (banks etc.) check clients ip address and terminates it's connection if it changes. When you use the source-hashing algorithm, IPVS checks the client against its local table and forwards connection always to same squid real server, so the client always accesses the web through same squid. source-hashing can become unbalanced when you have few clients and one of them use squid more frequently than others. With more clients, it's statistically balanced.
LVS with mark tracking: fwmark patches for multiple firewalls/gateways If the LVS is protected by multiple firewall boxes and each firewall is doing connection tracking, then packets arriving and leaving the LVS from the same connection will need to pass through the same firewall box or else they won't be seen to be part of the same connection and will be dropped. An initial attempt to handle the firewall problem was sent in by Henrik Nordstrom, who is involved with developing web caches (squids). This code isn't a scheduler, but it's in here awaiting further developements of code from Julian because it addresses similar problems to the SH scheduler in the next section. Julian 13 Jan 2002
Unfortunately Henrik's patch breaks the LVS fwmark code. Multiple gateway setups can be solved with routing and a solution is planned for LVS. Until then it would be best to contact Henrick, hno (at) marasystems (dot) com for his patch.
Here's Henrick's patch and here's some history. Henrik Nordstrom hno (at) marasystems (dot) com 13 Jan 2002
My use of the MARK is for routing purposes of return traffic only, not at all related to the scheduling onto the farm. This to solve complex routing problems arising in borders between networks where it is impractical to know full routing of all clients. One example of what I do is like this: I have a box connected to three networks (firewall, including LVS-NAT load balancing capabilities for published services) a - Internet b - DMZ, where the farm members are c - Large intranet For simplicity both Internet and intranet users connect to the same LVS IP addresses. Both networks 'a' and 'c' is complex, and maintaining a complete and correct routing table covering one of the networks (i.e. the 'c' network in the above) is on the border to impossible and error prone as the use of addresses change over time. To simplify routing decisions I simply simply want return traffic to be routed back the same way as from where the request was received. This covers 99.99% of all routing needed in such situation regardless of the complexity of the networks on the two (or more) sides without the need of any explicit routing entries. To do this I MARK the session when received using netfilter, giving it a routing mark indicating which path the session was received from. My small patch modifies LVS to memorize this mark in the LVS session, and then restore it on return traffic received FROM the realservers. This allows me to route the return traffic from the farm members to the correct client connection using iproute fwmark based routing rules. As farm distribution algorithms I use different ones depending on the type of application. The MARK I only use for routing of return traffic. I also have a similar patch for Netfilter connection tracking (and NAT), for the same purpose of routing return traffic. If interested search for CONNMARK in the netfilter-devel archives. The two combined allows me to make multihomed boxes who do not need to know the networks on any of the sides in detail, besides it's own IP addresses and suitable gateways to reach further into the networks. Another use of the connection MARK memory feature is a device connected to multiple customer networks with overlapping IP addresses, for example two customers both using 192.168.1.X addresses. In such case making a standard routing table becomes impossible as the customers are not uniquely identified by their IP addresses. The MARK memory however deals with such routing at ease since it do not care about the detailed addressing as long as it possible to identify the two customer paths somehow. i.e. interface originally received on, source MAC of the router who sent us the request, or anything uniquely identifying the request as coming from a specific path. The two problems above (not wanting to known the IP routing, or not being able to IP route) are not mutually exclusive. If you have one then the other is quite likely to occur.
Here's Henrik's announcement and the replies. Henrik Nordstrom 14 Feb 2001
Here is a small patch to make LVS keep the MARK, and have return traffic inherit the mark. We use this for routing purposes on a multihomed LVS server, to have return traffic routed back the same way as from where it was received. What we do is that we set the mark in the iptables mangle chain depending on source interface, and in the routing table use this mark to have return traffic routed back in the same (opposite) direction. The patch also moves the priority of LVS INPUT hook back to infront of iptables filter hook, this to be able to filter the traffic not picked up by LVS but matchin it's service definitions. We are not (yet) interested of filtering traffic to the virtual servers, but very interested in filtering what traffic reaches the Linux LVS-box itself.
Julian - who uses NFC_ALTERED ?
Netfilter. The packet is accepted by the hook but altered (mark changed).
Julian - Give us an example (with dummy addresses) for setup that require such fwmark assignments.
For a start you need a LVS setup with more than one real interface receiving client traffic for this to be of any use. Some clients (due to routing outside the LVS server) comes in on one interface, other clients on another interface. In this setup you might not want to have a equally complex routing table on the actual LVS server itself. Regarding iptables/ipvs I currently "only" have three main issues. As the "INPUT" traffic bypasses most normal routes, the iptables conntrack will get quite confused by return traffic.. Sessions will be tracked twice. Both by iptables conntrack and by IPVS. There is no obvious choice if IPVS LOCAL_IN sould be placed before or after iptables filter hook. Having it after enables the use of many fancy iptables options, but instead requires one to have rules in iptables for allowing ipvs traffic, and any mismatches (either in rulesets or IPVS operation) will cause the packets to actually hit the IP interface of the LVS server which in most cases is not what was intended.
Wensong's SH scheduler Scheduling is based on the IP of the client. Other than the few comments here, no-one has used the -sh scheduler. The -sh (source hash) scheduler was originally intended for directors with multiple firewalls, with the balancing based on hashes of the MAC address of the firewall. However with the scheduling based on the client IP, it would solve some of the problems that currently require persistence (i.e. having a client always go to the same realserver). Here's Wensong's announcement: Wensong Zhang wensong (at) gnuchina (dot) org 16 Feb 2001 Please see "man ipvsadm" for short description