LVS-HOWTO

Table of Contents

1. LVS: Introduction

1.1. Thanks

1.2. About the HOWTO

1.3. Nomenclature/Abbreviations

1.4. Minimal knowledge required

1.5. Free Technical Help

1.6. After you've Got Technical Help

1.7. Paid technical help

1.8. Mailing list: subscribing, unsubscribing, searching

1.9. Mailing list: posting to

1.10. Bug Fixes

1.11. Other load balancing solutions, GPL, opensource and commercial

1.12. Books on LVS

1.13. LVS in the news

1.14. Software/Information/HOWTOs useful/related to LVS

2. LVS: What is an LVS? Can I use an LVS?

2.1. What is a VIP?

2.2. Where do you use an LVS?

2.3. Client/Server relationship is preserved in an LVS

2.4. LVS director is an L4 switch

2.5. LVS forwards packets to realservers

2.6. LVS runs on Linux and FreeBSD directors

2.7. Code for LVS is different for each kernel series

2.8. kernels from 2.4.x series are SMP for kernel code

2.9. OS for realservers

2.10. LVS works on ethernet

2.11. LVS works on IPv6

2.12. LVS is continually being developed

2.13. LVS is 64 bit

2.14. Other documentation

2.15. LVS is not simple to install, get going or keep running

2.16. LVS Control (Failure, Thundering Herd, Sorry Servers)

2.17. clients on realservers

3. LVS: Install, Configure, Setup

3.1. Installing from Source Code

3.2. Ultra Monkey

3.3. Keepalived

3.4. ipvsman(d)

3.5. Alternate hardware: Soekris (and embedded hardware)

3.6. LVS on a CD: Malcolm Turnbull's ISO files

4. LVS: Ipvsadm and Schedulers

4.1. Using ipvsadm

4.2. Memory Requirements

4.3. sysctl documentation

4.4. Compile a version of ipvsadm that matches your ipvs

4.5. put realservers in /etc/hosts

4.6. RR and LC schedulers

4.7. Netmask for VIP

4.8. LBLC, DH schedulers

4.9. LVS with mark tracking: fwmark patches for multiple firewalls/gateways

4.10. SH scheduler

4.11. What is an ActiveConn/InActConn (Active/Inactive) connnection?

4.12. FAQ: ipvsadm shows entries in InActConn, but none in ActiveConn, connection hangs. What's wrong?

4.13. FAQ: initial connection is delayed, but once connected everything is fine. What's wrong?

4.14. unbalanced realservers: does rr and lc weighting equally distribute the load? - clients reusing ports

4.15. Changing weights with ipvsadm

4.16. Setting initial weights

4.17. Dynamically changing realserver weights

4.18. feedbackd

4.19. lvs-kiss

4.20. connection threshold

4.21. Flushing connection table

4.22. Thundering herd problem, Slow start code for realserver(s) coming on line

4.23. Handling kernel version dependant files e.g. System.map and ipvsadm

4.24. Limiting number of clients connecting to LVS

4.25. Who is connecting to my LVS?

4.26. experimental scheduling code

4.27. Ratz's primer on writing your own scheduler

4.28. changing ip_vs behaviour with sysctl flags in /proc

4.29. Counters in ipvsadm

4.30. Exact Counters

4.31. Scheduling TCP/UDP/SCTP/TCP splicing/

4.32. patch: machine readable error codes from ipvsadm

4.33. patch: stateless ipsvadm - add/edit patch

4.34. patch: fwmark name-number translation table

4.35. ip_vs_conn.pl

4.36. Luca's php monitoring script

4.37. ipvsadm set option

4.38. ipvsadm error messages

4.39. ipvsadm fast update bug with smp

4.40. Problems when no scheduler

5. LVS: LVS-NAT

5.1. Introduction

5.2. LVS-NAT bugs

5.3. Example 1-NIC, 2 Network LVS-NAT (VIP and RIPs on different network)

5.4. All packets sent from the LVS-NAT realserver to the client must go through the LVS-NAT director

5.5. Run the configure script

5.6. Setting up demasquerading on the director; 2.4.x and 2.2.x

5.7. rewriting, re-mapping, translating ports with LVS-NAT

5.8. masquerade timeouts

5.9. Julian's step-by-step check of a L4 LVS-NAT setup

5.10. How LVS-NAT works

5.11. In LVS-NAT, how do packets get back to the client, or how does the director choose the VIP as the source_address for the outgoing packets?

5.12. One Network LVS-NAT

5.13. re-mapping ports, rewriting is slow for 2.0, 2.2 kernels

5.14. Two instances of demon running on realserver

5.15. Performance of LVS-NAT

5.16. Various debugging techniques for routes

5.17. Connecting directly from the client to a service:port on an LVS-NAT realserver

5.18. A NAT router has no connections

5.19. Thoughts on extending NAT

5.20. Postings from the mailing list

5.21. LVS-NAT source routing patch (Brownfield, Sawari and Black)

5.22. LVS-NAT FTP Recipe

5.23. LVS-NAT vhosts with apache

5.24. LVS-NAT timeout problem

6. LVS: The ARP Problem

6.1. The problem

6.2. Put the VIP on the realservers lo device

6.3. The Cure(s)

6.4. The Cure: 2.0 kernels - nothing needed

6.5. The Cure: 2.2.x kernels - many options

6.6. The Cure: 2.4.x kernels - arp_ignore/arp_announce

6.7. The Cure: 2.6.x kernels - arp_ignore/arp_announce

6.8. arptables

6.9. The arp problem is on the realserver's VIP not the RIP

6.10. Testing an interface for replies to arp requests

6.11. Normal machines, Solaris, Novell Server

6.12. problems with switches

6.13. The ARP problem, the first inklings

6.14. A posting to the mailinglist by Peter Kese explaining the "arp problem"

6.15. arp bouncing

6.16. Lar's Method

6.17. Static Routing to Director

6.18. iproute2 arp on|off flag

6.19. Is the arp behaviour of 2.2.x kernel a bug?

6.20. The device doesn't reply to arp requests, the kernel does.

6.21. Properties of devices for the VIP

6.22. Topologies for LVS-DR and LVS-Tun LVS's

6.23. Why do all devices broadcast the arp replies

6.24. A discussion about the arp problem

6.25. ATM/ethernet and router problems

6.26. Same IP on multiple NICs

7. LVS: LVS-DR

7.1. LVS-DR example

7.2. How LVS-DR works

7.3. Handling the arp problem for LVS-DR

7.4. LVS-DR scales well

7.5. LVS-DR director as default gw for realservers, transparent proxy and Julian's martian and forward_shared patches

7.6. Accepting packets on LVS-DR director by fwmarks

7.7. security concerns: default gw(s) and routing with LVS-DR/LVS-Tun

7.8. routing to realserver from director

7.9. LVS-DR, LVS-Tun need rp_filter=0

7.10. Director as client in LVS-DR

7.11. from the mailing list

7.12. rewriting, re-mapping, translating ports with LVS-DR

8. LVS: LVS-Tun

8.1. LVS-Tun Intro

8.2. LVS-Tun example setup

8.3. You need a tunl0 device

8.4. the ARP problem with LVS-Tun

8.5. Reply packets appear to be spoofed

8.6. How LVS-Tun works

8.7. The RIP (not the tunl device) receives the ipip packet

8.8. Configure LVS-Tun

8.9. set rp_filter correctly

8.10. FreeBSD and Solaris realservers with LVS-Tun

8.11. Windows realservers with LVS-Tun

8.12. Realservers without ipip encapsulation

8.13. LVS-Tun has smaller MTUu: PMTU is disabled - handling fragmentation

8.14. MTU: early signs of problems

8.15. tunl mtu solved: Setting the MTU by MSS with iptables on the realserver

8.16. Setting the MTU by route

8.17. rewriting, re-mapping, translating ports with LVS-Tun

9. LVS: LocalNode

9.1. Two LocalNode Servers

9.2. Two Box LVS

9.3. Two Box LVS: both directors have active ipvsadm entries

9.4. Testing LocalNode

9.5. Localnode on the backup director

9.6. rewriting, re-mapping, translating ports with Localnode

10. LVS: You can't map (or rewrite) ports with LVS-DR, LVS-Tun or localnode (but you can with iptables)

10.1. You can't rewrite ports with localnode (but you can with iptables)

10.2. rewriting, re-mapping, translating ports with iptables in LVS-DR

10.3. can't port map with LVS

11. LVS: Non-LVS clients on Realservers

11.1. always NAT out clients through VIP

11.2. Masquerading clients on realservers to the outside world (SNAT)

11.3. Masquerading clients on LVS-NAT realservers

11.4. Masquerading clients on LVS-DR realservers

11.5. Masquerading clients on LVS-Tun realservers

11.6. Masquerading clients through the VIP on the director

11.7. 3-Tier LVS

11.8. Routes needed for 3-Tier LVS

11.9. Setting up routes using iptables and iproute2

11.10. from the mailing list

12. LVS: LVS clients on Realservers

12.1. Do you really need LVS clients on the realserver in a 3-Tier setup?

12.2. Realserver as LVS client in LVS-NAT

12.3. Realserver as LVS client in LVS-DR

12.4. Markus's thoughts

13. LVS: Non Linux Realservers

13.1. Loopback interface on Windows/Microsoft/NT/W2K

13.2. Windows Server 2008

13.3. Mac OS X (and Solaris)

13.4. Windows servers in Active Directory Domain

14. LVS: identd/authd

14.1. What is authd/identd?

14.2. authd/identd and other 3-Tier clients

14.3. symptoms of the identd problem

14.4. comp.os.linux.security FAQ on identd

14.5. Russ Nelson on identd

14.6. Why identd is a problem for LVS

14.7. tcpdumps of connections delayed by identd

14.8. There are solutions to identd problem in some cases

14.9. Turn off tcpwrappers

14.10. using iptables to handle identd

14.11. Identd and smtp/pop/qmail

15. LVS: Variants on LVS: Local Nodes (One Box LVS)

16. LVS: Variants on LVS: Peter Warasin's ip_vs() in PREROUTING

17. LVS: Variants on LVS: Sven Ulland's Two-node setup with overlapping client subnets

18. LVS-J: Ludo's reinJect Forwarder: using the director as a gateway to load balance connections to the internet

18.1. Introduction

18.2. reinJect setup with ipvsadm

18.3. The target LVS: sending packets with dst_addr=0/0 to ip_vs

18.4. setting up LVS-J forwarding

18.5. SNAT'ing the output

18.6. LVS-J discussion by Ludo

19. LVS: Services: general, setup, debugging new services

19.1. Single port services are simple

19.2. setting up a (new) service

19.3. services must be setup for forwarding type

19.4. Realservers present the same content: Synchronising (filesharing) content and config files, backing up realservers

19.5. cfengine for synchronising files

19.6. File Systems for (really big) Clusters: Lustre, Panasas

19.7. File Systems for Clusters: Samba waits for a commit and is slow, NFS fills buffers and is fast

19.8. Discussion on distributed filesystems

19.9. load balancing and scheduling based on the content of the packet: Cookies, URL, file requested, session headers

19.10. Timeouts for TCP/UDP connections to services

19.11. name resolution on realservers: running name resolution friendly demons on realservers

19.12. Debugging new services

19.13. "broken" services:servlets and j2ee

19.14. http logs, error logs

20. LVS: Services: single-port

20.1. ftp, tcp 21

20.2. ssh, sftp, scp, tcp 22

20.3. telnet, tcp 23

20.4. smtp, tcp 25; pop3, tcp 110; imap tcp/udp 143 (imap2), 220(imap3). Also sendmail, qmail, postfix, and mailfarms.

20.5. Mail Farms

20.6. dns, tcp/udp 53 (and dhcpd server 67, dhcp client 68)

20.7. http name and IP-based (with LVS-DR or LVS-Tun), tcp 80

20.8. http with LVS-NAT

20.9. httpd is stateless and normally closes connections

20.10. netscape/database/tcpip persistence (keepalives)

20.11. dynamically generated images on web pages

20.12. http: sanity checks, shutting down, indexing programs (modifying /etc/hosts), htpasswd, apache proxy and reverse proxy to look at URL, mod_backhand, logging

20.13. HTTP 1.0 and 1.1 requests

20.14. Large HTTP /POST with LVS-Tun

20.15. http keepalive - effect on InActConn

20.16. Fallback/Sorry pages with Apache

20.17. Testing http with apachebench (ab)

20.18. Apache setup for DoS

20.19. squids, tcp 80, 3128

20.20. authd/identd, tcp 113 and tcpwrappers (tcpd)

20.21. ntp, udp 123

20.22. https, tcp 443

20.23. name based virtual hosts for https

20.24. Obtaining certificates for https

20.25. Self made certificates

20.26. SSL Accelerators and Load Balancers

20.27. SSL termination at localnode: patch by Carlos Lozano, Siim Poder and Malcolm Turnbull

20.28. r commands; rsh, rcpi (and their ssh replacements), tcp 514

20.29. lpd, tcp 515

20.30. Databases

20.31. Databases: mysql

20.32. Using Zope with databases

20.33. Databases: Microsoft SQL server, tcp 1433

20.34. Databases: Oracle

20.35. Databases: ldap, tcp/udp 389, tcp/udp 636

20.36. nfs, udp 2049

21. LVS: Services: multi-port

21.1. Introduction

21.2. ftp general, active tcp 20,21; passive 21,high_port

21.3. ftp helper modules: ip_vs_ftp/ip_masq_ftp

21.4. ftp (active) - the classic command line ftp

21.5. ftp (passive)

21.6. ftp helper bug(s)

21.7. ftp is difficult to secure

21.8. ftps (ssl based ftp), tcp 21, 22?

21.9. dns, tcp/udp 53 (and dhcpd server 67, dhcp client 68)

21.10. samba, udp 137, udp 138, tcp 139, tcp 445

21.11. xdmcp, X-window, udp 177 (xdmcp), tcp 6000 (and ssh X-forwarding)

21.12. r commands; rsh, rcp, and their ssh replacements, tcp 513 (,514) and another connection

21.13. Streaming Media: RealNetworks, Quicktime, Windows Media Server, tcp/udp 554 (and other ports)

21.14. Radius, udp 1645,1646

22. LVS: Services that we haven't got to work with LVS yet

22.1. Kerberos

22.2. RMI

23. LVS: UDP Services - unique problems

23.1. SIP (Session Initiation Protocol)

23.2. UDP timeouts (SIP)

23.3. UDP timeouts (DNS)

23.4. Julian's One Packet Scheduler (OPS) for UDP, timeouts for DNS

23.5. icmp responses aren't generated by UDP timeouts on VIP-less directors

24. LVS: Routing and packet delivery to a director without a VIP (for fwmark and transparent proxy)

24.1. Introduction

24.2. Routing to and accepting packets by a VIP-less director

24.3. Routing to the MAC address of the director

24.4. Julian's iproute2 solutions

24.5. Ludos LVS target in iptables

24.6. Transparent proxy Q and A

24.7. Other tricks

25. LVS: Fwmarks (firewall marks)

25.1. Introduction

25.2. ipvsadm syntax for fwmark

25.3. setting up routing and packet delivery to the director

25.4. single-port service: telnet with fwmarks

25.5. Grouping services: single group, active ftp(20,21)

25.6. Grouping services: two groups, active ftp(20,21) and e-commerce(80,443)

25.7. passive ftp

25.8. fwmark with LVS-NAT

25.9. collisions between fwmark and VIP rules

25.10. persistence granularity with fwmark

25.11. fwmark allows LVS-DR director to be default gw for realservers

25.12. fwmark simplifies configuration for large numbers of addresses

25.13. Example: firewall farm

25.14. Example: LVS'ing a CIDR block

25.15. Example: forwarding based on client source IP

25.16. Example: load balancing multiple class C networks

25.17. Example: proxy server

25.18. Example: transparent web cache

25.19. Example: Multiply-connected router

25.20. httpd clients (browsers)

25.21. Example: dynamically generated images in webpages

25.22. Example: Balancing many IPs/services as one block

25.23. Example: Source controlled LVS - services and realserver customised by Client IP

25.24. Appendix 1: Specificiations for grouping of services with fwmarks

25.25. Appendix 2: Demonstration of grouping services with fwmarks

25.26. Appendix 3: Announcement of grouping services with fwmarks

25.27. fwmark examples from the mailing list

26. LVS: Transparent proxy (TP or Horms' method)

26.1. setting up routing and packet delivery to the director

26.2. General

26.3. How you use TP

26.4. The original 2.2 TP setup method

26.5. Transparent proxy for 2.4.x (and presumably 2.6.x)

26.6. Experiments showing that 2.4TP is different to 2.2TP

26.7. What IP TP packets arriving on?

26.8. Take home lesson for setting up TP on realservers

26.9. Handling identd requests from 2.4.x LVS-DR realservers using TP

26.10. Performance of Transparent Proxy

26.11. The difference between REDIRECT and TPROXY

27. LVS: Transparent Bridging

28. LVS: Persistent Connection (Persistence, Affinity in cisco-speak)

28.1. LVS persistence

28.2. Scheduling looks different under persistence

28.3. Persistent and regular (non-persistent) services together on the same realserver.

28.4. Tracing connections: where will the client connect next?

28.5. Bringing down persistent services.

28.6. Forcing a break in a persistent connection: expire_quiescent_template - Horms sysctl for quiescing persistent connections

28.7. what if a realserver holding a persistent (sticky) connection crashes

28.8. Load Balancing time constant is longer with persistence

28.9. The tcp NONE flag

28.10. Resetting the persistence timeout counter (persistence behaviour for short timeout values)

28.11. Why you don't want persistence for your e-commerce site: why you should rewrite your application

28.12. more about e-commerce sites: we used to think memory was the problem - it isn't

28.13. persistence with windows realservers

28.14. messing with the ipvsadm table while your LVS is running

28.15. Persistence for multiport services

28.16. Proxy services, e.g. AOL

28.17. key exchanges (SSL)

28.18. About longer timeouts

28.19. passive ftp and persistence

28.20. The Persistence Template (about port 0)

28.21. persistent clients behind a proxy or nat box

28.22. Rogue clients hidden by persistence

28.23. Long (1 day) persistence to windows terminal servers

29. LVS: Running a firewall on the director: Interaction between LVS and netfilter (iptables).

29.1. Start with no filter rules

29.2. Introduction

29.3. Netfilter hooks and LVS: the path/route of an ip_vs controlled packet

29.4. how to filter with netfilter

29.5. ipvs_nfct, netfilter connection tracking for ipvs

29.6. LVS-NAT netfilter conntrack example with ftp

29.7. tcpdump is LVS compatible

29.8. Writing Filter Rules

29.9. The Antefacto Netfilter Connection Tracking patches

29.10. The design of LVS as a netfilter module, pt1

29.11. The design of LVS for Netfilter and Linux 2.4, pt2

29.12. Example ip_tables filter scripts

29.13. performance hit on director with iptables/netfilter

29.14. Long sessions through LVS DR director terminated by icmp-host-prohibited (ICMP type 3 code 10)

29.15. stateful filtering: LVS-NAT

29.16. stateful filtering: LVS-DR

30. LVS: Cluster friendly versions of applications that need to maintain state

30.1. rewriting your application/service

30.2. Session Data, maintaining state in a cluster, from Andreas Koening

30.3. Single Session

30.4. IIS session management: how it works

30.5. Maintaining state with persistence

30.6. How others maintain state

31. LVS: Squid Realservers (poor man's L7 switch)

31.1. Terminology

31.2. Preview

31.3. Let's start assembling

31.4. One squid

31.5. Another squid

31.6. Combining pieces with LVS

31.7. Problems

32. LVS: Performance and Kernel Tuning

32.1. Performance Articles

32.2. Estimating throughput: Rule of Thumb

32.3. Estimating throughput: 100Mbps FE is really 8000packets/sec ethernet

32.4. Jumbo frames

32.5. Network Latency

32.6. Mixture of 100Mbps and GigE ethernet

32.7. NICs and Switches, 100Mbps (FE) and 1Gbps (GigE)

32.8. Ethernet,NIC Bonding

32.9. NIC problems - eepro100

32.10. NIC problems - tulip

32.11. dual/quad ethernet cards, IRQ sharing problems

32.12. Flakey Switch

32.13. performance testing tools

32.14. Max number of realservers

32.15. FAQ: What is the minimum hardware requirements for a director

32.16. FAQ: How fast/big should my director be?

32.17. SMP doesn't help, but 64 bit does

32.18. Performance Hints from the Squid people

32.19. realservers filling conntrack tables (LVS-DR)

32.20. Conntrack, effect on throughput

32.21. Don't use the preemptible/preemptable/preemptive kernels

32.22. 9.6Gbps served using LVS-DR with gridftp

33. LVS: Monitoring

33.1. CPU usage/load level on the director?

33.2. LVS throughput at the director with ipvsadm

33.3. Monitoring: LVS director throughput statistics from the /proc system (originally /proc/net/ip_vs_stats)

33.4. MRTG family: Intro

33.5. MRTG family: LVSGSP

33.6. MRTG

33.7. MRTG family: RRDtool

33.8. MRTG family: cacti

33.9. MRTG family: Ganglia (incl. INSTALL)

33.10. MRTG family: rrd images

33.11. Nagios

33.12. MIB/SNMP

33.13. home brew MIB/SNMP

33.14. Disks

33.15. Other output GUIs

34. LVS: Details of LVS operation, Security, DoS

34.1. Top 20 security vunerabilities

34.2. Top 75 security tools from the people at nmap

34.3. Network Testing with Abberant Packets

34.4. Do I need security, really?

34.5. What to do after a break-in, prevention strategies

34.6. More about syncookies

34.7. Can filter rules stop the intruder hopping to other machines?

34.8. Where filter rules act

34.9. /proc filesystem flags for ipv4, e.g.rp_filter

34.10. tcp timeout values, don't change them (at least yet)

34.11. /proc file system settings for LVS: security and private copies of tcp timeouts for LVS connections (you can change these)

34.12. timeouts the same for all services

34.13. Director Connection Hash Table

34.14. Hash table connection timeouts

34.15. Hash Table DoS

34.16. Hash table size, director will crash when it runs out of memory.

34.17. The LVS code does not swap

34.18. Other factors determining the number of connections

34.19. Port range: limitations, expanding port range on directors

34.20. Director does not have any ports (connections) open for an LVS connection

34.21. apps starved for ports

34.22. realserver running out of ports

34.23. Maximum number of NICs

34.24. DoS

34.25. DoS, from the mailing list

34.26. Testing DoS Strategies with testlvs: Creating large numbers of InActConn

34.27. Debugging LVS

34.28. realserver content: filesystem or database? (the many reader, single writer problem)

34.29. Developement: Supporting IPSec on LVS

35. LVS: ICMP

35.1. MTU discovery and ICMP handling

35.2. LVS code only needs to handle icmp redirects for LVS-NAT and not for LVS-DR and LVS-Tun

35.3. ICMP checksum errors

35.4. ICMP Timeouts

35.5. PMTUD (path MTU discovery)

35.6. Long sessions through LVS DR director terminated by icmp-host-prohibited (ICMP type 3 code 10)

36. LVS: High Availability, Failover protection

36.1. Introduction

36.2. Single Point of Failure (SPOF) - you can't protect against everything

36.3. Stateful Failover

36.4. Director failure

36.5. UltraMonkey and Linux-HA

36.6. Keepalived and Vrrpd

36.7. Using keepalived to failover routers

36.8. monitoring/failover messages should stay internal to LVS

36.9. Parsing problems with vrrpd config file

36.10. Two instances of vrrpd

36.11. HA MySQL

36.12. Failover of large numbers (say 1024) of VIPs

36.13. Some vrrpd setup instructions

36.14. Filter rules for vrrpd broadcasts

36.15. Vinnie's comparison between ldirectord/heartbeat and keepalived/vrrpd

36.16. Saru: All directors active at the same time

36.17. Active/Active by multipath: random musings

36.18. Server Load Balancing Registration Protocol

36.19. using iproute2 to keep demons running during failover, while link is down

37. LVS: Dynamic Routing, multiple gateways, realservers in multiple LVSs, dead gateway detection

37.1. Setting up multiple gateways: Realservers shared between two LVSs: ip route append

37.2. Connecting from clients through multiple parallel links: the dead gateway problem

37.3. Dynamic Routing to handle loss of routing in directors

37.4. Dynamic routing with gated: An LVS that connects to the outside world through two networks

37.5. flapping stemming from convergence time for spanning tree

38. LVS: Server State Sync Demon, syncd (saving the director's connection state on failover)

38.1. Intro

38.2. Release Notice

38.3. Expiration of Connection in Backup Director

38.4. Syncd boxes must have the same time

38.5. LVS and syncd do not use conntrack

38.6. Connection Synchronisation (TCP Fail-Over)

38.7. The synchd produces broadcast traffic

38.8. from the mailing list

38.9. Bug (fixed) in syncd: mixed endianness on directors

39. LVS: Realserver failure handled by Mon

39.1. Introduction

39.2. ethernet NIC failure, and channel bonding

39.3. Service/realserver failout: mon, ldirectord

39.4. Is ldirectord multithreaded? (ldirectord running high %CPU)

39.5. overriding ldirectord health checks from the command line

39.6. Mon for server/service failout

39.7. Monitoring the service running on the VIP on the realserver from the director

39.8. About Mon

39.9. Mon Install

39.10. Mon Configure

39.11. Testing mon without LVS

39.12. Can virtualserver.alert send commands to LVS?

39.13. Running mon with LVS

39.14. Why is the LVS monitored for failures/load by an external agent rather than by the kernel?

39.15. Running multiple directors (each with their own IP)

39.16. Mon scripts from Christopher DeMarco

40. LVS: Setting up Linux-HA for directors (mostly by using rpms)

40.1. linux-ha howto

40.2. Fix the (possible) ethernet alias issue.

40.3. Configure /etc/ha.d/. files.

40.4. Stop ldirectord from starting, ensure heartbeat starts on reboot

40.5. starting heartbeat and verifying functionality

40.6. Test your fail-over features, understand HA.

40.7. Configuration of mon - recommended

41. LVS: Director failover using heartbeat

41.1. Introduction

41.2. On using serial and ethernet connections for heartbeat

41.3. Ard van Breeman's replacement for IPaddr using ip and arping

42. LVS: Running LVS under UML (User Mode Linux), by Brett Elliot

42.1. Introduction

42.2. Ethernet bridging

42.3. Putting it all together: UML + LVS examples (not finished)

43. LVS: Newer networking tools: Policy Routing

43.1. Introduction

43.2. Policy Routing and ifconfig

43.3. Various debugging techniques for routes

43.4. checking source routed packets

43.5. handling arp problem with iproute2

43.6. ip commands you mightn't know about

43.7. Ratz's corrections on common iproute2/aliases misconceptions

43.8. Ratz's wrappers (for iproute2)

44. LVS: Weird hardware (and software)

44.1. Arp caching defeats Heartbeat switchover

44.2. Weird Hardware I: cisco catalyst routers gratuitously cache arp data (failover is slow)

44.3. Weird Hardware II: autonegotiation failure on cisco CSS 11050

44.4. Weird Hardware III: Watchguard firewall at client site

44.5. Weird Hardware IV: wrong device gets MAC address

44.6. Weird Hardware V: SonicWAll firewall rewriting sequence numbers

44.7. Weird Hardware VI: cisco 2924XL switch

44.8. Weird Hardware VII: unknown switches don't defragment

44.9. Weird Hardware VIII: bad routers/routing tables at ISP

44.10. Possible Wierd Hardware (or driver) IX: Broadcom GigE card

44.11. slow nics

44.12. PCI-X nics

44.13. Microsoft http clients and servers violate the RFC for TCP/IP

44.14. MSIE SSL bugs

45. LVS: Misc/FAQ/Wisdom from the mailing list

45.1. Having one director handling multiple LVS sites, Multiple VIPs

45.2. Setting up a fake service on the realserver with inetd

45.3. How to bring down a realserver for maintenance (eg swap disks)

45.4. keepalived: temporarily removing a realserver from view of keepalived; abnormal termination of keepalived

45.5. Howto turn your single node ftp/http server into an LVS without taking it off-line

45.6. shutdown of LVS

45.7. Other projects like LVS - Beowulf

45.8. Projects like LVS - Eddie

45.9. Recommendations for a redundant file system, RAID

45.10. on the need for extended testing

45.11. Bringing down aliased devices

45.12. Multiple IPs on the Director

45.13. Testimonials

45.14. Transport Layer Security(TLS)

45.15. Setting up a hot spare server

45.16. An LVS of LVSs

45.17. LVS on a Linux/IBM mainframe

45.18. mqseries

45.19. LVS log files

45.20. LVS and linux vlan

45.21. multi-home, multi-router LVS

45.22. Horror story, mostly from slow file system with disk intensive application

45.23. RTNETLINK answers:

45.24. LVS chokes on 600+ connections

45.25. Anti load balancing: all traffic required to go to one realserver

46. LVS: L7 Switching

46.1. Introduction

46.2. KTCPVS

46.3. DRWS

46.4. Alexandre's (unamed) L7 code

46.5. UltraMonkey-L7

46.6. from the mailing list about L7 switching

46.7. What is TCPSP?

47. LVS: Geographically distributed load balancing

47.1. Determining Location from the IP

47.2. Supersparrow

47.3. sharing/separate routers

47.4. Other uses of BGP4 with LVS

47.5. Geographically remote nodes connected by Bridging

47.6. Load Balancing by DNS (round robin DNS)

47.7. BIND, BGP with load balancing (more ideas from Horms)

47.8. Commercial Geographically Distributed Servers

47.9. from the mailing list

48. LVS: Loadbalancing with unmodified realservers

48.1. F5-SNAT

48.2. NetScaler

48.3. Using MASQ with REDIRECT to accept packet on realserver to replace a NetScaler

48.4. Using HAProxy with LVS to substitute for the remote server failover of a NetScaler

49. LVS: Virtualised Hosts in a Linux Virtual Server

49.1. Introduction

49.2. Virtualised Realsevers: VMWare/Xen

49.3. Running a test LVS (director, backup director and realservers) on one box (UML, VMWare)

49.4. VMWare problems with ntp

49.5. Xen tcpip checksum bug

49.6. Random observations thrashing around trying to get Xen/LVS-NAT working

50. LVS: Linux Distributions prepatched with LVS, Unsupported LVS addons

50.1. Distributions prepatched with LVS

50.2. PB's Nutshell HOWTO for Piranha/LVS-NAT

50.3. Horms advice for installing on RedHat systems

50.4. Recipe and LVS binaries for RedHat from Alex Kramarov

50.5. recipes for installing with RedHat from the mailing list

50.6. Hidden RPMs

51. LVS: Useful things that have no other place

51.1. Ramdisk

51.2. cscope

51.3. Neutral currents in multiphase power lines with non-linear loads (like computers with switching power supplies)

51.4. netcat/phatcat

52. LVS: FAQ

52.1. When will LVS be ported to Solaris, xxxBSD...?

52.2. Is there a HOWTO in Japanese, French, Italian, Mandarin...?