1st LVS Trivia Quiz, Jul 2000 To coincide with Wensong's talk at the Ottawa Linux Symposium 19-22 Jul 2000. Disclaimer: All questions have been painstakingly researched. The answers have not and will be posted after I return from Ottawa. (some answers are in the "it depends" category. All correct answers are acceptable). Alternate answers and flames from sore losers will be hotly disputed on the mailing list after the official answers are posted. Questions apply to the state of the LVS art at the end of Jun 2000. Rules: Mark all answers that are correct Score: 1 point for every correct answer or factoid. I had to pose some questions so they didn't give away the answers to other questions. If you thought some questions were ambiguous and steered you away from the correct answer, you're right. Have you ever read a man page or got an error message that gave you a straight answer? Of course not, and we're not going to have any of that here either. This test was for people who code in 1's and 0's and not for weenies who need compilers. Competition Divisions: You can do this 1. from memory only 2. using any means at your disposal. People using external information are required to drink 1 free beer for every piece of help, before going to the next question. Warning: This quiz may contain references to violence, sex, incitement to illegally overthrow the goverment, foul language and bad grammar. Then on the other hand it may not. You have been warned. Personalities Category 1. Wensong lives in Changsha, China where he's a. an academic at the National Laboratory for Parallel & Distributed Processing b. a student c. unemployable A. Wensong is a PhD student at the NLPDP. both (b) and (c) are acceptable 2. Who is "hidden"? A. Julian Anastasov Julian wrote the patch to hide the lo:0 device from arp requests in the 2.2 kernels 3. According to an unofficial count of the postings to the net, the most prolific posters are all born in the one country. a. What's the country b. Who are these blabbermouths? A. Australia, Horms and Joe 4. What is Horms real name? A. Simon Horman (I didn't know this. For more than you'd ever want to know about Horms see http://www.us.vergenet.net/~horms/about_me.html) 5 The machine that hosts the LVS primary website (www.linuxvirtualserver.org) and mailing list a. is in what country? b. is provided by which LVS person? A. Germany, Lars Marowsky-Bree (who works for SuSE). There are many mirror sites (http://www.linuxvirtualserver.org/mirrors.html) Total World Domination Category 1. Name commercial products based on LVS code. A. (in order of release dates) TurboLinux Cluster Server Red Hill Networks' WebMux RedHat's Linux Clustering Solution (any others?) Alinka (added after quiz) Heroics Category 1 point for each of the following 1. Set up a working LVS by any method. 2. Set up a working LVS completely from the command line. 3. Posted to the mailing list (1 point each) a. anything at all b. something useful c. nothing because there is too much noise there already. 4. Have earned money for LVS work. Lifestyle Questions 1. Have you ever programmed through to sunrise, because you couldn't stop? 1 point for yes 2. What is the normal number of ball point pens (biros (R)) that you can put into a plastic pocket protector in a standard business shirt, if you take your calculator out? A. This was for the engineers. Correct answer: the number you can grasp in your hand when grabbing them out of a bucket. For everyone else the correct response is "What's a business shirt?" 3. What are the advantages of no-iron shirts? A. This was a trick question. The correct response is bewilderment or panic realising there was no chance of an answer (1 point). For those that had any answer at all or tried to think of an answer, subtract 1 point. 4. How many females have posted to the mailing list? a. 0 b. 1-100 c. many A. (a) (as far as I know) If you're a female, the give yourself a bonus point for having survived in this statistically unlikely grouping of humanity. If you're male, give yourself a bonus point for choosing to join this statistically unlikely grouping of humanity. Techical Questions 1. How many penguins are in the LVS logo? A. 4 2. LVS has 3 distinct methods of getting a packet from the director to the realservers, VS-NAT, VS-DR, VS-TUN. a. which put the most load on the director b. which have the lowest latency A. (a) VS-NAT (b) VS-DR, VS-TUN (same low latency) 3. Which of these pieces of hardware/software/companies are/make an L4 switch a. F5 Y - makes BIG/ip b. cisco Y - makes Cisco Local Director c. mon N (program used to monitor realservers) d. BIG/ip Y made by F5 above e. Matterhorn N (mountain in Europe) f. Alteon Y g. Redwood N (maker of tape drives) h. Kudzu N (imported plant, an environmental disaster in the US south east) i. lvs Y j. SGI N (maker of computers) k. ldirectord N (program used to monitor realservers) 4. Does LVS work on non-Intel Linux directors (if yes, which hardware is known to work) a. yes b. no A. (a) DEC Alpha. At Redhat, one director in their Piranha setup is DEC alpha, the other director being Linux Intel. Another poster had a Dec Alpha director. (ie LVS is 64 bit tested). 5. Does LVS look inside the ethernet frames (or equivalent for other transport) or does it look at the contents (data) of the packet before deciding what to do with it. a. ethernet frame b. data A. (a) ethernet frame. LVS is a layer 4 (L4) switch and only looks at the IP headers. A switch that looks at the data (payload) of the packet is an application layer (L7) switch. Such capability would be nice to have in LVS and would allow session management. We hope that a future version of LVS will have L7 switching. 6. Can a VS-DR LVS use realservers running a. NT Y b. other non-linux unices Y 7. Does the "arp problem" affect a. VS-NAT no b. VS-DR yes c. VS-TUN yes 8. Under Linux, hidden interfaces may be established that will not be advertised via ARP, whether directly connected or otherwise. One method of avoiding the ARP problem with VS-DR and VS-TUN is to make the interface with the VIP on which host(s) hidden: a. the director b. the realservers c. all machines in the LVS A. (b) Only the director can arp. This allows the director to get the connect request from the client for the VIP. 9. What can you do to handle the "arp problem" on an LVS running 2.2.x kernels (1 point for each method). a. Julian's patch to stop the VIP on the realserver from arping (now part of the kernel) b. Steve WIlliams patch to do the same thing (for older kernels) (setting up hidden interfaces is acceptable here for (a) and (b), as this can now be done out of the box with 2.2.x kernels.) c. Have the router forward all packets for the VIP to the director but not to the realservers d. Have the realservers on a different network to the director VIP (used for VS-TUN, Lars method) e. Have the VIP on another NIC on the realserver f. Hardwire the MAC of the director into the router (arp -f) as the MAC of the VIP g. accept packets on the realserver by transparent proxy (Horms method) 10. A VS-DR LVS with identical realservers and unweighted round robin scheduling for the service telnet, is setup with the VIP on ethernet devices. _All_ devices with the VIP reply to arp requests. You connect to the LVS'ed service telnet many times in succession from a client connected directly to the director and you observe which realserver you connect to. Which of these are possible: a. you will connect to each realserver in the order listed by ipvsadm (ie the LVS works perfectly) b. you will connect to each realserver in random order c. you will connect to a subset of the realservers in random order d. you will always connect to the same realserver e. the telnet connect request will hang. A. (a),(b),(c),(d) all correct. Depending on the relative speed of the replies to arp requests of the director compared to the realservers, you can get any of (a),(b),(c),(d). My first LVS worked fine with all machines replying to arp requests as the MAC entry for the VIP in the client's arp cache was always the director's. If one of the realservers always gets its MAC address into the client's arp cache, then you will always connect to that realserver (answer (d)). In intermediate cases you could get (b) or (c). 11. A VS-DR LVS can be made to operate if the VIP is on a. both the director and the realservers b. the director only c. the realservers only d. none of the machines A. all are correct. (a) is the standard VS-DR in which the VIP is carried on an ethernet device (eg eth0 or alias) on the director and on lo:0 on the realservers. You can use transparent proxy or policy routing to replace the need for a device with the VIP on one or both of the director and realservers giving (b),(c),(d). fwmark also allows you to _not_ have the VIP on the director. 12. You set up a demonstration LVS using some generic Linux boxes on hand. The LVS handles telnet using round robin scheduling on 2 realservers, but when you attempt to connect, you get "connection refused". What could be wrong? A. The service is not available. ipvsadm forwards the connection attempt to a realserver:port that doesn't have the service. Either the ipvsadm table was not setup properly or the realserver doesn't have the service running on the expected port. 13. You fix this problem and next time the connection attempt hangs (forever). What is likely wrong if the LVS is a. VS-NAT b. VS-DR A. The reply packets aren't getting back from the realserver. The usual cause of this is the wrong default gw for the realservers The default gw for the realserver for a. VS-NAT - director b. VS-DR for normal setup - router (not director). With Julian's martian modification patch the default gw is the director. c. VS-TUN - router (not director) 14. You fix this problem and and instead of connecting immediately, the connection hangs for a while and then connects. On checking you find that connecting to the realserver directly completes immediately. What's wrong? A. The service on the realserver is running inside auth/identd. Identd attempts to find out the owner of the process on the client that made the connect request. The auth/identd request fails (times out) because tcp connections started on the realservers cannot get back to the realservers. The timeout in the RFC is 30 secs, but Linux (and most unices) set the timeout to 6 secs. Services affected are anything running under tcpwrappers (eg telnet, ftp) and sendmail (see HOWTO. sendmail throughput is abysmal unless identd is turned off). 15. Your pointy haired boss is beginning to think you are crazy and wants to buy the TurboLinux Cluster Server, but you doggedly start again and setup a VS-DR LVS from scratch. You get a gratifying immediate connection to one of the realservers. However after a few minutes, you realise that you're connecting to the same realserver every time, rather than alternating between the two realservers. What is wrong? A. The realservers are replying to arp requests for the VIP. This particular realserver has got its MAC address into the client's arp table first. 16. In failover setups where another director can replace a failed director, during failover, the connection between the client and realserver is a. maintained b. dropped c. hangs A. This is an "it depends" question. The connection is not explicitely maintained and state of the link will depend on the service that was interrupted and what is was doing at the time. The correct answer then is not (a), and hence it's going to be one of (b) or (c) depending on your luck. See next question for examples. 17. What will the client in the previous question see on director failover if the connection is a. idle telnet b. active ftp doing a file transfer c. idle http, the browser reloads just after the failover completes. d. http and is downloading a page at the time the director fails. A. (a). probably won't notice anything. Idle telnet sessions stay up for quite a while (eg down the NIC connecting you to a telnet session on a remote machine and then bring it up again, you'll still have your telnet session). If you hit a keystroke or a tcp keepalive packet is sent during the downtime, your connection will hang. (b). The session will almost certainly hang. If you're lucky a shower of icmp packets and resets will drop your connection but it's never happened to me. (c). The http client will reconnect and you won't notice anything. (d). An http download of a file should restart where it left off. 18. VS-DR has a different path for packets coming from the client and for those returning. The result of this is that services like ftp, which have large reply packets and small request packets, have a. higher maximum throughput b. the same maximum thoughtput c. lower maximum througput at the director than services like lpd, which have large request packets and small replies. A. (b) see http://www.linuxvirtualserver.org/Joseph.Mack/performance/single_realserver_performance.html 19. An LVS can recognise/use the fwmark (firewall mark) on a packet. The fwmark is put on the packet by the a. client b. routers on the internet c. router/firewall just outside the director d. director e. realserver A. (d). (Thanks to Horms) The fwmark is put on by ipchains when a packet from the correct network or host IP enters the netfilter. The fwmark is only internal to the sk_buff in the director, it does not get attached to the packet in a form that leaves the director. 20. The fwmark is recognised/used by the a. client b. routers on the internet c. router/firewall just outside the director d. director e. realserver A. (d). ipvs inserts itself into the forwarding rules. Instead of looking for packets destined for the VIP (classic VS-DR), ipvs looks for packets with the correct fwmark number to forward to the realservers. 21. The fwmark allows a. the director not to have a VIP b. the director to accept LVS requests destined for a subnet of addresses c. security precautions to block DoS attacks A. (a),(b). fwmark allows the director to accept packets destined for an arbitrary set of IPs or range (network) of IPs. Each network or IP can be marked with the same or different fwmarks. The director then sends the marked packets to realservers based on their fwmark. Code to handle SYN attacks is part of ipvs. You activate the code by setting switches in the /proc filesystem. 22. A VS-DR realserver is doing an ftp transfer with a client which goes down during the transfer. A router near the client sends back a "host unreachable" icmp packet. a. What LVS machine handles this packet? b. What does this machine do with the icmp packet (eg accept, drop, reject)? c. What is the LVS's response to the icmp packet? A. (from Julian) a. The packet is routed by the LVS code on the director to the correct realserver. The ICMP message from client encapsulates the datagram which caused this message (the router must encapsulate at least the first 576 bytes from the TCP packet and to send it to the director as ICMP message). The director's job is to look in the encapsulated header and to see if the original TCP packet is from an LVS connection. If it is, we forward the ICMP message to the appropriate real server via its RIP. Determining the RIP is the tricky part. The ICMP packet has encapsulated VIP:VPORT->CIP:CPORT/PROTO, but no information about the RIP. Each LVS hash table entry has: CIP; VIP; RIP; proto and ports. The proto is one field (same for all 3 addresses). We can lookup any two unique entries (CIP, port) and determine the other (the RIP). This works for TCP and UDP. b. If the realserver is Linux: if there is no traffic in the next 2 minutes, report it as an error (the TCP reaction to ICMP errors is not always immediate). c. The LVS handles the icmp packet in the same way as a single server at the VIP. Score: >90 : there aren't that many points. Have another beer and recount. >70 : Wensong wants to talk to you 60-69: kung-fu level 8 LVS master 50-59: LVS gold guru 40-49: LVS silver guru 30-39: LVS tin guru 20-29: LVS lead guru <20 : The HOWTO maintainer wants to talk to you. (C) Joseph Mack and the LinuxVirtualServer Project 2000. May be used anywhere with acknowlegement. ---------------------------------------------------------------------------