|AustinTek LVS page||| LVS homepage | AustinTek homepage ||
Copyright © 2003,4 Joseph Mack.
Aug 2003 and updated Feb 2004
Table of Contents
This meeting, and the many satellite meetings that occur before and after, is the main annual technical meeting for Linux. There are no vendors or sales people. There is a limit of 500 attendees.
OLS has grown in large part because of the location of Ottawa.
Although OLS started 5 yrs ago as a single meeting, satellite meetings are now conducted before or after the main meeting, to take advantage of the presence of people working on common projects e.g. the (by invitation only) Linux Kernel Summit is held in the two days before OLS.
The Linux Virtual Server project (LVS) met before (and during and after) the meeting. Unlike the Kernel Summit, the LVS people are quite inclusive about who we'll talk to and only expect you to be able to drink beer. LVS team members are from China, Japan, Switzerland, France, Australia, Bulgaria. I'm the only American . Apart from e-mail, this is our only chance to meet face-to-face. This year Lars, Horms, Ratz and myself attended.
The priority scheduler, originally proposed by Ratz, was worked on in hotel rooms and on steps leading into the conference center, anywhere that laptops could be put down. Ratz and Horms did their LVS development, simulating multiple machines on their laptops, with VM-ware.
Ratz needs the priority scheduler to handle sudden peaks in load, when a client will be sent to a "Please wait 1 min" with state information being held about the client. This scheduler is needed for servers, where an SLA requires that users always be able to connect. This is a problem when a server has to handle I/O in bursts, e.g. when bookings open for a rock concert (the Rolling Stones concerts in Canada with 500,000 tickets, sold out in several minutes, requiring the servers to hold 100,000 connections for the time needed for the bookings).
Horms LVS tutorial was the first talk of the conference.
Horms talk included a description of his implementation of the LVS synch demon and on his Saru code (http://www.ultramonkey.org/papers/active_active/), which allows multiple directors to be active at the same time. Horms showed how the synch state demon didn't need to run in client-server mode, but would work better in peer-to-peer mode, where it would broadcast the state of all connections it knew about. If the director was in backup mode, then it would be silent. Since only ESTABLISHED connections are relevent to failover, only ESTABLISHED state information is broadcast.
Linux Virtual Memory Management: The Linux VM is not in good shape; a Linux machine with load average of 80 will likely crash. The VM for *BSD is quite good. (I don't know why the *BSD VM isn't used in Linux.) Linux developers are trying various algorithms for VM with the hope of developing an O(1) VM. They aren't there yet, but Linux VM is better than it was a year ago. Academics regard the VM problem as solved, but the original VM algorithms were developed for machines that were much nearer to Von Neuman architecture than current machines. Although most people think that computer speed is increasing with Moore's Law, not all hardware is advancing at the same rate. The slews have accentuated the non-von properties of modern computers.
Table 1. Scaling of Speeds of Hardware with Moore's Law
|scaling with Moore's Law||Hardware examples|
|faster than Moore's Law||disk size|
|with Moore's Law||CPU speed, memory size, number of processes|
|slower than Moore's Law||disk speed, memory speed|
|much slower than Moore's Law||number of CPUs|
|do not scale with Moore's Law||speed of light (affects distance separating chips on a board)|
These changes have meant that machines that not so long ago were running 2000 processes are now handling 50000 processes, where the algorithms for VM do not scale. In particular, the ratio of job size to disk speed means that it's now expensive to swap.
The VM problem shows up with one of the beowulfs I run. For an overtemperature event at 100% CPU on 32 CPUs, it takes 45secs to schedule an orderly shutdown (we're scheduling a disorderly shutdown which takes about 10 secs).
|In my 2.4.24 kernel, I now notice that at high load, the CPU runs at 100% (sys, user, nice), whereas previously there would always be idle time (5-20%) no matter how high the load average. Presumably this is a result of better job scheduling.|
These are put on by sponsors and follow welcoming speeches by organisers etc. Although the talks are not given by Linux developers, the sponsors usually put some effort to make their talk relevant to the audience. Last year Wayne Meretsky, a senior designer from AMD, talked about their new 64bit CPU, its pipeline and multithreading design and showed that many-way SMP systems would be possible with near UMA performance (the access time to memory on another CPU was only twice that to local memory, meaning that there was no point in programming the compiler to treat memory accesses to other CPUs any differently to local memory accesses). This talk was received well.
This year a marketer from Intel gave a long talk on the "Ecology of Linux" the message being that "working together we can get it done". Very quickly the noise level at the tables rose as people returned to their conversations.
On another evening, IBM sponsored a talk by Ian Foster (Mr "Grid"). Ian Foster has a book out on Grid which reviewers on amazon.com regard as "cheer leading". Instead of a technical talk (say on the design of Globus), he plugged the latest edition of his book and gave a zero content power point presentation on the similarities between Grid computing and the electrical grid.
Much of the information transfer occured outside the talks. Problems that I've walked around and accepted as part of living would come up when someone would say "I found out why the glibc `make test` fails for machines with disks mounted with 'noatime'", or "anyone know why there are 3 drivers for device 'foo'?". The last problem is usually due to the person, who started the project, not accepting code/patches/new ideas from other people. These ("difficult to work with") people are unfortunately common in Linux.
There is a lot of commonly used code out there that won't compile in my hands (gnome, gnucash) and people use the precompiled binaries, negating most of the point of having the source code. I found that other people can't compile these packages either. While reassured that I'm not incompetent, I'm distressed that code is being released in such shape and being incorporated into other code.
I've spent some time trying to compile gnome parts on my machines, to find that instead of looking for the file they want (e.g. some library) they look for an ascii *.la file, that is supposed list the location of the file, but often doesn't. Why they hell doesn't gnome use ldconfig to find the libraries, just like everyone else does?
Disks are really cheap (Come on guys - you're supposed to say "Joe, how cheap are they?"). Well, they are so cheap ($1/G) that if you're working with people with reasonable disk hygiene, it's cheaper to install another disk than to send a sysadmin who costs $1/min to free up some diskspace.
People usually think free code is a good idea because it gives them a whole lot of code that does exactly what they want for free. This, of course, is completely wrong. The main expense of code is not writing it, but is debugging, maintaining and upgrading it. The real reason then that you want to give your code away, is so that you'll quickly have 1000s of users who will send in patches, bug fixes and new ideas and you won't have to do a anything except take credit for it all.
The Europeans are amazed that power lines in USA are above ground, particularly in areas of ice or hurricanes (or where I live, both) (Sydney with neither, has power underground). They also aren't happy that NA uses cell phone standards incompatible with those the rest of the world and are left standing around unable to get dial-tones. Of course you patiently explain to them that the phone providers here are only responding to the voice of the consumer exercising their right to choose, as you would expect in a free country. Here the consumer is asking for, nay demanding, incompatible protocols in adjacent zones and that people not be allowed to send data over cell phones. The situation in the rest of the world, interoperability from country to country and people who've been using their cell phones as laptop modems for a decade, is exactly what you'd expect when governments step in and over-regulate and strangle innovation in a free-enterprise system.
I mentioned to Ratz, that I had UPSs at home. Why? (he asked in astonishment). Because the power goes out occasionally and even if it doesn't, you get power bumps (<1sec) now and again, enough to bring down a machine and derange the phone answering machine, VCR and clocks. (In Maryland, I used to get a 1sec power bump once a week). This brought cries of amazement. His machine at home didn't have a UPS and had an uptime of over 400 days. He had 100's of machines at work, all of which were required to be up all year, and none of which had a UPS or had ever needed them. Why, if you couldn't rely on the power, no-one would be able to do business. It sounds like there isn't a UPS in the whole of Europe and you have to places like Iraq before you get anything comparable to the state of the power system in USA.
We had the first near-arrest in 5yrs of OLS. One of the younger speakers was found by the RCMP in a state of geographical confusion and gravitational challenge, while attempting to return to his hotel one night following discussions about the future of open source code. After engaging in some light conversation, he was taken in hand by the Mounties, who safely chauffered him without further incident to his hotel.
|A Conservative Prime Minister of Australia was once found in the foyer of a hotel in another country in a similar condition, but without his trousers. The newspapers in Australia described him as being found in an "emotional state", adding a phrase to the australian lexicon. The whereabouts of his trousers is still a mystery.|
 :-) For those who haven't met me, I was born in Australia and still have an Australian accent. Although I have US citizenship, it's not obvious to most people that I'm an American.
|AustinTek LVS page||| LVS homepage | AustinTek homepage ||