16. LVS: Variants on LVS: Peter Warasin's ip_vs() in PREROUTING

Peter Warasin peter (at) endian (dot) com 11 Sep 2007

I made some modifications on the lvs specific kernel code, which now leads into kernel oops. Could someone give me some pointers about how to find the bug? I am not very familiar with the kernel code, so maybe I missed some simple tricks which routined people know and me not. Basically I altered the lvs code in order to make it catch packets within the PREROUTING chain instead of the INPUT chain. My setup works, but sometimes I have a kernel oops. I think somewhere it lacks some sort of spinlock, but I not really know where to begin in order to find where it must be inserted.

My setup:

The setup works. But sometimes I have a kernel oops (fatal exception in interrupt, ip_rcv, ip_rcv_finish is involved). I tried to narrow down the problem, by removing patch nr 2, but the problem still exists. So the problem must be with the 1st patch. But what could cause this. I simply let LVS catch packets within PREROUTING chain instead INPUT chain. That seems not too different to me.

I think somewhere it lacks some sort of spinlock, but I not really know where to begin in order to find where it must be inserted.

Horms 13 Sep 2007

As Joe mentioned in a subsequent email, being able to move LVS from one chain to another is something that we are interested in. In particular I am of the believe that the FORWARD chain would be a much more logical home than LOCAL_IN as in some ways would allow LVS to act more like a router than a proxy (not that it is a proxy, but it kind of behaves like one in some ways because of its home on LOCAL_IN).

As I recall, I did try moving the code to the FORWARD chan a long time ago. I believe that the change was very similar to the LOCAL_IN to PRE_ROUTING snippet that you have below. I'm not sure that I ever posted the change, as I never tested it thorougly. So perhaps it too broke occasionally. In any case, this was a long time ago, and the kernel has changed significantly then, so any testing done at that time wouldn't really hold water now (incidently 2.6.9 is also pretty old, though I'm not sure what patches RHEL include to modernise it).

As for debugging your problem. Providing the oops message - if any - might help. Hopefully there is a stack trace in there and that should start to point to where the problem is.

Some portions of the locking schemantics of LVS are non-trivial and I have a deep suspicion that there are some races in there anyway. By which I mean, don't be surprised if things get a bit hairy as you are tracing through what is going on.

If your kernel is compiled with IP_VS_DEBUG then you can enable and adjust the verbosity of debugging messages that LVS produced by tweaking /proc/sys/net/ipv4/vs/debug_level as documented in Documentation/networking/ipvs-sysctl.txt in the kernel source tree.

Also, if you are doing development work, I do recommend considering using a more up to date kernel. Perhaps the latest rc kernel, currently 2.6.23-rc6. I'm not suggsting that you neccessarily drop this into production. But for development work, it is much easier to work with the kernel guys if you are on the same page as them.