Friday, January 14, 2011

Selective load balancing with Squid and iproute2

Recently I was asked by a client to develop a load balancing solution over two of their ADSL lines, but only for http traffic, and only for specific users. Sounds like a pain in the ass, right? Well, it was.

Essentially they've got four ADSL links:

  • eth0: (main uncapped line)
  • eth1:
  • eth2:
  • eth3:

eth0 is the default route where all traffic goes through, and eth3 is used for something else.

They're using Squid as their proxy server, and want all traffic for a specific Squid ACL (let's call it "employees") to be load balanced between eth1 and eth2.

My biggest worry was in how to set this up without having to alter the default route on the system, but - as is always the case with Linux - there is a way if you look hard enough.

My first thought was to use iptables to mark packets, then set up an iproute2 rule to pick up on those packets and forward them to the relevant routing table. Unfortunately, the best one could do with this method is manipulate all http traffic, however, as mentioned, only the traffic for specific users must be load balanced.

Obviously, this method won't work.

The key to the solution is Squid's TCP_OUTGOING_TOS configuration directive. This directive allows one to set the TOS value in outgoing IP packets on a per-ACL basis. What this means is that if you have an ACL called "employees", you can have all traffic generated by the users in that ACL have the TOS set to an arbitrary value. You can then use an iproute2 rule to pick up on all packets with that value set and do whatever you want with them. Bingo!

Firstly, we need to create three routing tables: ADSL1, ADSL2 and BALANCE. This is usually done in /etc/iproute2/rt_tables.

Then we begin adding routing information:

# Assumes is the address of the router on eth1
ip route add dev eth1 src table ADSL1
ip route add default via table ADSL1

# Assumes is the address of the router on eth2
ip route add dev eth2 src table ADSL2
ip route add default via table ADSL2

Now we add some rules

# eth1 traffic goes to table ADSL1
ip rule add from table ADSL1

# eth2 traffic goes to table ADSL2
ip rule add from table ADSL2

# Squid sets desired traffic TOS. Marked traffic goes to table BALANCE
ip rule add tos 0x0c table BALANCE

And finally we create the multilink route in the table BALANCE.

ip route add default scope global table BALANCE nexthop via dev eth1 weight 1 nexthop via dev eth2 weight 1

Then finally, in your Squid configuration, simply add the configuration directive

tcp_outgoing_tos 0x0c employees

That's pretty much it. iptraf confirms that all http traffic for the desired ACL gets balanced over eth1 and eth2. Obviously the load balancing won't be perfect due to route caching, etc. But it probably is the most beautiful solution possible without splashing out on a pricey Cisco router or some such.

For more information, be sure to read the Linux Advanced Routing & Traffic Control HOWTO.

1 comment:

elmanytas said...

A very useful post. You saved my day.
Thank you very much