[LARTC] Balancing multiple connections and NAT

[Raj Mathur]

Hi,

I have a client connected to the 'net through 3 ISP's.  Have set up a
Linux box to do routing and load sharing for the 3 connections.  A
fourth interface is connected to the LAN with private IP addresses.
Am using iptables to SNAT traffic to the appropriate IP depending on
the interface the packet gets routed onto.  The setup looks something
like this:

Interface       IP               Gateway	  Table	  Network
---------       --               -------          -----   -------
intA            ipA              gwA		  tableA  netA
intB            ipB              gwB		  tableB  netB
intC            ipC              gwC		  tableC  netC
[intD is the LAN interface]
intD            ipD (private)    no gateway	  global  netD

This works fine most of the time, except that once in a while (every
5-10 minutes or so) packets going out on (e.g.) intB suddenly start
getting NAT'ed to source address ipA (i.e. the address of another
interface).  Obviously this plays hell with the existing connections
on that link!

The ip commands I'm using are:

/sbin/ip route add netA dev intA src ipA table tableA
/sbin/ip route add netA dev intA src ipA
/sbin/ip route add default via gwA table tableA
/sbin/ip route add netB dev intB src ipB table tableB
/sbin/ip route add netB dev intB src ipB
/sbin/ip route add default via gwB table tableB
/sbin/ip route add netC dev intC src ipC table tableC
/sbin/ip route add netC dev intC src ipC
/sbin/ip route add default via gwC table tableC
/sbin/ip route add default scope global nexthop via gwB dev intB weight 1 nexthop via gwC dev intC weight 2 nexthop via gwA dev intA weight 2
/sbin/ip rule add from ipA table tableA
/sbin/ip rule add from ipB table tableB
/sbin/ip rule add from ipC table tableC

The iptables commands are:

/sbin/iptables -P FORWARD DROP
# Enable full flow on the LAN
/sbin/iptables -I FORWARD -s netD -i intD -j ACCEPT
/sbin/iptables -I FORWARD -d netD -o intD -j ACCEPT
# Allow all packets to go out
/sbin/iptables -I OUTPUT -o intA -j ACCEPT
/sbin/iptables -I OUTPUT -o intB -j ACCEPT
/sbin/iptables -I OUTPUT -o intC -j ACCEPT
/sbin/iptables -I OUTPUT -o intD -j ACCEPT
/sbin/iptables -I INPUT -i intD -j ACCEPT
/sbin/iptables -I INPUT -i lo -j ACCEPT
/sbin/iptables -P INPUT DROP
/sbin/iptables -A INPUT -i ! intD -m state --state RELATED,ESTABLISHED -j ACCEPT
# Hmmm, why is this one there?
/sbin/iptables -A INPUT -i intD -m state --state RELATED,ESTABLISHED -j ACCEPT
# NAT depending on outbound interface
/sbin/iptables -t nat -A POSTROUTING -s netD -o intA -j SNAT --to-source ipA
/sbin/iptables -t nat -A POSTROUTING -s netD -o intB -j SNAT --to-source ipB
/sbin/iptables -t nat -A POSTROUTING -s netD -o intC -j SNAT --to-source ipC

Any idea why connections that are flowing perfectly would suddenly
decide to start getting NAT'ed to the wrong source?  Or some place on
the 'net I can start looking?

Regards,

-- Raju
-- 
Raj Mathur                raju at kandalaya.org      http://kandalaya.org/
       GPG: 78D4 FC67 367F 40E2 0DD5  0FEF C968 D0EF CC68 D17F
                      It is the mind that moves