One thought at the back of my mind, is that what you have set up is known in IP parlance as a "dual homed host". Linux's default ARP behaviour is "odd". It's of no consequence if you have only one network interface, but can cause issues with more interfaces.
In essence the default Linux behaviour stems from an oddity of Sun networking. The concept is that the MAC address belongs to the system, not the interface. Sun's used to have only one MAC address, so that never gave issues. With more interfaces, and other systems that expect an MAC address to be related to a specific interface that is not true.
By default a Linux system will respond to an ARP request on an interface for any IP that the system owns, with the MAC address that corresponds to the interface with that IP , even if the IP address is bound to another interface on the box.
explanation here, from notes on kernel.org:
arp_filter - BOOLEAN
1 - Allows you to have multiple network interfaces on the same
subnet, and have the ARPs for each interface be answered
based on whether or not the kernel would route a packet from
the ARP'd IP out that interface (therefore you must use source
based routing for this to work). In other words it allows control
of which cards (usually 1) will respond to an arp request.
0 - (default) The kernel can respond to arp requests with addresses
from other interfaces. This may seem wrong but it usually makes
sense, because it increases the chance of successful communication.
IP addresses are owned by the complete host on Linux, not by
particular interfaces. Only for more complex setups like load-
balancing, does this behaviour cause problems.
arp_filter for the interface will be enabled if at least one of
conf/{all,interface}/arp_filter is set to TRUE,
it will be disabled otherwise
arp_announce - INTEGER
Define different restriction levels for announcing the local
source IP address from IP packets in ARP requests sent on
interface:
0 - (default) Use any local address, configured on any interface
1 - Try to avoid local addresses that are not in the target's
subnet for this interface. This mode is useful when target
hosts reachable via this interface require the source IP
address in ARP requests to be part of their logical network
configured on the receiving interface. When we generate the
request we will check all our subnets that include the
target IP and will preserve the source address if it is from
such subnet. If there is no such subnet we select source
address according to the rules for level 2.
2 - Always use the best local address for this target.
In this mode we ignore the source address in the IP packet
and try to select local address that we prefer for talks with
the target host. Such local address is selected by looking
for primary IP addresses on all our subnets on the outgoing
interface that include the target IP address. If no suitable
local address is found we select the first local address
we have on the outgoing interface or on all other interfaces,
with the hope we will receive reply for our request and
even sometimes no matter the source IP address we announce.
The max value from conf/{all,interface}/arp_announce is used.
Increasing the restriction level gives more chance for
receiving answer from the resolved target while decreasing
the level announces more valid sender's information.
arp_ignore - INTEGER
Define different modes for sending replies in response to
received ARP requests that resolve local target IP addresses:
0 - (default): reply for any local target IP address, configured
on any interface
1 - reply only if the target IP address is local address
configured on the incoming interface
2 - reply only if the target IP address is local address
configured on the incoming interface and both with the
sender's IP address are part from same subnet on this interface
3 - do not reply for local addresses configured with scope host,
only resolutions for global and link addresses are replied
4-7 - reserved
8 - do not reply for all local addresses
The max value from conf/{all,interface}/arp_ignore is used
when ARP request is received on the {interface}
arp_notify - BOOLEAN
Define mode for notification of address and device changes.
0 - (default): do nothing
1 - Generate gratuitous arp requests when device is brought up
or hardware address changes.
arp_accept - BOOLEAN
Define behavior for gratuitous ARP frames who's IP is not
already present in the ARP table:
0 - don't create new entries in the ARP table
1 - create new entries in the ARP table
Both replies and requests type gratuitous arp will trigger the
ARP table to be updated, if this setting is on.
If the ARP table already contains the IP address of the
gratuitous arp frame, the arp table will be updated regardless
if this setting is on or off.
Coupled with that Linux these days does not permit asymmetric routing:
Notes on asymmetric routing
Systems with their ARP cache polluted by the "generous" ARP arrangement that Linux uses by default, can find that they are blocked from communicating with a host as their traffic is sent to the "wrong" interface (ie not the one that sent the initial request) and is silently dropped by the target hosts routing filter.
My solution is to put this file in /etc/sysctl.d/sane-arp.conf
netadmin@backdoorpi:/etc/sysctl.d $ cat sane-arp.conf
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.all.arp_ignore = 1
setting the ARP configuration to something less "generous" , and more in line with the rest of the universes assumption that mac addresses relate to specific interfaces.
I'm by no means sure that this is in any way related to the issue you had, but it is one of the ways that very odd network behaviour can be triggered on dual home hosts.
It's also probably impossible to reproduce, since whatever event triggered the ARP cache pollution, if that was the cause, could be timing related, and or order of connection, or restart of one of the hosts.
Harry
