Bringing up my Linux router laptop: # /etc/init.d/network-manager stop (or: service network-manager stop) # ifconfig eth0 192.168.57.1 up this brings up the eth0 interface my ssh daemon is aleady running, so I can log in from 192.168.57.10, manually configured. Note that neither machine has a gateway/default route at this point, the SSH connection is just on a LAN. Looking at the output of tcpdump -xev -i eth0 -n : Packet header diagrams for IP: https://nmap.org/book/tcpip-ref.html Turns out IP headers are pretty readable even in hex. # less /etc/protocols => ICMP is "1" # printf "%d\n" 0x39 => 57 # printf "%d\n" 0xa8 => 168 # printf "%d\n" 0xc0 => 192 # ifconfig # tcpdump -vxe -i eth0 -n icmp # tshark -i eth0 icmp # tshark -V -i eth0 icmp # tcpdump -vxe -i eth0 -n icmp # ifconfig # route -n # ping 129.170.17.4 => this fails, no route to network Now inserting the first USB Ethernet adapter. It became eth2 # dmesg => shows the USB subsystem recognizing the device & loading the driver for it # ifconfig eth2 # ifconfig # ifconfig eth2 up => we don't have an IP address yet, but we can sniff traffic # ethtool eth2 => link detection says yes # ifconfig # tcpdump -i eth2 -n # arp -a => MAC/IP table of the kernel # ip neigh show => same, with a different tool Now let's get connectivity. We'll do it by asking, nicely. # dhclient -v eth2 => gets me a free IP, a default route, and DNS server configs, too. # ps ax | grep dh => dhclient remains running in the background to request again before IP lease expires # route -n => shows the default route that came in with the DHCP response # /etc/init.d/openvpn stop => my VPN came up, seeing disabled it # route -n # ping 129.170.17.4 => yay, global IP connectivity! # route del default gw 129.170.212.1 => deleting the default route, just to see what the errors will be # route -n # ping 129.170.17.4 => fails, "Network unreachable" # cat /etc/resolv.conf => these DNS server IPs came in on the same DHCP response, too # ping 129.170.212.67 => still works, because the LAN route is still present # ping 129.170.213.128 => same # ping 129.170.212.26 => same # ping 129.170.192.44 => doesn't work, not on the LAN! Now it's time to insert another interface. It will become eth1. It is facing the laptop that will be .100 # dmesg # ifconfig eth1 # ifconfig eth1 192.168.57.2 up => let's give it an IP on the _same_ LAN (not normally done). # route -n => too bad, now there are two competing routes for 129.168.57.0/24 # route del -net 192.168.57.0 dev eth1 => killing one; that didn't work # route del -net 192.168.57.0 netmask 255.255.255.0 dev eth1 => correct syntax, rute now removed # route -n # route add -net 192.168.57.100 netmask 255.255.255.255 dev eth1 => let's have a more specific route here # route -n # ping 192.168.57.100 => keeps sending ARP who-has 192.168.57.100, tell 192.168.57.2" . No reply, => hence no ICMP sent yet (Configuring the other side: "ifconfig eth0 192.168.57.100 up" so it would reply to ARP. # route -n # ping 192.168.57.100 => now it works # ifconfig Now we try pinging .10 from .100. It fails: ARP requests aren't responded to # tcpdump -i eth1 -n icmp => nothing at all, no icmp sent # tcpdump -i eth1 -n arp => ARP requests who-has 192.168.57.10, tell 192.168.57.100, no replies. # tcpdump -i eth1 -n => same ARPs, plus some broadcast MDNS traffic from .100 # cat /proc/sys/net/ipv4/ip_forward # echo 1 > /proc/sys/net/ipv4/ip_forward => now are are forwarding, but still nothing: we need some way of telling => .100 the MAC of .10 Note: setting default gw to .2 or .1 doesn't help, because the more specific route for 192.168.57.0/24 still causes ARP requests to be sent (which still fail). Reworked the routing on the .100, "route del -net 192.168.57.0 netmask 255.255.255.0 dev eth0", "route add -host 192.168.57.2 netmask 255.255.255.255 dev eth0", "route add -net 192.168.57.2 netmask 255.255.255.0 gw 192.168.57.2 dev eth0"---i.e., gave .2 as the gateway for all of 192.168.57.0/24 . The middle command is necessary, because a route to the gateway must exist before the gw is used in another route. This _almost_ works: # tcpdump -i eth1 -n => now ICMP requests are sent to the MAC of 192.168.57.2 # tcpdump -i eth0 -n => woah, that's a lot of packets! Oh, our SSH goes over that link, that's why. # tcpdump -i eth0 -n icmp => ... and are forwarded through eth0, and reach .10 # tcpdump -i eth0 -n arp => but now .10 needs to know .100's MAC to reply, and is sending ARP who-has for it; => now answer (ARPs are not forwarded by default # cat /proc/sys/net/ipv4/conf/all/proxy_arp # echo 1 > /proc/sys/net/ipv4/conf/all/proxy_arp => let's start ARP proxying on all interfaces: if => a MAC is learned on one, router now responds to ARP who-has => for it on _all_ interfaces # tcpdump -i eth1 -n icmp => this finally works! .10 and .100 can ping each other. Now let's configure the router to let .100 ping the Internets. # ping 8.8.8.8 => this fails even on our router. # route -n => Oh yes, I did not restore the default route after I killed it. # route add default gw 129.170.212.1 => restoring the default gw on router. # tcpdump -i eth1 -n icmp => pings are coming in from .100 # tcpdump -i eth2 -n icmp => ...and going our to 8.8.8.8, but no response Of course: these ping requests go out to the 129.170.212.1 (the Sudi gateway) with the non-routable "RFC 1918" private source address 192.168.57.100. These get dropped by the first border router, of course, as per RFC, https://tools.ietf.org/html/rfc1918 Now to configure NAT: change src IP is each outgoing forwarded packet from 192.168.57.100 to my routable IP on eth2, and change them back for incoming forwarded packets. # iptables-save => nothing much there # iptables -t nat -A POSTROUTING -s 192.168.57.0/24 -j MASQUERADE => I forgot to add "-o eth2", actually. => it still worked, but with a little surprise. => what was the surprise? Now ICMP is flowing: # tcpdump -i eth1 -n icmp => it seems like 8.8.8.8 is replying straight to 192.168.57.100. Can't happen, of course. # tcpdump -i eth2 -n icmp => instead, outgoing ICMP requests have eth2's routable IP for source, and => the incoming replies from 8.8.8.8 have the same routable IP for destination. So it works.