IP Encapsulation within IP, Linux-to-Linux ========================================== Linux has the ability to do IP-in-IP tunnels. This document is for Linux on both ends of a tunnel. It's really quite simple, too. I've used it with both 2.4 and 2.6 kernels. IP encapsulation was established as a standard in 1996 by RFC 2003, but I found Google results suggesting a Linux implementation of ipip dating back well before then. I only know how to do this on Linux, but contrary to what the [seriously outdated] LARTC page[1] says, I would think that any other IP-capable OS which implements RFC 2003 should be able to establish a tunnel with a Linux peer. Requirements: ============ * iproute2 package (ip(8), usually installed at /sbin/ip) * Kernel ipip module and whatever options it requires ** In 2.6 it pulls in a tunnel4 module ** In 2.4 it was standalone, but LARTC says it needed a new_tunnel module ** Oh, sure, if you're so cool (read:foolish) that you compiled everything into your kernel, that works too. * root access on each end * Your requirements: ** Basic competence in Unix user skills ** Basic competence in Linux administrative skills ** Basic understanding of IP networking (you are now crossing over into advanced IP networking, woo hoo!) Why/why not? =========== What I did with these tunnels: I gave my home machine an IP address at a business/colo site. There are lots of other possibilities as well. You could make a low-security VPN (a VPVPN, or virtually private virtual private network) between sites, for example. IP-in-IP is very simple and very flexible. It requires little extra memory, all of it in kernelspace, and can run well on low-powered embedded devices. If you're afraid that openvpn or IPSEC is too much CPU/memory overhead for your endpoint, and you only need a few static peer tunnels, and are not worried about the traffic being secured, this might be a good choice for you. Drawbacks, as hinted above, include possible extra bandwidth use for the outer encapsulating packet. It's hard to gauge how much this would mean in the real world, but obviously, the additional IP packet header takes some space, and if an inner packet is close to the size of the MTU, encapsulatiion could result in packet fragmentation. Other choices, also as hinted above, include VPN implementations such as OpenVPN, and IPSEC. GRE encapsulation. See the "mode" section of your "ip tunnel help" output to see what other sort of IP tunneling your Linux can do. HOWTO, step-by-step: =================== ** Step 1: Load kernel module[s] modprobe -v ipip That's all it takes, on each end, and again, if you compiled it in, skip this. ** Step 2: ip tunnel ip tunnel add Moe mode ipip remote other.ip.add.ress local our.ip.add.ress And on the other end: ip tunnel add Moe mode ipip remote our.ip.add.ress local other.ip.add.ress This needs some discussion. I'll go through it all just to be thorough. First, "ip" is of course /sbin/ip, and "tunnel" is the subcommand. Take a moment to look over "ip tunnel help". For those who like to minimize keystrokes, this can be done as "ip t h". Any ip(8) subcommand and its options can be abbreviated to the point of not being ambiguous with other subcommands or options. See ip(8) man page if you want to know more (and really, you should look in there eventually.) We are adding a tunnel, thus "add", and we are giving it a name of "Moe". Moe will be the name of our interface. Pick something that makes sense to you, something that reflects the use, purpose, and/or destination of this tunnel. For me, Moe was the DNS name of the IP address I was setting up to use via the tunnel. Next, "mode ipip" simply says we are using this type of Linux-to-Linux IPv4 encapsulation tunnel. There are other modes; this document is not about those. "remote" and "local" refer to the machine on which the command runs. The command machine is local, the other endpoint is remote. "other.ip.add.ress" is the address of the other endpoint -- as we can reach it NOW. We must have some kind of way to reach there. It could be via our default route to the Internet, or it could be any other kind of route. "our.ip.add.ress" is one of our existing IP addresses on THIS end. It would have to be the address which we would use to reach the remote. If it's going to be NAT'ed on the way to the remote, that can work too, if protocol 94 ("ipip")[2] is forwarded to us from the NAT router for the other end to reach us. (I'm not going to go into the NAT setup here.) For example, suppose our remote is at 192.0.2.2. Our interface of the default route is 192.0.2.42. We need: ip tunnel add Moe mode ipip remote 192.0.2.2 local 192.0.2.42 And on the other end: ip tunnel add Moe mode ipip remote 192.0.2.42 local 192.0.2.2 (The name "Moe" does not have to be the same. Again, pick a name which makes sense to you.) At this point we should be able to list our tunnels on both ends: ip -s tunnel They're not active yet, so not much to see there. Ignore that tunl0 interface, we are not using it. ** Step 3: Assign IP addresses ip address add our.tunnel.ip.address/32 peer other.tunnel.ip.address dev Moe And on the other end: ip address add other.tunnel.ip.address/32 peer our.tunnel.ip.address dev Moe Again, I'll go through it all. Here "address" is our subcommand. It can be shortened to "a" or anything you like. I usually use "addr" because it helps with mental recognition of what's going on. And again, we're using the "add" action to add an IP address to our new interface. "our.tunnel.ip.address" is the address that the tunnel will use on our end. The idea is that anyone who speaks IP with our.tunnel.ip.address (which is located at the other site) will actually be speaking/connecting to our local machine. Likewise, "other.tunnel.ip.address" is located at our end, but will be the address of the remote peer. At this point we should be able to list our addresses on both ends: ip address They're still not active yet. ** Step 4: Activate the links ip link set Moe up Hmmm, sounds like something Larry and Shemp might have tried to do before being abused! All this does is activate the link and the peer-to-peer routing. We should be able to ping across the tunnel from each endpoint, but it's otherwise not usable. (Unless, of course, that was all you wanted to do with the tunnel, in which case you're probably done!) It's the same command on each end, substituting the name you used, if it differs. Now if you list the tunnels (see the end of Step 2) after having pinged, you will see counters incremented. ** Step 5: Name routing tables $EDITOR /etc/iproute2/rt_tables or: echo "42 Moe" >> /etc/iproute2/rt_tables I like nice descriptive names, so I'm also going to name the routing tables which will be used for tunnel traffic. ip(8) keeps table names in /etc/iproute2/rt_tables by default. Your distro might have changed that somehow. It's up to you to read your distro's documentation in that case. The number you choose can be any that's not reserved nor already in use. Again, use a name that makes sense, and if it's not the same name on both ends, adjust accordingly. Neither name nor number has to be the same, but it might help you to keep them the same. ** Step 6: Default tunnel route ip route add default via other.tunnel.ip.address table Moe And on the other end: ip route add default via our.tunnel.ip.address table Moe On our end, this says that other.tunnel.ip.address will be the default gateway for any packets on our side which use the "Moe" route table. Right now, nothing is doing that. Likewise, our.tunnel.ip.address will be the default gateway for the "Moe" table on the peer. This is an essential step, but by itself, it didn't accomplish much. While some of this can be done out of order, this one requires that the tunnel is already up and active, i.e., a direct route to other.tunnel.ip.address must exist. ** Step 7: Basic routing policy ip rule add from our.tunnel.ip.address table Moe And on the other end: ip rule add from other.tunnel.ip.address table Moe We're getting close now. The ip(8) rule subcommand decides policy for any given network packet. Here, we say that if a packet uses our.tunnel.ip.address as its source IP, it goes to the "Moe" route table. There, you will recall, it finds the tunnel peer as its default gateway. ** Step 8: IP Forwarding and Proxy ARP echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp echo 1 > /proc/sys/net/ipv4/ip_forward This needs to be done on either or both ends as applicable, depending on a few factors. * Do both sides need to do forwarding and proxy ARP? If tunnel packets are not intended to go beyond the endpoint, no, they do not. * Is either side already doing either of these? If so, these commands will not hurt anything, but they're moot. * The example shows proxy ARP being activated for eth0. It should be the name of the interface of which our.tunnel.ip.address is in the same subnet. At this point, IT WORKS! Offer void where taxed or prohibited by law, or if your firewall is blocking packets in the filter table, FORWARD chain. In the latter case, read on. ** Step 9: Firewall iptables -vI FORWARD -i Moe -j ACCEPT iptables -vI FORWARD -o Moe -j ACCEPT Do note that this is probably not what you want. You quite likely want to restrict traffic going through the tunnel in some way, or else you might open yourself up to denial of service attacks. But securing a tunnel was not the goal of this document; establishing the tunnel was the goal. Here's what I did for mine, in the filter table: :FORWARD DROP [13884:966814] :State - [0:0] [ ... ] -A FORWARD -j State -A FORWARD -i Moe -j ACCEPT -A FORWARD -s place.to.test.from -j ACCEPT -A FORWARD -s one.other.test.site -j ACCEPT [ ... ] -A State -m conntrack --ctstate INVALID -j DROP -A State -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT This means I can initiate outbound connections at will, and replies are allowed back through. The two test sites can connect to me, and everything else is dropped. I wasn't running any servers on my end of the tunnel. But I do have httpd(8) at the other endpoint, so in the nat table: -A PREROUTING -d our.tunnel.ip.address -i eth0 -p tcp --dport 80 -j REDIRECT This intercepts any HTTP request going to our.tunnel.ip.address and sends it to the peer's HTTP daemon. ** Step 10: Using the tunnel ... This is mostly beyond the scope of this document. It is assumed that since you decided you needed a tunnel, that you would know how to make use of it. But a few suggestions won't hurt. nc(1) bills itself as a networking version of a Swiss Army knife. It does everything. It can do source routing, or it can bind a specific local source address with -s. I'm not going to try to document everything nc can do; wouldn't even be possible, since there are multiple implementations of it available. My "guinea pig" application was irssi(1), an IRC client. I used this: irssi -h Moe.mydomain.example.com It was then able to connect to an IRCd, and sure enough, it showed me coming from the right IP address. Servers such as Apache httpd(8) which typically bind to 0.0.0.0 will Just Work, when your routing and firewall is correct. Test them from outside your network/LAN. Servers such as ISC BIND named(8) might not work immediately, because they have to bind each IP address specially. Check logs or netstat(8) output to see if they have caught up with your additional IP address, and then test. As always, consult documentation for your server software. tracepath(8) and traceroute(8) are tools which can show you the routes followed by packets from your host. Try it from outside, too. You should see only one hop between your peer's upstream router and your tunnel IP address. That last hop might show significantly higher latency, since it has to traverse the Internet in encapsulation. (The path of the encapsulated packets is not shown.) Author: Rob McGee Date: 2010-03-29 (revision 1) Date: 2012-01-29 (revision 2: changed email address) * Suggestions for improvement are welcome. Suggestions that I should list all $DISTRO quirks are not improvement. Over time the scope of this document might broaden, but it's not likely to ever provide support for proprietary operating systems. It most likely will remain purely for generic Linux. * Requests for personal assistance are billable, but I might be available to help you out for a reasonable one-time fee. I do like doing this stuff! Hire me! :) Footnotes: ========= [1] http://lartc.org/howto/lartc.tunnel.ip-ip.html [2] Some Google results from years ago suggest that ipip was formerly assigned protocol 93 by IANA. I don't know (did not try to look up) when this changed. A very old Linux might show ipip as 93.