Question

I’m writing a testing tool that requires known traffic to be captured from a NIC (using libpcap), then fed into the application we are trying to test.

What I’m attempting to set-up is a web server (in this case, lighttpd) and a client (curl) running on the same machine, on an isolated test network. A script will drive the entire setup, and the goal is to be able to specify a number of clients as well as a set of files for each client to download from the web server.

My initial approach was to simply use the loopback (lo) interface… run the web server on 127.0.0.1, have the clients fetch their files from http://127.0.0.1, and run my libpcap-based tool on the lo interface. This works ok, apart from the fact that the loopback interface doesn’t emulate a real Ethernet interface. The main problem with that is that packets are all inconsistent sizes… 32kbytes and bigger, and somewhat random… it’s also not possible to lower the MTU on lo (well, you can, but it has no effect!).

I also tried running it on my real interface (eth0), but since it’s an internal web client talking to an internal web server, traffic never leaves the interface, so libpcap never sees it.

So then I turned to tun/tap. I used socat to bind two tun interfaces together with a tcp connection, so in effect, i had:

10.0.1.1/24 <-> tun0 <-socat-> tcp connection <-socat-> tun1 <-> 10.0.2.1/24

This seems like a really neat solution… tun/tap devices emulate real Ethernet devices, so i can run my web server on tun0 (10.0.1.1) and my capture tool on tun0, and bind my clients to tun1 (10.0.2.1)… I can even use tc to apply shaping rules to this traffic and create a virtual WAN inside my linux box… but it just doesn’t work…

Here are the socat commands I used:

$ socat -d TCP-LISTEN:11443,reuseaddr TUN:10.0.1.1/24,up &
$ socat TCP:127.0.0.1:11443 TUN:10.0.2.1/24,up &

Which produces 2 tun interfaces (tun0 and tun1), with their respective IP addresses.

If I run ping -I tun1 10.0.1.1, there is no response, but when i tcpdump -n -i tun0, i see the ICMP echo requests making it to the other side, just no sign of the response coming back.

# tcpdump -i tun0 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 65535 bytes
16:49:16.772718 IP 10.0.2.1 > 10.0.1.1: ICMP echo request, id 4062, seq 5, length 64
<--- insert sound of crickets here (chirp, chirp)

So am I missing something obvious or is this the wrong approach? Is there something else i can try (e.g. 2 physical interfaces, eth0 and eth1???).

The easiest way is just to use 2 machines, but I want all of this self-contained, so it can all be scripted and automated on a single machine, without and other dependencies…

UPDATE:

There is no need for the 2 socats to be connected with a tcp connection, it’s possible (and preferable for me) to do this:

socat TUN:10.0.1.1/24,up TUN:10.0.2.1/24,up &

The same problem still exists though…

Answer 1

OK, so I found a solution using Linux network namespaces (netns). There is a helpful article about how to use it here: http://code.google.com/p/coreemu/wiki/Namespaces

This is what I did for my setup….

First, download and install CORE: http://cs.itd.nrl.navy.mil/work/core/index.php

Next, run this script:

#!/bin/sh

core-cleanup.sh > /dev/null 2>&1
ip link set vbridge down > /dev/null 2>&1
brctl delbr vbridge > /dev/null 2>&1

# create a server node namespace container - node 0
vnoded -c /tmp/n0.ctl -l /tmp/n0.log -p /tmp/n0.pid > /dev/null
# create a virtual Ethernet (veth) pair, installing one end into node 0
ip link add name veth0 type veth peer name n0.0
ip link set n0.0 netns `cat /tmp/n0.pid`
vcmd -c /tmp/n0.ctl -- ip link set n0.0 name eth0
vcmd -c /tmp/n0.ctl -- ifconfig eth0 10.0.0.1/24 up

# start web server on node 0
vcmd -I -c /tmp/n0.ctl -- lighttpd -f /etc/lighttpd/lighttpd.conf

# create client node namespace container - node 1
vnoded -c /tmp/n1.ctl -l /tmp/n1.log -p /tmp/n1.pid > /dev/null
# create a virtual Ethernet (veth) pair, installing one end into node 1
ip link add name veth1 type veth peer name n1.0
ip link set n1.0 netns `cat /tmp/n1.pid`
vcmd -c /tmp/n1.ctl -- ip link set n1.0 name eth0
vcmd -c /tmp/n1.ctl -- ifconfig eth0 10.0.0.2/24 up

# bridge together nodes using the other end of each veth pair
brctl addbr vbridge
brctl setfd vbridge 0
brctl addif vbridge veth0
brctl addif vbridge veth1
ip link set veth0 up
ip link set veth1 up
ip link set vbridge up

This basically sets up 2 virtual/isolated/name-spaced networks on your Linux host, in this case, node 0 and node 1. A web server is started on node 0.

All you need to do now is run curl on node 1:

vcmd -c /tmp/n1.ctl -- curl --output /dev/null http://10.0.0.1

And monitor the traffic with tcpdump:

tcpdump -s 1514 -i veth0 -n

This seems to work quite well… still experimenting, but looks like it will solve my problem.