Question

Okay, here’s my problem. Please forgive me as it’s a little bit complicated. I’m almost 100% sure it’s caused by a MacOS 10.6 kernel bug, but since we can’t count on the bug being fixed, I need a workaround.

I learned earlier that “ipfw … fwd” rules don’t work correctly on MacOS 10.6 (it works on 10.5) unless you first do

sysctl -w net.inet.ip.scopedroute=0

However, it turns out that this solution isn’t perfect either; about 10 to 15 minutes after making this change, my Mac basically stops talking to the Internet. pinging anything outside my local network starts saying “no route to host”, even though I absolutely have a valid default route. I have traced the problem to incorrect ARP entries. Before I run the above command, my arp table looks like this:

# arp -a
router (192.168.42.1) at 0:1c:10:b0:d4:79 on en1 ifscope [ethernet]

After running the above sysctl and then doing ping google.com, it looks like this:

# arp -a
dd-wrt (192.168.42.1) at 0:1c:10:b0:d4:79 on en1 [ethernet]
dd-wrt (192.168.42.1) at 0:1c:10:b0:d4:79 on en1 ifscope [ethernet]

So far, so harmless. But a while later, the original arp entry times out and all we have left is the new one. MacOS tries to refresh the old entry, but it never comes back. tcpdump shows repeated ARP requests originating from my Mac, with correct ARP responses coming back from the router, but the answer never gets put into the ARP table. I suspect that the answer is just updating the other ARP entry for the same IP, since they both have the same key in some kind of hash table.

Running “arp -a -d” (or any variant of “arp -d” I’ve tried) does not successfully delete both ARP entries – only one of them. And not the right one, apparently.

Any of the following workarounds make the problem go away, but are undesirable:

• instead of changing the sysctl at runtime, edit sysctl.conf and reboot.
• after changing the sysctl, bring down the interface and bring it back up again.
• after changing the sysctl, delete all the routes through that interface (with the route command) and re-create them.

However, each of these options leaves the system temporarily in a state where packets aren’t routable. Moreover, since I don’t actually know what this sysctl does (can someone point me at documentation for it?) I’d really like my program to be able to change it back to normal on exit. But if I do that, then it’ll be broken again next time my program starts.

I think what I really need to do is just clear out the ARP table, but maybe I’m missing something obvious. Is there an easy way to solve this problem or do I have to resort to something ugly?

(BTW, the program I’m working on is open source called sshuttle. If you try it on a fresh Mac with the sysctl set to its default of 1, you should be able to easily replicate the problem.)

Thanks for any suggestions.

Answer 1

Have you tried the form arp -d <ip> ifscope <interface>?
From your example, try arp -d 192.168.42.1 ifscope en1

This is what I use to get rid of the ifscope’d arp entries.
Note that you must be root (or use sudo) to avoid permission errors.