Shallow Thoughts : tags : networking

Akkana's Musings on Open Source Computing and Technology, Science, and Nature.

Mon, 03 Sep 2018

Raspberry Pi Zero as Ethernet Gadget Part 3: An Automated Script

Continuing the discussion of USB networking from a Raspberry Pi Zero or Zero W (Part 1: Configuring an Ethernet Gadget and Part 2: Routing to the Outside World): You've connected your Pi Zero to another Linux computer, which I'll call the gateway computer, via a micro-USB cable. Configuring the Pi end is easy. Configuring the gateway end is easy as long as you know the interface name that corresponds to the gadget.

ip link gave a list of several networking devices; on my laptop right now they include lo, enp3s0, wlp2s0 and enp0s20u1. How do you tell which one is the Pi Gadget? When I tested it on another machine, it showed up as enp0s26u1u1i1. Even aside from my wanting to script it, it's tough for a beginner to guess which interface is the right one.

Try dmesg

Sometimes you can tell by inspecting the output of dmesg | tail. If you run dmesg shortly after you initialized the gadget (either by plugging the USB cable into the gateway computer, you'll see some lines like:

[  639.301065] cdc_ether 3-1:1.0 enp0s20u1: renamed from usb0
[ 9458.218049] usb 3-1: USB disconnect, device number 3
[ 9458.218169] cdc_ether 3-1:1.0 enp0s20u1: unregister 'cdc_ether' usb-0000:00:14.0-1, CDC Ethernet Device
[ 9462.363485] usb 3-1: new high-speed USB device number 4 using xhci_hcd
[ 9462.504635] usb 3-1: New USB device found, idVendor=0525, idProduct=a4a2
[ 9462.504642] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 9462.504647] usb 3-1: Product: RNDIS/Ethernet Gadget
[ 9462.504660] usb 3-1: Manufacturer: Linux 4.14.50+ with 20980000.usb
[ 9462.506242] cdc_ether 3-1:1.0 usb0: register 'cdc_ether' at usb-0000:00:14.0-1, CDC Ethernet Device, f2:df:cf:71:b9:92
[ 9462.523189] cdc_ether 3-1:1.0 enp0s20u1: renamed from usb0

(Aside: whose bright idea was it that it would be a good idea to rename usb0 to enp0s26u1u1i1, or wlan0 to wlp2s0? I'm curious exactly who finds their task easier with the name enp0s26u1u1i1 than with usb0. It certainly complicated all sorts of network scripts and howtos when the name wlan0 went away.)

Anyway, from inspecting that dmesg output you can probably figure out the name of your gadget interface. But it would be nice to have something more deterministic, something that could be used from a script. My goal was to have a shell function in my .zshrc, so I could type pigadget and have it set everything up automatically. How to do that?

A More Deterministic Way

First, the name starts with en, meaning it's an ethernet interface, as opposed to wi-fi, loopback, or various other types of networking interface. My laptop also has a built-in ethernet interface, enp3s0, as well as lo0, the loopback or "localhost" interface, and wlp2s0, the wi-fi chip, the one that used to be called wlan0.

Second, it has a 'u' in the name. USB ethernet interfaces start with en and then add suffixes to enumerate all the hubs involved. So the number of 'u's in the name tells you how many hubs are involved; that enp0s26u1u1i1 I saw on my desktop had two hubs in the way, the computer's internal USB hub plus the external one sitting on my desk.

So if you have no USB ethernet interfaces on your computer, looking for an interface name that starts with 'en' and has at least one 'u' would be enough. But if you have USB ethernet, that won't work so well.

Using the MAC Address

You can get some useful information from the MAC address, called "link/ether" in the ip link output. In this case, it's f2:df:cf:71:b9:92, but -- whoops! -- the next time I rebooted the Pi, it became ba:d9:9c:79:c0:ea. The address turns out to be randomly generated and will be different every time. It is possible to set it to a fixed value, and that thread has some suggestions on how, but I think they're out of date, since they reference a kernel module called g_ether whereas the module on my updated Raspbian Stretch is called cdc_ether. I haven't tried.

Anyway, random or not, the MAC address also has one useful property: the first octet (f2 in my first example) will always have the '2' bit set, as an indicator that it's a "locally administered" MAC address rather than one that's globally unique. See the Wikipedia page on MAC addressing for details on the structure of MAC addresses. Both f2 (11110010 in binary) and ba (10111010 binary) have the 2 (00000010) bit set.

No physical networking device, like a USB ethernet dongle, should have that bit set; physical devices have MAC addresses that indicate what company makes them. For instance, Raspberry Pis with networking, like the Pi 3 or Pi Zero W, have interfaces that start with b8:27:eb. Note the 2 bit isn't set in b8.

Most people won't have any USB ethernet devices connected that have the "locally administered" bit set. So it's a fairly good test for a USB ethernet gadget.

Turning That Into a Shell Script

So how do we package that into a pipeline so the shell -- zsh, bash or whatever -- can check whether that 2 bit is set?

First, use ip -o link to print out information about all network interfaces on the system. But really you only need the ones starting with en and containing a u. Splitting out the u isn't easy at this point -- you can check for it later -- but you can at least limit it to lines that have en after a colon-space. That gives output like:

$ ip -o link | grep ": en"
5: enp3s0:  mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000\    link/ether 74:d0:2b:71:7a:3e brd ff:ff:ff:ff:ff:ff
8: enp0s20u1:  mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000\    link/ether f2:df:cf:71:b9:92 brd ff:ff:ff:ff:ff:ff

Within that, you only need two pieces: the interface name (the second word) and the MAC address (the 17th word). Awk is a good tool for picking particular words out of an output line:

$ ip -o link | grep ': en' | awk '{print $2, $17}'
enp3s0: 74:d0:2b:71:7a:3e
enp0s20u1: f2:df:cf:71:b9:92

The next part is harder: you have to get the shell to loop over those output lines, split them into the interface name and the MAC address, then split off the second character of the MAC address and test it as a hexadecimal number to see if the '2' bit is set. I suspected that this would be the time to give up and write a Python script, but no, it turns out zsh and even bash can test bits:

ip -o link | grep en | awk '{print $2, $17}' | \
    while read -r iff mac; do
        # LON is a numeric variable containing the digit we care about.
        # The "let" is required so LON will be a numeric variable,
        # otherwise it's a string and the bitwise test fails.
        let LON=0x$(echo $mac | sed -e 's/:.*//' -e 's/.//')

        # Is the 2 bit set? Meaning it's a locally administered MAC
        if ((($LON & 0x2) != 0)); then
            echo "Bit is set, $iff is the interface"
        fi
    done

Pretty neat! So now we just need to package it up into a shell function and do something useful with $iff when you find one with the bit set: namely, break out of the loop, call ip a add and ip link set to enable networking to the Raspberry Pi gadget, and enable routing so the Pi will be able to get to networks outside this one. Here's the final function:

# Set up a Linux box to talk to a Pi0 using USB gadget on 192.168.0.7:
pigadget() {
    iface=''

    ip -o link | grep en | awk '{print $2, $17}' | \
        while read -r iff mac; do
            # LON is a numeric variable containing the digit we care about.
            # The "let" is required so zsh will know it's numeric,
            # otherwise the bitwise test will fail.
            let LON=0x$(echo $mac | sed -e 's/:.*//' -e 's/.//')

            # Is the 2 bit set? Meaning it's a locally administered MAC
            if ((($LON & 0x2) != 0)); then
                iface=$(echo $iff | sed 's/:.*//')
                break
            fi
        done

    if [[ x$iface == x ]]; then
        echo "No locally administered en interface:"
        ip a | egrep '^[0-9]:'
        echo Bailing.
        return
    fi

    sudo ip a add 192.168.7.1/24 dev $iface
    sudo ip link set dev $iface up

    # Enable routing so the gadget can get to the outside world:
    sudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'
    sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
}

Tags: , ,
[ 18:41 Sep 03, 2018    More linux | permalink to this entry | comments ]

Fri, 31 Aug 2018

Raspberry Pi Zero as Ethernet Gadget Part 2: Routing to the Outside World

I wrote some time ago about how to use a Raspberry Pi over USB as an "Ethernet Gadget". It's a handy way to talk to a headless Pi Zero or Zero W if you're somewhere where it doesn't already have a wi-fi network configured.

However, the setup I gave in that article doesn't offer a way for the Pi Zero to talk to the outside world. The Pi is set up to use the machine on the other end of the USB cable for routing and DNS, but that doesn't help if the machine on the other end isn't acting as a router or a DNS host.

A lot of the ethernet gadget tutorials I found online explain how to do this on Mac and Windows, but it was tough to find an example for Linux. The best I found was for Slackware, How to connect to the internet over USB from the Raspberry Pi Zero, which should work on any Linux, not just Slackware.

Let's assume you have the Pi running as a gadget and you can talk to it, as discussed in the previous article, so you've run:

sudo ip a add 192.168.7.1/24 dev enp0s20u1
sudo ip link set dev enp0s20u1 up
substituting your network number and the interface name that the Pi created on your Linux machine, which you can find in dmesg | tail or ip link. (In Part 3 I'll talk more about how to find the right interface name if it isn't obvious.)

At this point, the network is up and you should be able to ping the Pi with the address you gave it, assuming you used a static IP: ping 192.168.7.2 If that works, you can ssh to it, assuming you've enabled ssh. But from the Pi's end, all it can see is your machine; it can't get out to the wider world.

For that, you need to enable IP forwarding and masquerading:

sudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'
sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Now the Pi can route to the outside world, but it still doesn't have DNS so it can't get any domain names. To test that, on the gateway machine try pinging some well-known host:

$ ping -c 2 google.com
PING google.com (216.58.219.110) 56(84) bytes of data.
64 bytes from mia07s25-in-f14.1e100.net (216.58.219.110): icmp_seq=1 ttl=56 time=78.6 ms
64 bytes from mia07s25-in-f14.1e100.net (216.58.219.110): icmp_seq=2 ttl=56 time=78.7 ms

--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 78.646/78.678/78.710/0.032 ms

Take the IP address from that -- e.g. 216.58.219.110 -- then go to a shell on the Pi and try ping -c 2 216.58.219.110, and you should see a response.

DNS with a Public DNS Server

Now all you need is DNS. The easy way is to use one of the free DNS services, like Google's 8.8.8.8. Edit /etc/resolv.conf and add a line like

nameserver 8.8.8.8
and then try pinging some well-known hostname.

If it works, you can make that permanent by editing /etc/resolv.conf, and adding this line:

name_servers=8.8.8.8

Otherwise you'll have to do it every time you boot.

Your Own DNS Server

But not everyone wants to use public nameservers like 8.8.8.8. For one thing, there are privacy implications: it means you're telling Google about every site you ever use for any reason.

Fortunately, there's an easy way around that, and you don't even have to figure out how to configure bind/named. On the gateway box, install dnsmasq, available through your distro's repositories. It will use whatever nameserver you're already using on that machine, and relay it to other machines like your Pi that need the information. I didn't need to configure it at all; it worked right out of the box.

In the next article, Part 3: more about those crazy interface names (why is it enp0s20u1 on my laptop but enp0s26u1u1i1 on my desktop?), how to identify which interface is the gadget by using its MAC, and how to put it all together into a shell function so you can set it up with one command.

Tags: , ,
[ 15:25 Aug 31, 2018    More linux | permalink to this entry | comments ]

Fri, 02 Feb 2018

Raspberry Pi Console over USB: Configuring an Ethernet Gadget

When I work with a Raspberry Pi from anywhere other than home, I want to make sure I can do what I need to do without a network.

With a Pi model B, you can use an ethernet cable. But that doesn't work with a Pi Zero, at least not without an adapter. The lowest common denominator is a serial cable, and I always recommend that people working with headless Pis get one of these; but there are a lot of things that are difficult or impossible over a serial cable, like file transfer, X forwarding, and running any sort of browser or other network-aware application on the Pi.

Recently I learned how to configure a Pi Zero as a USB ethernet gadget, which lets you network between the Pi and your laptop using only a USB cable. It requires a bit of setup, but it's definitely worth it. (This apparently only works with Zero and Zero W, not with a Pi 3.)

The Cable

The first step is getting the cable. For a Pi Zero or Zero W, you can use a standard micro-USB cable: you probably have a bunch of them for charging phones (if you're not an Apple person) and other devices.

Set up the Pi

Setting up the Raspberry Pi end requires editing two files in /boot, which you can do either on the Pi itself, or by mounting the first SD card partition on another machine.

In /boot/config.txt add this at the end:

dtoverlay=dwc2

In /boot/cmdline.txt, at the end of the long list of options but on the same line, add a space, followed by: modules-load=dwc2,g_ether

Set a static IP address

This step is optional. In theory you're supposed to use some kind of .local address that Bonjour (the Apple protocol that used to be called zeroconf, and before that was called Rendezvous, and on Linux machines is called Avahi). That doesn't work on my Linux machine. If you don't use Bonjour, finding the Pi over the ethernet link will be much easier if you set it up to use a static IP address. And since there will be nobody else on your USB network besides the Pi and the computer on the other end of the cable, there's no reason not to have a static address: you're not going to collide with anybody else.

You could configure a static IP in /etc/network/interfaces, but that interferes with the way Raspbian handles wi-fi via wpa_supplicant and dhcpcd; so you'd have USB networking but your wi-fi won't work any more.

Instead, configure your address in Raspbian via dhcpcd. Edit /etc/dhcpcd.conf and add this:

interface usb0
static ip_address=192.168.7.2
static routers=192.168.7.1
static domain_name_servers=192.168.7.1

This will tell Raspbian to use address 192.168.7.2 for its USB interface. You'll set up your other computer to use 192.168.7.1.

Now your Pi should be ready to boot with USB networking enabled. Plug in a USB cable (if it's a model A or B) or a micro USB cable (if it's a Zero), plug the other end into your computer, then power up the Pi.

Setting up a Linux machine for USB networking

The final step is to configure your local computer's USB ethernet to use 192.168.7.1.

On Linux, find the name of the USB ethernet interface. This will only show up after you've booted the Pi with the ethernet cable plugged in to both machines.

ip a
The USB interface will probably start eith en and will probably be the last interface shown.

On my Debian machine, the USB network showed up as enp0s26u1u1. So I can configure it thusly (as root, of course):

ip a add 192.168.7.1/24 dev enp0s26u1u1
ip link set dev enp0s26u1u1 up
(You can also use the older ifconfig rather than ip: sudo ifconfig enp0s26u1u1 192.168.7.1 up)

You should now be able to ssh into your Raspberry Pi using the address 192.168.7.2, and you can make an appropriate entry in /etc/hosts, if you wish.

For a less hands-on solution, if you're using Mac or Windows, try Adafruit's USB gadget tutorial. It's possible that might also work for Linux machines running Avahi. If you're using Windows, you might prefer CircuitBasics' ethernet gadget tutorial.

Happy networking!

Update: there's now a Part 2: Routing to the Outside World and Part 3: an Automated Script.

Tags: , ,
[ 14:53 Feb 02, 2018    More linux | permalink to this entry | comments ]

Fri, 09 Nov 2012

How to talk to your Rapsberry Pi over an ethernet crossover cable with IP masquerading

I've been using my Raspberry Pi mostly headless -- I'm interested in using it to control hardware. Most of my experimenting is at home, where I can plug the Pi's built-in ethernet directly into the wired net.

But what about when I venture away from home, perhaps to a group hacking session, or to give a talk? There's no wired net at most of these places, and although you can buy USB wi-fi dongles, wi-fi is so notoriously flaky that I'd never want to rely on it, especially as my only way of talking to the Pi.

Once or twice I've carried a router along, so I could set up my own subnet -- but that means an extra device, ten times as big as the Pi, and needing its own power supply in a place where power plugs may be scarce.

The real solution is a crossover ethernet cable. (My understanding is that you can't use a normal ethernet cable between two computers; the data send and receive lines will end up crossed. Though I may be wrong about that -- one person on #raspberrypi reported using a normal ethernet cable without trouble.)

Buying a crossover cable at Fry's was entertaining. After several minutes of staring at the dozens of bins of regular ethernet cables, I finally found the one marked crossover, and grabbed it. Immediately, a Fry's employee who had apparently been lurking in the wings rushed over to warn me that this wasn't a normal cable, this wasn't what I wanted, it was a weird special cable. I thanked him and assured him that was exactly what I'd come to buy.

Once home, with my laptop connected to wi-fi, I plugged one end into the Pi and the other end into my laptop ... and now what? How do I configure the network so I can talk to the Pi from the laptop, and the Pi can gateway through the laptop to the internet?

The answer is IP masquerading. Originally I'd hoped to give the Pi a network address on the same networking (192.168.1) as the laptop. When I use the Pi at home, it picks a network address on 192.168.1, and it would be nice not to have to change that when I travel elsewhere. But if that's possible, I couldn't find a way to do it.

Okay, plan B: the laptop is on 192.168.1 (or whatever network the wi-fi happens to assign), while the Pi is on a diffferent network, 192.168.0. That was relatively easy, with some help from the Masquerading Simple Howto.

Once I got it working, I wrote a script, since there are quite a few lines to type and I knew I wouldn't remember them all. Of course, the script has to be run as root. Here's the script, on github: masq.

I had to change one thing from the howto: at the end, when it sets up security, this line is supposed to enable incoming connections on all interfaces except wlan0:

iptables -A INPUT -m state --state NEW -i ! wlan0 -j ACCEPT

But that gave me an error, Bad argument `wlan0'. What worked instead was

iptables -A INPUT -m state --state NEW ! -i wlan0 -j ACCEPT
Only a tiny change: swap the order of -i and !. (I sent a correction to the howto authors but haven't heard back yet.)

All set! It's a nice compact way to talk to your Pi anywhere. Of course, don't forget to label your crossover cable, so you don't accidentally try to use it as a regular ethernet cable. Now please excuse me while I go label mine.

Update: Ed Davies has a great followup, Crossover Cables and Red Tape, that talks about how to set up a subnet if you don't need the full masquerading setup, why non-crossover cables might sometimes work, and a good convention for labeling crossover cables: use red tape. I'm going to adopt that convention too -- thanks, Ed!

Tags: , , , ,
[ 16:57 Nov 09, 2012    More hardware | permalink to this entry | comments ]

Sat, 24 Mar 2012

Find out what processes are making network connections

A thread on the Ubuntu-devel-discuss mailing list last month asked about how to find out what processes are making outgoing network connectsion on a Linux machine. It referenced Ubuntu bug 820895: Log File Viewer does not log "Process Name", which is specific to Ubuntu's iptables logging of apps that are already blocked in iptables ... but the question goes deeper.

Several years ago, my job required me to use a program -- never mind which one -- from a prominent closed-source company. This program was doing various annoying things in addition to its primary task -- operations that got around the window manager and left artifacts all over my screen, operations that potentially opened files other than the ones I asked it to open -- but in addition, I noticed that when I ran the program, the lights on the DSL modem started going crazy. It looked like the program was making network connections, when it had no reason to do that. Was it really doing that?

Unfortunately, at the time I couldn't find any Linux command that would tell me the answer. As mentioned in the above Ubuntu thread, there are programs for Mac and even Windows to tell you this sort of information, but there's no obvious way to find out on Linux.

The discussion ensuing in the ubuntu-devel-discuss thread tossed around suggestions like apparmor and selinux -- massive, complex ways of putting up fortifications your whole system. But nobody seemed to have a simple answer to how to find information about what apps are making network connections.

Well, it turns out there are a a couple ofsimple way to get that list. First, you can use ss:

$ ss -tp
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port   
ESTAB      0      0                     ::1:58466                  ::1:ircd     users:(("xchat",1063,43))
ESTAB      0      0             192.168.1.6:57526       140.211.166.64:ircd     users:(("xchat",1063,36))
ESTAB      0      0                     ::1:ircd                   ::1:58466    users:(("bitlbee",1076,10))
ESTAB      0      0             192.168.1.6:54253       94.125.182.252:ircd     users:(("xchat",1063,24))
ESTAB      0      0             192.168.1.6:52167       184.72.217.144:https
users:(("firefox-bin",1097,47))

Update: you might also want to add listening connections where programs are listening for incoming connections: ss -tpla
Though this may be less urgent if you have a firewall in place.

-t shows only TCP connections (so you won't see all the interprocess communication among programs running on your machine). -p prints the process associated with each connection.

ss can do some other useful things, too, like show all the programs connected to your X server right now, or show all your ssh connections. See man ss for examples.

Or you can use netstat:

$ netstat -A inet -p
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 imbrium.timochari:51800 linuxchix.osuosl.o:ircd ESTABLISHED 1063/xchat      
tcp        0      0 imbrium.timochari:59011 ec2-107-21-74-122.:ircd ESTABLISHED 1063/xchat      
tcp        0      0 imbrium.timochari:54253 adams.freenode.net:ircd ESTABLISHED 1063/xchat      
tcp        0      0 imbrium.timochari:58158 s3-1-w.amazonaws.:https ESTABLISHED
1097/firefox-bin

In both cases, the input is a bit crowded and hard to read. If all you want is a list of processes making connections, that's easy enough to do with the usual Unix utilities like grep and sed:

$ ss -tp | grep -v Recv-Q | sed -e 's/.*users:(("//' -e 's/".*$//' | sort | uniq
$ netstat -A inet -p | grep '^tcp' | grep '/' | sed 's_.*/__' | sort | uniq

Finally, you can keep an eye on what's going on by using watch to run one of these commands repeatedly:

watch ss -tp

Using watch with one of the pipelines to print only process names is possible, but harder since you have to escape a lot of quotation marks. If you want to do that, I recommend writing a script.

And back to the concerns expressed on the Ubuntu thread, you could also write a script to keep logs of which processes made connections over the course of a day. That's definitely a tool I'll keep in my arsenal.

Tags: , , ,
[ 12:28 Mar 24, 2012    More linux | permalink to this entry | comments ]