Experience on Setting Up Alpine Linux

Starting out as one of the little unknown GNU/Linux distros, Alpine Linux has gain a lot of traction due to its featureful yet tiny size and the emergence of Linux Container implementation like Dockers and LXC. Although I came across it numerous time while testing out Dockers and LXC, I didn't pay much attention until recently while troubleshooting LXD. To summarize it, I really like the minimalist approach of Alpine Linux as for server or hardware appliance usage, nothing beats the simple direct approach.

My setup is based on the LXC container in Fedora 23. Unfortunately, you still can't create unprivileged container in Fedora. Hence, I have no choice but to do everything as root user. Not the best outcome but I can live with that. Setup and creation is pretty much straight forward thanks to this guide. The steps as follows:

Install and necessary packages and make sure the lxcbr0 bridge interface is up.
$ sudo dnf install lxc lxc-libs lxc-extra lxc-templates
$ sudo systemctl restart lxc-net
$ sudo systemctl status lxc-net
$ ifconfig lxcbr0

Create our container. By default, LXC will download apk package manager binary and all necessary default packages to create the container. Start the 'test-alpine' container once the container has been set up successfully.
$ sudo lxc-create -n test-alpine -t alpine
$ sudo lxc-start -n test-alpine

Access to the container through the console and press 'Enter'. Login as 'root' user but without any password, just press enter. Note to exist from the console, press 'Ctrl+q'.
$ sudo lxc-console -n test-alpine

Next, bring up the eth0 interface we can obtain an IP and making connection to the Internet. Check your eth0 network interface once done. Instead of SysV or Systemd, Alpine Linux is using OpenRC as its default init system. I've a hard time adjusting changes from SysV to Systemd and glad Alpine Linux did not jump to the Systemd bandwagon.
test-alpine:~# rc-service networking start
 * Starting networking ... *   lo ...ip: RTNETLINK answers: File exists
 [ !! ]
 *   eth0 ... [ ok ]

test-alpine:~# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:16:3E:6B:F7:8B  
          inet addr:  Bcast:  Mask:
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1562 (1.5 KiB)  TX bytes:1554 (1.5 KiB)

Next, configure out system. Similarly to Debian's dpkg-reconfigure, Alpine have a list of setup commands to configure your system. However, I prefer the consistent and sensible naming used here. This is something that other GNU/Linux distros should follow. I'm looking at you CentOS/Red Hat/Fedora.
test-alpine:~# setup-
setup-acf        setup-bootable         setup-hostname      setup-mta     setup-timezone
setup-alpine     setup-disk             setup-interfaces    setup-ntp     setup-xen-dom0
setup-apkcache   setup-dns              setup-keymap        setup-proxy   setup-xorg-base
setup-apkrepos   setup-gparted-desktop  setup-lbu           setup-sshd

Next, setup the package repository and let the system pick the fastest mirror. I like that we can pick the fastest mirror in the console, which is something impossible to do so in Debian/Ubuntu.
# setup-apkrepos

1) nl.alpinelinux.org
2) dl-2.alpinelinux.org
3) dl-3.alpinelinux.org
4) dl-4.alpinelinux.org
5) dl-5.alpinelinux.org
6) dl-6.alpinelinux.org
7) dl-7.alpinelinux.org
8) distrib-coffee.ipsl.jussieu.fr
9) mirror.yandex.ru
10) mirrors.gigenet.com
11) repos.lax-noc.com
12) repos.dfw.lax-noc.com
13) repos.mia.lax-noc.com
14) mirror1.hs-esslingen.de
15) mirrors.centarra.com
16) liskamm.alpinelinux.uk
17) mirrors.2f30.org
18) mirror.leaseweb.com

r) Add random from the above list
f) Detect and add fastest mirror from above list
e) Edit /etc/apk/repositores with text editor

Enter mirror number (1-18) or URL to add (or r/f/e/done) [f]: 
Finding fastest mirror... 
  3.07 http://nl.alpinelinux.org/alpine/
  4.43 http://dl-2.alpinelinux.org/alpine/
  4.18 http://dl-3.alpinelinux.org/alpine/
  4.43 http://dl-4.alpinelinux.org/alpine/
  7.56 http://dl-5.alpinelinux.org/alpine/
  4.45 http://dl-6.alpinelinux.org/alpine/
ERROR: http://dl-7.alpinelinux.org/alpine/edge/main: No such file or directory
 12.75 http://distrib-coffee.ipsl.jussieu.fr/pub/linux/alpine/alpine/
  3.27 http://mirror.yandex.ru/mirrors/alpine/
  3.55 http://mirrors.gigenet.com/alpinelinux/
 27.07 http://repos.lax-noc.com/alpine/
  3.87 http://repos.dfw.lax-noc.com/alpine/
 20.34 http://repos.mia.lax-noc.com/alpine/
  3.55 http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/
ERROR: http://mirrors.centarra.com/alpine/edge/main: network error (check Internet connection and firewall)
  4.96 http://liskamm.alpinelinux.uk/
  4.45 http://mirrors.2f30.org/alpine/
  5.61 http://mirror.leaseweb.com/alpine/
Added mirror nl.alpinelinux.org
Updating repository indexes... done.

Update our system. Even though there are more than five thousands packages, it is still not comparable to massive Debian list of available packages. But this is understandable due to the small number of contributors and their limited free time.
test-alpine:~# apk update
fetch http://dl-6.alpinelinux.org/alpine/v3.2/main/x86_64/APKINDEX.tar.gz
fetch http://nl.alpinelinux.org/alpine/v3.2/main/x86_64/APKINDEX.tar.gz
v3.2.3-104-g838b3e3 [http://dl-6.alpinelinux.org/alpine/v3.2/main]
v3.2.3-104-g838b3e3 [http://nl.alpinelinux.org/alpine/v3.2/main]
OK: 5289 distinct packages available

Let's continue by installing a software package. We'll use Git version control as our example. Installation is straight forwards with enough details.
test-alpine:~# apk add git
(1/13) Installing run-parts (4.4-r0)
(2/13) Installing openssl (1.0.2d-r0)
(3/13) Installing lua5.2-libs (5.2.4-r0)
(4/13) Installing lua5.2 (5.2.4-r0)
(5/13) Installing ncurses-terminfo-base (5.9-r3)
(6/13) Installing ncurses-widec-libs (5.9-r3)
(7/13) Installing lua5.2-posix (33.3.1-r2)
(8/13) Installing ca-certificates (20141019-r2)
(9/13) Installing libssh2 (1.5.0-r0)
(10/13) Installing curl (7.42.1-r0)
(11/13) Installing expat (2.1.0-r1)
(12/13) Installing pcre (8.37-r1)
(13/13) Installing git (2.4.1-r0)
Executing busybox-1.23.2-r0.trigger
Executing ca-certificates-20141019-r2.trigger
OK: 23 MiB in 28 packages

So far, I love the simplicity provided by Alpine Linux. In coming months, there will be more post on this tiny distro in coming months. Stay tuned.

Troubleshooting Dynamic Host Configuration Protocol (DHCP) Connection in LXD, Part 1: The Dnsmasq Server

While testing LXD, the GNU/Linux container hypervisor, one of the issue I've encountered was certain containers failed to obtain an IP address after booting up. Hence, for the past few days, while scratching my head investigating the issue, I've gained some understanding on how DHCP works and learned a few tricks on how to troubleshoot a DHCP connection.

DHCP is a client/server where the client obtain an IP address from the server. Thus, to troubleshoot any connection issue, we should look in two places, the server and the client side.

Is Dnsmasq up and running?
First, the server end. As I mentioned in my previous post, in LXD, the lxcbr0 bridge interface is basically a virtual switch, through Dnsmasq, provides network infrastructures services like Domain Name System (DNS) and DHCP services. If DHCP is not working, first things to check whether the Dnsmasq has been started correctly. Pay attention to all lines that contains the word 'dnsmasq' and check for any errors.
$ sudo systemctl status lxc-net -l
● lxc-net.service - LXC network bridge setup
   Loaded: loaded (/usr/lib/systemd/system/lxc-net.service; enabled; vendor preset: disabled)
   Active: active (exited) since Wed 2015-11-18 21:04:24 MYT; 1s ago
  Process: 21863 ExecStop=/usr/libexec/lxc/lxc-net stop (code=exited, status=0/SUCCESS)
  Process: 21891 ExecStart=/usr/libexec/lxc/lxc-net start (code=exited, status=0/SUCCESS)
 Main PID: 21891 (code=exited, status=0/SUCCESS)
   Memory: 408.0K
      CPU: 39ms
   CGroup: /system.slice/lxc-net.service
           └─21935 dnsmasq -u nobody --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --listen-address --dhcp-range, --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative

Nov 18 21:04:24 localhost.localdomain dnsmasq[21935]: started, version 2.75 cachesize 150
Nov 18 21:04:24 localhost.localdomain dnsmasq[21935]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify
Nov 18 21:04:24 localhost.localdomain dnsmasq-dhcp[21935]: DHCP, IP range --, lease time 1h
Nov 18 21:04:24 localhost.localdomain dnsmasq-dhcp[21935]: DHCP, sockets bound exclusively to interface lxcbr0
Nov 18 21:04:24 localhost.localdomain dnsmasq[21935]: reading /etc/resolv.conf
Nov 18 21:04:24 localhost.localdomain dnsmasq[21935]: using nameserver
Nov 18 21:04:24 localhost.localdomain dnsmasq[21935]: read /etc/hosts - 2 addresses
Nov 18 21:04:24 localhost.localdomain systemd[1]: Started LXC network bridge setup.

As LXD is still actively under development, there are still many pending issues, you may want to walk through the '/usr/libexec/lxc/lxc-net' script to investigate more. Although from my experience, is simple service restart 'systemctl restart lxc-net' should be sufficient.

Failed to create listening socket?
Few days back, one of the issue I've experienced is that the Dnsmasq server failed to start due to failure in creating listening socket.
Nov 14 20:43:18 localhost.localdomain systemd[1]: Starting LXC network bridge setup...
Nov 14 20:43:18 localhost.localdomain lxc-net[24314]: dnsmasq: failed to create listening socket for Cannot assign requested address
Nov 14 20:43:18 localhost.localdomain dnsmasq[24347]: failed to create listening socket for Cannot assign requested address
Nov 14 20:43:18 localhost.localdomain dnsmasq[24347]: FAILED to start up
Nov 14 20:43:18 localhost.localdomain lxc-net[24314]: Failed to setup lxc-net.
Nov 14 20:43:18 localhost.localdomain systemd[1]: Started LXC network bridge setup.

Alternately, you can also check through the Systemd journal log.
$ journalctl -u lxc-net.service 
$ journalctl -u lxc-net.service | grep -i 'failed to'

The question we should raise when looking into this error is which other process is trying to bind to port 53, the default DNS port. There are several ways ways to check this.

Are there any other running Dnsmasq instances? Note that output was formatted to improve readability. Besides the one started by lxc-net service. The other two instances were created by libvirt and vagrant-libvirt.
$ ps -o pid,cmd -C dnsmasq
 2851 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/vagrant-libvirt.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

 2852 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/vagrant-libvirt.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

 2933 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

 2934 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

21935 dnsmasq -u nobody --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --listen-address --dhcp-range, --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative

Is there any process currently listening to port 53 using the same IP address of
$ sudo netstat -anp | grep :53 | grep LISTEN
tcp        0      0   *               LISTEN      21935/dnsmasq       
tcp        0      0*               LISTEN      2933/dnsmasq        
tcp        0      0*               LISTEN      2851/dnsmasq        
tcp6       0      0 fe80::fc7b:93ff:fe7a:53 :::*                    LISTEN      21935/dnsmasq   

For my case, which I didn't manage to capture the output, is that another orphaned Dnsmasq instance preventing the 'lxc-net' service from launching a new Dnsmasq instance on lxcbr0 interface. If I remember correctly, this was due to the left over instances by me while debugging the '/usr/libexec/lxc/lxc-net' script.