Linux Containers (LXC) in Fedora 22 Rawhide - Part 1

While Docker, an application container is widely popular right now, I've decided to try LXC, a machine container that hold a virtual machine like VirtualBox or WMWare but with near bare-metal performance. As I was running on Fedora Rawhide (F22), let's try to install and setup LXC in this distro.

Installation is pretty much straight forward.
$ sudo dnf install lxc lxc-templates lxc-extra

Checking our installed version against the latest available version. Our installed version on par with the current release.
$ lxc-ls --version
1.1.2

The first thing to do is to check our LXC configuration. As emphasized in red below, the Cgroup memory controller is not enabled by default as it will incur additional memory. This can be enabled through by adding boot parameter cgroup_enable=memory to the Grub boot loader. For now, we will keep that in mind and stick to the default.
$ lxc-checkconfig

Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /boot/config-4.0.1-300.fc22.x86_64
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: missing
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

Before we can create our container, let's find out the available templates or GNU/Linux distros we can create.
$ ll /usr/share/lxc/templates/
total 348K
-rwxr-xr-x. 1 root root  11K Apr 24 03:22 lxc-alpine*
-rwxr-xr-x. 1 root root  14K Apr 24 03:22 lxc-altlinux*
-rwxr-xr-x. 1 root root  11K Apr 24 03:22 lxc-archlinux*
-rwxr-xr-x. 1 root root 9.5K Apr 24 03:22 lxc-busybox*
-rwxr-xr-x. 1 root root  29K Apr 24 03:22 lxc-centos*
-rwxr-xr-x. 1 root root  11K Apr 24 03:22 lxc-cirros*
-rwxr-xr-x. 1 root root  17K Apr 24 03:22 lxc-debian*
-rwxr-xr-x. 1 root root  18K Apr 24 03:22 lxc-download*
-rwxr-xr-x. 1 root root  48K Apr 24 03:22 lxc-fedora*
-rwxr-xr-x. 1 root root  28K Apr 24 03:22 lxc-gentoo*
-rwxr-xr-x. 1 root root  14K Apr 24 03:22 lxc-openmandriva*
-rwxr-xr-x. 1 root root  15K Apr 24 03:22 lxc-opensuse*
-rwxr-xr-x. 1 root root  40K Apr 24 03:22 lxc-oracle*
-rwxr-xr-x. 1 root root  11K Apr 24 03:22 lxc-plamo*
-rwxr-xr-x. 1 root root 6.7K Apr 24 03:22 lxc-sshd*
-rwxr-xr-x. 1 root root  25K Apr 24 03:22 lxc-ubuntu*
-rwxr-xr-x. 1 root root  13K Apr 24 03:22 lxc-ubuntu-cloud*

Let's proceed ahead by create our first container, a CentOS 6 distro. Unfortunately, as seen below, the creation failed due to deprecation of the Yum command which was redirected to DNF command.
$ sudo lxc-create -t centos -n centos-test

Host CPE ID from /etc/os-release: cpe:/o:fedoraproject:fedora:22
This is not a CentOS or Redhat host and release is missing, defaulting to 6 use -R|--release to specify release
Checking cache download in /var/cache/lxc/centos/x86_64/6/rootfs ... 
Downloading centos minimal ...
Yum command has been deprecated, redirecting to '/usr/bin/dnf -h'.
See 'man dnf' and 'man yum2dnf' for more information.
To transfer transaction metadata from yum to DNF, run:
'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate'

Yum command has been deprecated, redirecting to '/usr/bin/dnf --installroot /var/cache/lxc/centos/x86_64/6/partial -y --nogpgcheck install yum initscripts passwd rsyslog vim-minimal openssh-server openssh-clients dhclient chkconfig rootfiles policycoreutils'.
See 'man dnf' and 'man yum2dnf' for more information.
To transfer transaction metadata from yum to DNF, run:
'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate'

Config error: releasever not given and can not be detected from the installroot.
Failed to download the rootfs, aborting.
Failed to download 'centos base'
failed to install centos
lxc-create: lxccontainer.c: create_run_template: 1202 container creation template for centos-test failed
lxc-create: lxc_create.c: main: 274 Error creating container centos-test

The above error is a good example on why the transition from YUM to DNF command was unnecessary and caused breakage. It turned out that /usr/bin/yum is a shell script that display notification message. To resolve this, we need to point /usr/bin/yum to the actual yum program. There are way to bypass this step where we'll discuss about this in Part 2.
$ sudo mv /usr/bin/yum /usr/bin/yum2dnf
$ sudo ln -s /usr/bin/yum-deprecated /usr/bin/yum
$ ll /usr/bin/yum
lrwxrwxrwx. 1 root root 23 May  5 23:40 /usr/bin/yum -> /usr/bin/yum-deprecated*

Let's us try again. Although there is notification, the creation of the container will run smoothly. Since we're creating this for the first time, it will took a while to download all the packages.
$ sudo lxc-create -t centos -n centos-test
......
Complete!
Download complete.
Copy /var/cache/lxc/centos/x86_64/6/rootfs to /var/lib/lxc/centos-test/rootfs ... 
Copying rootfs to /var/lib/lxc/centos-test/rootfs ...
Storing root password in '/var/lib/lxc/centos-test/tmp_root_pass'
Expiring password for user root.
passwd: Success

Container rootfs and config have been created.
Edit the config file to check/enable networking setup.

The temporary root password is stored in:

        '/var/lib/lxc/centos-test/tmp_root_pass'

The root password is set up as expired and will require it to be changed
at first login, which you should do as soon as possible.  If you lose the
root password or wish to change it without starting the container, you
can change it from the host by running the following command (which will
also reset the expired flag):

        chroot /var/lib/lxc/centos-test/rootfs passwd

Checking our newly created container.
$ sudo lxc-ls
centos-test  

Checking the container status.
$ sudo lxc-info -n centos-test
Name:           centos-test
State:          STOPPED

Start our newly created container. Yet again, another error.
$ sudo lxc-start -n centos-test
lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 346 To get more details, run the container in foreground mode.
lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.

Let's try again, but with foreground mode (-F).
$ sudo lxc-start -F -n centos-test
lxc-start: conf.c: instantiate_veth: 2672 failed to attach 'vethM9Q6RT' to the bridge 'lxcbr0': Operation not permitted
lxc-start: conf.c: lxc_create_network: 2955 failed to create netdev
lxc-start: start.c: lxc_spawn: 914 failed to create the network
lxc-start: start.c: __lxc_start: 1164 failed to spawn 'centos-test'
lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.

I was quite surprised that Fedora did not create the lxcbr0 bridge interface automatically. Instead, we will use the existing virbr0 provided by libvirtd.
$ sudo yum install libvirt-daemon
sudo systemctl start libvirtd

Check the bridge network interface.
$ brctl show
bridge name     bridge id               STP enabled     interfaces
virbr0          8000.525400c28250       yes             virbr0-nic

Edit our container config file and change the network link from lxcbr0 to virbr0.
$ sudo vim /var/lib/lxc/centos-test/config
lxc.network.link = virbr0

Try to start the container again, this time, another '819 Permission denied' error.
$ sudo lxc-start -F -n centos-test
lxc-start: conf.c: lxc_mount_auto_mounts: 819 Permission denied - error mounting /usr/lib64/lxc/rootfs/proc/sys/net on /usr/lib64/lxc/rootfs/proc/net flags 4096
lxc-start: conf.c: lxc_setup: 3833 failed to setup the automatic mounts for 'centos-test'
lxc-start: start.c: do_start: 699 failed to setup the container
lxc-start: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2
lxc-start: start.c: __lxc_start: 1164 failed to spawn 'centos-test'
lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.

After struggled and googled for answer for the past hours, it actually dawned to me that the '819 Permission denied' error is related to SELinux policy. I did a quick check by disabled SELinux and reboot the machine and was able to start the container.

Also, just to confirm the SELinux error for lxc-start.
$ sudo grep lxc-start /var/log/audit/audit.log | tail -n 1
type=AVC msg=audit(1430849851.869:714): avc:  denied  { mounton } for  pid=3780 comm="lxc-start" path="/usr/lib64/lxc/rootfs/proc/1/net" dev="proc" ino=49148 scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dir permissive=0

Start the SELinux Alert Browser and run the below commands to add the security policy.
$ sealert

$ sudo grep lxc-start /var/log/audit/audit.log | audit2allow -M mypol
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i mypol.pp

$ sudo semodule -i mypol.pp

Start our container again and check it status.
$ sudo lxc-start -n centos-test 
[kianmeng.ang@localhost ~]$ sudo lxc-info -n centos-test
Name:           centos-test
State:          RUNNING
PID:            6742
CPU use:        0.44 seconds
BlkIO use:      18.55 MiB
Memory use:     12.14 MiB
KMem use:       0 bytes
Link:           veth4SHUE1
 TX bytes:      578 bytes
 RX bytes:      734 bytes
 Total bytes:   1.28 KiB

Attach to our container. There is no login needed.
$ sudo lxc-attach -n centos-test
[root@centos-test /]# uname -a
Linux centos-test 4.0.1-300.fc22.x86_64 #1 SMP Wed Apr 29 15:48:25 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@centos-test /]# cat /etc/centos-release 
CentOS release 6.6 (Final)

3 comments:

  1. Thanks for the help especially the bit about '819 Permission denied' having to do with SELinux

    ReplyDelete
  2. Thanks very much for an *extremely* helpful post!

    ReplyDelete
  3. Concise, accurate and very helpful. Thanks!

    ReplyDelete