Polipo - Tiny Caching Proxy

During provisioning a new virtual machine (VM), you will need to repeatedly destroy and rebuild the VM. One of the bottleneck is you've to re-download all the GNU/Linux distro packages. While you can use packaging tool like APT or YUM to cache your packages, it still can be shared by different VMs. To solve this, you can set up a caching proxy at your host machine to be shared among all the guest VMs.

Instead of default caching proxy, Squid, I've opted for Polipo, a smaller and simpler caching proxy. Setting up was quite straightforward with additional minor changes.

Install the packages.
$ sudo yum install polipo

Enable the service after reboot.
$ sudo systemctl enable polipo.service

Check the status of the service. One of the benefit of systemctl is that it show a lot of crucial details of the daemon or service. Thus, helps a lot when we're troubleshooting the server.
$ sudo systemctl status polipo.service 

Set up the proxy connection details as environment variables so that the console app, for example, wget or curl, can use this.
$ export {http,https,ftp,rsync}_proxy="http://localhost:8123"
$ export no_proxy=localhost,127.0.0.1
$ env | grep proxy
http_proxy=http://localhost:8123
ftp_proxy=http://localhost:8123
rsync_proxy=http://localhost:8123
https_proxy=http://localhost:8123

To test our proxy server using either curl or wget. Using curl. Option -sv is to show server header verbosely.
$ curl -sv www.google.com 2>&1 | grep 8123
* About to connect() to proxy localhost port 8123 (#0)
* Connected to localhost (127.0.0.1) port 8123 (#0)

Using wget. Using -S --spider so that wget don't download anything.
$ wget -S --spider www.google.com 2>&1 | grep 8123
Connecting to localhost (localhost)|::1|:8123... failed: Connection refused.
Connecting to localhost (localhost)|127.0.0.1|:8123... connected.
Connecting to localhost (localhost)|127.0.0.1|:8123... connected.

To test the caching while download a large file. First time download took around 3 minutes and subsequent download of similar file took less than 1 second.
$ time wget http://libguestfs.org/download/builder/cirros-0.3.1.xz
......
2014-09-28 04:17:40 (55.7 KB/s) - ‘cirros-0.3.1.xz’ saved [11419004/11419004]
real    5m52.107s
user    0m0.096s
sys     0m0.461s

$ time wget http://libguestfs.org/download/builder/cirros-0.3.1.xz
......
2014-09-28 04:17:51 (478 MB/s) - ‘cirros-0.3.1.xz.1’ saved [11419004/11419004]
real    0m0.028s
user    0m0.002s
sys     0m0.023s

Buggy Nouveau Driver in Fedora 20

Regardless what GNU/Linux distros, I still faced the same issue regarding the Nouveau, the open-sourced driver Nvidia graphic cards.

While using the 3.16 kernel, I can't seemed to boot into the graphical login. Is all blank page. It was perfectly fine using the 3.11 kernel.
$ uname -sr
Linux 3.16.2-201.fc20.x86_64

In the end, I've to switch to the console login by Ctrl-Alt-F2, and checking the systemd journal log. Sample error message relating the Nouveau driver as shown.
$ journalctl -r | grep nouveau
Sep 27 11:23:24 butterfly kernel: nouveau E[   PFIFO][0000:01:00.0] CACHE_ERROR - ch 0 [DRM] subc 2 mthd 0x0130 data 0x0000000
......
Sep 27 10:16:43 butterfly kernel: nouveau E[     DRM] GPU lockup - switching to software fbcon

Some quick search revealed that to stop the X from freezing during startup, you've to disable Nouveau acceleration, which is the common, typical, and conventional solution. There are two ways.

First, adding 'nouveau.nofbaccel=1' to the kernel parameter. This can be done during Grub2 bootup by pressing 'e' and append, after the 'rhgb quiet' the option line to the kernel parameter. It looks something like below.
...rhgh quiet nouveau.nofbaccel=1

Later, just press F10 key to continue booting the system. However, you'll have to do this everything you boot up your machine.

Second, to make this a permanent solution, you'll need to modify the Grub2 configuration. Again, there are two ways to do so.

Edit the /etc/default/grub and append 'nouveau.nofbaccel=1' to the line GRUB_CMDLINE_LINUX and update the Grub2.
$ sudo grub2-mkconfig --output=/boot/grub2/grub.cfg

Another approach is the set the configuration through module configuration file when the kernel is loading. I prefer this option as it is easier to change than using the Grub2 way.
$ sudo sh -c 'echo "options nouveau nofbaccel=1' > /etc/modprobe.d/nouveau.conf "

You may ask, why not using the proprietary Nvidia driver? Well, unfortunately, I can't get it to work correctly, especially with latest 3.16 kernel. And, I don't want to waste my time on troubleshooting the same issue again and again.

Stuck with Nouveau driver forced me to switch my desktop environment from Gnome 3 to Xfce4. I've learned that without well-supported graphic card driver, Gnome 3 experience still a lot to be desired. Mind you, this workstation is running on 20GB ram and yet, it does not help.

While not as fancy as Gnome 3, Xfce4 seems acceptable for my daily usage compare to other desktop environment.

Yum in Fedora 20

What is the first thing you do upon first login to any GNU/Linux distros? You update the whole system. In our case for F20, was using Yum.

Nothing fancy here, similar to Debian-based distros as well.
$ sudo yum update
......
Total download size: 559 M
Is this ok [y/d/N] : y
Downloading packages:
updates/20/x86_64/prestodelta
Delta RPMs reduced 361 M of updates to 96 M (73% saved)
.......

What interested me is use of Delta RPMs (DRPMs) or Presto to speed up downloading time and save bandwidth. As the name implied, DRPMs contains the binary differences between old and new RPM packages.

Similarly, Debian-based distros have something similar but not integrated with apt by default called debdelta which I haven't try out yet. The only con with such tool is that you will need quite a lot of processing power to calculate the differences.

After a few days of using F20, I've noticed that Yum was dog slow. Everything when I install or update the system, it will always requery the packages metadata. Until I realized, caching of downloaded RPM packages was disabled by default.
$ cat yum.conf | grep cache
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0

Just change the keepcache=1 and you'll notice less lagging when carry out any Yum actions.

To speed up downloading, you can also use the fastestmirror plugin. Make sure you change enabled=1 to enable the plugin.
$ sudo yum install yum-plugin-fastestmirror
$ cat /etc/yum/pluginconf.d/fastestmirror.conf | grep enabled
enabled=0

Yum groupinstall. While I really using aliases to install related group of RPM packages, unfortunately, it was quite buggy. One particular group I really love is the "Minimal Install" as illustrated below.
$ sudo yum groupinstall "Minimal Install"

This is quite helpful when you want to create a base cloud image or JEOS and change the role of the installation accordingly like as a Web or database server.

Unfortunately, I can't find something quite to install LAMP-stack quickly and conveniently like in Debian-based distros as shown.
$ sudo apt-get install lamp-server^

The only equivalent using Yum groupinstall is as shown here.
$ sudo yum groupinstall "Web Server"
$ sudo yum groupinstall "MySQL Database client"
$ sudo yum groupinstall "MySQL Database server"
$ sudo yum groupinstall "PHP Support"

Comparing both Yum and Apt, I still prefer Apt, which for me, far more stable and faster.

Fedora 20 Installation

After hoping through different GNU/Linux distros, I can't still have a smooth and painless installation with Fedora 20. There are always some tweaks and googling here and there to make things work. When you think of it, you can't blame it, as it was not designed to be used as a desktop operating system.

Some thoughts on the Fedora 20 installation and setup.

1. Resizing LUKS encrypted LVM partition. I tried but failed to do it correctly in my default Debian installation. Missed out certain step and corrupted the whole partition. Luckily, I've backup all the important stuff elsewhere. But somehow I was surprised that GParted 0.19.1 can ONLY detect luks-crypt partition. No worry, I'll learn about this later once I've setup the virtualization in the workstation.

2. Installation using netinstall. Supposed to speed up the installation process but I was stuck waiting to set up the installation source with no indication what so ever. I was under the assumption that the installer, Anaconda, somehow crashed or freeze but it turned out my slow Internet connection was the cause of it. In the end, I just switched to full DVD installation instead.

3. Partition Scheme. Following the default standard scheme but with encrypted partition using Butterface file system (Btrfs). Heard a lot about this filesystem but never really try it, will explore more after this.

4. No initial login screen, to be exact, GDM (Gnome Display Manager) was missing, and just a blank wallpaper. This happened after the first boot up upon finishing all the installation. Switched to another console by Ctrl+Alt+F2 and checked the /var/log/boot.log file. Nothing particular unique suggesting any issue. Suspected must be related to X or buggy Nouveau graphic card driver. Reboot the machine again and it seems to work.

The init Wars

"uselessd (the useless daemon, or the daemon that uses less... depending on your viewpoint) is a project which aims to reduce systemd to a base initd, process supervisor and transactional dependency system, while minimizing intrusiveness and isolationism. Basically, it’s systemd with the superfluous stuff cut out, a (relatively) coherent idea of what it wants to be, support for non-glibc platforms and an approach that aims to minimize complicated design."
-- uselessd.darknedgy.net
Via Phoronix. The never ending drama to the init wars or if you look at the positive side, the freedom of forking a software project to prove a point. Never underestimate the power of a single determined programmer. I just wish more resource should be poured towards getting a stable and optimize open sourced graphic card drivers like Nouveau or Radeon.

Switching to Fedora

It has been a while I've touch any RPM-based GNU/Linux distro, decided to try out Fedora as I'm going to learn more about virtualization.

Trying wodim, which stands for "Write Optical DIsk Media", to burn our ISO file to DVD.
$ sudo apt-get install wodim
$ wodim --version
Cdrecord-yelling-line-to-tell-frontends-to-use-it-like-version 2.01.01a03-dvd 
Wodim 1.1.11
Copyright (C) 2006 Cdrkit suite contributors
Based on works from Joerg Schilling, Copyright (C) 1995-2006, J. Schilling

Interesting history of Cdrkit, which was a fork of latest release of GPL-licensed cdrtools.
$ wodim --devices
wodim: Overview of accessible drives (1 found) :
----------------------------------------------------------------------------------
 0  dev='/dev/sg1'      rwrw-- : 'hp' 'DVDROM DH40N'
----------------------------------------------------------------------------------

$ sudo wodim -v dev=/dev/sg1 speed=4 -eject Fedora-20-x86_64-DVD.iso

Shockwave Flash has crashed

Kept getting this error message while trying to play any Flash video these past few days in Google Chrome Version 37.0.2062.120 in Debian Wheezy. According the this bug report, it was caused by the 'erroneous GLIBC_2.14 requirement'.

From all the recommended temporary solution, the best is still revert to the previous working version or downgrading. Luckily, apt kept a cache copy of the deb package.
$ sudo dpkg -i /var/cache/apt/archives/google-chrome-stable_37.0.2062.94-1_amd64.deb
$ killall chrome
$ google-chrome

Oz - Virtual Machine Builder

Stumbled upon this program, oz while trying different kind of virtual machine image builder. Unfortunately, there is not Deb package for Debian 7, hence I've to build one for myself.

Following the instruction here, creating deb package is quite straightforward.

Install the prerequisite packages.
$ sudo apt-get install debhelper python-all build-essential git-core gdebi

Clone the Git repo.
$ mkdir /tmp/oz
$ cd /tmp/oz
$ git clone https://github.com/clalancette/oz oz-git

Build the deb package.
$ cd oz-git
$ dpkg-buildpackage -us -uc
$ cd ../

Install the software with all the necessary dependencies using Gdebi installer.
$ sudo gdebi oz_*_all.deb

However, to get this tool to work, you'll have to install and setup KVM virtualization.
$ sudo apt-get install qemu-kvm libvirt-bin
$ sudo adduser kianmeng kvm
$ sudo adduser kianmeng libvirt

Refresh and update your user groups without manually logout from the system.
$ exec su -l $USER

As non-root user, there should be no permission denied error when running below command.
$ sudo virsh list --all

Network stuff.
$ sudo virsh net-list --all

Name                 State      Autostart
-------------------------------------------------------
default              inactive   yes      

$ sudo virsh net-start default

error: Failed to start network default
error: internal error Child process (/usr/sbin/dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/default.pid --conf-file= --except-interface lo --listen-address 192.168.122.1 --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases --dhcp-lease-max=253 --dhcp-no-override) unexpected exit status 2: 
dnsmasq: failed to create listening socket for 192.168.122.1: Address already in use

Apparently, running your own dnsmasq with have conflict with libvirt. To solve it, make sure dnsmasq bind to certain interface only. Edit /etc/dnsmasq.conf and uncomment these lines.
interface=wlan1
bind-interface

$ sudo service dnsmasq restart
$ sudo virsh net-start default
$ sudo virsh net-autostart default
$ sudo $ /sbin/brctl show
bridge name     bridge id                            STP enabled     interfaces
virbr0                 8000.525400d0634b       yes                     virbr0-nic

On a related note, based on my few days experience, if you want to try out anything related to cloud or virtual machine, Fedora seems to be a more suitable and supported GNU/Linux distro. I'm thinking whether to move away from Debian to Fedora as my base distro.

PHP Version Manager (PHPENV)

With the recently release PHP 5.6, I stumbled upon these tools, phpenv, (inspired by rbenv) and php-build which let you build different PHP versions without messing up your existing installation. For installation steps, we're following Kobito's setup guide.

1. To setup both phpenv and php-build, just type these commands. Be extra careful when running downloaded shell script directly from the net.

$ curl https://raw.github.com/CHH/phpenv/master/bin/phpenv-install.sh | bash
$ git clone git://github.com/CHH/php-build.git ~/.phpenv/plugins/php-build
$ echo 'export PATH="$HOME/.phpenv/bin:$PATH"' >> ~/.bashrc
$ echo 'eval "$(phpenv init -)"' >> ~/.bashrc
$ exec $SHELL -l

2. Checking our installation. Second command should give you a long list of PHP versions.
$ phpenv --version
rbenv 0.4.0-98-g13a474c

$ phpenv install -l

3. As I was running Debian, install all the necessary packages so we can compile the source code.
$ sudo apt-get install make ccache re2c libcurl libcurl-dev bison libcurl4-gnutls-dev libjpeg62-dev libmcrypt-dev libtidy libtidy-dev libxslt1-dev apache2-prefork-dev

4. Compilation. Took me around 15 minutes.
$ CFLAGS="-g" phpenv install 5.6.0

$ which php
/home/kianmeng/.phpenv/shims/php

5. Switching between installed version (system) and your compiled version.
$ phpenv versions
* system (set by /home/kianmeng/.phpenv/version)
5.6.0

$ phpenv global
system

$ php --version | grep ^PHP
PHP 5.4.4-14+deb7u14 (cli) (built: Aug 21 2014 08:36:44)

$ phpenv rehash
$ phpenv global 5.6.0

$ phpenv global
5.6.0

$ php --version | grep ^PHP
PHP 5.6.0 (cli) (built: Sep  1 2014 03:26:46)

Unfortunately, to install phpdbg, you'll need to create another shell script, as a plugin to php-build in order to build it. Someday perhaps.