Polipo - Tiny Caching Proxy

During provisioning a new virtual machine (VM), you will need to repeatedly destroy and rebuild the VM. One of the bottleneck is you've to re-download all the GNU/Linux distro packages. While you can use packaging tool like APT or YUM to cache your packages, it still can be shared by different VMs. To solve this, you can set up a caching proxy at your host machine to be shared among all the guest VMs.

Instead of default caching proxy, Squid, I've opted for Polipo, a smaller and simpler caching proxy. Setting up was quite straightforward with additional minor changes.

Install the packages.
$ sudo yum install polipo

Enable the service after reboot.
$ sudo systemctl enable polipo.service

Check the status of the service. One of the benefit of systemctl is that it show a lot of crucial details of the daemon or service. Thus, helps a lot when we're troubleshooting the server.
$ sudo systemctl status polipo.service 

Set up the proxy connection details as environment variables so that the console app, for example, wget or curl, can use this.
$ export {http,https,ftp,rsync}_proxy="http://localhost:8123"
$ export no_proxy=localhost,127.0.0.1
$ env | grep proxy
http_proxy=http://localhost:8123
ftp_proxy=http://localhost:8123
rsync_proxy=http://localhost:8123
https_proxy=http://localhost:8123

To test our proxy server using either curl or wget. Using curl. Option -sv is to show server header verbosely.
$ curl -sv www.google.com 2>&1 | grep 8123
* About to connect() to proxy localhost port 8123 (#0)
* Connected to localhost (127.0.0.1) port 8123 (#0)

Using wget. Using -S --spider so that wget don't download anything.
$ wget -S --spider www.google.com 2>&1 | grep 8123
Connecting to localhost (localhost)|::1|:8123... failed: Connection refused.
Connecting to localhost (localhost)|127.0.0.1|:8123... connected.
Connecting to localhost (localhost)|127.0.0.1|:8123... connected.

To test the caching while download a large file. First time download took around 3 minutes and subsequent download of similar file took less than 1 second.
$ time wget http://libguestfs.org/download/builder/cirros-0.3.1.xz
......
2014-09-28 04:17:40 (55.7 KB/s) - ‘cirros-0.3.1.xz’ saved [11419004/11419004]
real    5m52.107s
user    0m0.096s
sys     0m0.461s

$ time wget http://libguestfs.org/download/builder/cirros-0.3.1.xz
......
2014-09-28 04:17:51 (478 MB/s) - ‘cirros-0.3.1.xz.1’ saved [11419004/11419004]
real    0m0.028s
user    0m0.002s
sys     0m0.023s

No comments:

Post a Comment