Showing posts with label network. Show all posts
Showing posts with label network. Show all posts

This Week I Learned 2018 - Week 51

Last week post or something else from the past years.

Are we at the end of hardware virtualization performance? Yes, according to the trend of the Amazon EC2 Virtualization Types. However, in the end, we just go back to bare metal somehow. The rapid improvement in virtualization made setting up homelab and data hoarding possible, cheap, and fast.

Meanwhile, what the heck is Firecracker (official announcement from Amazon)? New virtualization tool based on Kernel-based Virtual Machine (KVM). Interestingly, checking its Git repo indicates that the project was written in Rust, due to its origin started from Chrome OS Virtual Machine Monitor (crosvm), which was written in Rust. Why? Serverless platform, and for Amazon, the removal of VM like Fargate which leads to further cost reduction. Similar, Nitro, the Amazon latest hypervisor, also leverages on KVM but only the core modules to achieve near bare metal performance.

How do you automatically clean up orphaned Docker containers, instances, volumes, networks, or images? If you use Docker for your daily development, your environment accumulates these leftover artifacts unless you're diligent enough to do the clean up yourself. My "research" (ahem, googling) found two tools, docker-gc and docker-clean. The former is written in Golang and thus make it more portable compare to the later in Bash. But why such feature is not built into Docker itself?

What the heck is MVC-L? A concept popularized by OpenCart. Nothing fancy, just an additional Language (L) layer added to the pattern. Combine with another existing extension pattern to MVC, HMVC, we will have HMVCL. Is software pattern still a thing these days?

Is being an independent ISP still a thing in 2018? Yes, it still is, especially in rural area. Whole infrastructure is based on Ubiquiti and Microtik hardware.

How to update parent state from child component in React? Callback in the parent component as a prop to the child component. Treat each component as a class and props as parameters passed to the instance of the class itself. The basis concept is quite straight forward and what was I thinking?

In the parent component.
render() {
    return <Child action={this.handler} />

In the child component.
render() {
    return <Button onClick={this.props.action} />

Bridging a Wireless NIC?

In our previous post, we have setup Pi-hole in LXD through bridging of macvlan network adapter. Thus, our containers share the network segment with the host's machine network. The limitation of such setup is network bridging only works for Ethernet network adapter instead of Wifi network adapter. Because "many wireless cards don't allow spoofing of the source address" (shown in example later) and also a limitation of 802.11. Read this answer for more complete explanation.

Following this guide, I tried to create a bridge using Bridge Control tool, `brctl` and add the wireless network interface, `wlp3s0` to it.
$ sudo brctl addbr wbr0

$ brctl show wbr0
bridge name     bridge id               STP enabled     interfaces
wbr0            8000.000000000000       no

$ sudo brctl addif wbr0 wlp3s0
can't add wlp3s0 to bridge wbr0: Operation not supported

To resolve the shown error above, we need to enable `4addr` option to our Wifi adapter. The `4addr` is used so that "IEEE 802.3 (Ethernet) frame gets encapsulated in a IEEE 802.11 (WLAN) frame".
$ sudo iw dev wlp3s0 set 4addr on
$ sudo brctl addif wbr0 wlp3s0

Trying to obtain an IP from our bridge interface, `wbr0`.
$ sudo dhclient -d wbr0
Internet Systems Consortium DHCP Client 4.3.5
Copyright 2004-2016 Internet Systems Consortium.
All rights reserved.
For info, please visit

Listening on LPF/wbr0/12:34:56:78:90:01
Sending on   LPF/wbr0/12:34:56:78:90:01
Sending on   Socket/fallback
DHCPDISCOVER on wbr0 to port 67 interval 3 (xid=0x444ed350)
DHCPDISCOVER on wbr0 to port 67 interval 3 (xid=0x444ed350)

And suddenly, we lost connectivity to our Wifi connection and can't find any Wifi network adapter anymore. Because of security reason, it's hard to spoof source MAC address.

To undo this, let's remove all changes we've made. Also, we may need to reboot the machine to regain the Wifi connectivity.
$ sudo brctl delbr wbr0
bridge wbr0 is still up; can't delete it

$ sudo iw dev wlp3s0 set 4addr off
command failed: Device or resource busy (-16)

$ sudo brctl delif wbr0 wlp3s0
$ sudo iw dev wlp3s0 set 4addr off

$ sudo ifconfig wbr0 down
$ sudo brctl delbr wbr0

$ sudo systemctl restart NetworkManager

Nevertheless, there are still different ways to make this works although far more complicated. Since macvlan does not work with wireless adapter, there is an alternative way using ipvlan. However, this was proposed to be included to LXD but postponed since macvlan provides similar features. Furthermore, DHCP will not works in both methods anyway.

This Week I Learned - 2016 Week 06

Previous post.

#1 PatternCraft. Learning Design Pattern through StarCraft. Never underestimate the importance of Software Metaphors in abstracting software engineering concepts.

#2 Ask HN: Best curated newsletters? Need a way to reduce your time from the net but at the same time still fear of missing out? Pick your favourite curated newsletters. Cron.weekly seems to have plenty of links which I've found interesting if you're looking into system administration. Mandarin Weekly caught my attention as well.

#3 How Git Merging turns you into a GITar Hero. Till today, I still don't understand why developers still fail to see the benefit of Git Rebasing. Maybe the complexity of the merged trees indicates productivity or sense of accomplishment? You know, software engineers tends to over-analyze and over-engineer.

#4 Linux Performance Analysis in 60,000 Milliseconds. Using uptime, dmesg, vmstat, mpstat, pidstat, iostat, free, sar, and top command, you can have an overview of the resource usage of a system. Don't want to go through the hassle of all these commands? Just use Glances, web or console-based monitoring tools written in Python. Perhaps, htop, an interactive process viewer or iotop, disk I/O status monitoring tool.

#5 Ping Sweep. Fun activity to do with nephews during CNY. We all learn how to find all available hosts that were connecting to the Access Point (AP). From the list of IP addresses, divide the these these hosts into mobile and computing devices. Have fun times scanning the network where they both overloaded the Wifi router by "nmapping" the network. The seed of learning have been planted, is really up to them to explore further. Hopefully, by the next CNY, they will move ahead even further and know which particular field in IT they want to venture into.

#6 Janice Kaplin: "The Gratitude Diaries". Is time to reflect and appreciate on what we have and where we are. How? Keep a gratitude journal.

#7 Today I Learned (TIL) is a famous subreddit. For technology related (programming or system administration), there are TIL collections created by Josh Branchaud, hashrocket, Jake Worth, and thoughtbot.

Troubleshooting Dynamic Host Configuration Protocol (DHCP) Connection in LXD, Part 1: The Dnsmasq Server

While testing LXD, the GNU/Linux container hypervisor, one of the issue I've encountered was certain containers failed to obtain an IP address after booting up. Hence, for the past few days, while scratching my head investigating the issue, I've gained some understanding on how DHCP works and learned a few tricks on how to troubleshoot a DHCP connection.

DHCP is a client/server where the client obtain an IP address from the server. Thus, to troubleshoot any connection issue, we should look in two places, the server and the client side.

Is Dnsmasq up and running?
First, the server end. As I mentioned in my previous post, in LXD, the lxcbr0 bridge interface is basically a virtual switch, through Dnsmasq, provides network infrastructures services like Domain Name System (DNS) and DHCP services. If DHCP is not working, first things to check whether the Dnsmasq has been started correctly. Pay attention to all lines that contains the word 'dnsmasq' and check for any errors.
$ sudo systemctl status lxc-net -l
● lxc-net.service - LXC network bridge setup
   Loaded: loaded (/usr/lib/systemd/system/lxc-net.service; enabled; vendor preset: disabled)
   Active: active (exited) since Wed 2015-11-18 21:04:24 MYT; 1s ago
  Process: 21863 ExecStop=/usr/libexec/lxc/lxc-net stop (code=exited, status=0/SUCCESS)
  Process: 21891 ExecStart=/usr/libexec/lxc/lxc-net start (code=exited, status=0/SUCCESS)
 Main PID: 21891 (code=exited, status=0/SUCCESS)
   Memory: 408.0K
      CPU: 39ms
   CGroup: /system.slice/lxc-net.service
           └─21935 dnsmasq -u nobody --strict-order --bind-interfaces --pid-file=/run/lxc/ --listen-address --dhcp-range, --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative

Nov 18 21:04:24 localhost.localdomain dnsmasq[21935]: started, version 2.75 cachesize 150
Nov 18 21:04:24 localhost.localdomain dnsmasq[21935]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify
Nov 18 21:04:24 localhost.localdomain dnsmasq-dhcp[21935]: DHCP, IP range --, lease time 1h
Nov 18 21:04:24 localhost.localdomain dnsmasq-dhcp[21935]: DHCP, sockets bound exclusively to interface lxcbr0
Nov 18 21:04:24 localhost.localdomain dnsmasq[21935]: reading /etc/resolv.conf
Nov 18 21:04:24 localhost.localdomain dnsmasq[21935]: using nameserver
Nov 18 21:04:24 localhost.localdomain dnsmasq[21935]: read /etc/hosts - 2 addresses
Nov 18 21:04:24 localhost.localdomain systemd[1]: Started LXC network bridge setup.

As LXD is still actively under development, there are still many pending issues, you may want to walk through the '/usr/libexec/lxc/lxc-net' script to investigate more. Although from my experience, is simple service restart 'systemctl restart lxc-net' should be sufficient.

Failed to create listening socket?
Few days back, one of the issue I've experienced is that the Dnsmasq server failed to start due to failure in creating listening socket.
Nov 14 20:43:18 localhost.localdomain systemd[1]: Starting LXC network bridge setup...
Nov 14 20:43:18 localhost.localdomain lxc-net[24314]: dnsmasq: failed to create listening socket for Cannot assign requested address
Nov 14 20:43:18 localhost.localdomain dnsmasq[24347]: failed to create listening socket for Cannot assign requested address
Nov 14 20:43:18 localhost.localdomain dnsmasq[24347]: FAILED to start up
Nov 14 20:43:18 localhost.localdomain lxc-net[24314]: Failed to setup lxc-net.
Nov 14 20:43:18 localhost.localdomain systemd[1]: Started LXC network bridge setup.

Alternately, you can also check through the Systemd journal log.
$ journalctl -u lxc-net.service 
$ journalctl -u lxc-net.service | grep -i 'failed to'

The question we should raise when looking into this error is which other process is trying to bind to port 53, the default DNS port. There are several ways ways to check this.

Are there any other running Dnsmasq instances? Note that output was formatted to improve readability. Besides the one started by lxc-net service. The other two instances were created by libvirt and vagrant-libvirt.
$ ps -o pid,cmd -C dnsmasq
 2851 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/vagrant-libvirt.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

 2852 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/vagrant-libvirt.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

 2933 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

 2934 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

21935 dnsmasq -u nobody --strict-order --bind-interfaces --pid-file=/run/lxc/ --listen-address --dhcp-range, --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative

Is there any process currently listening to port 53 using the same IP address of
$ sudo netstat -anp | grep :53 | grep LISTEN
tcp        0      0   *               LISTEN      21935/dnsmasq       
tcp        0      0*               LISTEN      2933/dnsmasq        
tcp        0      0*               LISTEN      2851/dnsmasq        
tcp6       0      0 fe80::fc7b:93ff:fe7a:53 :::*                    LISTEN      21935/dnsmasq   

For my case, which I didn't manage to capture the output, is that another orphaned Dnsmasq instance preventing the 'lxc-net' service from launching a new Dnsmasq instance on lxcbr0 interface. If I remember correctly, this was due to the left over instances by me while debugging the '/usr/libexec/lxc/lxc-net' script.