Oh crap ! I shouldn't have done that?!

Don't multitask. Seriously, don't multitask while handling important stuff. As usual, I was busy coding and monitoring the Apache web server log at the same time. Due to my wonky laptop battery, I've decided to switch to another backup battery and rebooted the machine. Exit from my remote connection and type this command at my console to reboot my lappy.
$ sudo reboot

Weird. Nothing happens. No programs was closed at my desktop environment. Nothing was shutdown. Checked again at my terminal. To my horror, I just realized that I didn't log out properly from the live production server and I had rebooted the live production server instead !

Sh*t! Holy double sh*t!

Rushed out to tell my boss about it. He laughed about it in a cool as cucumber way. Not surprised. He and those long serving employees had experienced far worse scenarios before. (They have enough war stories for generations) Not for me, this is so effing embarrassing stupid. Server rebooted, double checked everything again. Thank goodness, everything seemed okay and back to normal.

Post mortem analysis. Can we prevent accidental reboot in operational critical server ? Yes, you can. Just use Molly-guard.

In a non-IT term, Molly-guard is actually a cover (normall in red) of a button to prevent accidentally triggering of unwanted event like firing nuclear missiles. As a software, Molly-guard is a shell-script that check for existing SSH session and if any of these shutdown, reboot, halt, or poweroff commands were invoked. The script will prompt you to key in the hostname to confirm before proceeding with the intended critical action.

How do set this up and get it to work? In Ubuntu/Debian-based distros, is effing easy. Just apt-get it.

1. Install the package
$ sudo apt-get install molly-guard

2. Run a simulation. Note default configuration only works over a SSH session.
$ ssh localhost
$ sudo reboot
W: molly-guard: SSH session detected!
Please type in hostname of the machine to reboot:
Good thing I asked; I won't reboot servername ...
W: aborting reboot due to 30-query-hostname exiting with code 1.

3. What if you also want this program to work on non-SSH session ? Just edit the config file (/etc/molly-guard/rc) and set ALWAYS_QUERY_HOSTNAME to true.
# ALWAYS_QUERY_HOSTNAME
# when set, causes the 30-query-hostname script to always ask for the
# hostname, even if no SSH session was detected.
ALWAYS_QUERY_HOSTNAME=true

How about Centos/Redhat-based distros? Not in the official repositories. (Now you know why I dislikes rpm-based distros, limited softwares selection). You can download packages from these sites. Installation and setup should be the same.

Fun and exciting times these days. I blamed it all on the bloody wonky battery.

No comments:

Post a Comment