Showing posts with label bash. Show all posts
Showing posts with label bash. Show all posts

This Week I Learned 2018 - Week 01

Last year post or something from the past.

Happy new year 2018! Wedding is always an interesting and peculiar event, especially for someone like me. Nevertheless, it was a good short and simple ceremony.

Down with illness. Most likely due to infection of trying to culture Infusoria for your tiny frys. Wash your hand thoroughly. A few unlucky fellows didn't manage to survive the overcrowded tank and succumbed to the stress and infection leading to Dropsy (YouTube) and Fin rot. While the S1 solution (Malachite Green. Warning, it was banned for food fish and the dye is carcinogenic. You will notice that no ingredient was stated behind the bottle label.) or sea salts (boosts up the immune systems) did help but not for all cases. There is no cure for late stage infection. Still, prevention is better than cure. However, lack of experience, no quarantine procedures, diligence, it's very hard to maintain good water quality and acceptable temperature for each species in the tank.

What are the available methods of spawning Bettas? The are so many and each one of them have pros and cons. While the local experience breeders prefer the Thai way, those living in the city without access to open spaces have to stick with the Shallow Method. The one thing we learned was patience and preparation are crucial for a successfully spawning without killing the Betta pairs as well as the frys. There are so many steps in conditioning the Betta pairs and make sure male don't kill the female and eat those hatched frys.

The Looming Storm. It has been a while since the last good suspend thriller film coming from mainland China.

Kong kong kong kong kong kong kong kong kong kong kong. Being a native Hokkien speaker, I still can't get it right. This reminded me of the Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo, a good example of homonyms and homophones.

To people who are doing startups, read the story of how DropBox started as a Minimal Viable Product (MVP). The one particular slide (screenshot captured below) contains enough good advice on how to release your MVP to test the market, adjust, and re-release to grab the market share. An MVP maybe even be an Excel sheet with clickable user interface. Yes, there are people who use Excel sheet as good prototyping tool. Did I mentioned that most web apps are glorified Excel sheet?


What did you work on in 2017 and what are your 2018 resolutions? The usual annual reflection and future projection. The GarageScript and Hacker Dojo looks interesting but make a world a better place? Doubt so, do we need more software engineers? Meanwhile, this should be in the bucket list of most tech person, take a sabbatical year, move to smaller town, down sizing everything, and do things you want to do there. Best way to summarize THE 2018 resolution.
"Less consuming, more creating. Doesn't matter what it is, doesn't matter if it's bad."
piu piu SH. Horizontal shooter game written in Bash! Yes, Bash! Impressive indeed.

This Week I Learned - 2017 Week 22

Last week's post or the whole series.

Slow week. Lots of pending stuff needs to be cleared off.

One of the best thing in learning, always be the most stupid person in a class and ask a lot of questions. You never know you will learn something new or relearn something old. Or it may reaffirm some beliefs you always held truly to yourself.

#1 If you need to display CJK characters and emojis in MySQL, use 'utf8mb4_unicode_520_ci'. Historically (another good write-up), MySQL support 'utf8mb3', later changed to 'utfmb4', and changed again to 'utf8mb4_unicode_520_ci' (since MySQL 5.7), and lastly 'utf8mb4_0900_ai_ci' (since MySQL 8.0). Sometimes you wonder why you still stuck with the curse of MySQL where there are better solution out there.

`utf8mb4` is the superset of `utf8`. Meaning that the former also includes the later character set.

To show the supported character set in your MySQL installation.

First check your MySQL version.
mysql> SHOW VARIABLES LIKE "%version%";
+-------------------------+-------------------------+
| Variable_name           | Value                   |
+-------------------------+-------------------------+
| innodb_version          | 5.7.18                  |
| protocol_version        | 10                      |
| slave_type_conversions  |                         |
| tls_version             | TLSv1,TLSv1.1           |
| version                 | 5.7.18-0ubuntu0.17.04.1 |
| version_comment         | (Ubuntu)                |
| version_compile_machine | x86_64                  |
| version_compile_os      | Linux                   |
+-------------------------+-------------------------+
8 rows in set (0.33 sec)

Next list our the available character sets for `utf8mb4`.
mysql> SHOW COLLATION WHERE Charset = 'utf8mb4';
+------------------------+---------+-----+---------+----------+---------+
| Collation              | Charset | Id  | Default | Compiled | Sortlen |
+------------------------+---------+-----+---------+----------+---------+
| utf8mb4_general_ci     | utf8mb4 |  45 | Yes     | Yes      |       1 |
| utf8mb4_bin            | utf8mb4 |  46 |         | Yes      |       1 |
| utf8mb4_unicode_ci     | utf8mb4 | 224 |         | Yes      |       8 |
| utf8mb4_icelandic_ci   | utf8mb4 | 225 |         | Yes      |       8 |
| utf8mb4_latvian_ci     | utf8mb4 | 226 |         | Yes      |       8 |
| utf8mb4_romanian_ci    | utf8mb4 | 227 |         | Yes      |       8 |
| utf8mb4_slovenian_ci   | utf8mb4 | 228 |         | Yes      |       8 |
| utf8mb4_polish_ci      | utf8mb4 | 229 |         | Yes      |       8 |
| utf8mb4_estonian_ci    | utf8mb4 | 230 |         | Yes      |       8 |
| utf8mb4_spanish_ci     | utf8mb4 | 231 |         | Yes      |       8 |
| utf8mb4_swedish_ci     | utf8mb4 | 232 |         | Yes      |       8 |
| utf8mb4_turkish_ci     | utf8mb4 | 233 |         | Yes      |       8 |
| utf8mb4_czech_ci       | utf8mb4 | 234 |         | Yes      |       8 |
| utf8mb4_danish_ci      | utf8mb4 | 235 |         | Yes      |       8 |
| utf8mb4_lithuanian_ci  | utf8mb4 | 236 |         | Yes      |       8 |
| utf8mb4_slovak_ci      | utf8mb4 | 237 |         | Yes      |       8 |
| utf8mb4_spanish2_ci    | utf8mb4 | 238 |         | Yes      |       8 |
| utf8mb4_roman_ci       | utf8mb4 | 239 |         | Yes      |       8 |
| utf8mb4_persian_ci     | utf8mb4 | 240 |         | Yes      |       8 |
| utf8mb4_esperanto_ci   | utf8mb4 | 241 |         | Yes      |       8 |
| utf8mb4_hungarian_ci   | utf8mb4 | 242 |         | Yes      |       8 |
| utf8mb4_sinhala_ci     | utf8mb4 | 243 |         | Yes      |       8 |
| utf8mb4_german2_ci     | utf8mb4 | 244 |         | Yes      |       8 |
| utf8mb4_croatian_ci    | utf8mb4 | 245 |         | Yes      |       8 |
| utf8mb4_unicode_520_ci | utf8mb4 | 246 |         | Yes      |       8 |
| utf8mb4_vietnamese_ci  | utf8mb4 | 247 |         | Yes      |       8 |
+------------------------+---------+-----+---------+----------+---------+
26 rows in set (0.00 sec)

However conversion from `utf8` to `utf8mb4` will encounter these issue of `ERROR 1709 (HY000): Index column size too large. The maximum column size is 767 bytes`. There is a DAMN good guide (ironically better than official documentation) written to resolve that issue and planning for the migration. However, the SQL queries only works for MySQL 5.6.

Meanwhile, this is one good example of writing Perl unit test cases to evaluate `utf8mb4` character set.

#2 Found the solution to my long unanswered question. When writing Bash script, how do we skip certain parameter. For example, we want to skip first parameter but takes the second $2 parameter onward? `shift` my friend. Is a universal concept that I've looked at it almost every day in Perl but never occur to me that this is the same in Bash script as well.

#3 PostgreSQL 10 with examples. The only worthy stuff produced by HP besides the printer. People, embrace the most advanced FOSS database system. Stop playing with MySQL.

#4 Time your console commands but cannot capture the output of `time` to a file? You're not using the command correctly. You should surround it with parenthesis. Either one will work. Depends on how many output files you want to have.
$ (time ls) > outfile 2>&1
$ (time ls) > ls_results 2> time_results

#5 How do you lost 1kg per week or the perfect diet? Both gave the same good advice. It actually make sense without actually starving yourself. The key take here is to first figure out the daily needed calories. Then reduce 500 calories from your food intake and add activities that burn 500 calories as well. The second part make sense. You just don't focus on diet, you should incorporate physical activities as well.

#6 "How do I get started with artificial intelligence?" and "What can I do with AI in my own product or company?" Read this.

#7 The technology stack of craiglist. Surprising, they are using Sphinx Search Server. Thought they might be using Elasticsearch. Why? Based on the comparison on both search servers, Elasticsearch scores way higher than Shpinx. Maybe Sphinx fits the requirement usages in craiglist?

This Week I Learned - 2016 Week 27

Looking back to the last week post or you might want to check out the whole series.

What is Modern Perl? (via HN). The transition of Perl development motto from TMTOWTDI (pronounced Tim Toady) to TIMTOWTDIBSCINABTE (Tim Toady Bicarbonate). It took them (the Perl community) quite a long time to realize that consistency is not a bad things. Some approaches are really preferable but at least there should be a consistency way of coding, hence coding convention should and would exists in a project or a team.

REPL for Perl? Well, you will have to install Devel::REPL with some customization and workaround. To get started, just install these modules and run these commands.
# Prevent Data::Dump::Streamer failed to install error.
# See https://rt.cpan.org/Public/Bug/Display.html?id=100488
$ cpanm PadWalker 

# Enable command history
# See http://stackoverflow.com/a/28818133
$ cpanm Term::Readline::Gnu 

# Install the REPL
$ cpanm Devel::REPL

# Run the REPL
$ re.pl

# Find the Perl version through '$]' and '$^V' Perl's variables.
$ $]
5.024000

$ $^V
v5.24.0

# Quit from the REPL.
$ exit

How X11 works. Wonderful guide on underlying structure of X Windows System.

Free Perl programming books. Didn't realize there are still a lot of unknown, although quite old, Perl books.

Do you suffer from compulsive hoarding? Then r/datahoarder is for you. Be warned, you will burn some extra money to collect these data.

Probably the minimum Vim settings (compare to this and this) to have a good usage experiences, especially when administrating a server through the console.

How to write defensive Bash script. There is always one rule that I never follow, whic is to explicitly make all variables 'local'. Not a good habit if you want to be proficient at console.

It's the future. Be aware of over-engineered and complex solutions for your application before you decide to adopt it to your technology stack.

Awesome list of Perl resources. While we're at it, should you still use Perl?

It has been a while since I last follow with any CSS standard. For me, the existence of CSS frameworks means that the CSS is a solved issue for web development. One thing I noticed that the CSS box-model have changed these days till we have International box-sizing awareness day which the default 'box-sizing' should be 'border-box'. It seems that the Internet Explorer was rights about the default box calculation.

This Week I Learned - 2016 Week 15

Last week post or the whole series.

Stop using numbering system to make the post feels more like an essay. It seemed to change my writing style as well. Variations is good for writing and it took me 14 weeks to make necessary changes. The next step is to build a writing ritual to write at any time and any where.

The recent released of native Bash (HN thread) through Ubuntu in Windows (think of this as the reverse of Wine, mapping Linux system calls to Win APIs) though Windows Subsystem for Linux (WSL) caught everyone by surprise or not (some see this as the return of Embrace, extend, and extinguish by Microsoft). Historically, Apple and Oracle have done this (different approach) and continue to do so and I was surprised it took Microsoft this long. Compare to last time, this strategy basically a response to recapture the developer base rather than the user base.

And also, it proved that calling GNU/Linux now make more sense now than ever and and how significant importance of GNU system. Now that Microsoft and Canonical slapped the GNU system (user space) on top of Windows kernel, should we address this combination as GNU/kWindows (HN thread) instead, similar to GNU/kFreeBSD? Note the 'k' stands for kernel. How will this affect other GNU/Linux distros and FSF itself? Red Hat responded by letting developers download its distro for free. However, there is always the issue of lock-on when free software depends on nonfree. Since the embrace phase have started, the extend phase will follow suite. Is history going to repeat itself again?This shall be seen. Interesting days ahead in the technology scene in coming years.

Biggest regret as a programmer? Andrew Wulf's reflection did raise some interesting and thoughtful discussion in HN, Reddit, and Lobsters. In the end of the day, don't compare, do what you love but take care of your finance.

Hitler uses Docker. The parody-subtitled video making fun of the Docker, the overhyped container technology. If you don't understand the context of the video, read the annotated notes to the video. This reminds me of similar satire of MongoDB is web scale during the hype of NoSQL fad few years back.

Picking up Perl? While Peter Norvig said it's not possible to learn any programming languages in 21 days, but you can learn enough Perl in 2 hours and 30 minutes or in Y minutes instead. For starter, two things you will need to understand which are using strict and warnings as well as why use 'my'.

This Week I Learned - 2016 Week 14

Last week post or the whole series.

#1 Replace Git Bash with MinTTY. Even though you can run Bash on Ubuntu in Windows right now, the most acceptable way (without using the dreadful Windows Command line) before this is through Cygwin and MinTTY. Don't like MinTTY? Well you've Babun and MSYS2, both are based on Cygwin. But still, nothing beat a Vagrant emulated environment.

#2 12 years, 12 lessons working at ThoughtWorks. (HN thread, Reddit thread) Some beg to differ. His retrospective team approach, especially the four key questions, should be applied by any software team. Note that ThoughtWorks is both a software house and a consulting firm.

#3 BPF Compiler Collection. Efficient Linux kernel tracing and performance analysis. You can read the docs and try it out. Only for Linux kernel 4.1 and above though. Compliment to the Brendan Gregg's Linux performance material but at different approach.

#4 Brett Victor's bookshelf. Some people are just prolific book reader. I always love his idea of reactive documents, an implementation of his concept of Explorable Explanations.

#5 Startups in MontrĂ©al. E14N is the only one I'm aware of. Anyway, the discussion at HN is far more interesting regarding the place. Language racism is true and alive there, culturally and systematically forced upon you.

#6 Effective code review or faults finding and blames? Why do you need code review in the first place if trivial matter such as coding convention still cannot be properly enforced? Note that there are tools exists to fix most of these issues and is a no-brainer to rectify this (is just a command away). Root cause is still there is lack of healthy culture that values quality but instead more towards faster delivery.  Or maybe because the software industry itself does not promote integrity (Lobsters thread)?  Or maybe we applied the wrong approach?

#7 perlootut - Object-Oriented Programming in Perl Tutorial. Holy Macaroni! I've never realized that Perl's built-in Object-Oriented feature is so limited. In other words, object in Perl is a glorified hashes. Yes, you have to write your own classes from scratch!

#8 How to start gnome-terminal in fullscreen. Nobody bother to add or enable this feature as sensible default and you have to resort to multiple ways to get it to work. While I can understand of reducing the UI clutters (or dumbing down)in GNOME, but nobody actually use the gnome-terminal in fullscreen mode? It seems that GKH also have issue with gnome-terminal itself.

Case-insensitive Tab Completion in Bash

In Ubuntu, there are several default folders created when you added a new user, listing as shown. All this while, whenever I need to access one of these folders, I always need to capitalize the first character to trigger the tab completion.
$ ls
Desktop/  Documents/  Downloads/  Music/  Pictures/  Public/  Templates/  Videos/  examples.desktop

That day, while setting up another Ubuntu installation, it dawned on me to check whether case-insensitive works for tab completion in Bash? As usual, Google and AskUbuntu did not fail me. Yes, we can do that in Bash. So many years of using Ubuntu and it never occurs for me to raise such question?

To enable case-insensitive, just set completion-ignore-case for the Readline init file, .inputrc.
$ if [ ! -a ~/.inputrc ]; then echo "\$include /etc/inputrc" > ~/.inputrc; fi
$ echo "set completion-ignore-case On" >> ~/.inputrc

While we're looking into this, there are also another option, completion-map-case, which treats hypens ('-') and underscores ('_') as equivalent.

The final .inputrc file should look like below.
$include /etc/inputrc
set completion-ignore-case on
set mark-symlinked-directories on

Expanding RPM's Build-in Macro Values.

While looking into Drupal directory structures in Fedora, I stumbled upon this file, /usr/lib/rpm/macros.d/macros.drupal7, which seems to be the configuration file for RPM command.
$ cat /usr/lib/rpm/macros.d/macros.drupal7 
%drupal7            %{_datadir}/drupal7
%drupal7_modules    %{drupal7}/modules
%drupal7_themes     %{drupal7}/themes
%drupal7_libraries  %{_sysconfdir}/drupal7/all/libraries

# No-op macro to allow spec compatibility with RPM < 4.9 (no fileattrs)
%drupal7_find_provides_and_requires %{nil}

Let's try to expand the values of the above built-in RPM's macros.
$ cat /usr/lib/rpm/macros.d/macros.drupal7  | awk '{print $1}' | grep ^% | xargs -I % sh -c 'echo -en "%\t"; rpm --eval %' | column -t
%drupal7                             /usr/share/drupal7
%drupal7_modules                     /usr/share/drupal7/modules
%drupal7_themes                      /usr/share/drupal7/themes
%drupal7_libraries                   /etc/drupal7/all/libraries
%drupal7_find_provides_and_requires

Why not create a Bash function for the above command instead? Put this in your $HOME/.bashrc file.
# expanding the value of the rpm's built-in macros.
function rpm_macro() {
    if [[ -z "$1" ]]; then
        echo "No filename supplied"
    else
        cat $1 | awk '{print $1}' | grep ^% |\
        xargs -I % sh -c 'echo -en "%\t"; rpm --eval %' | column -t  
    fi  
}

Try out our newly created bash function.
$ rpm_macro macros.drupal7 
%drupal7                             /usr/share/drupal7
%drupal7_modules                     /usr/share/drupal7/modules
%drupal7_themes                      /usr/share/drupal7/themes
%drupal7_libraries                   /etc/drupal7/all/libraries
%drupal7_find_provides_and_requires

Note to self - 2014-06-08

One of the benefit of writing a journal is that you'll have the opportunity to reflect back and monitor your progress of your life. Especially how you got where you are right now.

Due to nature of my workplace, I'm slowly getting into the habit of weekly review and keep track of my time usage, something that may be frowned upon by others, but I believe is the right approach if you want to be very aware of you time usage.

For the past two days, I've switched my working style. No more coding with a lappy while lying down. Just sat at my table with dual-monitors and code and write. The noisy workstation still bother me but I'm slowly getting used to it. Productivity seems to increase and you seemed to accomplish something. Whether this is the right thing? Not sure.

While I'm a firm believer in computer user freedom and even set up a monthly donation to them, but what's my Free Software Activities? Sadly, none. However, what if I allocate two to three hours per week for such activities. Let's see how this goes.

More on Bash scripting. I've learned and relearned quite a few things today while writing a small script for setting up my development machine.

More on Bash Scripting

1. For anything that don't needs to be escaped a lot, sudo sh -c is good enough.

2. I've not even to begin to explore the power of sed, the stream editor. Example here is to insert a line to the beginning of a file. More reading on the one-line scripts for Sed in coming months.
$ sudo ls -l /root | sudo tee /root/ls.txt > dev/null
$ sudo sh -c 'ls -l > /root/ls.txt'

3. "Syntax error: "(" unexpected". That's the error you got when you tried to run a Bash script through Dash without the shebang line, which is the default system shell for Ubuntu or Debian.

4. It will take a while to appreciate the concise syntax of Bash. Let's look at these two examples [9] where we restart a daemon upon a missing file.

Ex1: Typical version
if [ ! -f /tmp/daemon.pid ]; then
python daemon.py restart
fi

Ex2: Concise version
[ -f /tmp/daemon.pid ] || python daemon.my restart

The [] constructor check the existence of the file and return *0 on success* and any number than 0 as failure. Yup, zero as success status compare to other popular programming language. Reason is you can return many different codes as error status but only one code to represent success.

Meanwhile, the || construct will execute the command on the right only if the command on the left fail.

Note to self - 2014-06-07

Most chassis and CPU fans use either a three pins or four pins connector. Not for HP Proliant server, different and proprietary. Their fan connector has six bloody pins. To make matter worse, these fans, sounds like a bloody vacuum cleaner. Imagine sitting next to that machine with a dbA of 79! No, you can't unplug the fan, Power-on self-test (POST) will fail!

Next time be extra careful when you're looking for a battle station. Remember to check the motherboard carefully, especially the noise and parts. Most server board contains proprietary pins and it's not easy and cheap to source these replacement parts.

If I can't still find another way to resolve this noise issue, I'll need to start another fund to get another Xeon-based quad-core workstation. Why Xeon and not i7/i5 ? Well, is not a gamer gig and furthermore, is nice to have make -j N.
                                     
Always find a way to speed up your daily routine. Pick anything around you that can be automated. As someone who spent roughly 50/50 of his time between the console and a browser, it never occurs to me that you can google from command line.

Lots of Bash coding today. Not sure why, I'm kind of have a certain liking to Bash script. Today I learned about shell bultin, passing all arguments to a function using "[email protected]", and ``backtick and $(cmd) for command substitution. The latter is preferred because you can have nested commands.