Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

This Week I Learned 2018 - Week 25

Week 24 post and something from archive.

中国最后的剑圣, 于承惠。在中文影坛里,在也找不到另一个演员能诠释演出这个角色,从霸气的恶人角色到闭山修行的一代宗师。可惜,晚年在影坛没参与任何武侠片。

How to Survive Your 40s? (via KH). As someone who going to take a leap into this new decade, I can probably relates (the screenshot below tells quite a lot as well) to the author experience. Since few years back, younger people have started to call me "uncle" (my choice of clothing did contribute to that as well). It's a sudden but natural shift that comes with your age. The article reminded me of a Korean movie (can't remember the name) I've watched few weeks back. Basically the protagonist (someone in his 50s) said you need to see this milestone as the second 20s. The second time for you to reflect or follow up with what you've done (differently this time) in your 20s. The to-do list since so many years ago is still so long and it will keep me occupied for so some times.



What the difference between Perl and Python? If you need a comparison between both programming languages, this book, "Scripting with Objects: A Comparative Presentation of Object-Oriented Scripting with Perl and Python", while quite dated (it was written in 2008), provides some insights on differences between these both programming languages. In the end, the rising popularity of Python and emergence of Perl 6 shown that, opinionated or there should be a standard way of doing things won.

Why you need to set default value in `sub` in Moo or Moose? Because having a subroutine wrapper returns a unique reference every time you create a new object.

How do you boot from USB thumb drive from Grub itself? Yes, this is possible (do read the whole discussion). You must go to the Grub console by pressing 'c'. Remember that you can tab to find out which removable media and partition to use. It's quite annoying that sometimes the BIOS cannot detect the removable media (thumb drive) and can't boot from the device itself.
grub> ls
grub> set root=(hd1, msdos2)
grub> chainloader +1
grub> boot

On a related note, migration from Fedora 27 to Fedora 28 was such a painful experience. The keyboard and mouse did not work and were very lagging. I'm not sure, but Fedora 28 was such a let down. In the end, have to wipe out the whole installation and replaced it with Ubuntu 18.04 and everything works as intended. Seriously, Fedora, what is going there with 28 release?

Why they said Perl is a more advanced scripting language for system administrator? See App::GitHubPullRequest, a Perl console tool that glues together three different console tools of git, stty, and curl.

How to train your kids to do house chores voluntarily? (via HNEmpowerment since toddler.

Dreadful tasks? Just try, give it a while.

Which Perl modules to use when making HTTP requests? There are so many.

How does you do dispatch table in Perl? Found an old discussion (2010) in HN. The book High-Order Perl have a whole chapter (PDF) on this topic.

This Week I Learned 2018 - Week 16

Last week post or something from the archive.

看了一套墨西哥的戏,《行走距离》。电影讲述一个三百公斤的胖子,寄托于摄影以解生活的苦闷。间接中,也结交了几位好友,让他生命有所改善。没有惯性的情节,简单不过及现实的故事。健康是宝,平淡是福。

Another week of learning new English vocabulary. One issue with making sentences using these words was that the sentences was either too forceful or too literally (直译的) translated.
Sandy alleviates (缓解) the excruciating pain on her lower body part by elevating both legs into a bucket with warm water. By doing so, they hope it can obliterate (忘却) the pain she was going through. Both legs were swelling due to her falling into liquescent (易液化的) soil and stabbed by tree trunk while trekking in the wood. The swelling have worsen and everyone worry that the muscle may liquefy (使液化) from within. 

Stuck with an interesting MySQL problem, which was quite easy but I can't seem to get it right after several tries. Basically we want to update certain column in t1 to the maximum value from a group of similar records. The SQL query was quite straightforward, just join with the second table, t2 with maximum record with the first table, t1.
UPDATE t_transaction tu
JOIN (
    SELECT code, MAX(flag) AS flag
    FROM t_transaction
    GROUP BY code
) t
ON tu.code = t.code
SET tu.flag = t.flag

Modern backend developer in 2018? (via HN) A checklist and decision making tree are always a good choice to make good decision making when picking up new technology stack.

Perl constant array. Sometimes the simplest and easiest way is the only way instead of going through and abusing quirky Perl syntax.
use constant FRUITS => ['apple', 'orange', 'banana'];

my $fruit = 'apple';
if (!grep $fruit eq $_, @{+FRUITS}) {
    print "Is not a fruit";
} else {
    print "Is a fruit";
}

use constant FRUITS => qw(apple orange banana); 
use constant FRUITS => (apple orange banana);  
my $fruit = 'apple';
if (!grep $fruit eq $_, FRUITS) {
    print "Is not a fruit";
} else {
    print "Is a fruit";
}

Modern PHP without a framework. (via HN) Nah, still hate it after all these years. Maybe I'm just prefer a more prettier programming language.

回顾自己的工作生涯,真的必须自我检讨未来的工作方向,看看那些是自己重视的
所以,你得先釐清有哪些條件是你重視的,像薪水、公司品牌、職位、工作內容等,一一把它寫出來。列出條件後,再排定先後順序,之後,和你手邊的 offer(拿到手的工作)比較,做成表格。縱軸是你重視的條件,並依照先後順序,由上而下排列;橫軸則是你拿到的 offer。當你完成表格後,該選擇的工作就會自動「跳出來」。
當我們在思考應該向哪家公司投遞履歷時,其實已經在篩選公司了,但如果是以「錢多事少離家近」為標準,長遠而言卻未必是最適合自己的公司。所謂「知己知彼,百戰百勝」,此時應該要先釐清自己的需求,包括期望的上下班時間、喜歡的工作環境、想做內勤或外務、理想的管理方式、工作如何分配、會議如何進行、同事如何相處、以及有哪些學習機會等。再依這些喜好排出先後順序,然後評估要應徵哪些公司。
工作上要留下那个”人才“? 当然是人品态度优先,能力可以慢慢去培养。

What are some tech companies that do not use open floor plan? Interesting question for those who are looking to justify cubicles. The office at Fog Creek was in between, open space for collaboration and private office for working.

Data class in Python 3.7. (via HN) What took them so long? Syntactic sugar or not, this should have been in any fundamental data type for any programming that supports OOP.

Vipassana for Hackers. (via HN) A book on how a tech guy perceived and practiced Vipassana meditation without involving religion, just the practice. Contrary to typical meditation book, lots of pictures explaining the core concepts.

Moving for good. (via HN) Good write-up when you're moving to a new place. Key quote here (emphasis added),
Ask lots of questions. Ask them to explain things, and show you how it’s done. When they state a fact, ask how they know. When they state an opinion, ask for examples.
How to be a systems thinker. (via HN) I still can't get my head to understand this and the list of reading materials is quite long. Worth spend time on this? We shall see then.

This Week I Learned - 2017 Week 44

Last week post or the past stuff.

SMS bulk notification is a sunset industry right now in MY. I doubt people are still using SMS for communication except to receive notification from the banks, telcos, or something else. Companies or these value-added service providers are struggling to sustain their business and finding ways to pivot their core business to something else. However, I still fail to see how they going to evolve from there.

I've been travelling to a certain part of KL these days and every time I'm around that area, these service providers will spam me with SMS message relating to the services in that area. Suspected my telco provider have been sharing my preferences and location to these third party value-added service providers. Not to mention recent mobile data leaked.

Diet change seems to work somehow. Will continue with it and keep monitoring the progress.

Recent events, unfortunately, reinforced my prejudice regarding certain beliefs. The research summary shown below (emphasis added) best described it. Lesson learned. Move on.
"People are quick to change their moral values depending on which rule means more cash for them instead of others, a study shows. The researchers conclude that the "Pursuit of self-interest is tempered by the constraints of coordination. People seek not only to benefit themselves but also to persuade other people that they are morally right in doing so."
The "Rules' of Writing. (via HN) Definitely agree with the minimize the usage of adjectives and adverbs or better, kill it. Overuse is a sign of lazy writing, for examples:
"The man moved slowly."

Where you can rewrite using more vivid and stronger verbs.
"The man sneaked."
"The man strolled."
"The man limped."

Mastery is the side effect of deliberate and constant practices. You may or may not achieve it but nevertheless, if you enjoy doing it, then just continue as it. You can always start with technical writing.

What Python stack should you use? AWS Lambda, Python, AWS Cognito, Postgres and ReactJS. Interesting conclusion that goes against the bare-bone frameworks like Flask or Bottle.

Nature is both brutal and weird. What will happen when these eggs grow into mature adult bettta fish? (emphasis added by me)
"During and after spawning, the male uses his mouth to retrieve sinking eggs and deposit them in the bubble nest (during mating the female sometimes assists her partner, but more often she simply devours all the eggs she manages to catch). Once the female has released all of her eggs, she is chased away from the male's territory, as she will likely eat the eggs. If she is not removed from the tank then she will most likely be killed by the male."

This Week I Learned - 2017 Week 18

Last week post or you want to read through the whole series.

Since the prevalent of Internet these days, we're more connected that ever. And yet, we're even more isolated. More and more shopping malls and residential areas are being built especially around the transit stations. Basically all looks the same and people are discourage to travel to another town anymore because everything is available either online or at your nearest malls.

Sick. Down with fever. It has been a while since last time. No one but myself to blame. For people around my age, good rest and food is a way to healthy life. We're not seeking prolonged life but prolonged health.

#1 So, what's your Plan B? Good discussion from the comments. While there is some sort of discrimination against older programmers, but the reality is older programmers may just burn out or bore doing the same mind-numbling stuff over and over again. Yes, the same sh*t but different day. Not everyone have the opportunity to work meaningful projects (yes, subjective view), but most of us just work on some glorified spreadsheet or database skin. You see the same mistakes repeated over and over again from different systems and it's just pointless or bother to raise it up again.

#2 On becoming a low-level programmer. There is a curated list of information especially on the hardware part.. There is also another list of learning the fundamental of computer science. So many things to learn and explore and yet, so little time. Sometimes the reality of life commitments limit our choice on the thing we want to pursuit. I need to finish everything by July before I can start to clear the list from my bucket list. One step at a time. Follow the system.

#3 First, make it work, then make it right, and finally, make it fast. Get the thing to work first before you're thinking about making it right (giving the right name) and optimize it. Fulfill the requirements of the stakeholders first (prototyping) before you're looking into doing it right and fast. Something I need to remind myself constantly as we're often obsess with perfection. What's the point if your build it the right way and fast but does not meet the stakeholder needs?

#4 How to write a REST client in Perl. Most Perl modules have limited documentation, for example, little example on how to use the APIs. This is where PHP (it has been so long since I mentioned something about PHP) really shines. As the language is slowing dying, there are little or no good example on how to start something up fast.

Meanwhile, something rather interesting about Perl. I've seen quite a few Perl codes with subroutines that prepended with ampersand (&) operator. Basically, the ampersand is needed if you're calling a subroutine before its declaration. Off course, there are many other reasons and usages to use the ampersand, but commonly as reference to a subroutine.
&hello;

sub hello {
    print "hello";
}

When using OO with Perl, you will need to instantiate the constructor or method invocation through the arrow (->) operator (a reference). Good examples given below.

Using the arrow notation or virtual method.
my $apple = Fruit->new("apple");
$apple->get_price('usd');

Using the double colons notation or static method.
my $apple = Fruit::new("apple");
Fruit::get_price($apple, "usd");

However, to invoke the method either statically or virtually, you'll need to write the constructor as follow.
sub new {
    my $self  = shift;
    my $class = ref($self) || $self;
    return bless {}, $class;
}

#5 "error: src refspec master does not match any." Silly me. You need to commit something locally before pushing to remote Git server.

#6 Tiny Python Notebook. Definitely not tiny but quite comprehensive guide to Python 3.6.

#7 How to unstage all staged files in Git? It's so easy that the best answer is not upvoted enough.
$ git reset

#8 Postgres Weekly. When come to FOSS database choices for web application for Intranet application, I'm strong opinion that PostgreSQL should be the default choice. Feature like Range Types and others solves quite a lot of issues and save lots of developer-hours. Yes, MySQL has its own usage, but for any business application which involves monetary and strong ACID compliance, it's not a right choice.

This Week I Learned - 2017 Week 12

Last week post or you can read the whole series.

9 hours of writing and only 3 pages done.

#1 The important of morning rituals in developing a system (install a habit) to achieve your life goals. Is good to be morning person as you can achieve quite a lot when your mind is fresh. There are these 7 morning rituals which I will slowly introduce to my daily life. These are:
  1. Plan a day before. I've been slowly adopting this habit by going through the list of items in my bag and write down what I need to do tomorrow. Not consistent though as for some day I failed to plan for anything.
  2. Wake up early. Either from 4:30 AM till 6:30 AM. This is doable provided that you sleep quite early on the previous day. According to this sleep calculator, these are the corresponding time you should sleep if you want to wake up at a particular hours.
    1. 04:30 AM - 7:30 PM or 9:00 PM or 10:30 PM or 12:00 AM
    2. 05:30 AM - 8:30 PM or 10:00 PM or 11:30 PM or 1:00 AM
    3. 06:30 AM - 9:30 PM or 11:00 PM or 12:30 AM or 2:00 AM
  3. Start the day with exercise.
  4. Do your highest priority projects. Study in the morning and reviewing again in the evening before bedtime. This is good for learning as you can recall and rethink what you've learned in the morning. Also? Before even you start your journey to work, you've already accomplished so much for the day.
  5. Work on your side projects. Definitely need to try this. Do not work on your side projects late at night. Do it in the morning.
  6. Have a uninterrupted conversation with your partner. Do we actually having any serious and uninterrupted conversation with our partner, even for just two minutes? Doubt so.
  7. Meditate. I've been doing this since the moment I woke up in the morning. Although for just a mere 10 seconds, slowly but surely I can really concentrate on it.
#2 Beautiful SICP. There is another Python version as well. The discussion is always revolves whether it's good or not. I haven't yet finish the damn book since I bought it 20 years ago. Is this year the year for me to finish it? Doubt so. Even though I placed the book next to my work place and look at it on daily basis. Also, SICP is also the recommended book (debatable) on the subject on programming in Teach Yourself CS. (via HN) While we on that, someone should start another list on Teach Yourself SE. My personal experiences is that Software Engineering knowledge far outweight the Computer Science knowledge.

Since we can all access good quality CS education through the Internet, does local CS degree still relevant today? Or does this means that universities or colleges have been rendered to just another platform for face-to-face socializing instead of gaining knowledge?

#3 Almost 15 years of Design by Contract. I can relate to the author experience and still is a firm believer of Design by Contract (DbC). Bertrand Meyer still have a huge influence on me. Like most things in life, moderation is the key. Use DbC only when necessary.

#4 Real-time LED strip visualization (see Gif animation below) using ESP8266 or Raspberry Pi. Not only the project was interesting, the detail documentation by the developer was impressive. Definitely one of the most solid Github project I've encountered.


#5 Alone in the Wilderness. (via Reddit) A documentary (search YouTube for the video) about Richard Proenneke, who upon his retirement at age of 51 years, build an ultimate man cave to live a life of solitude in Aslaka. Carpentry is such a underestimate skill a man could have.

#6 Interesting Git graph generated by Bit-Booster, the offline commit graph drawing tool from one of my personal Git repository. I'm not sure how to interpret the relationship of my commits, seems somehow broken?


This Week I Learned - 2016 Week 42

Last week post or the whole series.

Interesting week indeed. It has been a while since I last encountered so many different type of personalities who want or don't want to be a developer.

As usual, what have I learned this week? The usual stuff.

If you're running on GNU/Linux and want a way to manage different Windows OS through Vagrant, you can try this Vagrantfile. Installation and setup is pretty much straightforward, just make sure the Vagrantfile is downloaded. Unfortunately, the login still fail to work.
$ sudo install virtualbox vagrant
$ vagrant plugin install winrm winrm-fs

$ mkdir vagrant_win
$ cd vagrant_win
$ wget https://goo.gl/ATpaEY
$ IE=Win7IE8 vagrant up

Sanic, Python 3.5+ asynchronous web server. The discussion at HN seems rather interesting. While this is nothing new, asynchronous database layer like asyncpg seems rather userful to improve your DB query speed.

Issue with Babun's memory conflict after Windows updates? Try rebasing, not that Git rebasing thought. Cygwin still is the better and prefered choice for Unix experience in Windows. Yes, I know there is Bash on Windows.
1) Exit babun.
2) cmd /c %SYSTEMDRIVE%\Users\%USERNAME%\.babun\cygwin\bin\dash.exe -c '/usr/bin/rebaseall -v'

Customozing HTML's file inputs. Probably the most comprensive guide on different techniques to change the default behavour.

Web framework benchmarks. The Round 10 has one of the best humourous write-up.
The project returns with significant restructuring of the toolset and Travis CI integration. Fierce battles raged between the Compiled Empire and the Dynamic Rebellion and many requests died to bring us this data. Yes, there is some comic relief, but do not fear—the only jar-jars here are Java.
What happens when you rename a branch in Git? Plenty of things. First, you rename it locally. Next, you rename it remotely (is the same as remove the old branch and add a new branch). After that, either you update your upstream URL or checkout a fresh copy of the said new branch. Lastly, you may needs to batch update your commit messages.
$ git branch -m new_name
$ git branch -m old_name new_name
$ git push origin :old_name
$ git push --set-upstream origin new_name
$ git filter-branch -f --msg-filter 'sed "s/foo/bar/"' master..HEAD

The database schema for StackOverflow is publically accessible. I was surprised that it's such a straight forward design and nothing fancy at all. Well, is just a CRUD app with some additional tweaks here and there. However, the ranking formulae is far more interesting when compare to different algorithms used by other popular forum-like sites.

So may ways to iterate through the Perl's array. Implementation 1, 4, and 5 is what I normally used but the 5th method is still my favourite.

Source code syntax higlighter through Javascript? Just found out today, besides highlight.js, there is also Prism.js. The former seems to have more languages support but the later is used for quite a few popular projects.

NBA season going to start soon, maybe is time for me to learn some Statistics through certain API? Can't wait what surprises the 2016/2017 season will give us.

This Week I Learned - 2016 Week 24

Last week write-up or you might want to read the whole series.

Typosquatting programming language package managers. Mostly likely inspired by the Npm's left-pad incident.

Print debugging on steroid in PHP? Try Kint.

Due to the decommission of CPU processor by big Internet companies, the used market was flooded with Xeon E5-2670. Hence, there are many who tried to build a powerful Dual-Xeon machine. While the processor is roughly around USD 70 and highly soughtable, the motherboard is very costly. It's reasonable that everyone want to assembly the most powerful machine with minimum cost. However, judging by the TDP, it recommended to build smaller and more economical home lab. Again, it still depends on what the main purpose you want to build your own home lab.

tl;dr: Maintenance code: 3008.

Have an Unifi account with the Huawei EC6108V8 media player? Want to make the most of this tiny device? The essential configuration and setup steps are in post #10, #14, #32 (enable you to login to Play Store), #69, #90, #98, #102, #103, #178 (if #32 doesn't work), #190, #208 (similar to #178), #218 (maintenance code), #252, #263, #278 (similar to #178 and #208), #287, #291, #311, #319, #359, #377 (like #178), #387 (device specification), and #414.

This Week I Learned - 2016 Week 08

Last week post.

#1 NameError: name 'basestring' is not defined. Surprisingly there is still conflict with Ansible when installed using pip for Python 2 and Python 3.

#2 GNU/Linux Performance. Poster of tools you can use to investigate performance issues with your system.

#3 Container as Python module. (HN discussion) Interesting concept indeed. I've been looking at Docker for the past three weeks and this is probably best interesting use of container. It's useful when you want to build up an actual test environment from your Python apps or scripts. Instead of Mock object, you can test against the actual system, for example, an actual database system.

#4 Xamarin sold to Microsoft. (HN discussion). What took them so long? I read (can't remember where), it was sold for 400 millions. Interesting to see how this unfold in coming future.

#5 Non Zero Day. (HN discussion) Effective way to build a new habit through chain-method or streak. No, Jerry Seinfield did not create the Seinfield productivity program. For me, almost daily Git commit. You have to get started on something, the baby step..

Experiences On Using Static Code Analysis Tools for Python

Static code analysis, as the name implies, is the analysis of the non-running source code of a program. This can be done manually through code reviews where an experienced developer will inspect and walk through the code to find potential programming mistakes. However, such manual process are time consuming and can be improved through automated static code analysis tool.

In Python programming languages, Pyflakes, PyChecker, and Pylint is the common static code analysis tool. This post discusses the experiences on applying these tool to Subdown, an open-sourced image scraper console tool written in Python.

To evaluate and compare different these three static code analysis tool for the Python programming language, I've pick an open-sourced project called Subdown. This program is a image downloader console for the Reddit, an online news sharing community. The site is organized into multiple SubReddits, a breakdown of the smaller communities grouped by topics or interests. The program consist of a single file Python script which crawls a targeted SubReddits for its external URLs of images and asynchronous download these images. First, download the sample script.
$ wget https://raw.githubusercontent.com/radiosilence/subdown/master/subdown.py

Pyflakes
Pyflakes is a very basic fundamental tools. It only parse the Python source files to check for any errors. However, this tool does not check for any coding style violation. The warning shown below indicates that the Subdown program imports additional unused module named ‘mimetypes’. Loading unnecessary resources will slow down program execution and utilize additional memory resources.
$ pyflakes subdown.py 
subdown.py:11: 'mimetypes' imported but unused

PyChecker
Similar to Pyflakes, PyChecker also parse and check for source files for errors, hence shares similar warning with Pyflakes. Furthermore, Pychecker also import and executing Python modules for additional validation. This result illustrated below shows that the warning indicated that the same Python module, gevent was being imported to the application in two separate ways where it should be done once.
$ pychecker subdown.py
Processing module subdown (subdown.py)...
 ImportError: No module named _winreg
       
 Warnings...
 :28: self is not first method argument
 subdown.py:11: Imported module (mimetypes) not used
 subdown.py:17: Using import and from ... import for (gevent)

However, this is false positive warning. As shown from its code below, the Subdown program import all the methods from the gevent module. However, in second line, it reimport again from gevent module but only the monkey class so it can “monkey patch” the existing behaviours to work around the limitation of the standard socket module. Monkey patching is one of the feature of dynamic typed programming languages where we can extend and modify the existing behaviours of the methods, attributes, or functions during run-time. This technique is used typically to work around the constraints of no able to modify existing libraries.
16 import gevent
17 from gevent import monkey; monkey.patch_socket()

Pylint
Pylint, the next static code analysis tool in our evaluation, is the most comprehensive with lots of additional features. See Appendix A for the details output when ran against Subdown program. Instead of just checking for Python code errors like the previous two tools, it also check the coding style violation and code smells. Code style is validated against Python’s PEP 8 style guide. Meanwhile, code smells is a piece of inefficient code, while may run correctly, still have room for improvement through code refactoring. All these are categorized into five message types as shown below:
  • (C) convention, for programming standard violation
  • (R) refactor, for bad code smell
  • (W) warning, for python specific problems
  • (E) error, for probable bugs in the code
  • (F) fatal, if an error occurred which prevented pylint from doing further processing

Comparing the sample result below with previous two tools, we get similar warning of unused import. However, there is a new warning not found which is ‘Unreachable code’.
W:198,12: Unreachable code (unreachable)
C:200, 0: Missing function docstring (missing-docstring)
W: 11, 0: Unused import mimetypes (unused-import)

Extracting out the portion of code in shown below which corresponds to the warning of ‘Unreachable code’, it shows that Line 198 will not be executed at all due to the raise statement in line 197. Upon raising an exception in Line 197, the program will halt and exit the execution. This is a good example where the static code analysis tool can help to uncover incorrect assumption made by the developer.
194         try:
195             get_subreddit(subreddit, max_count, timeout, page_timeout)
196         except Exception as e:
197             raise
198             puts(colored.red(str(e)))

While writing this post, I've found another tool called Pylama, which is a helper tool that wraps several code linters like PyFlakes, Pylint, and others. However, there is an issue integrating with Pylint. You may give it a try but YMMV.

Switching Between Different Commits in Git

PyVim, an implementation of Vim editor in Python caught my attention while browsing through the HackerNews recently. After trying and installing it through Python's pip installer, I've decided to install the latest version from its Github repository instead.

Before that, let's setup the Python's Virtual Environment.
$ cd /tmp
$ mkdir pyvim
mkdir: created directory ‘pyvim’

$ cd pyvim/
$ virtualenv -p /usr/bin/python2.7 venv
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in venv/bin/python2.7
Also creating executable in venv/bin/python
Installing setuptools, pip...done.

$ source venv/bin/activate
(venv)$ which python
/tmp/pyvim/venv/bin/python

Clone the PyVim Git repository with the folder and the Virtual Environment you've created in previous step.
(venv)$ git clone https://github.com/jonathanslenders/pyvim.git
Cloning into 'pyvim'...
remote: Counting objects: 196, done.
remote: Compressing objects: 100% (51/51), done.
remote: Total 196 (delta 29), reused 0 (delta 0), pack-reused 143
Receiving objects: 100% (196/196), 597.72 KiB | 165.00 KiB/s, done.
Resolving deltas: 100% (93/93), done.
Checking connectivity... done.

You'll obtain below directory structure.
(venv)$ tree -L 2
.
├── pyvim
│   ├── CHANGELOG
│   ├── docs
│   ├── examples
│   ├── LICENSE
│   ├── pyvim
│   ├── README.rst
│   ├── setup.py
│   └── tests
└── venv
    ├── bin
    ├── include
    ├── lib
    └── lib64 -> lib

10 directories, 4 files

Next, install latest PyVim and all the necessary Python packages within the Virtual Environment.
(venv)$ cd pyvim
$ python setup.py install
......
Finished processing dependencies for pyvim==0.0.2

Run the PyVim program and the output below shows that there is breakage with the latest committed version.
(venv)$ pyvim --help
Traceback (most recent call last):
  File "/tmp/pyvim/venv/bin/pyvim", line 9, in 
    load_entry_point('pyvim==0.0.2', 'console_scripts', 'pyvim')()
  File "/tmp/pyvim/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 519, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/tmp/pyvim/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2630, in load_entry_point
    return ep.load()
  File "/tmp/pyvim/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2310, in load
    return self.resolve()
  File "/tmp/pyvim/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2316, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/tmp/pyvim/venv/lib/python2.7/site-packages/pyvim-0.0.2-py2.7.egg/pyvim/entry_points/run_pyvim.py", line 17, in 
  File "/tmp/pyvim/venv/lib/python2.7/site-packages/pyvim-0.0.2-py2.7.egg/pyvim/editor.py", line 27, in 
  File "/tmp/pyvim/venv/lib/python2.7/site-packages/pyvim-0.0.2-py2.7.egg/pyvim/layout.py", line 17, in 
ImportError: No module named reactive

Let's find any tagged stable working versions but it seems that the author does not create any tagged branch.
(venv)$ git tag
(venv)$

Since there is not tagged branch, then we'll need to find out the hash of the last stable commit. In our case here, is version 0.0.2. Results as shown using the git log command with summarized output.
(venv)$ git log --oneline --decorate | cut -c -80
1ec47f1 (HEAD, origin/master, origin/HEAD, master) Command functions rename
d842f06 add docopt to install_requires in setup.py
1fdd937 Override ControlT from prompt-toolkit: don't swap characters before curs
c944a28 Implemented the :cq command.
eaa4b1e Fix: use accepts_force also for bw/bd
4920b74 Added :bd as keybinding to buffer close
f179bd6 Implemented scroll offset.
6c160ce Show 'No \! allowed' when used for commands not supporting it.
b1d9813 Fix python 3/2 compatibility for urllib.
892188c Fixed typo in README.txt
2409ad7 Some rephrasing in the README.
40cfe66 Reload option for :edit and :open
9bb4975 Added ':open' as an alias for ':edit'.
e33db19 Added ':h' alias for ':help'
a010ea8 Implemented ControlD and ControlU key bindings, for scrolling half a pag
39c72b1 Implemented ControlE and ControlY key bindings
ad880c1 Auto closes new/empty buffers when they are hidden. This solves the :q i
082ce60 Added accept_force parameter to commands decorator. 'bp'/'bn' now also a
dfed3a3 Fix a bug where a user could leave a buffer with unsaved changes by issu
1ff1bac fix ctrl-f shortcut
5369f5d Abstraction of I/O backends. Now it is possible to open .gz files and ht
75d3a3b Mention alternatives in README.rst
10dcb2d Added ControlW n/v key bindings for splitting windows.
d13f5e6 Added PageUp/PageDown key bindings.
4009f8b New screenshot for cjk characters.
34b6175 Fixed NameErrors in .pyvimrc example.
cc0b333 Adding shorthands for split and vsplit
78c2225 Pypi release 0.0.2 -- (0.0.1 release failed)
5083d0b Pypy release 0.0.1
df71609 Usable pyvim version. - Layouts: horizontal/vertical splits + tabs. - Ma
fb129f5 Initial ptvim version.
a60a0b6 Initial commit

From the result above, commit hash id 78c2225 is the first public release version. Let's switch our HEAD to that commit.
(venv)$ git checkout 78c2225
Note: checking out '78c2225'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 78c2225... Pypi release 0.0.2 -- (0.0.1 release failed)

Re-install and re-run the program again. It seemed this committed version work without any issues.
(venv)$ python setup.py install
(venv)$ pyvim --help
pyvim: Pure Python Vim clone.
Usage:
    pyvim [-p] [-o] [-O] [-u ] [...]

Options:
    -p           : Open files in tab pages.
    -o           : Split horizontally.
    -O           : Split vertically.
    -u  : Use this .pyvimrc file instead.


To reset back the HEAD back to the origin/master.
(venv) $ git reset --hard origin/master
HEAD is now at 1ec47f1 Command functions rename

Confirm we're at the latest HEAD through git log command.
(venv) $ git log --oneline --decorate -1
1ec47f1 (HEAD, origin/master, origin/HEAD, master) Command functions rename

Instead of searching through the log, we can tag particular commit.
(venv)$ git tag -a v0.0.2 -m "Release 0.0.2" 78c2225

Let's check again throught the git log and git tag command.
(venv) $ git log --oneline --decorate | grep 'HEAD\|tag'
1ec47f1 (HEAD, origin/master, origin/HEAD, master) Command functions rename
78c2225 (tag: v0.0.2) Pypi release 0.0.2 -- (0.0.1 release failed)

$ git tag
v0.0.2

Instead of switching to particular commit hash id, we can switch directly by using tag name.
$ git checkout v0.0.2
Previous HEAD position was 1ec47f1... Command functions rename
HEAD is now at 78c2225... Pypi release 0.0.2 -- (0.0.1 release failed)

$ git status
HEAD detached at v0.0.2
nothing to commit, working directory clean

Static Variable in Python

The solution, as shown below, is actually quite straight forward. Basically by using Python attribute, you can emulate static variable in a function without using a global variable.
def myfunc():
if not hasattr(myfunc, "counter"):
    myfunc.counter = 0  # it doesn't exist yet, so initialize it
    myfunc.counter += 1

Coming from C-based programming languages, PHP in my case, it going to take a while for me to adapt to Python. Yes, I still code Python like a PHP, hence, more practice needed.

Python and Makefiles

"Besides building programs, Make can be used to manage any project where some files must be updated automatically from others whenever the others change."
-- Wikipedia on Make
One of the issue I've noticed while doing a Django project was there are a bunch of shell scripts in the project folder. These tiny shell scripts were mostly related to creating virtualenv, resetting database, and others. Would that be nice if we can combine and group all these scripts into one file?

That is possible and easy through Make and makefiles, provided that you're willing to pick up the rules. The only minor annoyance is those running Windows need to install Make for Windows.

Something I learned while working on my Makefile.

1. A rule tells the make program on how to build your program. The syntax, as shown below, is straight forward and consists of a target, dependencies, and commands.
target: dependencies/target
commands

2. Prepend the build target name with '_' if you don't want the build targets to show up autocompletion in your shell.

3. Build script too noisy or verbose? For example, printing of working directory. To silence it, use make -s or prepend '@' before the commands.

On Django Grappelli

When adding any new framework or library to your development, you'll eventually encounter the 80/20 rule in software development. In which you've finish the 80% of the work but stuck almost infinitely with the remaining 20%.

Several things I learned the hard way about Django Grappelli.

Sequence of loading the INSTALLED_APPS is very important. The Grappelli module must comes before django.contrib.admin module. Failing to do so and the changes to the admin layout will not take effect.
INSTALLED_APPS = (
    'grappelli',
    'django.contrib.admin',
)

If you want to customize the layout and use the default CSS styling, read the documentation on templates. Unfortunately, not googlable and must be access locally though your Django instance at http://localhost:8000/grappelli/grp-doc/. Oh boy, so much time wasted on googling for the tutorial or documentation on customization.

Where is the bloody documentation on nav-global block?

Note to self - 2014-06-05

1. Taobao, slowly but surely, will replace the local ecommerce players as the go to destination for online purchasing. Nothing interesting among them, just another reseller of items procured from merchants in Taobao. The real money or profit still flow through logistic, or to be specific, freight forwarder.

2. Watching items being purchased online in real-time was quite an additive activity. You will be surprised by the types of item purchased online and some of these items are dirt cheap! I'm so going to create a dashboard to load these purchases using my underutilized monitor.

3. What do you call a PHP programmer who are in the midst of switching to another programming language like Python? A better PHP programmer. Ouch.

4. PHP vs. Python. Python is a better when you want to learn more about programming in general or be a better developer. While PHP is great when you want to create something fast for the web. Contrary to popular belief, Python is not a beginner friendly language. By the way, PHP is a web template language. Which one makes you happy? Neither. But is good to finally escape from PHP after all these 10 years. For me, PHP is "cari makan" and Python is to stimulate my bored mind.

5. export PYTHONDONTWRITEBYTECODE=1
Yup. No more pyc files scatter around. Only for Python 2.6 though.

6. Read the bloody change logs. Seriously, when you're upgrading any software or libraries, please remember to read the bloody change logs.

7. Awareness, acceptance, and courage are the keys to become a better estimator. First, you must aware of your current situation. Second, you must accept the current limitation either in your or your surrounding. Third, you must have the courage to make hard decision and mistakes to go ahead.

Autoload Module for Python Shell and IPython

In PHP, print_r is a very useful function to display variable in a human-readable format. Similarly, both pprint and awesome_print provides equivalent function in Python. Although the former has more extra features.

Since most of my Python time is spend on either the Python shell or IPython, it will be nice if we can autoload these two modules upon starting the shell.

Python shell

First we need to set the environment variable PYTHONSTARTUP so it can autoload the file. You should put this into your .bashrc file and reload it.
export PYTHONSTARTUP=$HOME/.pythonstartup

The content of .pythonstartup file, which is just a normal Python script as shown. Besides that, we also enable tab completion as the default shell has limited features.
# two use modules for pretty print variables
from awesome_print import ap
from pprint import pprint

import rlcompleter
import readline

# enable tab completion
readline.parse_and_bind("tab: complete")

To test our autoloading, just start the Python shell and type the sample code which will list all the attributes of the ap function.
$ python
Python 2.7.5+ (default, Feb 27 2014, 19:37:08)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> ap(dir(ap))
[
  [ 0] __call__,
  [ 1] __class__,
  [ 2] __closure__,
  [ 3] __code__,
  [ 4] __defaults__,
  [ 5] __delattr__,
  [ 6] __dict__,
  [ 7] __doc__,
  [ 8] __format__,
  [ 9] __get__,
  [10] __getattribute__,
  [11] __globals__,
  [12] __hash__,
  [13] __init__,
  [14] __module__,
  [15] __name__,
  [16] __new__,
  [17] __reduce__,
  [18] __reduce_ex__,
  [19] __repr__,
  [20] __setattr__,
  [21] __sizeof__,
  [22] __str__,
  [23] __subclasshook__,
  [24] func_closure,
  [25] func_code,
  [26] func_defaults,
  [27] func_dict,
  [28] func_doc,
  [29] func_globals,
  [30] func_name
]

IPython

Again, for IPython, the setup is similar. First, you must set the export IPYTHONDIR environment variable in your bash file. In my Ubuntu 14.10, the default data path was set to $HOME/.config which contains other unnecessary configuration files to be added to my Git’s repository.
export IPYTHONDIR=$HOME/.ipython

Next, we instantiate and create the profile data.
$ ipython profile create
[ProfileCreate] Generating default config file: u'/home/kianmeng/.ipython/profile_default/ipython_config.py'
[ProfileCreate] Generating default config file: u'/home/kianmeng/.ipython/profile_default/ipython_qtconsole_config.py'
[ProfileCreate] Generating default config file: u'/home/kianmeng/.ipython/profile_default/ipython_notebook_config.py'

Following the directory structure shown below, create the startup script file 10-default.py.
$ tree .ipython
.ipython
├── profile_default
│   └── startup
│       └── 10-default.py

Add these to the 10-default.py file.
$ cat .ipython/profile_default/startup/10-default.py
from awesome_print import ap
from pprint import pprint

Start our IPython session and try to test for our autoloaded modules.
$ ipython
Python 2.7.5+ (default, Feb 27 2014, 19:37:08)
Type "copyright", "credits" or "license" for more information.

IPython 0.13.2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: ap??
Type:       function
String Form:
File:       /usr/local/lib/python2.7/dist-packages/awesome_print/awesome_print.py
Definition: ap(*args)
Source:
def ap(*args):
    for arg in args:
        print format(arg)

In [2]:

Which Python Framework?

As I was revisiting Python again, I was wondering which Python web framework should I look into ? Denis Krienbühl shared his experiences on using all four popular Python frameworks and which one to use. To expand a bit of his tl;dr; summary.

a) Flask.
Simple and one-page app like dashboard status.

b) Django.
Quick CRUD app with built-in authentication and administration. Opinionated.

c) Pyramid.
Similar to Django but less opinionated and has the flexibility of mixing different libraries.

d) Plone.
For Content Management System (CMS) only.

Maybe I should look into Flask instead of Bottle and Pyramid instead of Django. My friend, Kamal, recommended that I stick with PeeWee and Bootle.

Detect Nudity in Images

It has been a while since I wrote any Python  script. Found this interesting Python library, nude.py which detects nudity in images. Some googling here and there just to refresh my memory on how to code in Python again.

First, install the library.
$ sudo apt-get install python-imaging
$ sudo pip install nudepy

Then, I wrote a tiny script to scan my Pictures folder for any nudity. The script is dog slow and can be optimized using multithreading. Next time, perhaps. On #1, you'll will to expand the expand the tilde manually.
import os, nude
from nude import Nude
from os import listdir
from os.path import expanduser

#1 expand tilde manually
path = expanduser("~/Pictures/")
files = listdir(path)
for file in files:
    n = Nude(path + file)
    n.parse()
    # ordinal is optional since Python 2.7 [3]
    tpl = "image {} has {} nudity"
    print tpl.format(file, "no" if n.result == False else "")

Result time.  Detection is quite accurate even with baby photos. However, occasionally the library will throw exception for certain black and white photos.

Note to self : Code more in Python.

Extract Hyperlinks Using Python and PHP

Is always a great fun if you can rewrite certain code from one programming language to another. I was looking at this short snippet of Python code by unconscionable which request a page and dump certain comments links.

Sample Python code reproduced here.
import urllib2, re
headers = {'User-agent': 'I promise I\'m not doing this a lot',}
req = urllib2.Request("http://www.reddit.com/r/BuyItForLife/search?q=headphones&restrict_sr=on", None, headers)
website = urllib2.urlopen(req)

html = website.read()

links = re.findall('"((http|ftp)s?://.*?)"', html)
for i in links:
    if 'http://www.reddit.com/r/BuyItForLife/comments/' in i[0]:
        print i[0]

My rewrite using PHP using file_get_contents and stream_context_create, something new for me.
<?php

$options['http']['header'] = "User-agent: I promise I'm not doing this a lot'\r\n";
$context = stream_context_create($options);

$url  = "http://www.reddit.com/r/BuyItForLife/search?q=headphones&restrict_sr=on";
$html = file_get_contents($url, TRUE, $context);
preg_match_all('/"((http|ftp)\s?:\/\/.*?)"/i', $html, $links);
foreach ( $links[1] as $link )
{
    if ( strstr($link, 'http://www.reddit.com/r/BuyItForLife/comments') )
        echo $link, "\n";
}

Comparison of both code snippet.
  1. Regex is simpler and more readable in Python. You don’t need to escape certain character (example is forward slash /) like in PHP. API is simpler and make more sense, result are returned instead of using callback in PHP where you have two sets of array.
  2. file_get_contents() is awesome and dangerous as well for reading both offline and online file. Nothing equivalent is found in Python.
  3. Finding and matching string is way more readable in Python.

Python Malaysia February Meetup 2013

Updates : Malcolm passed away on March 17, 2013.

First meetup this year. The guest speaker is Malcolm Tredinnick, one of the core developers of Django Web framework. He shared his experiences on the framework and how to contribute to the project. These are some notes of the talk.

On contributing to the Django project or how you can help.
  1. Check out the project from github, run the unit test, and submit bug report. According to him, almost 50% of Django codecase are unit testing code.
  2. Alternatively, you can start helping by searching through the bug reports for new and easy pickings tickets. Besides, you can help by verifying new ticket.
  3. Subscribe to the mailing list, read and understand the discussion, become a "silent expert". Also, try to answer questions in the mailing list or stackoverflow.

On learning Python.
  1. Stick to Python 3.3 (latest version), especially you're a beginner.
  2. Use the Six [9] module to make sure Python code run fine in both 2.x and 3.x.

That's all I can remember and jot down.

LPTHW - Day 186 Evaluating Your Game

Reading exercise 44 of LPTHW. Mostly about what type of naming coding convention should be used and advice on improving yourself as programmer. Interesting part from this chapter are:

1) Name your function or methods as command rather than how it does something. Example of command way is list.pop() and how way is `list.remove_from_end_of_list()`. I tend to follow the later way because I always follow the rule of turning comment into code.

2) Agree with using CamelCase for class name and under_score naming convention. Sadly I wish PHP should follow this style rather than more Javaish like style. But this is just a matter of reference.

3) One of the way to determine that you're writing readable code is read it out loud especially during coding. Code readability is one of the next thing I am working on. "The Art of Readable Code" is on my next reading list after I finish all the  #LPtHW  exercise.

4) Still a beginner in Python, is best for me now is find good Python projects to read, study, learn, and mimic the style.

5) Strongly agree that comments should explain the why rather than the how. These days, I wrote business rules and special cases as comments in the code.

6) I have done evaluating exercise 43. Pretty much like the way I code in PHP but not sure I am using Python correctly, suspect there exists a more Pythonic way rewriting the code. More exercises and Python programs should help me to develop this Pythonic sense.