Showing posts with label http. Show all posts
Showing posts with label http. Show all posts

This Week I Learned 2018 - Week 48

Last week post or read the old stuff instead.

What is this console app that always gave me a conflicting experience every time I use it? ImageMagick. Besides the forking of GraphicsMagick, the use of complex XML format (surprising for a 28-years program at the time of writing) as its configuration settings, it always failed when processing large number of files that exceed its default threshold values. Tweaking it with different settings or disabled it did not resolve the crash issue. Switching to GraphicsMagick yielded the same result. The workaround was convert each image file to PDF in parallel manner and merge all PDF files as single large PDF file.
find . -name '*.jpg' | parallel --progress convert {} {.}.pdf
pdfunite *.pdf scanned_doc.pdf

Does Webpack, Docker Machine, through Virtual Box, and Unix-based host OS work well? No, not really when come to watching changes of files and hot reloading. First, Inotify, the Linux kernel subsystem which notify file changes to application is not and will not be supported in Virtual Box. How about we switch to VMWare? Nope, not supported either. Switching Webpack's watch method to polling have its own issue as well, CPU usage and hot reloading depends on frequency of polling. The key is find the acceptable polling interval. Off course, there are workarounds but not to my liking. If you read carefully, optimal solution using Inotify does not work because the limitation of shared folders through Network File System (NFS) and how file changes event is not communicated between Docker's host and container.

What to consider when designing RESTful API for third parties usage? There are four rules: (1) Use API keys for every request endpoint, (2) Regulate usage through rate limiting using HTTP 429 and API key instead of IP address, (3) Revoke API key if there are usage violation but provides API for client to check rate limits, and (4) Use other means to validate authentication and authorization beside API key.

What are the differences between COALESCE and IFNULL in MySQL? There are several but COALESCE is the preferred choice because (1) It's a standard and should works across multiple DBMS (but do we switch DB that often?), (2) COALESCE support multiple arguments until it can find the first non-NULL value but IFNULL only  support two arguments, and lastly (3) IFNULL is slightly faster than COALESCE. Interestingly, undefined value (1/0) is considered as NULL or missing value.
mysql > SELECT IFNULL(1/0,'yes');
mysql > yes

mysql> SELECT COALESCE(1/0,'yes');
mysql > yes

What is the equivalent of `which` in Windows? `where` as shown below.
C:\where notepad

This Week I Learned 2018 - Week 32

Post from previous week or from the archive.


What are the available wild Betta species? We're looking into Betta Brownorum as our next breeding project. Since this is a wild species, we can't find any stock from local breeders. It seemed you will have to capture it from the wild, in Sarawak.

Why there is a sign in Perl's hash key? Legacy reason.

Getting excited when buying stuffs (the highs is in the anticipation and it's more fun planning a vacation instead of the actual vacation) and have buyer remorse later? There is a quick tip on handling that. Take the amount you're going to spend and pay off any pending debts or loans. In the end, it's a divert and replace strategy. Any impulse purchase can be put off (delay strategy) if you set a goal before succumb to your materialistic needs. For example, lose a KG, indulge yourself into the stuff you desire for so long. Remember to reward yourself as you deserve it. Note that this is applicable to stuff and not really experience, especially spending on good experience with friends and family (moderation applies here as well). Another way, love shopping? Why not spends money on investment. Shop investment, not liabilities. Instead of getting the Nike shoes, buy Nike stocks. You have to ask yourself, which is more motivating? Watching your bills accumulates or your saving grows?

How to spend $0 each day? Meal prep. My friend. Meal prep.

How Europe Learnt to Swim? Like any forms of knowledge, observation, self-learned, trial and error, consolidation, and sharing of knowledge. I'm quite surprised that most island inhabitants can't swim properly to save their own life. Seriously?

What cloths to keep and throw away when cleaning up your closet? One key question to ask yourself is "If I saw this at the store right now, would I buy this again?" Throw away the cloth if the answer is no.

What is JSON PATCH? A combination of JSON and HTTP PATCH. You will see the HTTP content-type have been set to application/json-patch+json.

What was your most 'can't-put-down' book that you've ever read? Here are the list in random order. Overwhelm by the list? The quick way to filter out is to read the first few pages and if it caught your attention, proceed ahead. If not, ignore.
  1. Watership Down by Richard Adams (discussion)
  2. One Hundred Years of Solitude by Gabriel García Márquez (discussion)
  3. The Historian by Elizabeth Kostova (discussion)
  4. Night Watch by Terry Pratchett (discussion)
  5. 11/22/63 by Stephen King (discussion)
  6. The Book Thief by Markus Zusak (discussion)
  7. Killing Floor by Lee Child (discussion)
  8. Lonesome Dove by Larry McMurtry (discussion)
  9. Jurassic Park by Michael Crichton (discussion)
  10. Neverwhere by Neil Gaiman (discussion)

This Week I Learned 2018 - Week 20

Week 19 post or something from the past.

Interesting development on the local scene. Everyone is overwhelmed by the endless good news which some seemed too good to be true. Still too soon to tell but nobody think it will get worst than the current mess. On a side note, at least now we can read articles in Medium from our mobile devices.

The completion of BSL20180124. Our second successful spawn. As usual, write-up on the whole process and retrospection on our breeding process. Both of us are getting more experienced, selective, and bolder when breeding Bettas. So many things learned during these few months and what we learned can definitely can help us to improve our other spawning projects. We can now confidently buy better grade (ahem, more expensive) Betta fishes and breed them. But right now, the main focus is to change our breeding method from leaving fry with father to removing fry after free swimming. The former method produces limited number of fry and the later will yield large (till 500 fry) spawn. We shall see the result in coming months.

The difference between `application/xml` and `text/xml`. Encounter this when making RESTful request and the existing CPAN module does not recognizes `text/xml`.

Good sample Dockerfile to setup your Perl application in Docker instance.
FROM perl:5.26

RUN cpanm Carton && mkdir -p /usr/src/app
WORKDIR /usr/src/app

ONBUILD COPY cpanfile* /usr/src/myapp
ONBUILD RUN carton install

ONBUILD COPY . /usr/src/app

Detect whether an item exists in a Perl's array. Why such simple stuff needs to be so complicated in Perl?
# $value can be any regex. be safe
if ( grep( /^$value$/, @array ) ) {
    print "found it";

A Practical Guide to (Correctly) Troubleshooting with Traceroute (PDF). We have been using traceroute wrongly all this while.

This Week I Learned - 2017 Week 18

Last week post or you want to read through the whole series.

Since the prevalent of Internet these days, we're more connected that ever. And yet, we're even more isolated. More and more shopping malls and residential areas are being built especially around the transit stations. Basically all looks the same and people are discourage to travel to another town anymore because everything is available either online or at your nearest malls.

Sick. Down with fever. It has been a while since last time. No one but myself to blame. For people around my age, good rest and food is a way to healthy life. We're not seeking prolonged life but prolonged health.

#1 So, what's your Plan B? Good discussion from the comments. While there is some sort of discrimination against older programmers, but the reality is older programmers may just burn out or bore doing the same mind-numbling stuff over and over again. Yes, the same sh*t but different day. Not everyone have the opportunity to work meaningful projects (yes, subjective view), but most of us just work on some glorified spreadsheet or database skin. You see the same mistakes repeated over and over again from different systems and it's just pointless or bother to raise it up again.

#2 On becoming a low-level programmer. There is a curated list of information especially on the hardware part.. There is also another list of learning the fundamental of computer science. So many things to learn and explore and yet, so little time. Sometimes the reality of life commitments limit our choice on the thing we want to pursuit. I need to finish everything by July before I can start to clear the list from my bucket list. One step at a time. Follow the system.

#3 First, make it work, then make it right, and finally, make it fast. Get the thing to work first before you're thinking about making it right (giving the right name) and optimize it. Fulfill the requirements of the stakeholders first (prototyping) before you're looking into doing it right and fast. Something I need to remind myself constantly as we're often obsess with perfection. What's the point if your build it the right way and fast but does not meet the stakeholder needs?

#4 How to write a REST client in Perl. Most Perl modules have limited documentation, for example, little example on how to use the APIs. This is where PHP (it has been so long since I mentioned something about PHP) really shines. As the language is slowing dying, there are little or no good example on how to start something up fast.

Meanwhile, something rather interesting about Perl. I've seen quite a few Perl codes with subroutines that prepended with ampersand (&) operator. Basically, the ampersand is needed if you're calling a subroutine before its declaration. Off course, there are many other reasons and usages to use the ampersand, but commonly as reference to a subroutine.

sub hello {
    print "hello";

When using OO with Perl, you will need to instantiate the constructor or method invocation through the arrow (->) operator (a reference). Good examples given below.

Using the arrow notation or virtual method.
my $apple = Fruit->new("apple");

Using the double colons notation or static method.
my $apple = Fruit::new("apple");
Fruit::get_price($apple, "usd");

However, to invoke the method either statically or virtually, you'll need to write the constructor as follow.
sub new {
    my $self  = shift;
    my $class = ref($self) || $self;
    return bless {}, $class;

#5 "error: src refspec master does not match any." Silly me. You need to commit something locally before pushing to remote Git server.

#6 Tiny Python Notebook. Definitely not tiny but quite comprehensive guide to Python 3.6.

#7 How to unstage all staged files in Git? It's so easy that the best answer is not upvoted enough.
$ git reset

#8 Postgres Weekly. When come to FOSS database choices for web application for Intranet application, I'm strong opinion that PostgreSQL should be the default choice. Feature like Range Types and others solves quite a lot of issues and save lots of developer-hours. Yes, MySQL has its own usage, but for any business application which involves monetary and strong ACID compliance, it's not a right choice.

This Week I Learned - 2016 Week 49

Last week post or the whole series.

When a screenshot says a lot. The 14-plus hours uptime is something to be concerned about. Is best to be away from your machine from time to time. As you age, there is no needs to tweak your environment, just use the default settings for almost everything.

Certain unfortunate requirements led me to buy a USB-based high powered 300Mpbs Wifi adapter, TP-Link TL-WN8200ND. Unfortunately, while the driver seems to load properly, I still can't connect through Wifi in Ubuntu 16.10.

Understanding htop. Comprehensive guide to htop and its equivalent console commands. It will make you realize how much htop have aggregate and collect all the necessary information. The same author also wrote another useful guide on HTTP headers.

While we're on HTTP. Encountered this error with Nginx few weeks back where the error log shows that "upstream sent too big header while reading response header from upstream". In other words, your proxy server, Nginx does not like the data sent over from the application server (upstream). Several reasons may cause this like large cookies size, cookies with way old timestamp, or mismatch of response size and content length. Several ways to resolve this, either fix the issue at the upstream , disable proxy buffer, or increase proxy buffer sizes. Example as shown below (do not follow this values, adjust accordingly). Don't understand these settings? You can read details explanation and an excellent guide on these directives.
http {
    proxy_buffers           8 4k;
    proxy_buffer_size         8k;
    proxy_busy_buffers_size   16k;

On Perl. Nothing much pickup for the last two weeks, mostly just test cases and test cases. Interesting behaviour when returning value from subroutine. Being Perl, implicit is better than explicit as compare to Python. For example, there is this rule of do no return `undef`, just use the bareword `return`.
use Dumper;
sub a { return undef; }
sub b { return; }

my @aa = a();
my $a = a();

my @bb = b();
my $b = b();

Dumper(\@aa); # [undef], not false
Dumper($a); # undef, false

Dumper(\@bb); # [], false value because empty array
Dumper($b); # undef, false

How to implement Test-Driven Design (TDD) in Perl? So many good links given in those answers to the questions. Unfortunately, most of the links are quite dated and some may not be that relevant anymore. But since this is Perl, most stuff should be long standardized and stable.

This Week I Learned - 2016 Week 39

Last week post or the whole series.

The month of October have started. As we're closing in to the end of year 2016, I've still have long list of pending items to follow up. Sometimes, is best to take break, reflecting on the current situation, and re-prioritize your short and long term goals. Nevertheless, there is one thing for sure, regardless the outcome, discipline is one sure way to get closer to what you want to achieve.

Interesting stuff I've learned this week.

Few days back, I've managed to finish the book Modern Perl. I bought and starting reading the book on 25th June and finished it by 30th September. It took me roughly 98 days to wrap up the whole book page by page (not a good strategy according to How to Read a Book). My impression? Not a beginner book and not quite a suitable book for any developer switching from other languages. It's more like a reference book on how to code according to the Modern Perl convention. The correct sequence of Perl books to read are as follow:

Language FundamentalsReferences
Learning Perl
Intermediate Perl
Mastering Perl
Programming Perl / the "Camel book"
Modern Perl

While we're in the discussion on book. I'm quite disappointed with Eloquent Javascript. Someone suggested that I should look into `You Don't Know JS` book series instead. Maybe I should try it out instead. It's currently the next item in my reading list.

Again, on Javascript. There is an annoying behaviour of using Array with Object. Seriously? I was caught surprised by this.

On a related note, the popularity of Javascript leads to an issue of Same-origin policy where it's not permitted to request a resource from different domain. The Cross-Origin Resource Sharing (CORS) is a specification to resolve this. Solution exists for different popular web servers with support from different web browsers. For example, how to resolve this matter in Cloudfront.

Life as a Middle-Aged Geek (via HN). The advantage of being a middle-age developer is you probably know what you want or don't want out of your life. As time is catching up and you have plenty of life commitments elsewhere, if better you should constantly review and revise your own life goals. While ageism is an ongoing issue with technology sector, the Old Geek Jobs site was created for those developers whose are (35 and beyond). Or you can be an indie hacker instead?

Look-and-say sequence. Quite an interesting integer sequence, useful when you want to setup a new Wifi password but at the same time, want to confuse the hell of your users.

TLD used for local development? Mine is still using `.dev` but you must be aware that `.dev` is a generic Top Level Domain (gTLD).

Differences between Ansible's `command` and `shell` usages and when to use it. It seems I need to rewrite most of my dotfiles' steps. Didn't realize that `ansible-lint` actual exists.

What are the software stacks or cool tools used in the startup scene?

How to hire a programmer? While technical skills is a crucial requirement, cultural fit and attitude are equivalent important as well. But, as the article mentioned, nothing is guaranteed.

Polipo - Tiny Caching Proxy

During provisioning a new virtual machine (VM), you will need to repeatedly destroy and rebuild the VM. One of the bottleneck is you've to re-download all the GNU/Linux distro packages. While you can use packaging tool like APT or YUM to cache your packages, it still can be shared by different VMs. To solve this, you can set up a caching proxy at your host machine to be shared among all the guest VMs.

Instead of default caching proxy, Squid, I've opted for Polipo, a smaller and simpler caching proxy. Setting up was quite straightforward with additional minor changes.

Install the packages.
$ sudo yum install polipo

Enable the service after reboot.
$ sudo systemctl enable polipo.service

Check the status of the service. One of the benefit of systemctl is that it show a lot of crucial details of the daemon or service. Thus, helps a lot when we're troubleshooting the server.
$ sudo systemctl status polipo.service 

Set up the proxy connection details as environment variables so that the console app, for example, wget or curl, can use this.
$ export {http,https,ftp,rsync}_proxy="http://localhost:8123"
$ export no_proxy=localhost,
$ env | grep proxy

To test our proxy server using either curl or wget. Using curl. Option -sv is to show server header verbosely.
$ curl -sv 2>&1 | grep 8123
* About to connect() to proxy localhost port 8123 (#0)
* Connected to localhost ( port 8123 (#0)

Using wget. Using -S --spider so that wget don't download anything.
$ wget -S --spider 2>&1 | grep 8123
Connecting to localhost (localhost)|::1|:8123... failed: Connection refused.
Connecting to localhost (localhost)||:8123... connected.
Connecting to localhost (localhost)||:8123... connected.

To test the caching while download a large file. First time download took around 3 minutes and subsequent download of similar file took less than 1 second.
$ time wget
2014-09-28 04:17:40 (55.7 KB/s) - ‘cirros-0.3.1.xz’ saved [11419004/11419004]
real    5m52.107s
user    0m0.096s
sys     0m0.461s

$ time wget
2014-09-28 04:17:51 (478 MB/s) - ‘cirros-0.3.1.xz.1’ saved [11419004/11419004]
real    0m0.028s
user    0m0.002s
sys     0m0.023s


" approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare mininum of centralized management of these services, which may be written in different programming languages and use different data storage technologies."
-- James Lewis & Martin Fowler, emphasis added
The term have been lingered in my mind for the past two weeks but I didn't pay much attention to it until today. Yes again, another gimmicky development term which seems to be another a rebranding of Unix philosophy and simplified version of Service-Oriented Architecture (SOA). Sigh, the side effect of the trending butt, ahem, cloud technology these days.

How to implement this architecture style? Decompose and move each component in your monolithic system into its own service. Each service can be development using any platforms, programming languages, or data storages but communicates through JSON over HTTP or lightweight messaging bus. In short, a change of the communication style between each components from function calls to messages.

Nothing new here, old wine in a new bottle.

Quickest Mitigation to BREACH Attack

Before that, the best explanation I read so far on BREACH attack. To quote from Wikipedia,
"A BREACH attack can extract login tokens, email addresses or other sensitive information from TLS encrypted web traffic in as little as 30 seconds (depending on the number of bytes to be extracted), provided the attacker tricks the victim into visiting a malicious web link."
Maybe is just me but way better explained that the official disclosure. Reluctantly (since performance will be affected), I've opted the most effective mitigation method by disabling HTTP-compression at server side. Example given here is running on #Apache 2.2.22 in Ubuntu 13.04.

1. Check our Apache web server version.
$ apache2 -v
Server version: Apache/2.2.22 (Ubuntu)
Server built:   Jul 12 2013 13:18:14

2. Check if the dynamic module deflate is enabled.
$ sudo apachectl -t -D DUMP_MODULES | grep deflate
 deflate_module (shared)
Syntax OK

3. Double-confirm that the web server is sending compressed content to the client. We're using curl HTTP client. Look for Content-Encoding header field in the HTTP response returned from the web server.
$ curl -I -k --compress https://localhost
HTTP/1.1 200 OK
Date: Wed, 07 Aug 2013 16:36:35 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.4.9-4ubuntu2.2
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 20
Content-Type: text/html

4. Disable mod_deflate and restart the web server again.
$ sudo a2dismod deflate
Module deflate disabled.
To activate the new configuration, you need to run:
  service apache2 restart

$ service apache2 restart

5. Recheck the HTTP response Content-Encoding header. It should be missing from the result returned.
$ curl -I -k --compress https://localhost
HTTP/1.1 200 OK
Date: Wed, 07 Aug 2013 16:42:53 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.4.9-4ubuntu2.2
Content-Type: text/html

Google PageSpeed

"PageSpeed Service is an online service to automatically speed up loading of your web pages. PageSpeed Service fetches content from your servers, rewrites your pages by applying web performance best practices and serves them to end users via Google's servers across the globe."
-- Google's PageSpeed Service FAQ, emphasis added

Now I understand why and how certain major alternative news portal can handle all the traffic during the last general election.

Micro-Routing Framework

Interesting question on which micro-routing frameworks for PHP in r/php. Slim, dispatch, and klein are my favourites for their simplicity and procedural approach. Aura Router looks interesting too but feels too heavy.

As I mentioned before, someone should really add the missing routing functions to the PHP core. Thus, making PHP a fully-featured micro-framework with PDO as model and PHP itself as view. As they said, PHP as Template Engine (PATE). Two benefits for doing so. First, we have a standard way of doing routing in PHP. Second, no dependancy on any frameworks for single file script (think apc.php) or small size app (think simple REST app).

Can someone in php-internals add this to PHP 5.6?

The Missing apple-touch-icon-precomposed.png

And I thought I had fixed the 404 missing apple-touch-icon-precompose.png file issue few week ago but yet today the issue still haven't resolved. It seems that the image files must located at the document root of the web server.

Meaning that if you put your image file outside the document root like below
-- assets/
   -- img/
      -- apple-touch-icon-precomposed.png

You need to redirect them using mod_rewrite to the actual path. In you .htaccess, add these lines.

    RewriteEngine On
    RewriteRule ^apple-touch-icon-precomposed\.png$ assets/img/apple-touch-icon-precomposed.png [L]

When I checked my .htaccess file, I noticed I did the same for favicon.ico as well and it never cross my mind to do the same for this !

Redis : New Member to the Server Stack

Interesting article on the size and the technologies behind big porn sites. While some may not agree, porn industry do accelerate Internet advancement. To best of my knowledge, how I visualize the software stacks behind these sites.
[browser] --- [cdn | memcached] --- [haproxy | varnish] --- [nginx] --- [redis]

It seems that Redis, the key-valued storage engine has slowly replaces MySQL as the de facto primary data storage engine. Contrast to what the article said, Redis was not used to store and serve videos. Instead, it should be used for storing metadata. Which is consistency with the typical use cases for Redis.

One of the server stack you should look into in 2013. What's the next step ? Let's get it install in Ubuntu 12.10.
$ sudo apt-get install redis-server
$ redis-cli

For further reading, I will recommend you to read Ken Nejima's thought on Redis. Note that the article was written in 2010 but it is the best writing on Redis I read so far.

Unity Dash Online Video Search in Ubuntu 12.10

I was learning how to use ngrep, a command line tool that let you search and filter all the network traffic coming in and out from our machine. While testing to see what being transfer from and to port 80 ( see command below), I notice my newly upgraded Ubuntu 12.10 keep on making HTTP request to
$ sudo apt-get install ngrep
$ sudo ngrep -d any port 80

It seems, default settings in Ubuntu 12.10 enables unity video lens to periodically connect to this server to search information for video recommendation. It is advisable to remove or disable this unused and unnecessary feature. Just type this command and logout and re-login your session.
$ sudo apt-get --purge remove unity-scope-video-remote

I was curious of the content in the JSON file. So we going to make a manual HTTP query to capture the file. Note that direct browser query will return empty result (suspect user agent checking) so we have to use curl or wget.
$ curl -o videosearch.json

The returned JSON file contains a list of YouTube videos. Changes to the request parameters(q and sources) returns almost similar result. Nothing interesting here. Let's inspect the HTTP response header instead.
$ curl -D header videosearch.json

Note that I removed the timestamp and added some comments
$ cat header
HTTP/1.1 200 OK
Date: .....
Server: gevent/0.13.0 gunicorn/0.13.4

The response is coming from gunicorn, a lightweight Python WSGI HTTP server using gevent, a coroutine Python networking library.
Vary: X-Geo-Country

What HTTP request header fields (in this case X-Geo-Country) to be used in caching decision either to regenerate from application server or reload from proxy server.
Content-Type: application/json
Content-Length: 10309
Expires: ......

Age: 203

The age in seconds of the JSON file in the proxy cache.
X-Cache: HIT from
X-Cache-Lookup: HIT from

Requested content is found in the cache (HIT) of the caching server.
Via: 1.0 (squid/2.7.STABLE7)

Squid, popular web caching server.
Via: 1.1
Connection: close

The HTTP request has gone (back and forth) through these two servers. Note that is Ubuntu music search API server.