Showing posts with label plmotw. Show all posts
Showing posts with label plmotw. Show all posts

Perl Module(s) Of The Week - 2019 Week 11 - HTTP::AnyUA

In Perl, Tim Toady is a well known way of coding, which is there's more way to do it. Similarly in CPAN context, there's more modules that provides similar features. Good example is available of HTTP client or user agent modules where we have the de facto standard LWP::UserAgent, the modern non-blocking I/O and WebSocket Mojo::UserAgent, the small and tiny HTTP::Tiny, the lighting fast Furl, the event loop AnyEvent::HTTP, the legacy Curl-based wrapper Net::Curl::Easy, and numerous other modules that extend or wrap over the mentioned modules.

This the where HTTP::AnyUA module comes in. It provides a unification programming interface for all common HTTP requests based on the HTTP::Tiny interface. Think of this as the Database Independent Interface (DBI) for HTTP programming.

The installation as usual.
$ cpanm HTTP::AnyUA

Using the sample code below, we iterate using different HTTP client modules and request your truly website.
use strict;
use warnings;

use AnyEvent::HTTP;
use Furl;
use HTTP::AnyUA;
use HTTP::Tiny;
use LWP::UserAgent;
use Mojo::UserAgent;
use Net::Curl::Easy;

foreach my $ua (qw(Furl HTTP::Tiny LWP::UserAgent Mojo::UserAgent Net::Curl::Easy)) {
    my $any_ua = HTTP::AnyUA->new(ua => $ua->new, response_is_future => 1);
    print "Using: $ua\n";

    my $future = $any_ua->get('');
    $future->on_done(sub {
        my $response = shift;
        print "$response->{status} $response->{reason}\n\n";

    $future->on_fail(sub {
        my $response = shift;
        print STDERR "ERROR: $response->{status}\n\n"

Unfortunately, we can't seem to get consistent result from all different HTTP clients. This makes us wonder, while this module claimed to have unified interfaces but does it provides consistent results? If we have inconsistent HTTP response, wouldn't it be better we just use the individual HTTP client module instead?
$ perl 
Using: Furl
200 OK

Using: HTTP::Tiny
200 OK

Using: LWP::UserAgent
ERROR: 403

Using: Mojo::UserAgent
Using: Net::Curl::Easy
ERROR: 301

Nevertheless, let's continue with some code reading. Some interesting code that caught my attention.

Instead of using the environment variable, `PERL_HTTP_ANYUA_DEBUG` everywhere in the code to determine whether to print debugging info, just centralized in the logging subroutine. Good to use the environment variable to toggle this.
sub _debug_log { print STDERR join(' ', @_), "\n" if $ENV{PERL_HTTP_ANYUA_DEBUG} }
$self->_debug_log('Created with user agent', $self->ua);

The implementation on handling exception through `eval` that implement something similar like try and catch block.
my $resp = eval { $self->backend->request(uc($method) => $url, $args) };
if (my $err = [email protected]) {
    return $self->_wrap_internal_exception($err);

Check if the `_module_loader` object exists, if not just instantiate it.
# get a module loader object
sub _module_loader { shift->{_module_loader} ||= Module::Loader->new }

Perl Module(s) Of The Week - 2019 Week 10 - Mojo::Util

Yogi Betta once said "you can observe a lot just by watching".

I somehow agree with that statement, in a way, to a certain extend.

And one amusing thing I noticed among startups or local meetups here in MY, their web tech stack is always chasing the latest greatest in the industry. Fair enough, who don't like new shinny stuff? It does make you feels at the forefront of the industry and it looks good on your resume as well. Or maybe because we have to keep learning in a rapid changing industry? Or maybe we just fear of missing out?

The funny thing is, as I observed, poking fun at PHP is a common theme here. PHP is seen like a second class programming language and it's a sin to use PHP in a web stack these days. Ironically, most of their front facing website or landing page is hosted in LAMP stack and powered by Wordpress. Which some of them admitted themselves, should have been done in a system of their own chosen and favorite latest greatest web programming language. This is doable but the financial cost of hiring a freelancer to design and setup a professional looking website far outweigh the former choice. 

The thing is, PHP these days, PHP 7 to be exact, have move from the "Visual Basic" of the web to the "Java" of the web. Gone are the days where you would hack a site with tremendous amount of global utility functions. These days, it's more mature, with a standard packaging system, numerous web frameworks, and a more Object-Oriented (OO) and types support. Yes, you still can use back all the global helper functions like you used to.

Why all these discussions about PHP? Well, this is related to the Perl module we're reviewing this week, Mojo::Util, a module that contains a bunch of portable utility functions. These functions is not the same as global helper functions. According to C2 wiki, helper functions are:
"Functions that do not aid code reusability are helper functions; their sole purpose is to "help" a single function by cleaning the code and making the logic clearer."
As usual, the installation.
$ cpanm Mojo::Util

Let's try using some of the helper functions from the module. Example code using Punycode, a representation of Unicode characters in ASCII characters used in domain names, as shown below.
use diagnostics; # To display details of any warnings.
use strict;
use warnings;
use utf8; # Since we've Chinese characters, ensure the file is UTF8.
use feature qw ( say );
use Mojo::Util qw ( punycode_decode punycode_encode );

# Ensure our standard output is UTF8 since we're printing UTF8 characters.
# Otherwise we will encounter "Wide character in say at line 14."
# error.
binmode STDOUT, ':utf8';

my $string = qq|百度|;
my $punycode = punycode_encode $string;

say $punycode;
say punycode_decode $punycode;

# Expected output
$ perl    

Mojo::Utils contains a list of alias functions that reduce complexity of the code. Some of these are aliases to other modules. For example, these MD5 helper functions which reduce unnecessary typing of the fully qualified names.
sub md5_bytes { Digest::MD5::md5(@_) }
sub md5_sum   { Digest::MD5::md5_hex(@_) }

As mentioned earlier, utility functions is not the same as helper functions. As shown below, the `_unescape` helper function is used once for `html_escape` utility function.
# Helper for html_unescape
sub _unescape {
  if ($_[0]) {
    return chr hex $_[0] if substr($_[0], 0, 1) eq 'x';
    return chr $_[0];
  return exists $ENTITIES{$_[1]} ? chr $ENTITIES{$_[1]} : "&$_[1];";

While going through the code, I've found some interesting and funny The Simpsons quotes. The quotes and the relevant codes are pretty much self-explanatory.

(1) "Bart, stop pestering Satan!".
# "Bart, stop pestering Satan!"
our @EXPORT_OK = qw/b64_decode b64_encode camelize decamelize decode encode/;
push @EXPORT_OK, qw/get_line hmac_md5_sum hmac_sha1_sum html_escape/;
push @EXPORT_OK, qw/html_unescape md5_bytes md5_sum punycode_decode/;
push @EXPORT_OK, qw/punycode_encode qp_decode qp_encode quote/;
push @EXPORT_OK, qw/secure_compare sha1_bytes sha1_sum trim unquote/;
push @EXPORT_OK, qw/url_escape url_unescape xml_escape/;

(2) "Daddy, I'm scared. Too scared to even wet my pants. Just relax and it'll come, son."
# "Daddy, I'm scared. Too scared to even wet my pants.
#  Just relax and it'll come, son."
sub html_unescape {
  my $string = shift;
  $string =~ s/
          \d{1,7}             # Number
          x[0-9A-Fa-f]{1,6}   # Hex
      ([A-Za-z]{1,8})         # Name
  /_unescape($1, $2)/gex;
  return $string;

Perl Module(s) Of The Week - 2019 Week 09 - Carp and Carp::Always

This week we will look into three Perl modules, Carp and Carp::Always that alters or enhances the default behaviours of error handling using both `warn` and `die` function. Basically both functions will raise a warning and write an error message to standard error (STDERR) but `die` will throw an exception by exiting from the program through `exit` function.

The `die` function is commonly used when opening a file. When the opened file is not found, an exception will be thrown with an error displayed to STDERR and termination of the program. Example below illustrates this. We can substitute `die` with `warn` but that is not a good practice as open a missing file disrupts the normal flow of the program and an exception should be raised and the program should be terminated.
open(my $fh, "<", "input.txt")
    or die "Can't open < input.txt: $!";

Installation is straightforward. As usual.
$ cpanm Carp Carp::Always

What kind of different behaviours that `Carp` module provides? Let look at the below code example consists of a module `` and the calling script ``.
package Foo;
use Carp;

sub twarn { warn "test warn"};
sub tcarp { carp "test carp"};


use lib ".";
use Foo;


Running the above code will give us below output. Upon an exception, the `warn` function reports the file name (``) and the line number (`line 4`) where the error occurred. Using `carp` function with verbosity enabled, the function shows additional details on the caller's file name (``) and line number (`line 5`). If you're writing a Perl's CPAN module or subroutines, using `Carp` module is a preferred choice as the library user (the caller) is more interested in finding where in their own code that causing the exception.
$ perl -MCarp=verbose 
test warn at line 4.
test carp at line 5.
        Foo::tcarp() called at line 5

The `Carp` module provides several common functions that wrapped around both `warn` and `die` functions.

(1) `carp` - `warn` of errors (from perspective of caller).
(2) `cluck` - `warn` of errors with stack backtrace.
(3) `croak` - `die` of errors (from perspective of caller)
(4) `confess - `die` of errors with stack backtrace

What if you don't want or can't changed any existing code that uses the `warn` and `die` function but still want behaviour provided by `Carp` module with backtrace? This is where `Carp::Always` module comes in as shown in the result below.
$ perl -MCarp::Always               
test warn at line 4.
        Foo::twarn() called at line 4
test carp at line 5.
        Foo::tcarp() called at line 5

Perl Module(s) Of The Week - 2019 Week 06 - SMS::ClickSend

This week Perl module, SMS::ClickSend was chosen because after reviewing the source code, I personally felt that this module embodies what I consider the Perl 5 way of simple coding style. Regardless Perl's notorious TMTOWTDI, as I observed, there exists a standard coding style which includes (1) Use minimum dependencies for speed and simplicity, (2) Prefer plain old Object-Oriented (OO) using blessed instead of Moose or Moo, (3) Snake case subroutine naming convention, (4) lack and little validation, and (5) Brevity without overly verbose. Even through the module was written in 2014 (5 years ago), I've seen new CPAN module that was written in such style even in 2019. There are some CPAN modules which were written in such over-engineered way that spans multiple modules. But that is a discussion for another day.

As usual, Installation is simple.
$ cpanm SMS::ClickSend

Sample usage of the API based on the synopsis.
use SMS::ClickSend;
my $sms = SMS::ClickSend->new(
    username => 'username',
    api_key  => 'API_KEY...',
my $res = $sms->send(
    to => '+61411111111',
    message => 'This is the message',
print Dumper(\$res); use Data::Dumper;

Let us look at the constructor. The constructor, `new` accept an array of values that needs to be converted to a hash. Hence, the number of items in the array should be even, hence the modulo by 2 to detect any odd number of items passed. Converting an array to a hash is quite straight forward, just `%hash = @array` should be sufficient;
sub new {
    my $class = shift;
    my %args = @_ % 2 ? %{$_[0]} : @_;

Check whether the required parameters exists without using the `exists`. Just use short-circuit evaluation through logical or operator. Read more on logical disjunction.
    $args{username} or croak 'username is required.';
    $args{api_key}  or croak 'api_key is required.';

Use default user agent if nothing was set or defined through the logical-or assignment.
    $args{ua} ||= LWP::UserAgent->new;

Lastly, just make sure or bless the hash arguments to be part of an object in the `SMS::ClickSend` package.
    return bless \%args, $class;

To fully understand how the constructor works, we need to show how the constructor was being called by the calling code.
# We are passing an array of four elements
my $sms = SMS::ClickSend->new(username => $username, api_key => $api_key);

# We are passing a hash and this can be obtained from first element from the argument, `$_[0]`.
my $sms = SMS::ClickSend->new({username => $username, api_key => $api_key});

# This will not work and `$args` will become an empty hash.
my $sms = SMS::ClickSend->new($username);

# This will work partially as `api_key` argument is missing and trigger the `croak`.
my $sms = SMS::ClickSend->new(username => $username);

Instead of catering for all possible inputs, perhaps we can just stick to one way of an array instead of hash to accept arguments to the constructor.
sub new {
    my ($class, %args) = @_;

    $args{username} or croak "username os required";
    $args{api_key} or croak "api_key is required";

    return bless \%args, $class;

Another interesting module found within the code. Basically, `var` is "Perl pragma to predeclare global variable names". Basically a global variable within the scope of the package. Superseded by `our` function which is a "lexical alias to a package (i.e. global) variable".
use vars qw/$errstr/;
sub errstr { $errstr }

Perl Module(s) Of The Week - 2019 Week 05 - API::Google

While there are tons of web service API module in the CPAN, we will look into a rather simple and useful Perl's module, API::Google. As the name implies, this module allows us to connect to Google services via the API. There are not a lot of recent released modules written in non-Moo/Moose-way which allows us to look into using the Perl's plain old Object-oriented (OO) way.

Installation is quite straight forward. Usage wise, you will have to obtain the OAuth2 credential and generate the token file, `config.json` file.
$ cpanm API::Google

Let's move on the study and reading the code.

Firstly, I'm quite surprised that the default user agent or HTTP client module to make web request used is not the typical LWP::UserAgent but instead Mojo::UserAgent. There is nothing wrong with LWP::UserAgent, it's an established and works well. But if you need to make non-blocking I/O or Websocket web request, then Mojo::UserAgent is the preferred module, especially for modern web development. Moreover, the documentation is very comprehensive and helpful.

Next, since the author did not use Moo or Moose, therefor a helper subroutine was created to access the object attribute or the hash value.
sub ua {

Looking through the code, I'm constantly reminded that I forget how to join two hashes in Perl. Basically you have to understand the difference between a list and an array or a hash (an associative array) and numerous examples to illustrate that.
  $payload = { payload => $payload };
  %$params = (%$params, %$payload);

Perl Module(s) Of The Week - 2019 Week 04 - Reply

Is there a good read–eval–print loop (REPL) or language shell exists for Perl? Yeah, there are several and the Perl module of the week is going to look at Reply module, a REPL that can be customized with several plugins for additional features.

Installation and quick run of the program. The `0>` is the console prompt at the counter of number of lines.
$ cpanm Reply
$ reply

To install and use different plugins, for example, Reply::Plugin::Editor. Just install and update the default configuration file, `~/.replyrc`. To launch the editor when running the REPL, just use `#e` at the console.
$ cpanm Reply::Plugin::Editor
$ cat ~/.replyrc
editor = vi

Now for some code reading of the module. Surprisingly quite a small program compares to my initial assumption. To execute the program as source code, instantiate the class to load the configuration file and `run` it.
use Reply;
Reply->new(config => "$ENV{HOME}/.replyrc")->run;

First, code was written in old school Perl's OO as shown below. Codes are self-explanatory with clear variable names.
sub new {
    my $class = shift;
    my %opts = @_;
    my $self = bless {}, $class;
    return $self;

Next, run and loop the execution of the console input step by step until there is a termination.
sub run {
    my $self = shift;
    while (1) {
        my $continue = $self->step;
        last unless $continue;
    print "\n";

Plugins are loaded dynamically based on the configuration files using the Module::Runtime module.
    if (!blessed($plugin)) {
        $plugin = compose_module_name("Reply::Plugin", $plugin);
        die "$plugin is not a valid plugin"
            unless $plugin->isa("Reply::Plugin");
        my $weakself = $self;
        $plugin = $plugin->new(
            publisher => sub { $weakself->_publish(@_) },

Perl Module(s) Of The Week - 2019 Week 03 - MP3::Daemon

They said, old habits dies hard, especially if you're coming from console background. The tendency of trying to do everything at the console is hard to break. Console programs in *nix worlds tends to follow the Unix philosophy of do one thing and do it well, work well with other programs, and exchange data using text stream. The Perl module of this week, MP3::Daemon, a text front end to mpg123, was designed to follow such philosophy. For example, play part of a MP3 file repeatedly.
while true ; do mp3 j 1300 ; sleep 386 ; done

Installation and quick run of the program.
$ cpanm MP3::Daemon
$ mp3

Trying to run one of the command and we encountered these error messages. Note that this program was first written in 2001, 18 years ago, maybe there was some bugs or maybe it was due to my local installation. Furthermore, following recent naming convention, the module should have been named `App::MP3`. However, looking and reading old code will give you some insights on writing Perl code without resorting to external modules but focus on just basic Perl code. Nevertheless, we try to resolve these two issues.
 $ mp3 ls
Subroutine main::pod2usage redefined at (eval 16) line 1.
 main::BEGIN() called at (eval 16) line 1
 eval {...} called at (eval 16) line 1
 eval 'use Pod::Usage' called at /home/foobar/perl5/bin/mp3 line 8
No such file or directory at /home/foobar/perl5/lib/perl5/MP3/ line 84.

We first look into the `No such file or directory` error. Looking through the source code at line 84, educated guess that `mpg123` program (the program was name after the MPEG layer 1/2/3 format) have not been installed. Installed the console app will remove the error message.
$ sudo apt install mpg321

If you cannot access the source code, you can jump to the particular line using `less`. The `+84` option will jump to line 84 and `-N` will show line numbers.
$ less +84 -N /home/foobar/perl5/lib/perl5/MP3/

For second issue on subroutine redefined error. Looking through the source code as shown below. In Perl, without using external modules, we use `eval` for exception handling. The code checks for `Pod::Usage` availability, if not found, set the error message to `[email protected]`, the eval error variable.
eval "use Pod::Usage";
if ([email protected]) {
    sub pod2usage { system("perldoc mp3"); exit 1 }

To resolve this issue, first we need to enable `diagnostics` module to give us hints on what and how to resolve this warning. Add this line to the `mp3` executable script.
use diagnostics -verbose;

Run this program again. We will obtain full verbose details on the warning and how to resolve it.
    These messages are classified as follows (listed in increasing order of
        (W) A warning (optional).
        (D) A deprecation (enabled by default).
        (S) A severe warning (enabled by default).
        (F) A fatal error (trappable).
        (P) An internal error you should never see (trappable).
        (X) A very fatal error (nontrappable).
        (A) An alien error message (not generated by Perl).
    The majority of messages from the first three classifications above
    (W, D & S) can be controlled using the warnings pragma.
    If a message can be controlled by the warnings pragma, its warning
    category is included with the classification letter in the description
    below.  E.g. (W closed) means a warning in the closed category.
    Severe warnings are always enabled, unless they are explicitly disabled
    with the warnings pragma or the -X switch.
    Trappable errors may be trapped using the eval operator.  See
    "eval" in perlfunc.  In almost all cases, warnings may be selectively
    disabled or promoted to fatal errors using the warnings pragma.
    See warnings.

Subroutine main::pod2usage redefined at (eval 17) line 1.
 main::BEGIN() called at (eval 17) line 1
 eval {...} called at (eval 17) line 1
 eval 'use Pod::Usage' called at /home/foobar/perl5/bin/mp3 line 9 (#1)
    (W redefine) You redefined a subroutine.  To suppress this warning, say
            no warnings 'redefine';
            eval "sub name { ... }";

Made changes to the program as follow and the warning should be disabled.
  8 eval "use Pod::Usage";
  9 if ([email protected]) {
 10     {
 11         no warnings 'redefine';
 12         eval qq/sub pod2usage { system("perldoc mp3"); exit 1 };/                                          
 13     }
 14 }

Perl Module(s) Of The Week - 2019 Week 02 - Module::CoreList

One of the issue when using Perl is the abundant list of Perl's modules. Finding the right modules can be overwhelming and sometimes frustrating. Duplicating, deprecating, or abandon modules is typical situation for a programming language which is 31 years old. Hence, which recommended Perl's module should you uses then? Start with the modules that came default with the Perl installation, the core list. How? By using the Module::CoreList module.

Installation and quick check of the module.
$ cpanm Module::CoreList
$ perl -e 'use Module::CoreList'

Now, let's write some code (``) to list out all the core modules that came with Perl 5.29.6.
use strict;
use warnings;
use feature qw|say|;

use Module::CoreList;

say $_ foreach sort @{[Module::CoreList->find_modules(qr/.*/i, 5.029006)]};

Running the code and show the first ten results.
$ perl | head

So far so good. But do we need to write some code in order to make any query? No, there is a console utility, corelist that will do that. For corresponding command almost similar to our Perl's code before this.
$ corelist -v 5.29.6 | head

The following modules were in perl 5.29.6 CORE
Amiga::ARexx                                 0.04
Amiga::Exec                                  0.02
AnyDBM_File                                  1.01
App::Cpan                                    1.671
App::Prove                                   3.42
App::Prove::State                            3.42
App::Prove::State::Result                    3.42
App::Prove::State::Result::Test              3.42

Now some code reading. What interesting stuff can we learn from reading the source code of Module::CoreList?

(1) All the data, for example, Perl's version, starting from Perl 5 was hard-coded as large hash (see the `%released` hash).

(2) To prevent duplication, the changes (see `delta_from`, `changed`, and `removed` hash) of modules for each Perl version were stored as delta hash (see `%delta` hash).

(3) These use of `@_` pass through from one subroutine to another subroutine as shown below. See the `first_release_raw` subroutine.
sub a { say @_; }
sub b { &a; }
sub c { &a(); }
sub d { a(@_); }
sub e { &a(@_); }

b 1, 2, 3; # 123
c 1, 2, 3; # nothing is printed
d 1, 2, 3; # 123
e 1, 2, 3; # 123

(4) The END block is not a subroutine but a block of code that executed after all codes have been ran and before the Perl interpreter exited.

How about source code from corelist?

(1) *nix piping in Perl. See the example below.
    my @bundles =  map { $_->[0] }
                  sort { $b->[1] <=> $a->[1] }
                   map { [$_, numify_version($_)] }
                  grep { not /[^0-9.]/ }
                  keys %feature::feature_bundle;

Perl Module(s) Of The Week - 2019 Week 01 - Data::Money

Since this item was on my to-do list for quite some time and this is the first week of the new year, might as well proceed ahead and do it. Inspired by Python Module of the Week (PyMOTW), I will start a series of weekly blog posts to review and discuss any interesting Perl module(s) that caught my attention. Write up mostly will be on some code example, issue encounter, and what can we learn from the code, as in code reading.

Our Perl module of this week is Data::Money, which allows us to perform math operations on different currency in an object manner and let us understand how operator overloading works in Perl. Furthermore, this is a good example to showcase the actual implementation of PoEAA's Money class design (there are actual four Money class designs exists), which is a rather simple approach which does not allows multiple currencies operation. Nevertheless, an actual production ready implementation.

Installation and quick check of the module.
$ cpanm Data::Money

It seemed in latest Perl version (v5.28.0), the dependant modules, Locale::Currency and Locale::Codes will be removed from Perl core distribution.
$ perl -e "use Data::Money"
Locale::Currency will be removed from the Perl core distribution in the next major release. 
Please install it from CPAN. It is being used at /usr/local/lib/perl5/site_perl/5.28.0/Data/, line 27.
Locale::Codes will be removed from the Perl core distribution in the next major release. 
Please install it from CPAN. It is being used at /usr/local/lib/perl5/5.28.0/Locale/, line 22.

To remove the warnings, we have to install Locale::Currency manually.
$ cpanm Locale::Currency
$ perl -e "use Data::Money"

Let write some code (save as file ``) that create the new Money object or data type. `MYR` is the currency code where `MY` is two digits country code for `MalaYsia` and `R` is the name of the currency, in this case, `Ringgit`.
use strict;
use warnings;
use feature 'say';

use Data::Money;

my $price = Data::Money->new(value => 1.2, code => 'MYR');
say $price->as_string;
say $price->stringify;

Running the code will give us the same result twice. `RM` is the currency symbol that replaced our previous used `$` symbol.
$ perl

Let's look at these two subroutines, `as_string` and `stringify`. First, `as_string` is an alias to `stringify` subroutine. To create an subroutine alias in Perl, we will need to use typeglob (hence, the asterisk notation) to manipulate the Symbol Table, which stores all the variables in a package.
*as_string = \&stringify;

Second, as the name implied, `stringify` is typical subroutine naming convention for stringification in Perl, to convert an object into string or context of string. This is done through `overload` package as shown below, where it's implemented through anonymous subroutine. The subroutine will be invoked whenever the blessed package was called in a string context. You can read this tutorial for a sample working stringification using classical OOP way.
use overload
    '""' => sub { shift->stringify };