Showing posts with label subversion. Show all posts
Showing posts with label subversion. Show all posts

This Week I Learned - 2017 Week 28

Last week post or something from the archive.


Another long and outstanding issue resolved. While I'm not partially involved but glad it has finally been settled in a mature and non-violent manner. Long story short, is a long story. Nevertheless, everything back on track and are quite a few things I need to follow up and get it done by this year. Half year have gone and not much time left before the end of 2017.


When you're switching character set and column collation in MySQL, you will encounter certain limitation of the index length of Innodb storage engine, the error of 'The maximum column size is 767 bytes.' . There are three ways to resolve this issue. First, you resize the column size from VARCHAR(255) to VARCHAR(191). Second, you resize the the table index length. Third, you switching the storage engine from Antelope to Barracuda and then switch the table to dynamic row format. Similarly, MySQL utf8 to utf8mb4. This post summarized three essential SQL statements needed to do the migration.

MySQL, while is a pile of mess, is a good case study of Worse is better. Likewise, same goes to Craiglist. Say you want about the site, which still stuck in the early 90 design, but still making profit until this day. Sometimes, good enough should be sufficient.


Everyone's code basically sucks, including yours. Honest insight on what you will learn after many years into software development. Importantly, don't get too comfortable in any job. In the end of day, you're delivering software to solve problem, is all what the stakeholder wants, nothing more and nothing less. While some have more pessimistic view of moving to management instead of development and managing up (some said it's overwhelmingly cynical or Machiavellian).


The hierarchy of 'Troubleshooting Software Problems. Probably the best thing I've read so far on how to troubleshooting any problems. The hierarchy of problem as shown below:
  • data
  • code
  • workflow
  • design
  • architectural
  • team
  • project
  • organizational
  • leadership / existential

Is our eating habit, and thus eating habit influenced by "the array of microscopic organisms that live inside our intestinal tract." The whole thread was an interesting read with comments backed by scientific research. Maybe gastrointestinal microbiota is manipulating our eating behaviour. Which leads to the new type of obesity treatment through gut microbiome transplant. In other words, taking care of your gut is one of the essential way to stay healthy and fit.


Shelving and checkpointing in Subversion. It has been so long that finally someone look into adding local commit support. However, it's just the initial stage although there are many design documents available.

Where We Started. The movie and the guitar song. The former is a low budget movie with witty dialogue and the later made me want to pick up guitar again.

Teaching Subversion ?

After many years of using, encouraging, and observing developers using Subversion led me to a conclusion, it was never a technical issue but a people issue. Some lessons learned:

Management support
Unless the management, especially those without technical background, support the use of any source control to be the integral part of development process. Otherwise, you will be wasting your time and your breath.

Can anyone still practice software development without source control? Yes, you can. Short term project, especially those can be handled by single developer. Or one developer per module or folder approach where source code are shared through one centralized file server. Hence, you will encounter files in project that looks like this:
index.php
index_20131213.php
index_20131010.php

Yup, the developer is cooking his own source control. I cringes every time I see anyone does this. The saddest part is such practice is still so prevalent.

Company that don't value the importance of source control does not realize the benefit of safeguarding their intellectual property and other assorted gains in using it.

Use source control properly
Troy Hunt's 10 commandments lay down the rules on how to use it correctly. I will like to focus on two major difficulties I observed especially for those (me as well) who just starting to explore source control.

Write a meaningful and better commit message. You should explain the why and not really the how. Occasionally you will read empty commit, repeated (e.g. "latest update"), or just plain wrong commit message. These are useless and not helpful for anyone to review back any previous changes. Why so ? Because writing good commit message is hard, especially those whose mother tongue is not English. To solve this, set a standard log message format and encourage the developer to break each task into smaller chunk.

Commit early, often, and small / atomic. Break a large task into smaller sub-tasks. Unless the developer is stuck at something, there should be multiple commits per day. Also, this is to prevent those developers who like to commit everything in one shot.

In short, having the right tool and convention are not sufficient enough unless the developers are willing to use it correctly.

Invalid UTF-8 Sequence in Subversion Again

I remembered I encountered this issue before few months back in March. But yet, I can't remember how to solve it. My previous solution to detect the culprit filename with invalid encoding was using strace but this post recommended a better, more accurate approach to detect it. The detection is straight forward, just convert the hexadecimal code to ASCII.

Let's look at my previous example
$ svn up

svn: Valid UTF-8 data
(hex: 49 50 )
followed by invalid UTF-8 sequence
(hex: a0 2d 56 69)

My issue last time I did not fully grok the error message. What the svn client tried to tell me is there is a filename in hexadecimal sequence of 49 50 a0 2d 56 69 that is causing the corruption.

Convert this hexadecimal sequence to ASCII and the full culprit filename known as shown below.
$ echo "\x49\x50\xa0\x2d\x56\x69" | xargs -0 printf
IP�-Vi

What you can do right now is find any file that contains this sequence of characters of IP�-Vi and remove it.
$ ls IP�-Vi*
$ rm -rf IP�-Vi*

Whoala ! There you have it.

svn mergeinfo

We've slowly started to utilize Subversion branching and merging. Previously, due to some policy beyond my control, the project was branched, merged, and reviewed manually through me. But luckily these days, we've slowly started to truly utilize actual Subversion branching and merging. It will take a while for all of us to familiarize and settle down on such development style.

Several commands regarding branching and merging.

1. To create a branch
$ svn cp ^/trunk ^/branch/feature-x

2. To visualize the history of branching
$ svn mergeinfo ^/trunk ^/branch/feature-x

    youngest common ancestor
    |         last full merge
    |         |        tip of branch
    |         |        |         repository path

    100               120   
    |                  |      
  -------| |------------         trunk
     \                        
      \                       
       --| |------------         branches/feature_x
                       |      
                       120 

3. To merge from trunk to feature branch
$ svn merge ^/trunk ^/branches/feature-x

4. To check what have been merged into your branch
$ svn mergeinfo ^/trunk ^/branches/feature-x --show-revs merged

5. To check what have NOT been merged into your branch
$ svn mergeinfo ^/trunk ^/branches/feature-x --show-revs eligible

Create a .deb Package from Source Code

Due to the Subversion PPA still stuck at version 1.7.9. In order to use Subversion client version 1.8.3, I need to upgrade by source code compilation. As I mentioned before, I really don't like this installation method. Fortunately, I've found this tool called checkinstall, which let you create a software packages (deb or rpm compatible) so you can remove it later.

We will illustrate by compile and install the latest version of SQLite, a lightweight database management system.

1. First, download the latest SQLite source code. We're using the autoconf version. Later, extract the tarball.
$ wget https://www.sqlite.org/2013/sqlite-autoconf-3080002.tar.gz
$ tar zxvf sqlite-autoconf-3080002.tar.gz

2. After that, we need to install all the necessary software or libraries in order to compile it. apt-get has a wonderful option call build-dep which will install all these dependencies so you can build the software.
$ sudo apt-get build-dep sqlite3

3. Go into the source code folder and generate the Makefile.
$ cd sqlite-autoconf-3080002
$ ./configure

4. Before we compile the source code, let's install the checkinstall package.
$ sudo apt-get install checkinstall

5. Compile the source code and create a .deb package. Make sure you pass -j option [5] to   to speed up the compilation time. Rules of thumb is twice the number of your CPU cores. For my case, my lappy has 4 CPU cores using the nproc command. Hence, we should use -j8 option.
$ make -j8 && sudo checkinstall

6. Please take note that checkinstall will create a package and install it immediately. Buy you can remove it.
$ sudo dpkg -r sqlite-autoconf

7. Check the content the the deb package file. It will show you a list of files to be installed.
$ dpkg --contents sqlite-autoconf_3080002-1_amd64.deb

Revisit Subversion Branching and Merging Again

Was reading this presentation on version control in #Subversion . Relearning the whole branching and merging flow again. Due to some constraints (access right, data sensitivity, and my pure laziness as well), our flow is a bit effed up. No one but myself to blame since I am responsible for syncing all the works by different developers.

According to the slide, the flow is as follows:

1. Create a branch from trunk and commit.
$ svn cp ^/trunk ^/branches/mybranch
$ svn co ^/branches/mybranch

2. Inside your own branch, mybranch in this case, you need to keep in sync with the trunk.
$ svn merge ^/trunk
$ svn co

3. Later, once the features in mybranch is completed, you will need to reintegrate to the trunk. Inside a clean copy of trunk, run these commands.
$ svn merge --reintegrate ^/branch/mybranch
$ svn co

Once integrated, the branch is basically end-of-life. No more modification to it. This is one step which I did differently, basically we have a stable branch which actually act like trunk. Why ? I spend most of my time in stable branch and hardly touch trunk. That why.

However, the --reintegrate option has deprecated in version 1.8. Subversion will automatically decide a reintegration merge or not.

4. One integration is done you'll need to tag it.
$ svn cp ^/[email protected]^/tags/mytag
$ svn co

5. Rinse and repeat.

There is still one question lingering in my head right now. How do I prevent certain files from being modified during merging ? More on this in next post perhaps.

Back to the slides. One particular slide echo my sentiment about the practice of using Subversion or any version control. You need to commit small, commit early, and commit often for each small task. It keeps the momentum going and motivate the team to move forward. This was further enhanced by our practice of Kanban scheduling methodology and Podomoro time management technique. Our development process is getting better but there still room for improvement.

One thing for sure, I need to move the Subversion to a new server. The installation (by yours truly) is seriously way effed up. Should I just migrate to Mercurial or Git? Or maybe I should just use hgsvn or git-svn instead?

Common Commit Messages Format

Inspired by the recent change log of nginx, a HTTP and reverse proxy server. We have standardized on a common format for all commit messages to our Subversion repository. All commit messages are prefixed by these keywords:

- Change : If we modify the default behaviour of the system
- Feature : Add new stuff to the code base
- Bugfix : Fix existing issues with ticket id or example

Also, the title of our todo items or user stories (if you like to call it that way) also follow the same naming convention. Just for the sake of consistency. Why? To help us summarise and create weekly changelog in a readable and consistent format easily.

Unfortunately, we're still stuck in the Subversion mindset and can't commit and merge as often as we like like the other Distributed Version Control System (DVCS) like Git or Mercurial. Actually you can, just create a feature branch and commit as often to it and summarize your changes during merging.

Next steps ? Increase the test coverage and improve the schedule estimation.

Invalid UTF-8 sequence in Subversion

Encountered this when I tried to update the local working copy. Error message obtained as follow:
$ svn up

svn: Valid UTF-8 data
(hex: 49 50 )
followed by invalid UTF-8 sequence
(hex: a0 2d 56 69)

No idea which file(s) are causing this. Google around for quick solution. Found that you will need to go into every folder and type svn info to trace down the culprit file. Not a good solution.

Second approach. Try to limit down the scope. Using svn log -v to trace back recently committed files.

Google around for answer. Suspect some of the committed files was encoded in different charset. Checked using file -i command as shown. Not helpful at all.
$ file -bi config.php

text/x-c++; charset=us-ascii

Google again. Interesting solution proposed by cooper,
"strace svn status will give you the name of the offending file. unfortunately, svn care about name of files that are in one of its directories, even if it’s not under revision."
Try again using strace, an utility to monitor system calls [4]. Whola, offending file highlighted in bold.
$ strace svn status

*open("pdf", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 3*
fstat(3, {st_mode=S_IFDIR|0777, st_size=278528, ...}) = 0
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
getdents(3, /* 38 entries */, 4096)     = 4088
write(2, "svn: Valid UTF-8 data\n(hex: 49 5"..., 122svn: Valid UTF-8 data
(hex: 49 50)
followed by invalid UTF-8 sequence
(hex: a0 2d 56 69)
) = 122
close(3) 

It seems the offending files were located in the pdf folder. Remove that folder and try svn status again.

Whoa ! Problem solved. The root of the problem was related to certain PDF files generated with unsupported enconding file names.

Seriously, I fricking love strace!

Subversion Server Installation and Setup in CentOS 5.x

Is so fricking hard to find latest greatest official packages for CentOS. For example, the latest version of Subversion in CentOS 5.8 is 1.6.11. If you want the 1.7.x version, you have to either download from third party repository like wandisco, upgrade to CentOS 6.x, or just compile own your own. I always wary with unofficial rpm packages especially those that you can’t verify. Unless I need to set up something fast or just plain lazy, I will just go ahead and download these rpms. Direct upgrade from 5.x to 6.x is not supported and it is advisable to do a fresh install. Compile from source, been there, done that, not sure what I am doing. While I understand the CentOS team is severely lacking resources, how I wish there is something similar to Ubuntu PPA for CentOS.

Enough ranting. Let’s proceed with the installation. The installation steps are based on two very helpful guide, the official and unofficial. As I am currently stuck with 5.X and I badly need the centralized metadata storage (just one .svn hidden folder) in 1.7.X.

Download all the necessary packages and install them. Make sure you have remove the existing subversion packages. All command is run as root.
$ yum remove subversion*
$ yum install httpd
$ rpm -ivh subversion-1.7.7-1.x86_64.rpm
$ rpm -ivh mod_dav_svn-1.7.7-1.x86_64.rpm
$ rpm -ivh subversion-perl-1.7.7-1.x86_64.rpm
$ rpm -ivh subversion-tools-1.7.7-1.x86_64.rpm
$ rpm -ivh subversion-python-1.7.7-1.x86_64.rpm

Backup the original Subversion Apache module configuration file.
$ cd /etc/httpd/conf.d
$ cp subversion.conf subversion.conf.orig

Edit the configuration file (subversion.conf) as follows:
LoadModule dav_svn_module     modules/mod_dav_svn.so
LoadModule authz_svn_module   modules/mod_authz_svn.so

DAV svn
SVNListParentPath on
SVNParentPath /repo

SSLRequireSSL
AuthzSVNAccessFile /repo/acl
AuthType Basic
AuthName "Repositories"
AuthUserFile /repo/passwd
Require valid-user

Create the repository folder.
$ cd /
$ mkdir repo
$ cd repo

Create the password (passwd) with a sample user (melanie).
$ htpasswd -m passwd melanie
New password:
Re-type new password:
Adding password for user melanie

Create a sample ACL file with content of
[/]
melanie = rw

Create a sample project and change all folder permission to the apache user.
$ svnadmin create sample
$ chown apache.apache -R /repo
$ service httpd restart

Browse to https://localhost/repo/ or https://localhost/repo/sample. The server should prompt you for username and password.

Upgrade Subversion Client to Version 1.7 in Ubuntu 12.04

Before upgrade your local working copy, you’ll need to clean up your working directory to prevent any errors during upgrade.
$ svn cleanup .

Upgrade your Subversion client from 1.6.x to 1.7.x in Ubuntu 12.04
$ sudo apt-add-repository ppa:svn/ppa
$ sudo apt-get update
$ sudo apt-get dist-upgrade
$ svn --version
svn, version 1.7.5 (r1336830)
   compiled Jun 26 2012, 22:39:53

Upgrade your local working copy
$ svn upgrade .
$ tree -L 1 .svn
.svn
├── entries
├── format
├── pristine
├── tmp
└── wc.db

Didn’t realize this until I read in the [documentation][3],
"However, some of the new 1.7 features may not be available unless both client and server are the latest version. There are also cases where a new feature will work but will run less efficiently if the client is new and the server old."
Now I understand why my local working copy does not has version 1.7 features. Both client and server must has the latest version. Question now is how to upgrade those ancient old client in CentOS ?

Subversion Keyword Substitution

I don't know when and why this this started but it has become an old habit of mine when I inherit any legacy projects.

Put all $Id$ tag at head of the file after the <?php opening tag. Note that the keyword Id is a substitution for Revision, Date, and Author.
<?php
// $Id$

Set keywords property to all php files
$ svn -R propset svn:keywords "Id" *

Commit the property changes and update local copy
$ svn commit -m "set keyword property" *
$ svn up

See the result
$ cat test.php

<?php
// $Id: test.php 12 2011-08-28 13:09:48Z foobar $

How to Properly Set External Definition Using Subversion

$ svn propset svn:externals "local_folder http://remoteip/project_name" .
$ svn up
$ svn commit -m "Add external library link"

Two important points,
  1. You must quote the property value
  2. The dot (.), which is referring to the local directory, at the end of the command IS NECESSARY