manandbytes

Just clone it!

manandbytes — Wed, 29 Jun 2011 20:45:00 +0000

Some real-life example: someone had a problem changing a default port for Jenkin’s HTTP connector. I already had a similar system available building couple projects. While reading that post I’ve realised that it will cost me almost nothing to help this person and check out if this problem exist. How this could be that simple?

You can use complex systems. Locally!

First, because I run Jenkins locally. You can check out Dave Carver’s post Local Continuous Integration with Hudson and Git (sorry, I failed to find a link to original post on his blog) why and how to setup such environment. Here is his another post on how such setup affects your development workflow. Distributed CI: How it could work « Quality Software Development with Ease has a lot of details how to deal with distributed builds. My setup is slightly different as I use a vitrual machine to run Jenkins, with a Git, Subversion and some other tools like several versions of JDK etc. And all this stuff is completely independent from my main development tools.

You can freely experiment with such systems

Second, because running on a virtual machine gives my some other benefits like being able to pause it or clone at any moment. And I did exactly this: paused running VM, cloned it and started it back. Now I was able to experiment with a completely working system. It actually took about 15 minutes during a lunch to verify this without any risk to break my working system.

Summary

Here are two more reasons why using virtualization even on something that small as Lenovo ThinkPad X201 Tablet makes sence. You may use this system full-time. Or only from time to time. It may be critical for your daily activity. Or completely irrelevant and nothing will happen if it will be unavailable for a week. The main point while using virtualization is: it doesn’t matter!

Don’t know how to move to Git?

manandbytes — Fri, 29 Apr 2011 18:39:00 +0000

We started out with SVN mostly because that was what everyone was used to. A year ago or so, one of our remote developers started using a git-svn mirror to remove some of the latency issues they had with SVN. Someone else put together a ten page wiki on how to easily use a git-svn mirror and push back to the canonical SVN repo. As time went on, interest in git grew and new modules were created in git, instead of our SVN repositories. Finally there was a push to move everything to git.

It’s not a problem for a new codebase, but preserving all history (with authors as well) may be critical for existing ones. If you are not sure how to convert your existing SVN repositories into Git, Goodbye SVN, Hello Git from Sonatype provides a background why they moved to Git and a step-by-step instructions.

Why -source and -target not a convention

manandbytes — Wed, 27 Apr 2011 10:00:00 +0000

Convention over configuration is a simple concept. Maven incorporates this concept by providing sensible default behavior for projects. The Compiler Plugin is used to compile the sources of your project and provides some defaults for source/target levels for Java source code.

You may think that source/target levels is a part of this ‘convention over configuration’ concept. But actually they are not, so keep reading to learn why.

What problems to expect when building something that small as Kewin Sawicki‘s TimeAgo library with Maven? No, I’m not kidding since I’ve expected no problems at all but… Unfortunately, build of the current master branch fails:

[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Compilation failure

/org.github.timeago/src/main/java/org/github/timeago/TimeAgo.java:[129,23] cannot find symbol
symbol  : method format(java.lang.String,long)
location: class java.text.MessageFormat
...
[INFO] ------------------------------------------------------------------------

There are 5 such problems related to java.text.MessageFormat‘s method format(String pattern, Object... arguments). This method uses the varargs feature introduced in Java5, as well as the method itself, so the problem is obvious: Maven tries to compile TimeAgo with pre-Java5 JDK.

How is it posible?! It was taken for granted that Maven allows to avoid such problems but, for some reason, I can’t build it as is.

Here we have a problem, that Martin Fawler explained some time ago:

One of the prevailing assumptions that fans of Continuous Integration have is that builds should be reproducible. By this we mean that at any point you should be able to take some older version of the system that you are working on and build it from source in exactly the same way as you did then.

Let’s digg into some details and try to find out why it’s unreproducible and how easy it will be to find a configuration that ‘just works’.

Problem: unreproducible build

So, right now I have 3 different JDK installed:

usr/lib/jvm/java-6-openjdk
usr/lib/jvm/java-1.5.0-sun-1.5.0.22
usr/lib/jvm/java-6-sun-1.6.0.24

Lets add 4 versions of Maven2 to the mix:

Apache Maven 2.1.0 (r755702; 2009-03-18 21:10:27+0200)
Apache Maven 2.2.1 (rdebian-4)
Apache Maven 3.0.2 (r1056850; 2011-01-09 02:58:10+0200)
Apache Maven 3.0.4-SNAPSHOT (rNON-CANONICAL_2011-04-06_23-20_mn; 2011-04-06 23:20:51+0300)

and try to build TimeAgo with mvn clean compile using bash-matrix-project approach described earlier:


Apache Maven 2.1.0 (r755702; 2009-03-18 21:10:27+0200)	FAIL
Apache Maven 2.2.1 (rdebian-4)	FAIL
Apache Maven 3.0.2 (r1056850; 2011-01-09 02:58:10+0200)	SUCCESSFUL
Apache Maven 3.0.4-SNAPSHOT (rNON-CANONICAL_2011-04-06_23-20_mn; 2011-04-06 23:20:51+0300)	SUCCESSFUL

As build fails for Maven 2.x but succeeds for Maven 3.x, regardless of JDK been used, definetely, it’s not reproducible. Let’s try to find the reason for such strange behavior.

Why?

Fire build again but this time we’ll set source/target levels explicitly, via command-line options -Dmaven.compiler.source=1.5 and -Dmaven.compiler.target=1.5 and it succeeds for all environments.

#1: missing source/target levels in POM

Looking at the project’s POM, we see nothing related to compiler plugin’s settings. This way build depends on default values for source and target levels implemented in currently effective version of the compiler plugin. I strongly disagree with comments like

Maven has a strong preference for «convention over configuration» which means that requiring the source and target entries in the pom is not really reasonable IMO. Instead, the defaults should be documented better, and perhaps the error message could be adjusted to point to a FAQ entry that explains things for new users.

on MCOMPILER-57. Comments on MCOMPILER-80 like

Using default source/target levels in a build tool is not a good idea since it makes both the success of the build and its artifacts dependent on the environment in which it was run, contrary to the goals of a reproducible build. A given module or tree of modules will have a certain source level it requires in order to compile, so increments to this in the POM should be coversioned with source code changes to use new language features. Target level (usually but not necessarily the same as source level) is even more important to specify explicitly, because it is not obvious when you get it wrong – until your bytecode fails to load on an older customer JVM which you had intended to still support.

or this one

Just that, the source/target levels should be decided by its owners/developers, NOT BY Maven. It’s the project owner/developer’s responsibility to decided the expected source/target levels and set the values in the project POM. Thus to have reproducible builds across computers/jdks.

reflect a reality much better.

Maven2 help plug-in is not that helpful

I tryed to inspect the project’s effective POM with maven-help-plugin, as you may find answers like this. For Maven 2.x it’s quite small, due to some bug in Maven2 and not in plugin:


  
    
      maven-compiler-plugin
      2.0.2

This bug has been reported as fixed in Maven 3.x and shouldn’t affect maven-help-plugin:


  
    
      maven-compiler-plugin
      2.3.2
      
        
          default-testCompile
          test-compile
          
            testCompile
          
        
        
          default-compile
          compile
          
            compile

Yup, there are more deatils provided but still no effective settings for source and target. Since it doesn’t show maven-compiler-plugin’s effective source and target levels, don’t put your trust in maven-help-plugin.

#2: undefined version of the Maven Compiler Plugin

But which source and target options are effective right now? There are two problems with plugin’s version resolution. The first one, from Maven 3.x Compatibility Notes:

When a plugin was invoked without an explicit version given in the POM or on the command line, Maven 2.x used to pick the latest version available where the latest version could either be a release or a snapshot.

And the second, the default source and target levels are moving targets. According to our experiment, we actualy depend on two different versions of compiler plugin (2.0.2 and 2.3.2), declared for already released versions of Maven2. But what about intermediate versions and requests like change to 1.4, to 1.5 or even refuse building if they are missing?

Unspecified plugin’s version is as bad as a missing source level and is finally considered a bad-practice and scheduled for removal since Maven3.

Solutions

The best one

Source code’s source and target levels are not a convention, it’s a configuration (like groupId, artifactId or your packages’ name). You MUST specify them explicitly, it’s not a Maven’s (or any other tool’s) responsibility. So just specify them:

diff --git a/org.github.timeago/pom.xml b/org.github.timeago/pom.xml
index 1a3fe35..76157c5 100644
--- a/org.github.timeago/pom.xml
+++ b/org.github.timeago/pom.xml
@@ -27,6 +27,14 @@
    
      
        org.apache.maven.plugins
+        maven-compiler-plugin
+        
+          1.5
+          1.5
+        
+      
+      
+        org.apache.maven.plugins
        maven-jar-plugin

Even without specifying maven-compiler-plugin’s version, such small change makes maven-help-plugin more useful in both Maven2 and Maven 3 as well.

Pin plugin’s version

With Maven Enforcer plugin you can cope with a second problem and fail a build if any plugin don’t have a version:

diff --git a/org.github.timeago/pom.xml b/org.github.timeago/pom.xml
index 6701f31..dd7b3e4 100644
--- a/org.github.timeago/pom.xml
+++ b/org.github.timeago/pom.xml
@@ -61,6 +61,24 @@
          
        
      
+      
+        org.apache.maven.plugins
+        maven-enforcer-plugin
+        1.0
+        
+          
+            enforce-plugin-versions
+            
+              enforce
+            
+          
+        
+        
+          
+            
+          
+        
+

After this change, your build will fail if some plugins are missing valid versions, with a clear reason:

[INFO] ------------------------------------------------------------------------
[INFO] Building Time Ago
[INFO]    task-segment: [test-compile]
[INFO] ------------------------------------------------------------------------
[INFO] [enforcer:enforce {execution: enforce-plugin-versions}]
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequirePluginVersions failed with message:
Some plugins are missing valid versions:(LATEST RELEASE SNAPSHOT are not allowed )
org.apache.maven.plugins:maven-clean-plugin.    The version currently in use is 2.3
org.apache.maven.plugins:maven-resources-plugin.        The version currently in use is 2.3
org.apache.maven.plugins:maven-deploy-plugin.   The version currently in use is 2.5
org.apache.maven.plugins:maven-compiler-plugin.         The version currently in use is 2.0.2
org.apache.maven.plugins:maven-install-plugin.  The version currently in use is 2.3.1
org.apache.maven.plugins:maven-surefire-plugin.         The version currently in use is 2.7.2
org.apache.maven.plugins:maven-site-plugin.     The version currently in use is 2.0
org.apache.maven.plugins:maven-jar-plugin.      The version currently in use is 2.2

[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed.

However, you still have to be careful when choosing versions. Using wrong version, you may, by accident, narrow down a range of the tools available to your consumers.

For instance, one of the real projects I know about, is a Flex application that requires some specific version of a plugin to be used and that very version works with an exact version of Maven 2. So using any plugins that require anything higher than Maven 2.0.8, IIRC, is a ‘no-no’ for this project.

Recent changes in Jenkins is another example. It started using com.cloudbees.maven-license-plugin to check license headers in source files. Initially this plugin required Maven 3.x and introducing this dependency to Jenkins made it impossible to build Jenkins with Maven 2.x for no apparent reason. Fixing this problem, in this particular case, was easy for two reasons:

the author, Kohsuke Kawaguchi, has commit rights to both codebases;
it was easy to fix in both projects but required a new version of plugin to be released.

But this not true for most projects. I would rather be more consvervative in such situations and not use the latest (but not always greatest) versions without clear benefits.

Summary

One apple a day… When it refers to the Maven compiler plugin, you:

must explicitly define source and target levels
should specify version of plugin (but please, be sane)

These small changes will make your builds much more stable and resistant to changes in environments while obeying its contracts like ‘I’m Java5-compatible’.

Installing Apache Harmony on Debian

manandbytes — Sun, 17 Apr 2011 18:05:00 +0000

What’s Apache Harmony? It’s an open source Java SE implementation with an unclear future. But, please, don’t ask me ‘Why installing it?’ Because developers are curious creatures

I don’t like to install a software distributed as a tar.gz archives. If it is possible, I’ll use my distribution’s native packages. It turned out that Apache Harmony project provides prebuilt packages for Debian/Ubuntu. So you have to download packages one by one and just install with dpkg --install harmony-5.0-jre_0.0r991518-1_i386.deb.

There is nothing wrong if you’re interested in a one-time installation but there are manual actions required when (and if) Harmony project provides updates:

you have to be notified somehow about the fact that a new version is available;
you have to download a number of packages one by one;
you may have to (semi)manually resolve missing dependencies.

As my motto says,

less work for humans, more work for machines. just as it should be

so let my machines do some more work

Make packages’ location known to APT

Add these two lines to /etc/apt/sources.list:

deb http://mirrors.ibiblio.org/pub/mirrors/apache/harmony/milestones/6.0/debian/i386/ ./
deb http://mirrors.ibiblio.org/pub/mirrors/apache/harmony/milestones/5.0/debian/i386/ ./

and update the list of available packages with aptitude update.

For most Apache projects, all links on the download page point to the best mirror site based on your location, and Harmony’s Project Downloads page is no different. It suggested http://apache.vc.ukrtel.net/ as the best mirror site for my current location and, unfortunately, files mirrored to /harmony/milestones/6.0/M3/debian/i386. Chances are that the next milestone builds will be stored in other directory like /harmony/milestones/6.0/M4/debian/i386 and I have to touch sources.list. I think that layout provided by mirrors.ibiblio.org doesn’t have these drawbacks but only time will tell.

Install them

Installation is as simple as running aptitude install harmony-5.0-hdk harmony-6.0-hdk.

Check what’s installed

Let’s check what we have for main two executables, update-alternatives --display java:

java - manual mode
  link currently points to /usr/lib/jvm/java-6-sun/jre/bin/java
/usr/bin/gij-4.4 - priority 1044
/usr/bin/gij-4.5 - priority 1045
/usr/bin/java-harmony - priority 30
  slave java.1.gz: /usr/share/man/man1/java-harmony.1.gz
/usr/bin/java-harmony-6.0 - priority 30
  slave java.1.gz: /usr/share/man/man1/java-harmony-6.0.1.gz
/usr/lib/jvm/java-1.5.0-sun/jre/bin/java - priority 53
  slave java.1.gz: /usr/lib/jvm/java-1.5.0-sun/jre/man/man1/java.1.gz
/usr/lib/jvm/java-6-openjdk/jre/bin/java - priority 1061
  slave java.1.gz: /usr/lib/jvm/java-6-openjdk/jre/man/man1/java.1.gz
/usr/lib/jvm/java-6-sun/jre/bin/java - priority 63
  slave java.1.gz: /usr/lib/jvm/java-6-sun/jre/man/man1/java.1.gz
Current 'best' version is '/usr/lib/jvm/java-6-openjdk/jre/bin/java'.

and update-alternatives --display javac:

javac - manual mode
  link currently points to /usr/lib/jvm/java-6-openjdk/bin/javac
/usr/bin/ecj - priority 143
  slave javac.1.gz: /usr/share/man/man1/ecj.1.gz
/usr/bin/gcj-wrapper-4.5 - priority 1045
/usr/bin/javac-harmony - priority 30
  slave javac.1.gz: /usr/share/man/man1/javac-harmony.1.gz
/usr/bin/javac-harmony-6.0 - priority 30
  slave javac.1.gz: /usr/share/man/man1/javac-harmony-6.0.1.gz
/usr/lib/jvm/java-1.5.0-sun/bin/javac - priority 53
  slave javac.1.gz: /usr/lib/jvm/java-1.5.0-sun/man/man1/javac.1.gz
/usr/lib/jvm/java-6-openjdk/bin/javac - priority 1061
  slave javac.1.gz: /usr/lib/jvm/java-6-openjdk/man/man1/javac.1.gz
/usr/lib/jvm/java-6-sun/bin/javac - priority 63
  slave javac.1.gz: /usr/lib/jvm/java-6-sun/man/man1/javac.1.gz
Current 'best' version is '/usr/lib/jvm/java-6-openjdk/bin/javac'.

What about more deatiled verson information? /usr/bin/java-harmony -version reports:

Apache Harmony Launcher : (c) Copyright 1991, 2010 The Apache Software Foundation or its licensors, as applicable.
java version "1.5.0"
Apache Harmony (1.5.0)
DRLVM (1.5.0-r991518)
pre-alpha : not complete or compatible
svn = r991518, (Sep  7 2010), Linux/ia32/gcc 4.3.2, release build

http://harmony.apache.org

and /usr/bin/java-harmony-6.0 -version:

Apache Harmony Launcher : (c) Copyright 1991, 2010 The Apache Software Foundation or its licensors, as applicable.
java version "1.6.0"
Apache Harmony (1.6.0)
DRLVM (1.6.0-r991881)
pre-alpha : not complete or compatible
svn = r991881, (Sep  8 2010), Linux/ia32/gcc 4.3.2, release build

http://harmony.apache.org

Surprises

No alternatives for java-harmony nor javac-harmony
What is the Debian alternatives system:

The Debian alternatives system creates a way for several programs that fullfill the same or similar functions to be listed as alternative implementations that are installed simultaneously but with one particular implementation designated as the default.

It’s a minor issue, though.

Both Harmony 5 and 6 use the same Java compiler
/usr/bin/javac-harmony -version and /usr/bin/javac-harmony-6.0 -version report exactly the same version of the compiler:

Eclipse Java Compiler 0.972_R35x, 3.5.1 release, Copyright IBM Corp 2000, 2009. All rights reserved.

It seems to me as a quite serious issue, but what roadblocks to expect when cross-compiling and using different versions of compilers is a topic for another post.

Summary

I don’t have enough time to play with, so there are not so much to say, except: Apache Harmony installs, works in some cases, and may be uninstalled with ease.

How to emulate Jenkin’s ‘matrix build’ in command line ad-hoc

manandbytes — Thu, 07 Apr 2011 08:03:00 +0000

Why?

Jenkins presents a concept of the multi-configuration project (AKA matrix project). This feature is useful when you would like to test your application in several environments (with different versions of JDK) or package it for different platforms.

But what if you can’t use Jenkins for some reason? Scriptable shell comes to the rescue and here is a real-life example: I have to investigate an issue building a project in different environments (actually, for a combination of several versions of Maven2 and a number of different JDKs).

How?

So, I need two «axes» for my task:

version of Maven2. There are only 4 versions to run my tests against and I just enumerate all installations;
version of JDK. Thanks to a great package management system available in Debian, I still have Sun’s Java5 JDK installed. Not as a default one, however And Sun’s JDK 6 alongside OpenJDK 6… And some other are installed and removed occasionally. So I will just use find utility instead of enumerating all my JDK installations.

Solution

for m in \
    ~/bin/apache-maven-2.1.0/bin/ /usr/share/maven2/bin/ ~/bin/apache-maven-3.0.2/bin/ ~/bin/apache-maven-3.0-SNAPSHOT/bin/ ; do \
    for j in `find /usr/lib/jvm/ -maxdepth 2 -type d -name bin` ; do \
        echo "-----" ; \
        export JAVA_HOME=$j/../ ; \
        $m/mvn -version ; \
        $m/mvn clean compile --quiet ; \
    done \
done

Once executed, this runs given project using a) every version of Maven2 and b) for every Maven2 version using every JDK available. Output looks like:

-----
Apache Maven 2.1.0 (r755702; 2009-03-18 21:10:27+0200)
Java version: 1.6.0_22
Java home: /usr/lib/jvm/java-6-openjdk/jre
Default locale: uk_UA, platform encoding: UTF-8
OS name: "linux" version: "2.6.38-2-686-bigmem" arch: "i386" Family: "unix"
[ERROR] BUILD FAILURE
-----
Apache Maven 2.1.0 (r755702; 2009-03-18 21:10:27+0200)
Java version: 1.5.0_22
Java home: /usr/lib/jvm/java-1.5.0-sun-1.5.0.22/jre
Default locale: uk_UA, platform encoding: UTF-8
OS name: "linux" version: "2.6.38-2-686-bigmem" arch: "i386" Family: "unix"
[ERROR] BUILD FAILURE
-----
Apache Maven 2.1.0 (r755702; 2009-03-18 21:10:27+0200)
Java version: 1.6.0_24
Java home: /usr/lib/jvm/java-6-sun-1.6.0.24/jre
Default locale: uk_UA, platform encoding: UTF-8
OS name: "linux" version: "2.6.38-2-686-bigmem" arch: "i386" Family: "unix"
[ERROR] BUILD FAILURE

Mission accomplished.

The Eight Fallacies of Distributed Computing are going to vanish

manandbytes — Sat, 19 Mar 2011 07:51:00 +0000

The http://www.sun.com site will be decommissioned on June 1 of this year.

…according to OTN Garage, an official blog of the System Admin and Developer Community of the Oracle Technology Network. They promise that the content in blogs.sun.com will remain available. I’m not sure they will put redirects in place so that all the thousands of links across the web to blogs.sun.com remain functional. Actually, they wouldn’t as straight redirects cost so much money and their IT budget is tight. More proof that Oracle is only interested in saving a few pennies a year.

Oracle willingly saying «we don’t care» about this history of work that exists in the sun.com domain and The Eight Fallacies of Distributed Computing by Peter Deutsch is one of them:

Essentially everyone, when they first build a distributed application,
makes the following eight assumptions. All prove to be false in the
long run and all cause big trouble and painful learning experiences.

The network is reliable

Latency is zero

Bandwidth is infinite

The network is secure

Topology doesn’t change

There is one administrator

Transport cost is zero

The network is homogeneous

Отношение к фреймворкам: профи против… не-профи

manandbytes — Sat, 12 Mar 2011 09:07:42 +0000

Общеизвестно, что all frameworks sucks. Одни больше, другие… Или это зависит от персонального отношения?

Например, Алексей Тутубалин в О Qt и рисовании прямоугольничков:

Получается, все используемые типы объектов, которым это надо (штук пять) нужно сабклассить, переопределять в них один метод, добавлять еще парочку (get/setPainterCompositionMode), то бишь практически одинаковый код в пяти местах.

По моему опыту с Qt, это означает, что я что-то не заметил и это место уже должно быть украдено до меня…..

Jenkins is alive!

manandbytes — Sun, 30 Jan 2011 11:58:22 +0000

There is no more Hudson logo in Jenkins since this Kohsuke’s commit. However, Hudson’s logo in SVG format is still there.

Hudson is dead. Long live Jenkins!

manandbytes — Sat, 29 Jan 2011 22:00:34 +0000

Who’s driving this thing? Community, I hope. Andrew Bayer just posted the vote results on renaming Hudson to Jenkins (more details). What does this mean for me personally?

So what does this mean now? Well, it means Jenkins lives.

Well, no more «hudson» in my vocabulary, not a big deal

we will be renaming the existing Google Groups

Four mail filters should be updated in GMail, not a problem.

and renaming our organization at Github from hudson to jenkinsci.

Well, I’ve forked Hudson/Jenkins and some plugins on Github. Will Github rename base repositories or have I drop and fork them again? For sure, I’m on my own with my local repositories, I have to change couple of git remotes.

Hudson is dead. Long live Jenkins!

manandbytes

Just clone it!

You can use complex systems. Locally!

You can freely experiment with such systems

Summary

Don’t know how to move to Git?

Why -source and -target not a convention

Problem: unreproducible build

Why?

#1: missing source/target levels in POM

Maven2 help plug-in is not that helpful

#2: undefined version of the Maven Compiler Plugin

Solutions

The best one

Pin plugin’s version

Summary

Installing Apache Harmony on Debian

Make packages’ location known to APT

Install them

Check what’s installed

Surprises

Summary

How to emulate Jenkin’s ‘matrix build’ in command line ad-hoc

Why?

How?

Solution

More on the fallacies of distributed networking

The Eight Fallacies of Distributed Computing are going to vanish

Отношение к фреймворкам: профи против… не-профи

Jenkins is alive!

Hudson is dead. Long live Jenkins!