Karol Stasiak’s Blog

GraalVM’s native-image on Windows 10 x64 – guide and unscientific benchmarks

2019-05-12T19:20:00+00:00

Short-lived programs on JVM have a huge disadvantage – they start slowly and a huge part of their code runs interpreted before the JIT compiler kicks in. The solution is to compile the program to native code.

GraalVM is a new polyglot virtual machine, but it also comes with a decent native compiler for JVM programs, called native-image. My initial tests showed that it is worth the hassle for short-lived programs, like compilers and other command-line tools.

I compiled the Millfork compiler with native-image and got huge speedups and the resulting executable runs on Windows out of the box. Here is how I did it.

Prerequisites

I will assume that your program is of size similar to the Millfork compiler, which is about 11 MB. You will need the following:

a machine running your target operating system with at least 16 GB of RAM – no, this is not an exaggeration (I used Windows 10 x64)
a machine running Linux (it may be the same machine if you’re targeting Linux, or another machine; it can be less beefy, 2 GB of RAM should be enough)
copies of GraalVM and native-image on both those systems (on Windows, native-image comes with GraalVM; on Linux and macOS, you’ll need to install it using the gu utility); get it here: https://github.com/oracle/graal/releases
a fat jar with your program, compatible with Java 8 (your program should run with just java -jar myprogram.jar; GraalVM doesn’t support newer Javas yet)
on Linux or macOS: a relatively recent GCC
on Windows: a copy of Windows 7 SDK (see instructions below)

I will assume that your GraalVM is installed in /graalvm on both machines and native-image is available.

Installing Windows 7 SDK (Windows only)

This is the tricky part. I have only done this on Windows 10 x64, so I will only list the steps necessary on that platform. Other platforms may work differenly:

download the SDK file GRMSDKX_EN_DVD.iso from Microsoft
mount the ISO image, as for example E:
uninstall Microsoft Visual C++ 2010 Redistributables – yes, this is necessary!
run E:\Setup\SDKSetup.exe
follow the instructions, install at least the headers, the compilers and the redistributables

Analysing the program

First, you need to figure out which objects in your program are accessed via reflection and which resources in the jar are loaded. To figure this out automatically, run your program on your Linux machine with the following options:

/graalvm/bin/java -agentlib:native-image-agent=config-output-dir=/path/to/config-dir/ -jar myprogram.jar

where /path/to/config-dir/ is a path to a directory where GraalVM should store data gathered during the run. The run should cover as much code paths as possible. If necessary, you can run the program again and merge the results, using:

/graalvm/bin/java -agentlib:native-image-agent=config-merge-dir=/path/to/config-dir/ -jar myprogram.jar

(Note “merge” instead of “output” in the option name.)

The generated output will be several JSON files. Inspect them and manually add missing entries to jni-config.json, reflect-config.json, proxy-config.json and resource-config.json (the file names are links that lead to the official documentation).

Building the native binary

Put the config files into the META-INF/native-image directory in the resources of your program and rebuild the jar (this allows you to skip adding those files to the command line options for native-image). Files reflect-config.json and resource-config.json are obligatory, jni-config.json and proxy-config.json are needed only if they’re not empty.

On your target machine, free as much RAM as you can. If you’re using Windows, use the Windows SDK 7.1 Command Prompt instead of the normal command prompt.

In the command prompt, run:

/graalvm/bin/native-image -jar myprogram.jar

This step will take a while. Go make yourself a coffee.

You may see warnings about class locality, especially if the program was written in Scala. Ignore them.

If you see a message that native-image decided to create a fallback executable, then it means that you failed to provide the necessary config files. Unless you’re fine with shipping an entire JVM with your exe, abort the compilation, fix the config files and try again. I have not tested fallback executables, so I can’t provide any guidance on how to use them.

If the compilation succeeds, then congratulations, your program is now a standalone native binary. Next to your myprogram.jar you should see myprogram.exe (if on Windows).

Testing

You should now test that your executable works. You may have problems with class loading (which means non enough entries in reflect-config.json), reflection (ditto), resource loading (resource-config.json). If any problems occur, go back to the analysing step and try to add the missing entries, either automatically or manually.

Results

The Millfork compiler was a perfect target for native-image. It’s a program that runs through a set series of stages once and in a fixed order. The native binary ended up being 38 MB, which is smaller than even the smallest official JVM, and that’s without counting 11 MB of the original jar.

Speed increased. Launching the compiler without any command line options (which simply yields an error message) takes less than 0.1 seconds, while doing the same with a JVM takes about 1.9 seconds. Even a moderately long compilation task is faster with the native executable. Compiling Space Poker at -O1 takes about 1.0 seconds with the native executable and about 4.8 seconds with Java (either 8 or 11), and at -O4 it takes about 6.3 seconds and about 10–11 seconds respectively.

Overall, the experiment was a success. Future releases of Millfork will be shipped as both a cross-platform jar and a Windows executable.

Getting your Java 8 application to run on Java 11 – in 9 easy steps

2018-09-20T00:40:00+00:00

Recently, I had an opportunity to fix two applications to get them running on Java 10 while preserving compatibility with Java 8. Migrating to the module system was not on the table, the goal was just to run everything on a newer OpenJDK. It was not a trivial task: neither of programs worked on Java 10 out of the box. Here’s how I managed to fix the applications so the same binary code could run on Java 8, 10 and the release candidate version of Java 11.

Step 1 – try building/running your program with Java 10/11

Huge chances that it will fail to work. Huge chances it even fail to compile. Don’t worry, we’ll fix it later. Start by trying to run your automated tests on new Java (you do have them, right?) and see what’s failing. Keep analysing the errors after every fix you make.

Step 2 – remove unused or unimportant imports

Remove all unused imports, this includes imports in the .gradle files. There’s nothing more annoying than failing to compile a file that imports a nonexistent class which it doesn’t even use.

Also, if you’re just importing a random class from com.sun.* or javax.* to use just one static method, consider replacing it with an alternative. In our case, they were javax.xml.bind.DatatypeConverter#printHexBinary and javax.xml.bind.DatatypeConverter#parseHexBinary, which was used purely for debugging, so they were replaced with own implementations.

Step 3 – update your build tools to run against the new Java

We were using Gradle 3.x, which doesn’t work on new Java at all. To update it, you need to change the distribution-url property in your gradle-wrapper.properties, remove gradle-wrapper.jar and run ./gradlew --version to make it fetch the new version.

Optional: You may also want to remove old Gradle caches in your home directory (most likely at ~/.cache/gradle) to reclaim some disk space.

After your tools are updated, run the clean command of your build tool using the new Java. Keep running it from time to time whenever you suspect something weird is going on with the compiler.

Step 4 – add missing dependencies

Java 11 removes several packages from the standard Java distribution. In short, everything that starts with javax. is suspect. Here is what dependencies I had to add:

javax.annotation:javax.annotation-api
javax.validation:validation-api
javax.xml.bind:jaxb-api
com.sun.xml.bind:jaxb-core
com.sun.xml.bind:jaxb-impl
javax.xml.ws:jaxws-api
com.sun.xml.ws:jaxws-rt
com.sun.xml.ws:rt
javax.activation:activation

You might need a totally different set. For example, JavaFX is no longer a part of Java SE, so you might need to add that.

Don’t worry, adding those dependencies doesn’t cause trouble on Java 8.

Step 5 – check for name clashes between new Java classes and your classes

In our case, it was java.lang.Module vs com.fasterxml.jackson.databind.Module. The simplest solution was to use the fully qualified version of the latter.

Step 6 – fix all the other random compilation errors that might pop up

For some unexplained reason, a piece of code using a method handle as a parameter to Optional#map stopped wanting to compile. I just replaced it with a lambda:

     // worked on Java 8, but for some reason didn't on Java 11
     optional.flatMap(b -> foo(x, b)).map(Bar::getBarValue)

     // works on both Java 8 and 11
     optional.flatMap(b -> foo(x, b)).map(b -> b.getBarValue())

After this step, your program should build, but it might probably still work incorrectly.

Step 7 – update your dependencies

Anything that uses code generation, annotation processing, native code, reflection, and so on is the most likely suspect.

In our case, the main problem cause was Eclipselink. We were using the 2.6.x branch, but it does not support Java above 9. The solution was simply to replace org.eclipse.persistence:eclipselink:2.6.5 with org.eclipse.persistence:org.eclipse.persistence.jpa:2.7.3 and fix all the issues that popped up because of that.

Step 8 – run all automatic tests on both Java 8 and Java 11

I hope I didn’t surprise you with this one. Make it green for both.

Step 9 – deploy to one of your test environments and test it there

After doing all the fixes, I ran the application first locally, with some instances on Java 8 and some on 11, and when I verified it’s working, I pushed it to our test cluster which was a mix of Java 8 and Java 10. Everything worked fine.

Done. At least, that’s what it took me to do it, you might encounter some extra issues I luckily managed to dodge.

If you remember how easy was it to migrate from Java 7 to 8, this process felt much worse. I can really hope that most of it is just growing pains related to the module system and the new versioning scheme introduced in Java 9 and that it won’t be as problematic in the future. But that’s Oracle we’re talking about, who knows.

Adding support for page fragments to Google Analytics the quick and easy way

2017-12-02T23:40:00+00:00

By default, Google Analytics fires once per page load and therefore it only adds entries when the user navigates between different documents. But today, in the era of frequent single-page applications, this might not be enough. A web app might have multiple different sections, displayed conceptually from within the same document, and Google Analytics won’t track where the use is navigating.

In search of a solution for tracking user behaviour accross the document, I found several guides that required to set up the Google Tag Manager. But I couldn’t be bothered figuring yet another tool, so I decided to look for a more hacky and a simpler solution.

The solution is simple and as a prerequisite it requires that every change you wish to track causes a history event. Most SPA frameworks do that already to support the “back” button in browsers, so I’ll assume it works. I’m also not giving any guarantees about cross-browser support, but it works in Firefox and Chrome.

Just paste the following just after you Google Analytics code:

window.onpopstate = function(event) {
    ga('set', 'page', window.location.pathname + window.location.search + window.location.hash);
    ga('send','pageview');
};

(If you need to use more onpopstate events, you’ll need to join the handlers yourself.)

This way, every time a new history event is pushed onto the history stack, GA will send an event that contains both the document path, the query parameters and the fragment name (so, something like /document?query=parameters#fragment). If you don’t need some of those parts, or you want to track some more things, you can easily adapt the snippet for your needs.

Done! No need to learn about a new product and dive into unfamiliar territories, which would take hours, when a simple five-minute hack does the job.

A final detail: browsers differ on whether the event is fired on the initial page load, so you might want to take that into account.

Move from Octopress to Github-side Jekyll

2016-11-20T15:00:00+00:00

One of the main reasons I haven’t posted a lot on this blog is that the whole process of building the website was quite cumbersome and required a fully configured Ruby environment. Thankfully, Github offers its own Jekyll installation that automatically builds the website and allows people to submit files straight from the browser. So I decided to spend few hours and unshackle myself from the confines of my local and soon-to-be ditched Ubuntu 14.04 installation that hosted necessary tools. The result is, as you can see, a fully working and functionally-equivalent blog, but which I can edit in a browser. In fact, I’m writing this post in Firefox right now.

Of course, migration was not trivial. I had to make multiple changes:

I had to remove lots of plugins written in Ruby, since Github is not going to run arbitrary Ruby code.
Partials and layouts had to be modified to accomodate removed plugins and other differences between my old Octopress and current Jekyll.
I used Compass for compiling SCSS. Since Github doesn’t provide Compass integration, I just run it on my own computer and then upload compiled CSS.
The plugin I missed the most was category pages plugin, which I emulated following the guide at http://www.minddust.com/post/alternative-tags-and-categories-on-github-pages/
I rewrote the category cloud based on https://superdevresources.com/tag-cloud-jekyll/ but I adapted it to use categories as defined in the previous step. I also used the same method to inject category lists in other places.
I also fixed the Google search box, because if I make a query qith two q parameters and then get redirected from google.com to google.com.pl, then one of the parameters is ignored. I solved it by hardcoding the box to use google.co.uk.
I took some time to fix table styling, so that tables don’t look as hideous as they used to.

Incremental synchronization of ZIP files in bandwidth-constrained environments

2015-09-13T18:57:00+00:00

Suppose you have two ZIP archives on two different machines and you want to synchronize their contents with minimal network traffic, without much regard for the CPU usage, and you know that the target machine already contains the older version of the archive, which is mostly identical to the new version. For example, you have to upload a fat JAR with the new build over a metered connection, and the server literally sits in a garage at the other end of the country.

If the archive is small, uploading it in its entirety looks simple enough, but if it grows into dozens of megabytes, of which 90% is non-changing 3rd party code, it quickly becomes a waste of both time and bandwidth.

Time is money. And depending on your ISP, bandwidth can be money too.

Long story short, I’m going to assume that the target machine runs Linux, and the source machine runs either Windows or Linux. I am going to use rsync and fuse-zip.

Why those two? Because fuse-zip can convert a ZIP archive into a writeable filesystem, and rsync can sync a writeable filesystem with another. Details below.

Task 1 – Mount the archives

Create a mountpoint on the target machine, I’ll use /tmp/z. Then mount your archive using fuse-zip:

$ sudo fuse-zip -o allow_other,rw,uid=`id -u`,gid=`id -g` /stuff/target.zip /tmp/z

To test if it works, you create a file in the mounted filesystem. It would appear in the ZIP file after you unmount it.

$ echo TEST > /tmp/z/TEST_FILE_PLEASE_IGNORE
$ sudo umount /tmp/z
$ unzip -l /stuff/target.zip | grep TEST_FILE_PLEASE_IGNORE

Do not modify the ZIP archive while it’s being mounted and do not interrupt the unmounting process if the archive was mounted for writing, as this can corrupt it.

If your source machine runs Linux, you can do the same, but mount the archive read-only:

$ sudo fuse-zip -o allow_other,ro,uid=`id -u`,gid=`id -g` /stuff/source.zip /tmp/z

You should experiment a bit if sudo and usd/gid are necessary.

If your souce machine runs Windows, you can use Pismo File Mount to mount the archive as a drive and assign it a letter. It’s a GUI program, I haven’t found a scriptable alternative yet.

Task 2 – Synchronize the mounted filesystems

On the source machine:

$ rsync --delete -zrucv -e ssh /tmp/z/ remoteuser@targethost:/tmp/z/

Explanation for the parameters:

-c so that the files will be compared using checksums instead of modification dates. We do not care about nor trust the dates here, what matters is content.
-z for the compression. Do not use the flag if the files themselves won’t compress well.
-r for recursive synchronization.
-u so only modified files are uploaded.
--delete so the files that are absent in the source archive will be deleted from the target archive. We want the target archive to have the same content as the source one, not more.
-v stands for verbose. Used once, makes rsync list all the files as they are being sent. You can skip this option, or repeat it several times, depending on your preference.
-e ssh picks the shell used to synchronize the files. SSH is the recommended one for most cases.

Note that the paths end with slashes, so rsync synchronizes the contents of the directories, not the directories themselves.

If the source mashine is running Windows, you can use Cygwinhttps://www.cygwin.com/, just install appropriate packages (you’ll need rsync and most likely also openssh). If you used Pismo File Mount to mount the ZIP to e.g. J: drive, use /cygdrive/j/ as your source directory.

Task 3 – Unmount the filesystems

On Linux (both source and target machine), do

$ sudo umount /tmp/z

You have to unmount the target archive so the changes take place, and the source archive so the next time you mount it, it would be current again.

Putting it all together

Here is a small script that can be used to synchronize two ZIP archives:

#!/bin/bash

# put all the necessary values here
REMOTEUSER=...
REMOTEHOST=...
SOURCEZIP=...
TARGETZIP=...

OS=`uname -o`
if [ "$OS" = Cygwin ] ; then
	# depends on where you mount it; assuming here the J: drive
	P=/cygdrive/j
else
	P=/tmp/z
	mkdir -p "$P"
	sudo umount "$P"
	sudo fuse-zip -o allow_other,uid=`id -u`,gid=`id -g`,ro "$SOURCEZIP" "$P"
fi

ssh $REMOTEUSER@$REMOTEHOST sudo jar_mount_rw
rsync --delete -zrucv -e "ssh" "$P/" $REMOTEUSER@$REMOTEHOST:/tmp/z/
ssh $REMOTEUSER@$REMOTEHOST sudo umount /tmp/z

if [ "$OS" = Cygwin ] ; then
	echo 'Finished. Unmount the J: drive.'
else
	PID="`pgrep fuse-zip`"
	if [ -z "$PID" ] ; then
		exit
	else
		sudo umount "$P"
	fi
fi

It requires this script to be present as jar_mount_rw on the target machine:

#!/bin/bash

# put all the necessary values here
TARGETZIP=...

umount /tmp/z
mkdir -p /tmp/z
fuse-zip -o allow_other,rw,uid=`id -u`,gid=`id -g` "$TARGETZIP" /tmp/z

Results

I don’t have any hard data, but the overall effect is that a 40 MB JAR archive with only few files modified in it gets synchronized in few seconds, while it could take several minutes on slower connections to upload it whole.

Java 8 Functional Interfaces Cheatsheet

2015-08-08T22:37:00+00:00

Java 8 introduced basic support for first-class functions. The functions, unlike in other languages, aren’t represented by only handful of types. Instead, Java 8 uses dozens of various types depending on arity, parameter types and the return type. The standard documentation is pretty unwieldy, so for my and your convenience, I prepared a list of all functional interfaces in a more useful order.

Few things to have in mind:

All interfaces are in java.util.function package unless otherwise noted.
Interfaces are specialized only for boolean, int, long and double, and only sometimes. Other primitive types, and all primitive types in certain situations, will have to be boxed.

Nullary functions

function type	Java type
() → void	`java.lang.Runnable`
() → boolean	`BooleanSupplier`
() → int	`IntSupplier`
() → long	`LongSupplier`
() → double	`DoubleSupplier`
() → A	`Supplier<A>` or `java.lang.Callable<A>`

Whether to use Supplier or Callable, it’s mostly a matter of taste and semantics. Supplier has a method called get, Callable has call. This suggests that functions with larger side-effects should be Callable, and functions with small side-effects (mostly lazily-initialized values) should be Suppliers. Of course, this is only a suggestion.

Unary functions

function type	Java type
int → void	`IntConsumer`
long → void	`LongConsumer`
double → void	`DoubleConsumer`
A → void	`Consumer<A>`
int → boolean	`IntPredicate`
long → boolean	`LongPredicate`
double → boolean	`DoublePredicate`
A → boolean	`Predicate<A>`
int → int	`IntUnaryOperator`
long → int	`LongToIntFunction`
double → int	`DoubleToIntFunction`
A → int	`ToIntFunction<A>`
int → long	`IntToLongFunction`
long → long	`LongUnaryOperator`
double → long	`DoubleToLongFunction`
A → long	`ToLongFunction<A>`
int → double	`IntToDoubleFunction`
long → double	`LongToDoubleFunction`
double → double	`DoubleUnaryOperator`
A → double	`ToDoubleFunction<A>`
A → A	`UnaryOperator<A>`
A → B	`Function<A, B>`

Note the following:

The primitive type of the argument is attached directly, the primitive type of the result uses the prefix To.
If the result is boolean, then the function is called a predicate. It’ it’s void, then a consumer.
If the parameter type and the result type are the same, the function is called a unary operator.
There are no specialized interfaces from a primitive to a reference.

Binary functions

function type	Java type
(A, int) → void	`ObjIntConsumer<A>`
(A, long) → void	`ObjLongConsumer<A>`
(A, double) → void	`ObjDoubleConsumer<A>`
(A, B) → void	`BiConsumer<A, B>`
(A, B) → boolean	`BiPredicate<A, B>`
(int, int) → int	`IntBinaryOperator`
(long, long) → long	`LongBinaryOperator`
(double, double) → double	`DoubleBinaryOperator`
(A, A) → A	`BinaryOperator<A>`
(A, B) → int	`ToIntFunction<A, B>`
(A, B) → long	`ToLongBiFunction<A, B>`
(A, B) → double	`ToDoubleBiFunction<A, B>`
(A, B) → C	`BiFunction<A, B, C>`

Ternary functions and above

None.

Random ideas for Scala 3.0

2014-04-29T01:22:00+00:00

Here comes a list of things that I would love to see in Scala 3.0. Some of them are breaking changes, hence 3.0 not 2.13 or anything like that. Some of them are about the compiler, some of them are about the library, some of them are about the external tools. Some of those ideas are different solutions for the same problem.

Syntax

Treating number literals with leading zeroes as decimal (with a warning for a version or two).
Binary and octal literals in forms of 0b01010101 and 0o037.
Underscores in numeric literals: 1_000_000.
Some kind of byte array literal.
BigInt and BigDecimal literals: 100000000000000000000000000000000000N, 0.01m
Short and Byte literals, both accepting both signed and unsigned values: 20000s, 255y
Removal of XML literals and an xml macro string context as a replacement: xml"<p>Hello $world</p>"
Introducing true break, continue and goto.
A special syntax for monads, like Haskell’s do.
- do is already a keyword, it can cause problems with do...while loops.
- for is clunky, especially when it comes to if...else. Maybe fix for?

Standard library

Clean-up of the collections. Currently, the standard collections are a little mess. I think Paul Phillips sums it up nicely.
Standard scala-time library, being a wrapper for both Joda Time and Java 8 Time API.
- There would be two functionally identical implementations.
HLists.
Vectors with length known at compile-time.
A macro that includes C header files on compile time and generates corresponding JNA interfaces.
Removal of /: and :\ methods.
A clone of .NET’s dynamic type.
String.toInt and friends accepting a base.
A ^ operator for sets, returning the symmetric difference.

Value classes

Specialization for custom value classes.
Generating specialized classes on the fly by the compiler (so the compiler would take an existing class, either ours or from an external library, and specialize it).
- This would allow to specialize collections.
Unboxed arrays for custom value classes.
Different name mangling for methods taking value classes.
- Currently, if you have class A(x: Int) extends AnyVal, class B(x: Int) extends AnyVal and try to write both def method(a: A) and def method(b: B), you get an error.
Multi-field value classes.
- This could work as following: if the value is a field, a local variable, or a parameter, use several variables. If it’s a return value, box it.
- This of course wouldn’t improve performance much, unless there would be an alternative way to return such values.

Type system

Almost full type inference for private and local functions and variables.
Built-in type-level integers, booleans and strings, with some operations on them.
Compile-time null safety – detection of provable paths that could potentially lead to NPE’s.
Multimethods.

Fixing minor annoyances

Adding Scalaz’s some and none.
Adding Scalaz’s Equal and Monoid typeclasses.
@adt annotation on a trait would make the generated apply method of companion objects of case classes that implement that trait return that trait.
- For example, annotating Option with @adt and writing var x = Some(1) would make x of type Option[Int], not Some[Int].
- It would be nice to make it somehow work with case objects, especially in polymorphic cases, like with Option and None, or Either and either Left or Right (either of these fixes only one type parameter, not two).
Ignoring missing annotation definitions in external libraries.
Allowing for creating annotations with runtime retention in Scala.

Tooling

A decent, officially supported Findbugs plugin.
A configurable style checker, preferably including “powerlevels”.
A Java-to-idiomatic-Scala converter.
A Scala-to-idiomatic-Java converter.
Easier developing for Android.
- Built-in Proguard-like optimizer?
- Dalvik backend?

Implementing catamorphisms in Java

2014-04-29T01:20:00+00:00

Warning: I’m now going to wake up the ghosts of the past. The past which includes your undergraduate abstract algebra lectures.

This post is mostly a result of my boredom and my willingness to show that you can shoehorn almost every abstraction into almost every programming language, but it’s not exactly the best idea to do so.

Also, I simply wanted to say a word or two about several abstract mathematical concepts. It can be a worthwhile intellectual exercise.

Algebras

Before I talk about catamorphisms, mentioned in the title, I’d like to have a look at a very abstract and general mathematical structure: an algebra.

An algebra is a tuple that contains:

some sets (called carrier sets), most often one
usually some operations, each of them from some Cartesian product of the carrier sets to one of the carrier sets
sometimes some distinguished elements from those sets (they’re often superfluous, but they will be required later; you can also think about them as of nullary operations)

Examples:

a set of strings Str and the operation of concatenation +: Str × Str → Str
a set of natural numbers Nat and the operation of addition: +: Nat × Nat → Nat
a set of finite subsets of natural numbers Nat and a set of natural numbers P(Nat), a distinguished empty set Ø and the operations of union and intersection P(Nat) × P(Nat) → P(Nat) and the largest element operation max: P(Nat) → Nat (with max(Ø) = 0)

Note: I’m using the + operator for string concatenation because the article is supposed to end up with creating some Java code, and Java uses + for string concatenation.

Usually when talking about an algebra, one specifies properties those operations have, for example concatenation is associative, addition is commutative and associative, and so on.

When you look at it, an algebra can be considered equivalent to a program in a statically typed functional language.

Homomorphisms

The second important thing we need are algebra homomorphisms. Let’s assume we have two algebras A₁ and A₂ with the following properties:

They have the same number of carrier sets, e.g. X₁, Y₁, Z₁ and X₂, Y₂, Z₂.
They have the same number of distinguished elements from the corresponding carriers sets, e.g. ε₁ ∈ X₁, α₁ ∈ Y₁, β₁ ∈ Y₁ and ε₂ ∈ X₂, α₂ ∈ Y₂, β₂ ∈ Y₂.
They have the same number of operations from the correspoding carriers sets to the corresponding carrier sets, e.g. f₁: X₁×Y₁→Y₁, g₁: Y₁×Y₁→Y₁, h₁: X₁×X₁→X₁, k₁: Y₁→Z₁ and f₂: X₂×Y₂→Y₂, g₂: Y₂×Y₂→Y₂, h₂: X₂×X₂→X₂, k₂: Y₂→Z₂.

I’ll be calling such pairs of algebras algebras with matching signatures.

Now if we have a function H from carrier sets of one algebra to carriers sets of another algebra, and that function preserves the algebraic structure, e.g. in the example from above:

H: (X₁ ∪ Y₁ ∪ Z₁) → (X₂ ∪ Y₂ ∪ Z₂)
H(ε₁) = ε₂, H(α₁) = α₂, H(β₁) = β₂
H(f₁(x,y)) = f₂(H(x), H(y)), H(g₁(y,y')) = g₂(H(y), H(y')), H(h₁(x,x')) = h₂(H(x), H(x')), H(k₁(y)) = k₂(H(y))

then we call H a homomorphism.

You probably need an example. I’ll start with a simple one: the two first algebras I mentioned have a homomorphism between them: the length. Indeed, length(str₁ + str2) = length(str₁) + length(str₂).

There’s also a homomorphism back, let’s call it aaaa, which for a given natural number returns a string made of that number of a’s, e.g. aaaa(3) = "aaa". You can easy see that aaaa(n₁ + n₂) = aaaa(n₁) + aaaa(n₂).

Note: You have probably noticed, that instead of performing calculations on some operands in one algebra and finally converting the result to another algebra using a homomorphism, we can first convert the result to the second algebra and then perform the calculations. Indeed that is correct and usually it’s useful, especially if the operations in the second algebra are easier to perform. Haskell library HLearn converts an algebra of data samples to an algebra of statistics, allowing for instantaneous recalculating of statistics after adding some data to an enormous sample.

If the homomorphism is invertible, i.e. there is a homomorphism K from the second algebra back to the first, and K(H(x)) = x and H(K(y)) = y, then we call H an isomorphism and the two algebras isomorphic. Two isomorphic algebras can be considered equivalent for most purposes.

Since length and aaaa don’t have that property (aaaa(length("b")) = "a" ≠ "b"), nor any other pair of homomorphisms does, the algebra of strings with concatenation and the algebra of natural numbers with addition are not isomorphic.

Initial algebras

A special category of algebras are initial algebras. Their carrier sets are not given, but are defined from the given distinguished elements and operations. For example, the carrier set X for an initial algebra with distinguished element Z ∈ X and unary operation S: X → X is {Z, S(Z), S(S(Z)), S(S(S(Z))), S(S(S(S(Z))))...}. Operations in initial algebras are not commutative nor associative, but they are always injective. They’re sometimes called constructors, and like the constructors in programming languages, they always construct new, distinct values. When initial algebras are implemented, they’re usually called algebraic data structures.

Let’s implement the above initial algebra in Java:

public interface X {
  
}

public class Z implements X {
    public boolean equals(Object o){
        return o instanceof Z;
    }
}

public class S implements X {
  
  public final X field1;
  
  public S(X param1){
      this.field1 = param1;
  }
  
  public boolean equals(Object o){
      return o instanceof S && ((S)o).field1.equals(field1);
  }
}

Note: Implementing hashCode, providing null-safety and adding getters left as an exercise for the reader.

##Catamorphisms

The nice thing about the initial algebras is that for every algebra with a matching signature, there exists exactly one homomorphism from initial algebra to the given algebra. That unique homomorphism is called the catamorphism.

For example, given the algebra above, we can uniquely map it onto an algebra with natural numbers, number 0, and function x → x + 1: H(Z) → 0, H(S(x)) → H(x) + 1. While it may look tautological, keep in mind that the elements of the carrier set of an initial algebra are uniquely defined by the distinguished elements and operations that were used to create them.

Let’s implement it in Java!

public int h(X x){
    if (x instanceof Z) {
        return 0;
    } else if (x instanceof S) {
        return h(((S)x).field1) + 1;
    else {
        throw new IllegalArgumentException();
    }
}

It works, but it’s ugly.

Pattern matching… sort of

Let’s take another look at our homomorphism in Java. The code has several issues:

it explicitly checks type and casts the parameter;
it calls itself recursively;
it has this ugly unreachable code that will suddenly start throwing exceptions when we add another constructor to our X algebra.

Let’s address #1 and #3 first.

If we were using a functional language with first-class support for algebraic data types, we could write our homomorphism using pattern matching:

let h x = 
  match x with
  | Z -> 0
  | S field1 -> h field1 + 1

Java does not support pattern matching, that’s obvious and furthermore, the alternative solution using explicit casts is ugly and prone to bugs. To solve it in Java, the officially preferred way is to use the visitor pattern.

interface XVisitor<T> {
      T visit(Z z);
      T visit(S s);
}

public interface X {
      <T> T visit(XVisitor<T> visitor);
}

public class Z implements X {
    public boolean equals(Object o){
        return o instanceof Z;
    }
    public <T> T visit(XVisitor<T> visitor){
        return visitor.visit(this);
    }
}

public class S implements X {
  
    public final X field1;
    
    public S(X param1){
        this.field1 = param1;
    }
    
    public boolean equals(Object o){
        return o instanceof S && ((S)o).field1.equals(field1);
    }

    public <T> T visit(XVisitor<T> visitor){
        return visitor.visit(this);
    }
}

class H implements XVisitor<Integer> {

    public int visitZ(Z z){
        return 0;
    }

    public int visit(S s){
        return s.field1.visit(this) + 1;
    }
}

Note: As you can see, it’s not the typical Visitor pattern as described by the Gang of Four. First, here the visit methods actually return something instead of returning void, second, GoF suggests that the branching classes, like S here, always had the visitor visit their children and express that in the S.visit method.

Since the only thing we need from the instances of our classes is values of their fields, we can simplify it further:

interface XVisitor<T> {
    T visitZ();
    T visitS(X field1);
}

public interface X {
    <T> T visit(XVisitor<T> visitor);
}

public class Z implements X {
    public boolean equals(Object o){
        return o instanceof Z;
    }
    public <T> T visit(XVisitor<T> visitor){
        return visitor.visitZ();
    }
}

public class S implements X {
  
    public final X field1;
    
    public S(X param1){
        this.field1 = param1;
    }
    
    public boolean equals(Object o){
        return o instanceof S && ((S)o).field1.equals(field1);
    }

    public <T> T visit(XVisitor<T> visitor){
        return visitor.visitS(field1);
    }
}

class H implements XVisitor<Integer> {

    public int visitZ(){
        return 0;
    }

    public int visitS(X field1){
        return field1.visit(this) + 1;
    }
}

Looks way better. Too enterprisy for my taste, but hey, it’s Java.

The upside of this new approach is that whenever we add a new class implementing our base interface, the compiler will kick and scream until we fix our visitors.

The downside is that we still have to call visit in visitS. The burden of converting field1 to T should be on the S class. The visitor interface should be decoupled from the X interface. Let’s try a bit harder.

Catamorphisms and visitors, take two

Let’s have a look at our simple algebra so far:

interface X:
    Z()
    S(X param1)

As you can see, it’s recursive. That means it’s also infinite.

Let’s get rid of the recursion somehow. What happens if we replace X with some type parameter T?

interface XPrecursor<T>:
    ZPrecursor<T>()
    SPrecursor<T>(T param1)

(Sorry for weird syntax.)

As you can see, our algebra is no longer recursive. (It can still be infinite though, for example if some constructors use another infinite types.) Such family of non-recursive algebras is called the precursor of our initial algebra X. All algebras in this family are also initial.

Note: XPrecursor<X> is isomorphic to X. Here are the homomorphisms both ways:

X fromPrecursor(XPrecursor<X> p){
    if (p instanceof ZPrecursor) {
        return new Z();
    } else if (p instanceof SPrecursor {
        return new S(((SPrecursor<X>)p).param1);
    } else {
        throw new IllegalArgumentException();
    }
}

XPrecursor toPrecursor(X x){
    if(x instanceof Z){
        return new ZPrecursor();
    } else if (x instanceof X) {
        return new SPrecursor<X>(((S)x).param1);
    } else {
        throw new IllegalArgumentException();
    }
}

How would a visitor for XPrecursor<T> look like?

interface XPrecursorVisitor<T,U> {
    U visitZ();
    U visitS(T field1);
}

Note: Since XPrecursor<T> is not recursive, the visitor does not have to know anything about XPrecursor<T>.

Note: Remember: every XPrecursorVisitor<T,U> defines a homomorphism XPrecursor<T> → U.

And finally: can we convert an XPrecursorVisitor<T,T> into an XVisitor<T>? The answer is yes:

class XCata<T> implements XVisitor<T> {
    private final XPrecursorVisitor<T,T> underlying;
    
    public XCata(XPrecursorVisitor<T,T> u){
        underlying = u;
    }
    
    public T visitZ(){
        return underlying.visitZ();
    }
    
    public T visitS(X x){
        return underlying.visitS(x.visit(this));
    }
}

Now, let’s sum all that we know:

X is an initial algebra
for every T, XPrecursor<T> is an initial algebra
for every homomorphism f: XPrecursor<T> → T (which means XPrecursorVisitor<T,T>) exist at least one homomorphism g: X → T (which means XVisitor<T>), and only one that doesn’t incorporate any external constants into it

As some of you may have noticed, I called the converter XCata. Indeed, if we think of the XPrecursorVisitor<T,T> as of an algebra over T (where the methods represent the distinguished elements if zero parameters and the operations if not), XCata is the catamorphism from initial algebra X to the T algebra.

Armed with this knowledge, we can add <T> T visit(XPrecursorVisitor<T> visitor) to our X interface and inline the methods of XCata in the implementations of the new method:

public interface X {
    <T> T visit(XPrecursorVisitor<T> visitor);
}
public class Z implements X {
    public boolean equals(Object o){
        return o instanceof Z;
    }
    public <T> T visit(XPrecursorVisitor<T> visitor){
        return visitor.visitZ();
    }
}

public class S implements X {
  
    public final X field1;
    
    public S(X param1){
        this.field1 = param1;
    }
    
    public boolean equals(Object o){
        return o instanceof S && ((S)o).field1.equals(field1);
    }

    public <T> T visit(XPrecursorVisitor<T> visitor){
        return visitor.visitS(field1.visit(visitor));
    }
}

Some more initial algebra examples

Before we continue, we will need some more examples. Here are some unfinished Java snippets:

// a binary tree with leaves containing values of type T
interface Tree<T> { 
...

class Leaf<T> implements Tree<T> {
    Leaf(T leaf){ 
...

class Branch<T> implements Tree<T> {
    Branch(Tree<T> tree, Tree<T> right) {
...

// a simple integer arithmetic expression tree with both variables and constants
interface Expression {
...

class Variable extends Expression {
    Variable(String name) {
...

class Constant extends Expression {
    Constant(int value){
...

class Add extends Expression {
    Add(Expression expr1, Expression expr2) {
...

class Multiply extends Expression {
    Multiply(Expression expr1, Expression expr2) {
...

class Negate extends Expression {
    Negate(Expression expr) {
...

In fact, trees and expressions are the most common examples discussed in articles on similar topics.

Where the heck to use all that theoretical mumbo-jumbo?

Wikipedia says that functional programmers refer to catamorphisms as folds. Every time you have some recursive datatype, usually tree-like or list-like, folding takes such large value and folds it into some nice, small, single value. It does that by looking only at the nearest neighbourhood: the already folded child nodes and the type and other fields of the current node.

From now on, I’ll call things I called PrecursorVisitors before, Folders.

Given the tree defined above, how would a folder for a Tree<T> look like?

interface TreeFolder<T,U> {

    U visitLeaf(T leaf);

    U visitBranch(U left, U right);
}

And it’s used like this:

interface Tree<T> {
    <U> fold(TreeFolder<T,U> folder);
}

class Leaf<T> implements Tree<T> {

    final T leaf;

    Leaf(T leaf) {
        this.leaf = leaf;
    } 

    public <U> U fold(TreeFolder<T,U> folder) {
        return folder.visitLeaf(leaf);
    }
}

class Branch<T> implements Tree<T> {

    final Tree<T> left;
    final Tree<T> right;

    Branch(Tree<T> left, Tree<T> right) {
        this.left = left;
        this.right = right;
    } 

    public <U> U fold(TreeFolder<T,U> folder) {
        return folder.visitBranch(left.fold(folder), right.fold(folder));
    }
}

What can you use it for? For example, to get the leftmost element of the tree:

class Leftmost<T> extends TreeFolder<T,U> {

    public T visitLeaf(T leaf){
        return leaf;
    }

    public T visitBranch(T left, T right){
        return left;
    }
}

Or count leaves:

class LeftMost<T> extends TreeFolder<T,Integer> {

    public Integer visitLeaf(T leaf){
        return 1;
    }

    public Integer visitBranch(Integer left, Integer right){
        return left + right;
    }
}

As for the expressions, evaluating them is one way of folding them:

interface ExprFolder<T> {
    T visitConst(int value);
    T visitVariable(String name);
    T visitAdd(T expr1, T expr2);
    T visitMul(T expr1, T expr2);
    T visitNegate(T expr);
}

class Evaluate implements ExprFolder<Integer> {
    private final Map<String, Integer> vars;
  
    Evaluate(Map<String, Integer> vars){
        this.vars = vars;
    }

    public Integer visitConst(int value){
        return value;
    }

    public Integer visitVariable(String name){
        return vars.get(name);
    }

    public Integer visitAdd(Integer expr1, Integer expr2){
        return expr1 + expr2;
    }

    public Integer visitMul(Integer expr1, Integer expr2){
        return expr1 * expr2;
    }

    public Integer visitNegate(Integer expr){
       return -expr1;
    }
}

or convert to a string:

class Stringify implements ExprFolder<String> {

    public String visitConst(int value){
        return "" + value;
    }

    public String visitVariable(String name){
        return name;
    }

    public String visitAdd(String expr1, String expr2){
        return "(" + expr1 + " + " + expr2 + ")";
    }

    public String visitMul(String expr1, String expr2){
        return expr1 + "×" + expr2;
    }

    public String visitNegate(String expr){
        return "(-" + expr1 + ")";
    }
}

And so on.

So, is everything a fold?

In short, not quite.

Some operations, like for example expression optimisation, are not a fold, unless you pick a weird result type. Folds always progress from leaves to the root and never descend back. Optimisation of expression trees, on the other hand, is usually implemented as a descent from the root to the leaves.

In theory though, everything can be implemented as a fold. Please don’t do that though.

Furthermore, folding shown in this post is strict, which means it always descends to the end of a tree before returning back. If the data structure it was traversing was infinite, it would never yield an result.

Final remarks

First of all, I’d like to thank Bartosz Milewski for a nice write-up on the same topic: https://www.fpcomplete.com/user/bartosz/understanding-algebras1111

Second of all, Java is not the best suited programming language for working with algebraic data types. Lack of pattern matching, verbose class declaration syntax, defaulting to equality by reference, defaulting to mutable fields, requiring explicit copying from constructor parameters to fields, and few other things make this pattern bothersome. It works better in other languages though, see for example Milewski’s Haskell examples.

From explicit nested monads to Kleisli arrows over monad transformers

2014-04-29T01:15:00+00:00

Recently at work I had a task of developing a small tool that could display charts with progress of our data extraction over time.

After each iteration, results of data extraction were stored in several directories and compared against a handwritten reference set.

In short, this is the directory structure:

|--+ batch1
|  |--+ correct
|  |  `--- data.txt
|  |--+ guessed
|  |  |--- data.00000145a8ab68b9.txt
|  |  |--- data.00000145a92a530f.txt
|  |  `--- data.0000014594039f5b.txt
|  |--- input1.pdf
|  |--- input2.pdf
|  `--- input3.pdf
`--+ batch2
   |--+ correct
   |  `--- data.txt
   |--+ guessed
   |  |--- data.00000145a8ab68b9.txt
   |  |--- data.00000145a92a530f.txt
   |  `--- data.0000014594039f5b.txt
   |--- input4.pdf
   |--- input5.pdf
   `--- input6.pdf

The hexadecimal numbers are timestamps in milliseconds since Unix epoch.

Each file consisted of records in the format:

input1.pdf value from the first file
input2.pdf value from the second file
input3.pdf value from the third file

Long story short, there was no difference what programming language I would write the tool in, so I picked Haskell. Let’s ignore most details of the implementation. What’s important for this entry, is that I implemented the following functions:

allCorrectFiles :: [FilePath]  -> IO [FilePath]
allGuessedFiles :: [FilePath]  -> IO [(LocalTime, FilePath)]
readDataFile :: FilePath -> IO [(Entry, String)]
getAllData :: [FilePath] -> IO ([(Entry, String)], [(LocalTime, Entry, String)])

The initial implementation of the getAllData function was straightforward, yet a bit clunky:

getAllData subDirs = do
	correctFiles <- allCorrectFiles subDirs
	guessedFiles <- allGuessedFiles subDirs
	correctData <- mapM readDataFile correctFiles
	guessedData <- forM guessedFiles $ \(t,f) ->
		x <- readDataFile f
		return $ map (\(e,s) -> (t,e,s)) x
	return (concat correctData, concat guessedData)

This code is quite ugly. The especially jarring were the return $ map combination and concats in the final line.

After a while, I noticed that all the functions I call from the getData function are of type a -> IO [b]. A double monad. So, I added transformers library to my project and rewritten that function as:

getAllData subDirs = do
	correctData <- runListT $ do
		f <- ListT $ allCorrectFiles subDirs
		x <- ListT $ readDataFile f
		return x	
	guessedData <- runListT $ do
		(t,f) <- ListT $ allGuessedFiles subDirs
		x <- ListT $ readDataFile f
		return (t, fst x, snd x)
	return (correctData, guessedData)

Now it looks way more uniform.

This was my first piece of code using monad transformers, so I will explain what’s going on to everyone who was in the same situation as me before understanding it.

IO is a monad, so is []. For some monads, if you wrap them in another monad, you still get something that can be used like a monad. One of such monads is [].

For every monad m and type a, m [a] is a monad over a. But in order for Haskell to treat it as one singular monad, we need to wrap it. And that’s what ListT is for.

So allCorrectFiles subDirs returns IO [FilePath], and after wrapping it with ListT we get ListT IO FilePath.

This way, we stack our monadic effects: we both iterate over elements of a list, and track impure side effects. If the readDataFile or allGuessedFiles were pure, we would use [] monad. If they were impure, but returned only one element, we’d use IO monad. By wrapping IO [a] into ListT, we can use them both.

Since the results of inner do blocks are now of type ListT IO a, we unwrap them with runListT.

Note that since ListT allowed <- operator to unwrap not only the IO monad, but also the list monad, we didn’t end up with lists of lists and we no longer needed to concat those lists together.

The code got less ugly, but it also got a bit longer. Could I get it shorter?

The answer was yes, but only a teeny tiny bit, and involved some more complicated machinery. Arrows.

The trivial arrow instance for functions is easy to understand: >>> behaves like flipped function composition, and there are several combinators, like &&&, ***, first and second that make it easier to work with 2-element tuples.

Since the return value from the allGuessedFiles was a list of 2-element tuples in an IO monad, and I only operated on the second element before fusing them together, I decided to use arrows. Luckily, the base library contains definitions for monadic arrows, which are to >>= as trivial arrows are to ..

The code had to involve even more wrapping and unwrapping. Here it is:

getAllData subDirs = do
	correctData <- runKL subDirs $ 
		kl allCorrectFiles >>> kl readDataFile
	guessedData <- runKL subDirs $ 
		kl allGuessedFiles >>> second (kl readDataFile) >>^ \(a,(b,c)) -> (a,b,c)
	return (correctData, guessedData)
	where 
		kl f = Kleisli $ ListT . f
		runKL x ar = runListT $ runKleisli ar x

Now I do believe I owe you some explanations.

Let’s start with the definition of kl. f is of type a -> m [b], where m is a monad. After composing it with ListT, we get ListT . f of type a -> ListT m b, where ListT m is a monad now.

Kleisli wraps our function into a Kleisli arrow (of type Kleisli (ListT m) a b), so Haskell knows that when we join arrows together, we don’t want to abstract . but >>=. This way, kl f >>> kl g means the same as \x -> f x >>= g, where f >>> g would mean \x -> (g . f) x.

runKL applies our arrow and unwraps the ListT transformer. I reversed the argument order of runKleisli for convenience.

So how is the guessedData calculated? allGuessedFiles returns IO [(LocalTime, FilePath)]. second makes its argument to be applied only to the second argument of the input tuple, so second readDataFile would be of type (a, FilePath) -> IO[(a,(Entry, String))]*, but since we used kl, it’s actually Kleisli (ListT IO) (a, FilePath) (a,(Entry, String)).

* Note: I kinda simplified here a bit and technically this type is wrong.

The last function, which is a lambda expression, was lifted into a Kleisli arrow with the >>^ operator. In case of Kleisli arrows, such lifting is equivalent to \f -> (return . f). It turns a function of type a->b into a function of type a -> m b wrapped into a Kleisli.

I also took a peek at the arrow-list package, which provides its own definitions of equivalents for kl and runKL, but I decided to not use it, since it would give little value.

So, was it worth it?

In some sense, yes. I have learnt what monad transformers and Kleisli arrows can be used for. The main problem is that the problem I had to solve was too small for such complex machinery. The transformers made the code more uniform, which allowed application of arrows, while arrows would have made code mode compact if it was more complex.

hOCR4J 0.1 released

2014-01-03T00:11:00+00:00

I would like to announce the release of a Java library for parsing hOCR documents: hOCR4J. You can download it from here. I’m planning to get it to Sonatype too, so you may be able to get it from there in the near future.

hOCR is an output format used by OCR programs, including Tesseract. It contains information about all the OCR’d words, their position, and their assumed organisation into lines and paragraphs. Currently, hOCR4J was tested to work with Tesseract-generated hOCR’s, I plan to test other OCR programs in the future.

hOCR4J parses hOCR documents, creates an immutable model for them (nice when using functional programming style), and provides various tools to manipulate and modify them.

hOCR4J makes a good starting point when developing an application which extracts data from OCR’d documents that have non-trivial layouts.

The model of an hOCR document is simple: a page contains areas, an area contains paragraphs, a paragraph contains lines, a line contains words. Each of these objects has a bounding box, which defines its position on the scanned page. hOCR4J provides various operations on bounding boxes in the Bounds class, including scaling, resizing, translating, unions, intersections, and more.

Using hOCR4J is also simple. First, we need to get our hOCR file and read the hOCR into a string. Then we can parse it:

String hocr = ... ; // load hOCR here

List<Page> pages = HocrParser.parse(hocr);

Page page0 = pages.get(0);

We can now extract some text:

List<String> textLines = page0.getAllLinesAsStrings();

We can only extract lines that satisfy some conditions:

List<Line> lines = page0.findAllLines(LineThat.matchesRegex("^IMPORTANT:"));

for (Line line: lines){
  String text = line.mkString();
  // ...
}

We can look for words in italics:

List<Word> words = page0.getAllWords();

for (Word word: words){
  if (word.isItalic()){
    String w = word.getText();
    // ...
  }
}

We can look for location of a word or phrase (spaces are ignored, as OCR sometimes inserts more or less of them):

// let's censor the name of the culprit
Page censoredPage = page0.mapLines(new Function<Line,Line>(){
  public Line apply(Line line){

    // we check if the line mentions culprit's name
    final Bounds theCulpritIs = line.findBoundsOfWord("The culprit is");

    if (theCulpritIs != null) {
      // we take bounds of the entire line
      Bounds full = line.getBounds();

      // and we calculate bounds of a line
      // that doesn't extend beyond words "The culprit is"
      Bounds remain = new Bounds(
        full.getLeft(), 
        full.getTop(), 
        theCulpritIs.getRight(), 
        full.getBottom());

      // finally, we trim our line
      return line.createBounded(remain);
    } else {
      // all the other lines are left unmodified
      return line;
    }
  }
});

return censoredPage;

In the near future, I’m planning a tutorial on extracting text columns from hOCR using hOCR4J.