IDN use and abuse

JWZ blogged about the Unicode snowman. If you're running a proper browser, take a close look at the domain name:
☃.net

A brief, two sentence overview: For any domain name which begins with xn-- followed by some gobbledygook, certain clients like web browsers can interpret the gobbledygook as a Punycode representation of some Unicode string. So the snowman's real domain name is xn--n3h.net.

I was quite excited for a while since many of these Unicode dingbats and symbols are unregistered in combinations of two or more, but then I found that the killjoys at the IETF had put a stop to that with RFC 4690. So while the snowman registration can be continued, no new dingbats can be registered.

Nevertheless, we can still have fun abusing the simpler Chinese characters. For a laugh I registered 丄.com and 丿乀.com. These might not be active when you read this, and to be honest I'm not quite sure where I'll point them at the moment. The first looks like bottom, the symbol for non-terminating programs. Hmmm maybe that'd be good for some insightful blog about functional programming? The second is a total abuse of two characters together, but looks like the number 8 in Japanese (IETF rules forbid registering actual numbers, even non-Arabic ones).

Someone should do the world a favour and register 丅丨丅.com (xn--9gqa8h.com).

Update

Subdomains are of course not regulated by the IETF jobsworths. Here's another, prettier unicode snowman: http://☃.earthlingsoft.net/, and I can have http://☆☆☆.annexia.org/

MinGW: Compile software for Windows without leaving your Fedora machine

For the last few weeks I've been focused on the Fedora MinGW project. This project gives Fedora users a compelling new feature: you can build your software for Windows, without ever needing to leave the Fedora / Linux environment. In fact you can do everything, up to and including creating a Windows installer for your customers, without needing once to touch Windows.

To demonstrate how this works, I'm going to show you how to port a simple application to Windows, using Fedora MinGW. The app I've chosen is virt-viewer, a graphical console viewer for virtual machines, written in C.

First we install the cross-compiler environment and any libraries that our program requires. (Until the MinGW packages are accepted into Fedora, you'll have to get them from our temporary yum repository)

yum install mingw32-gcc mingw32-binutils \
  mingw32-gtk2 mingw32-gtk-vnc mingw32-libvirt mingw32-libxml2 \
  mingw32-nsis mingw32-nsiswrapper

With software such as virt-viewer that is based on the standard autoconf "configure" script, the cross-compiling step is simple. You just have to do:

./configure --host=i686-pc-mingw32

That's all you have to do to configure virt-viewer (and most other software) to cross-compile for Windows.

Now we just do make and discover ... ah, that it doesn't compile. This leads us to the hard part of porting software over to Windows. Windows uses the Win32 API instead of the usual POSIX / libc API found on Linux.

For virt-viewer there are several problems:

1. virt-viewer uses some header files like <sys/socket.h> which aren't found under Win32.
   
2. We need to include <windows.h> on Windows (but not on Linux). For Win32, this header file is analogous to <stdlib.h> or <unistd.h>, and almost every C source file should include it.
   
3. virt-viewer makes some Linux-specific system calls which aren't available in the Win32 API. The problematic calls are:
   
   
   • usleep (sleep for a specified number of microseconds)
     
   • fork (create a subprocess)
     
   • socketpair (create a pipe to communicate with the subprocess)
     

Problems (1) and (2), the missing header files, are easily solved in a very portable way. For each header file which is missing on Windows or Linux, we will just add a configure-time test and some #ifdef magic. Into configure.ac we put:

AC_CHECK_HEADERS([sys/socket.h sys/un.h windows.h])

and then into the C sources files we put:

#ifdef HAVE_SYS_SOCKET_H
#include <sys/socket.h>
#endif

and so on.

Problem (3) -- missing APIs -- are the hardest problems to solve. In general there are three strategies we could try:

(a) Try to find an equivalent but different API which is present on Linux and Windows. As an example here, Windows has a call which is very similar to pipe, and might be used to replace socketpair.
(b) Write a replacement function for each problematic API.
(c) Comment out the particular feature in the code which uses the missing calls. This is less satisfactory of course: Windows users will now be missing some feature.

We're going to fix problems in (3) with a mixture of strategies (b) and (c).

Windows doesn't have usleep, but looking at MSDN I see that it does have a function Sleep (DWORD milliseconds) which can be used as a replacement for usleep.

You can test and replace functions conditionally by adding this to configure.in:

AC_REPLACE_FUNCS([usleep])

Remember that you don't want to replace this on Linux and any platforms that have usleep, and that is what AC_REPLACE_FUNCS does.

The code to implement usleep is now placed into a single function in a file with the same name, usleep.c:

#ifdef WIN32
int
usleep (unsigned int usecs)
{
  unsigned int msecs = usecs / 1000;
  if (msecs < 1)
    Sleep (1);
  else
    Sleep (msecs);
}
#endif

The magic of autoconf will ensure this file will only be linked into the main program when it is needed.

As for fork and socketpair, it turns out we are quite lucky. These two calls are only used to implement a specific virt-viewer feature, namely tunneling connections over ssh. If you conclude, as I did, that ssh isn't that common on Windows machines, then you can do as I did and just comment out that feature conditionally when building on Windows.

With those changes, we have now completed our port of virt-viewer to Windows (full patch). After rerunnning:

autoconf
./configure --host=i686-pc-mingw32
make

we are left with virt-viewer.exe, a full Gtk application that runs on Windows.

Creating a Windows installer

To package up Windows applications into full-featured installers, that include menu shortcuts, desktop icons and an uninstaller, we wrote a little helper program called nsiswrapper. As its name suggests, it is a wrapper around the NSIS Windows Installer, which we also ported over to run natively under Fedora.

You'll need to wrap up not just virt-viewer.exe, but the Gtk-related DLLs and helper modules. With nsiswrapper you would do:

  nsiswrapper --run \
    --name "Virt-Viewer" \
    --outfile "Virt-Viewer-for-Windows.exe" \
    --with-gtk \
    /usr/i686-pc-mingw32/sys-root/mingw/bin/virt-viewer.exe

What can OCaml do that you can't do in other programming languages?

Depressingly predictable comments on the Haskell story in Slashdot yesterday such as:

"I have *never* seen it being used since. To my mind they both belong in the category 'interesting, but pointless'."

and

"The point is that there's nothing those languages can do that can't be done, often more easily, with the current crop of popular languages. Elegance cannot beat convenience in the workplace, or in most at any rate."

and so on.

Here are some useful things you can do in OCaml which you cannot do in the "current crop of popular languages".

1. Change the language to suit your task.

In our case the task was to parse binary formats.

If you have a few hours to spare, try writing a C program to parse tcpdump files. Don't forget that the endianness in these files is variable so you'll need lots of if (..) { field1 = htonl (field1); ... }. OK so that's a bit hard. Let's say you want to parse a 6 bit length field 'n' followed by an n+1 bit data field (as a 1-64 bit int). Go and write it in C now.

Using our bitstring extension to OCaml, parsing binary structures is really effortless:

let bits = Bitstring.bitstring_of_file "input.data" in
bitmatch bits with
| { n : 6;
    data : n+1 } -> data

This is the complete tcpdump parser, just 113 lines of code.

And it's fast too. The resulting code compiles down to machine code and inlines direct calls to C functions at every opportunity, so in practice you can parse data at speeds approaching C.

Another task we had was to check that hundreds of SQL statements in a modest sized web application actually matched with fields in the database, in other words that they didn't reference non-existent fields or treat a string field as an integer and so forth. Doing this by testing is almost impossible because many SQL statements are only used on rare error paths. We could contemplate doing it manually once but not routinely.

So instead we extended the OCaml language to do the checking for us every time we compiled the code. The resulting PG'OCaml project gives you compiled-time checked type-safe access to your database. It's used by mod_caml and ocsigen, or you can just use it in standalone programs. I won't go into detail because Dario Teixeira wrote a great introduction and tutorial to PG'OCaml.

Martin Jambon, author of Micmatch which adds regular expressions directly into the language, has an excellent tutorial. Browse the list of syntax extensions here.

2. Get the compiler to check for errors in your logic

Why can't your compiler check for errors like when you use a read-only database connection in a SQL INSERT statement, or you are supposed to call library function set_word_size because you try to call get_word?

In OCaml, using phantom types [tutorial] you can do exactly this.

3. Drop down to imperative code when you need speed like C

Scripting languages are expressive but slow ... and getting slower. But there's no need to compromise expressiveness to get real speed. Static typed languages with type inference (which excludes C, C++, Java, C# and most others, but includes SML, OCaml and F#) give you the expressiveness to write compact code but without compromising on speed.

OCaml programs are more than 10 - 15 times faster and less memory intensive than Python programs.

More about speed and optimizing OCaml programs. Comparison of a raytracer written in C++ and OCaml.

4. Large well-established standard libraries

OCaml comes with hundreds of packages for many tasks now.

If that's not enough you can directly call out to Perl, Java, Python and .Net [on Windows] libraries. Or you can call C functions directly.

You can compile the same code on Unix, Mac OS X and Windows.

So enough of this "elegance can't beat convenience" stuff please.

Tip: Read all lines from a file (the most common OCaml newbie question?)

Is this the most common OCaml beginners question? It comes up every few weeks on the OCaml beginners list, and I have tried to answer it before.

It seems that everyone who learns OCaml comes away with the impression that functional programming is the New Cool Thing, and imperative programming is Bad and Must Be Avoided.

I'm going to say it now: programming fashions are stupid and counterproductive. The only things that matter are that your program is short, easy to write, easy to maintain and works correctly. How you achieve this has nothing to do with programming fads.

Reading all lines from a file is an imperative problem, and the shortest solution (easy to write, easy to maintain and correct¹) uses a while loop, in OCaml or any other language:

let lines = ref [] in
let chan = open_in filename in
try
  while true; do
    lines := input_line chan :: !lines
  done; []
with End_of_file ->
  close_in chan;
  List.rev !lines

Actually, no, I'm lying. The best solution is this:

Std.input_list chan

which is supplied by extlib. Don't bother to duplicate functions which are already provided in commonly available libraries.

---

¹This is only strictly speaking correct if you handle clean-up if input_line throws some read error (exception). In the common case where you just exit the program, leaving the channel open is perfectly acceptable.

Home server, part 3, quick update

As I alluded to on the GLLUG mailing list, I got stuck installing Red Hat Enterprise Linux (RHEL) 5.2 on the Viglen MPC-L because of this bug when installing RHEL or CentOS on a Geode LX. Consequently I trashed the Xubuntu pre-install and managed to make the machine unbootable. The BIOS claims to support all sorts of useful boot methods such as booting from USB flash drives and over the network, but in practice nothing seemed to work for me except booting from the local hard disk.

This means I'll have to remove the hard disk (it's a 2.5" laptop-sized HDD) and transplant it into another machine, and to do this I'm waiting for the correct 2.5" - IDE adaptor cable from Maplin.

I think the most common question people have asked me is how fast the Viglen is, with its AMD Geode LX central processor. When I had Xubuntu on the machine and while I was failing to install RHEL, I played with it and the conclusions are mixed. Firstly it's fair to say that it is not a fast machine, but fast enough to be useful for (eg) light browsing. But I did observe that some specific things are really slow. For example the timezone / graphical map selection during the RHEL install is really slow, but the rest of the RHEL install was perfectly fine. According to Alan Cox the Geode is quite a strange beast. Some pieces of "hardware" are in reality emulated (certain PCI devices for one), so my theory is that when you hit a piece of code which uses these emulated devices in a certain way, suddenly everything slows down.

The true test of this will come when I get RHEL installed and have it running as a server.

Home server, part 2, just photos

I received the Viglen MPC-L which will be part of my home server.

The first, rather unfair impression is "cheap Mac Mini", but to Viglen's credit it is very much cheaper than a Mac, about one quarter of the price, at a mere £80 inc. tax and delivery.



Front and back:





The full set is shown below. I was surprised that Viglen even bothered supplying a mouse and keyboard, since they don't supply the obvious missing bits:



... but what about all this waste? Viglen could do better to reduce this:

MinGW - build Windows binaries from the comfort of your Fedora

MinGW is a Windows C/C++ compiler and toolchain based on gcc and binutils which runs on Linux, but generates Windows libraries (*.dll) and executables (*.exe). This means that when a customer asks you to write a Windows program, you can do it without leaving the comfort of your familiar Linux environment.

Today the Fedora MinGW Special Interest Group made a milestone release of the compiler toolchain and many basic libraries including Gtk and all its dependencies.

These aren't part of Fedora proper yet, but you can grab source RPMs and x86-64 binaries from my website here:

http://www.annexia.org/tmp/mingw/
(Still uploading as I write this)

If you prefer you can clone our Mercurial development repository. (Read the README file)

The current draft packaging guidelines may look complicated but if you start from our example spec file (mingw-example.spec) then that takes care of most packaging scenarios and dependency generation automatically.