New blog

I finally got sick of blogspot's inability to handle basic editing tasks, and moved to a new blog on wordpress.

Talk: Using the Fedora Windows cross-compiler

Date:	Sunday, 8th March 2009
Time:	18:15 UTC
Location:	Virtual -- #fedora-classroom on irc.freenode.net
Details:	https://fedoraproject.org/wiki/Classroom#The_Current_Timeline
Background:	https://fedoraproject.org/wiki/Features/Windows_cross_compiler

A talk and interactive session, "Using the Windows cross-compiler":

• API basics: POSIX, libc, Win32, gtk, Qt, etc.


• Cross-compiler basics


• Practical demonstration:
  setting up the cross-compiler in Fedora
  compiling a small Gtk program
  testing it in Wine
  building a Windows installer


• Future directions (Win64, Mac OS X ?)


• How to get involved

If you have root access to a Fedora 10 or rawhide install (i386 or x86-64 only) you can follow along with the practical part.

Photos from OCaml Users Meeting, Grenoble, 2009

We had a very successful OCaml User Meeting this week, with 45 people coming from around Europe to Grenoble.



Xavier Leroy on progress made over 2008

Sylvain Le Gall OCamlCore.org and OCaml as fast as C

Maxence Guesdon Chamo and Cameleon

Florent Ouchet VHDL symbolic simulation

David Teller Batteries Included

Christophe Troestler pa_do (delimited overloading) syntax

Richard Jones (me) OCaml Windows cross-compiler

Christophe Raffalli Parsing with dypgen

Xavier Leroy on the right:



Dinner the evening before:

Talk in London on Fedora & Windows cross-compilation

I am giving a talk and demonstration in London (UK) this Saturday afternoon, 31st January on the subject of the Fedora MinGW (Windows cross-compiler) project. Free entry, everyone is invited!

Previous MinGW postings on this blog ...

ocsigen (OCaml web framework) benchmarked

This benchmark compares the OCaml web framework ocsigen against Ruby on Rails and lighttpd+C with some excellent results for the OCaml framework:

	Reqs/sec	Mem usage
Rails with mongrel, 1 process	260	49MB
Rails with mongrel via nginx (rev proxy), 1 proc	220	~51MB
Rails with mongrel, 4 processes via nginx	430	~200MB
OCaml ocsigen (1 process)	5800	4.5MB
lighttpd with FastCGI app in C, 20 procs	9300	4.5MB

OCaml is not just an order of magnitude faster and an order of magnitude more memory efficient, but it also provides complete compile-time safety, catching multiple errors at compile time which would otherwise only show up after extensive testing.

More discussion on this reddit thread.

There's a Fedora ocsigen package waiting for review here.

Home server, part 4, installing the OS

For want of a cable, my home file server wasn't coming along very well, but today I hooked up the 2.5" IDE to PATA converter cable from Maplin with the hard drive from the disassembled Viglen MPC-L:


The Viglen was really easy to disassemble by the way. Two screws on the back hold on the backplate, and then the entire motherboard/hard disk assembly slides straight out. Another three screws let you remove the hard drive.

I settled on relatively simple route to install CentOS. I used Red Hat's KVM virtualization to run a VM, attaching the physical hard drive and the (virtual) CentOS DVD ISO. It sounds complicated, but all you need is this virt-install command line to do it (the host is Fedora 10):

virt-install --connect=qemu:///system \
  -n centos5 -r 512 \
  -v --accelerate \
  -c /root/CentOS-5.2-i386-bin-DVD.iso \
  -f /dev/sde \
  --vnc --vncport=5900

(Adjust the path to the CentOS DVD ISO, and the physical hard drive device as appropriate).

CentOS 5.2 installed in about 15 minutes:

Destroying old hard drives

This is a simple, cheap method for destroying old hard drives, making the data unrecoverable against casual attackers and identity fraudsters (although probably not some hypothetical government agency with multi-million dollar resources).

For this you will need a stack of old hard drives:


An electric drill with a twist drill bit (suitable for going through metal), and most importantly some eye protection:


Line up the hard drives against the wall and drill straight through them. I didn't show it in this picture, but in fact I drilled through from the other (PCB) side to ensure that I went through the PCB but didn't go through any components that might explode:


Sunlight where there's not supposed to be sunlight!


For a few of the drives, mainly older ones, I couldn't get all the way through, but I got through to the platters, which is the important part:


Now you can see why eye protection is not optional. This old IBM SCSI-LVD drive had glass platters which shattered into tiny, sharp shards of metal-plated glass when the drill went through:


For extra assurance, I will soak the drives in a bucket of water for a few days before disposing of them:

Stop spying on my encyclopedia reading

Recently the unaccountable UK "Internet Watch Foundation" added pages from Wikipedia to a secret list of censored pages.

I would like to make the point that no one should be prosecuted for reading an encyclopedia. Furthermore, no free, democratic society should tolerate authorities spying on people reading works of knowledge.

Let's together stop this spying now.

Here is a simple action that you can take right now, that won't cost you any time or money. When linking to any page on Wikipedia, use the secure URL:

https://secure.wikimedia.org/wikipedia/en/wiki/Main_Page

(Replace Main_Page with the name of the Wikipedia page as usual. You can also replace the language (en) or link to commons pages).

Fedora Rawhide has OCaml 3.11.0

Fedora Rawhide [the experimental/development version of Fedora] has been completely rebuilt with OCaml 3.11.0, and all library and application problems that were found have been patched.

List of packages: http://cocan.org/fedora#Package_status

Mailing list announcement

Slashdot groupthink

This may be the first time a comment of mine has been modded down to -1 on Slashdot. I'm questioning whether the inefficiency of glib outweighs the speed advantage of C. Very few of the replies get it. Perhaps this proves the people only read the first sentence of any posting ... tl;dr.

Fedora 10 & OCaml

You can join in the general Fedora 10 fun here, but a quick note that Fedora 10 comes with stable OCaml 3.10.2 and 68 OCaml packages, making us the fastest, best supported functional language in Fedora.

Common mistakes cross-compiling MinGW packages

Using the headers from /usr/include

The headers in /usr/include are for the native libraries installed on the system, and it's highly unlikely they will work for cross-compilation. By "won't work" I mean that types and structure fields could be different, resulting in a segfault.

The Fedora MinGW project takes two steps to avoid using native libraries by accident: Firstly GCC is configured so it looks in /usr/i686-pc-mingw32/sys-root/mingw/include and never looks in /usr/include (as long as you don't tell it to). Secondly we supply a replacement %{_mingw32_configure} RPM macro which sets PKG_CONFIG_PATH, so any pkg-config done will pick up the cross-compiled libraries' configuration instead of any native libraries' configuration.

$ PKG_CONFIG_PATH=/usr/i686-pc-mingw32/sys-root/mingw/lib/pkgconfig \
  pkg-config --cflags glib-2.0
-mms-bitfields -I/usr/i686-pc-mingw32/sys-root/mingw/include/glib-2.0
-I/usr/i686-pc-mingw32/sys-root/mingw/lib/glib-2.0/include  

One thing that can still go wrong is that you don't have the cross-compiled library installed and it then picks up the native library. For example, you missed a BuildRequires line. That mistake usually becomes evident when the program tries to link, because linking a cross-compiled Windows binary to a native Fedora library won't work.

Not setting --prefix

You likely don't want to install Windows binaries and libraries under /usr or /usr/local. For a start it's better to keep Windows things in one place, and the packaging guidelines have specified that place to be /usr/i686-pc-mingw32/sys-root/mingw. But mainly it's not a good idea to mix up native and cross-compiled libraries, which will cause all sorts of problems as in the point above.

If you use %{_mingw32_configure} in RPM specfiles, or the mingw32-configure command, then paths will be set correctly for you.

Not using a portability library

If you're writing the program yourself, or if you're doing the often difficult work of porting an existing application, use a portability library to help you. Which you choose is up to you and depends on many factors, but we would recommend that you look at these ones:

• glib2 and gtk2
• Gnulib
• apr
• QtCore, Boost, POCO or NSPR if you're using C++

Writing your own build system

While it's fashionable to dislike autoconf and m4 macros, it is still by far the easiest way to both build your program on multiple systems, and to cross-compile. So use autotools or cmake, and definitely don't write your own build system. Discourage other projects from writing their own build systems too.

This really comes down to bitter experience. Every project we have had to port that has used its own build system has been far more of a headache than those that just used autoconf or cmake.

Running programs during the build process

When cross-compiling, it's always a mistake to run programs during essential build steps. The problem is that you can't be sure that binaries can be made to work in the build environment. For Windows binaries, there is some chance of running them under Wine, but Wine itself is incompatible with autobuild environments like mock and Koji. Furthermore Wine only works on x86 platforms, and it's not possible to use it at all when cross-compiling from other architectures like PPC.

Running programs during make test is normal and useful though.

Egg & "Verified by Visa"

Message sent to Egg today about Verified by Visa:

Dear Sir/Madam,

I would like to permanently opt out of "Verified By Visa" when making purchases online. It just moves the liability on to me and the technical implementation of it is frankly crap. If not, I'll cancel my card (I expect you'll be happy about that) since it's no longer useful for purchases.

If however you are going to introduce some scheme which is really secure, such as a hardware token or one-time credit card numbers or authorization by SMS message, then let me know.

Update (2008-11-20) — a dull form reply from Egg:

The Secure online code service is supported by Verified by Visa and MasterCard Secure Code. It protects your card with a password, giving you added security when you shop online.

When you make purchases online with participating retailers, you'll be presented with a receipt at the end of the checkout process. The receipt includes details of your purchase, showing retailer name, purchase amount and date. You sign the receipt using your personal password and click 'Submit' to proceed with the purchase. Without your password the purchase can't be completed.

This is a system that's been put in place by Visa and MasterCard. It's to provide a more secure service, when making purchases online.

Unfortunately, this isn't something we can remove from your Egg Card.

Thanks for your message.

Emily Stirling
Internet Customer Services

Er yes, thanks for nothing Emily. You don't mention the idiotic implementation or the fact that they are passing liability over to their customers. I'm cancelling my credit card and looking for a secure alternative.

Update (2008-11-24) — I can't believe it, the fuckers cancelled my credit card.

LWN.net has an interview with us about MinGW Windows cross-compiler

Here is the article link if you are an LWN subscriber:

http://lwn.net/Articles/307732/

If you're not an LWN subscriber, you can use this free link to get to the article:

http://lwn.net/SubscriberLink/307732/0efc7b75c5696ae5/

Please consider subscribing to LWN!

OCaml Users Meeting, Feb 2009, Grenoble

Sylvain is already organizing the next OCaml Users Meeting 4th Feb 2009 in Grenoble, France.

The last meeting (rubbish photo I took below) was a great success, and since so much has happened in the community this year, I expect this one will be even bigger and better.



Update: Sylvain's announcement and the official photo

malloc failures

I can't put a comment on Debarshi's post, so I'll answer here. Debarshi complains about this comment by the "inimitable" Jeff Johnson:

You have to look at the usage case, malloc returning NULL is a "can't happen" condition where an exit call is arguably justified.

Returning an error from library to application when malloc returns NULL assumes:

1) error return paths exist [...]
2) applications are prepared to do something meaningful with the error

Another problem is that only about 1 in 10 memory allocations in a typical C program are mallocs. The rest are stack-allocated variables, and those aren't usually checked at all. If any of your 9 out of 10 stack allocations fail, your whole program fails hard.

This is the correct way to deal with those 1 in 10 memory allocations that you can check — provide a custom abort function that the main program can override in the very rare case that they can do anything useful other than exit:

void (*custom_abort) () = abort;

void
lib_set_custom_abort (void (*new_abort) ())
{
  custom_abort = new_abort;
}

void *
lib_malloc (int n)
{
  void *data = malloc (n);
  if (data == NULL) custom_abort ();
  return data;
}

Note that the main program can use longjmp (or exceptions in some cases) to "return" back to a safe point in the program, such as a transaction checkpoint. If the main program uses pool allocators — about the only safe and sensible way to deal with C's programming model — then the program has a chance of recovering.

Really the answer is to use a sensible programming language though. Programming languages invented before C had safer, faster memory allocation, dealt with 10 out of 10 memory allocation errors, and provided a mechanism to recover correctly. Those languages are now 30 years more advanced. In 2008 we're having these silly arguments about how to deal with malloc failures. That's a failure of ourselves as programmers.

MinGW: Inkscape cross-compiled from Fedora to Windows

Inkscape, cross-compiled for Windows from Fedora. We use Fedora and a completely open source tool chain for build, and Wine for testing. Windows is only required for final checks before deployment.

MinGW: It's got an icon and it works!

As you can see from the screenshot below, I addressed Nicu's complaint and added a simple icon to the virsh (virt shell) EXE file. Here's how to do that again using all open source tools. We also a fixed a rather embarrassing endianness bug in our XDR implementation, and so virsh/libvirt can talk to remote libvirtd servers.

MinGW: Screenshots

Previously I showed you how to build software on Fedora which will run on Windows. You can run this software on Fedora using Wine, but it's also nice to know that it even runs on a real Windows machine. Here are some screenshots of the installer running under Windows XP. Remember that this was entirely created on a Fedora Linux system, using completely open source software:


The menubar across the top of the screen comes from virt-viewer. Windows is running virtualized.




Notice the desktop shortcuts for each application, added by nsiswrapper automatically. In future we'll actually want to disable some of these since they don't really make sense for command line applications.


Who's the "surfer dude"?

IDN use and abuse

JWZ blogged about the Unicode snowman. If you're running a proper browser, take a close look at the domain name:
☃.net

A brief, two sentence overview: For any domain name which begins with xn-- followed by some gobbledygook, certain clients like web browsers can interpret the gobbledygook as a Punycode representation of some Unicode string. So the snowman's real domain name is xn--n3h.net.

I was quite excited for a while since many of these Unicode dingbats and symbols are unregistered in combinations of two or more, but then I found that the killjoys at the IETF had put a stop to that with RFC 4690. So while the snowman registration can be continued, no new dingbats can be registered.

Nevertheless, we can still have fun abusing the simpler Chinese characters. For a laugh I registered 丄.com and 丿乀.com. These might not be active when you read this, and to be honest I'm not quite sure where I'll point them at the moment. The first looks like bottom, the symbol for non-terminating programs. Hmmm maybe that'd be good for some insightful blog about functional programming? The second is a total abuse of two characters together, but looks like the number 8 in Japanese (IETF rules forbid registering actual numbers, even non-Arabic ones).

Someone should do the world a favour and register 丅丨丅.com (xn--9gqa8h.com).

Update

Subdomains are of course not regulated by the IETF jobsworths. Here's another, prettier unicode snowman: http://☃.earthlingsoft.net/, and I can have http://☆☆☆.annexia.org/