Destroying old hard drives

This is a simple, cheap method for destroying old hard drives, making the data unrecoverable against casual attackers and identity fraudsters (although probably not some hypothetical government agency with multi-million dollar resources).

For this you will need a stack of old hard drives:


An electric drill with a twist drill bit (suitable for going through metal), and most importantly some eye protection:


Line up the hard drives against the wall and drill straight through them. I didn't show it in this picture, but in fact I drilled through from the other (PCB) side to ensure that I went through the PCB but didn't go through any components that might explode:


Sunlight where there's not supposed to be sunlight!


For a few of the drives, mainly older ones, I couldn't get all the way through, but I got through to the platters, which is the important part:


Now you can see why eye protection is not optional. This old IBM SCSI-LVD drive had glass platters which shattered into tiny, sharp shards of metal-plated glass when the drill went through:


For extra assurance, I will soak the drives in a bucket of water for a few days before disposing of them:

Common mistakes cross-compiling MinGW packages

Using the headers from /usr/include

The headers in /usr/include are for the native libraries installed on the system, and it's highly unlikely they will work for cross-compilation. By "won't work" I mean that types and structure fields could be different, resulting in a segfault.

The Fedora MinGW project takes two steps to avoid using native libraries by accident: Firstly GCC is configured so it looks in /usr/i686-pc-mingw32/sys-root/mingw/include and never looks in /usr/include (as long as you don't tell it to). Secondly we supply a replacement %{_mingw32_configure} RPM macro which sets PKG_CONFIG_PATH, so any pkg-config done will pick up the cross-compiled libraries' configuration instead of any native libraries' configuration.

$ PKG_CONFIG_PATH=/usr/i686-pc-mingw32/sys-root/mingw/lib/pkgconfig \
  pkg-config --cflags glib-2.0
-mms-bitfields -I/usr/i686-pc-mingw32/sys-root/mingw/include/glib-2.0
-I/usr/i686-pc-mingw32/sys-root/mingw/lib/glib-2.0/include  

One thing that can still go wrong is that you don't have the cross-compiled library installed and it then picks up the native library. For example, you missed a BuildRequires line. That mistake usually becomes evident when the program tries to link, because linking a cross-compiled Windows binary to a native Fedora library won't work.

Not setting --prefix

You likely don't want to install Windows binaries and libraries under /usr or /usr/local. For a start it's better to keep Windows things in one place, and the packaging guidelines have specified that place to be /usr/i686-pc-mingw32/sys-root/mingw. But mainly it's not a good idea to mix up native and cross-compiled libraries, which will cause all sorts of problems as in the point above.

If you use %{_mingw32_configure} in RPM specfiles, or the mingw32-configure command, then paths will be set correctly for you.

Not using a portability library

If you're writing the program yourself, or if you're doing the often difficult work of porting an existing application, use a portability library to help you. Which you choose is up to you and depends on many factors, but we would recommend that you look at these ones:

• glib2 and gtk2
• Gnulib
• apr
• QtCore, Boost, POCO or NSPR if you're using C++

Writing your own build system

While it's fashionable to dislike autoconf and m4 macros, it is still by far the easiest way to both build your program on multiple systems, and to cross-compile. So use autotools or cmake, and definitely don't write your own build system. Discourage other projects from writing their own build systems too.

This really comes down to bitter experience. Every project we have had to port that has used its own build system has been far more of a headache than those that just used autoconf or cmake.

Running programs during the build process

When cross-compiling, it's always a mistake to run programs during essential build steps. The problem is that you can't be sure that binaries can be made to work in the build environment. For Windows binaries, there is some chance of running them under Wine, but Wine itself is incompatible with autobuild environments like mock and Koji. Furthermore Wine only works on x86 platforms, and it's not possible to use it at all when cross-compiling from other architectures like PPC.

Running programs during make test is normal and useful though.

malloc failures

I can't put a comment on Debarshi's post, so I'll answer here. Debarshi complains about this comment by the "inimitable" Jeff Johnson:

You have to look at the usage case, malloc returning NULL is a "can't happen" condition where an exit call is arguably justified.

Returning an error from library to application when malloc returns NULL assumes:

1) error return paths exist [...]
2) applications are prepared to do something meaningful with the error

Another problem is that only about 1 in 10 memory allocations in a typical C program are mallocs. The rest are stack-allocated variables, and those aren't usually checked at all. If any of your 9 out of 10 stack allocations fail, your whole program fails hard.

This is the correct way to deal with those 1 in 10 memory allocations that you can check — provide a custom abort function that the main program can override in the very rare case that they can do anything useful other than exit:

void (*custom_abort) () = abort;

void
lib_set_custom_abort (void (*new_abort) ())
{
  custom_abort = new_abort;
}

void *
lib_malloc (int n)
{
  void *data = malloc (n);
  if (data == NULL) custom_abort ();
  return data;
}

Note that the main program can use longjmp (or exceptions in some cases) to "return" back to a safe point in the program, such as a transaction checkpoint. If the main program uses pool allocators — about the only safe and sensible way to deal with C's programming model — then the program has a chance of recovering.

Really the answer is to use a sensible programming language though. Programming languages invented before C had safer, faster memory allocation, dealt with 10 out of 10 memory allocation errors, and provided a mechanism to recover correctly. Those languages are now 30 years more advanced. In 2008 we're having these silly arguments about how to deal with malloc failures. That's a failure of ourselves as programmers.

MinGW: Compile software for Windows without leaving your Fedora machine

For the last few weeks I've been focused on the Fedora MinGW project. This project gives Fedora users a compelling new feature: you can build your software for Windows, without ever needing to leave the Fedora / Linux environment. In fact you can do everything, up to and including creating a Windows installer for your customers, without needing once to touch Windows.

To demonstrate how this works, I'm going to show you how to port a simple application to Windows, using Fedora MinGW. The app I've chosen is virt-viewer, a graphical console viewer for virtual machines, written in C.

First we install the cross-compiler environment and any libraries that our program requires. (Until the MinGW packages are accepted into Fedora, you'll have to get them from our temporary yum repository)

yum install mingw32-gcc mingw32-binutils \
  mingw32-gtk2 mingw32-gtk-vnc mingw32-libvirt mingw32-libxml2 \
  mingw32-nsis mingw32-nsiswrapper

With software such as virt-viewer that is based on the standard autoconf "configure" script, the cross-compiling step is simple. You just have to do:

./configure --host=i686-pc-mingw32

That's all you have to do to configure virt-viewer (and most other software) to cross-compile for Windows.

Now we just do make and discover ... ah, that it doesn't compile. This leads us to the hard part of porting software over to Windows. Windows uses the Win32 API instead of the usual POSIX / libc API found on Linux.

For virt-viewer there are several problems:

1. virt-viewer uses some header files like <sys/socket.h> which aren't found under Win32.
   
2. We need to include <windows.h> on Windows (but not on Linux). For Win32, this header file is analogous to <stdlib.h> or <unistd.h>, and almost every C source file should include it.
   
3. virt-viewer makes some Linux-specific system calls which aren't available in the Win32 API. The problematic calls are:
   
   
   • usleep (sleep for a specified number of microseconds)
     
   • fork (create a subprocess)
     
   • socketpair (create a pipe to communicate with the subprocess)
     

Problems (1) and (2), the missing header files, are easily solved in a very portable way. For each header file which is missing on Windows or Linux, we will just add a configure-time test and some #ifdef magic. Into configure.ac we put:

AC_CHECK_HEADERS([sys/socket.h sys/un.h windows.h])

and then into the C sources files we put:

#ifdef HAVE_SYS_SOCKET_H
#include <sys/socket.h>
#endif

and so on.

Problem (3) -- missing APIs -- are the hardest problems to solve. In general there are three strategies we could try:

(a) Try to find an equivalent but different API which is present on Linux and Windows. As an example here, Windows has a call which is very similar to pipe, and might be used to replace socketpair.
(b) Write a replacement function for each problematic API.
(c) Comment out the particular feature in the code which uses the missing calls. This is less satisfactory of course: Windows users will now be missing some feature.

We're going to fix problems in (3) with a mixture of strategies (b) and (c).

Windows doesn't have usleep, but looking at MSDN I see that it does have a function Sleep (DWORD milliseconds) which can be used as a replacement for usleep.

You can test and replace functions conditionally by adding this to configure.in:

AC_REPLACE_FUNCS([usleep])

Remember that you don't want to replace this on Linux and any platforms that have usleep, and that is what AC_REPLACE_FUNCS does.

The code to implement usleep is now placed into a single function in a file with the same name, usleep.c:

#ifdef WIN32
int
usleep (unsigned int usecs)
{
  unsigned int msecs = usecs / 1000;
  if (msecs < 1)
    Sleep (1);
  else
    Sleep (msecs);
}
#endif

The magic of autoconf will ensure this file will only be linked into the main program when it is needed.

As for fork and socketpair, it turns out we are quite lucky. These two calls are only used to implement a specific virt-viewer feature, namely tunneling connections over ssh. If you conclude, as I did, that ssh isn't that common on Windows machines, then you can do as I did and just comment out that feature conditionally when building on Windows.

With those changes, we have now completed our port of virt-viewer to Windows (full patch). After rerunnning:

autoconf
./configure --host=i686-pc-mingw32
make

we are left with virt-viewer.exe, a full Gtk application that runs on Windows.

Creating a Windows installer

To package up Windows applications into full-featured installers, that include menu shortcuts, desktop icons and an uninstaller, we wrote a little helper program called nsiswrapper. As its name suggests, it is a wrapper around the NSIS Windows Installer, which we also ported over to run natively under Fedora.

You'll need to wrap up not just virt-viewer.exe, but the Gtk-related DLLs and helper modules. With nsiswrapper you would do:

  nsiswrapper --run \
    --name "Virt-Viewer" \
    --outfile "Virt-Viewer-for-Windows.exe" \
    --with-gtk \
    /usr/i686-pc-mingw32/sys-root/mingw/bin/virt-viewer.exe

What can OCaml do that you can't do in other programming languages?

Depressingly predictable comments on the Haskell story in Slashdot yesterday such as:

"I have *never* seen it being used since. To my mind they both belong in the category 'interesting, but pointless'."

and

"The point is that there's nothing those languages can do that can't be done, often more easily, with the current crop of popular languages. Elegance cannot beat convenience in the workplace, or in most at any rate."

and so on.

Here are some useful things you can do in OCaml which you cannot do in the "current crop of popular languages".

1. Change the language to suit your task.

In our case the task was to parse binary formats.

If you have a few hours to spare, try writing a C program to parse tcpdump files. Don't forget that the endianness in these files is variable so you'll need lots of if (..) { field1 = htonl (field1); ... }. OK so that's a bit hard. Let's say you want to parse a 6 bit length field 'n' followed by an n+1 bit data field (as a 1-64 bit int). Go and write it in C now.

Using our bitstring extension to OCaml, parsing binary structures is really effortless:

let bits = Bitstring.bitstring_of_file "input.data" in
bitmatch bits with
| { n : 6;
    data : n+1 } -> data

This is the complete tcpdump parser, just 113 lines of code.

And it's fast too. The resulting code compiles down to machine code and inlines direct calls to C functions at every opportunity, so in practice you can parse data at speeds approaching C.

Another task we had was to check that hundreds of SQL statements in a modest sized web application actually matched with fields in the database, in other words that they didn't reference non-existent fields or treat a string field as an integer and so forth. Doing this by testing is almost impossible because many SQL statements are only used on rare error paths. We could contemplate doing it manually once but not routinely.

So instead we extended the OCaml language to do the checking for us every time we compiled the code. The resulting PG'OCaml project gives you compiled-time checked type-safe access to your database. It's used by mod_caml and ocsigen, or you can just use it in standalone programs. I won't go into detail because Dario Teixeira wrote a great introduction and tutorial to PG'OCaml.

Martin Jambon, author of Micmatch which adds regular expressions directly into the language, has an excellent tutorial. Browse the list of syntax extensions here.

2. Get the compiler to check for errors in your logic

Why can't your compiler check for errors like when you use a read-only database connection in a SQL INSERT statement, or you are supposed to call library function set_word_size because you try to call get_word?

In OCaml, using phantom types [tutorial] you can do exactly this.

3. Drop down to imperative code when you need speed like C

Scripting languages are expressive but slow ... and getting slower. But there's no need to compromise expressiveness to get real speed. Static typed languages with type inference (which excludes C, C++, Java, C# and most others, but includes SML, OCaml and F#) give you the expressiveness to write compact code but without compromising on speed.

OCaml programs are more than 10 - 15 times faster and less memory intensive than Python programs.

More about speed and optimizing OCaml programs. Comparison of a raytracer written in C++ and OCaml.

4. Large well-established standard libraries

OCaml comes with hundreds of packages for many tasks now.

If that's not enough you can directly call out to Perl, Java, Python and .Net [on Windows] libraries. Or you can call C functions directly.

You can compile the same code on Unix, Mac OS X and Windows.

So enough of this "elegance can't beat convenience" stuff please.

Tip: Read all lines from a file (the most common OCaml newbie question?)

Is this the most common OCaml beginners question? It comes up every few weeks on the OCaml beginners list, and I have tried to answer it before.

It seems that everyone who learns OCaml comes away with the impression that functional programming is the New Cool Thing, and imperative programming is Bad and Must Be Avoided.

I'm going to say it now: programming fashions are stupid and counterproductive. The only things that matter are that your program is short, easy to write, easy to maintain and works correctly. How you achieve this has nothing to do with programming fads.

Reading all lines from a file is an imperative problem, and the shortest solution (easy to write, easy to maintain and correct¹) uses a while loop, in OCaml or any other language:

let lines = ref [] in
let chan = open_in filename in
try
  while true; do
    lines := input_line chan :: !lines
  done; []
with End_of_file ->
  close_in chan;
  List.rev !lines

Actually, no, I'm lying. The best solution is this:

Std.input_list chan

which is supplied by extlib. Don't bother to duplicate functions which are already provided in commonly available libraries.

---

¹This is only strictly speaking correct if you handle clean-up if input_line throws some read error (exception). In the common case where you just exit the program, leaving the channel open is perfectly acceptable.

Writing printf-like functions

Here's a quick and useful tip: failwith is the standard function to raise a general error, but it's a little bit clumsy to use because it only takes a fixed string.

if temp >= 100 then
  failwith "we've reached boiling point"

If you want to have the message contain useful debugging information, you need to use sprintf to generate the fixed string, like:

if temp >= 100 then
  failwith (sprintf "%d degC: boiling point reached" temp)

(I'm assuming here that you have open Printf at the top of your file, something which you should almost always do so you don't need to write Printf.sprintf all the time,).

With this simple tip we can turn failwith into a function that automatically takes a printf-like format string, and we can learn a little bit about the arcana of polymorphic types too.

First of all, here is the code:

let failwith format = ksprintf failwith format

You can see in the toplevel that it works:

# let failwith format = ksprintf failwith format ;;
val failwith : ('a, unit, string, 'b) format4 -> 'a = <fun>
# failwith "hello, %s" "world" ;;
Exception: Failure "hello, world".
# failwith "error code %d" 3 ;;
Exception: Failure "error code 3".

ksprintf is the key function here. Like sprintf it takes a format string and a variable number of parameters, and makes a fixed result string. Unlike sprintf it doesn't return the string, but passes it to the function which is its first parameter — in this case, the standard failwith function. So ksprintf is useful because it can turn almost any fixed string function into a printf-like function.

Now how about the lesson on type arcana? Well if you know anything about currying you might think that you could write the new failwith function even shorter, like this:

let failwith = ksprintf failwith

If you try this, you'll find the new function works some of the time, but fails to type-check at other times. In fact, the first time you use it, it seems to "remember" the type of all the arguments, and then refuses to work if any of those types change:

# failwith "hello, %s" "world" ;;
Exception: Failure "hello, world".
# failwith "error code %d" 3 ;;
This expression has type (int -> 'a, 'b, 'c, 'd, 'd, 'a) format6
but is here used with type
  (string -> 'e, unit, string, 'e) format4 =
    (string -> 'e, unit, string, string, string, 'e) format6

If we take a close look at the inferred types of the wrong definition, we can see why:

# let failwith = ksprintf failwith ;;
val failwith : ('_a, unit, string, '_b) format4 -> '_a = <fun>

'_a (with an underscore) is not a polymorphic type, but a single type that the compiler just hasn't been able to infer fully yet. As soon as you give it more information (eg. calling the function), the compiler infers that type into some concrete type (like string -> ... above) and won't let you change it later.

A more advanced question is to work out why type inference fails to infer the more general polymorphic type. I suspect this FAQ may have the answer.

Sharing a global variable between C and OCaml

I was asked today if it is possible to share a global variable between C and OCaml. This was my first response:

OCaml cannot just directly access C globals. At best you'd need to have a function that returns the address of the C global, _and_ the C global would need to be in a form that OCaml code could understand although that is pretty easy to arrange.

In a followup, I gave some example code:

------------------------------------------------------------ test_c.c
/* Variable shared between C and OCaml. */

#include <caml/mlvalues.h>
                                                                                
/* On the OCaml side, this will be made to look like a structure
 * containing a single int (well, provided you don't look *too*
 * closely at it). 
 */
value shared_var = Val_int (0); 

value
get_struct_addr (value unitv)
{
  /* Return the address of the 'structure'. */
  return ((value) &shared_var);
}

/* Increment the shared variable. */
value
increment_it (value unitv)
{
  int i = Int_val (shared_var);
  i++;
  shared_var = Val_int (i);
  return (Val_unit);
}
----------------------------------------------------------------------

------------------------------------------------------------ test.ml
(* Variable shared between C and OCaml. *)

type var = {
  shared_var : int;
}

external get_struct_addr : unit -> var = "get_struct_addr" "noalloc"
external increment_it : unit -> unit = "increment_it" "noalloc"

let var = get_struct_addr () ;;

while true do
  Printf.printf "value of the variable now is %d\n%!" var.shared_var;
  increment_it ();
  (* OCaml isn't expecting that increment_it modifies a variable, so
   * there is no guarantee that we will see the changed value next
   * time around.
   *)
  Unix.sleep 1;
done
----------------------------------------------------------------------

$ gcc -I /usr/lib/ocaml -c test_c.c
$ ocamlopt -c test.ml
$ ocamlopt unix.cmxa test_c.o test.cmx -o test
$ ./test
value of the variable now is 0
value of the variable now is 1
value of the variable now is 2
value of the variable now is 3
value of the variable now is 4
[etc.]

Before you use this code, be aware that it's a real hack.

Tip: Calling C functions directly with "noalloc"

In OCaml you can call a C function by writing:

external call_1 : float -> float = "call_1"

then just using call_1. However these calls are not direct. They go via an OCaml runtime function called caml_c_call. This is a tiny bit of assembler, so the overhead isn't large, but it does use a computed jump which on many processors is quite slow.

Luckily this indirection is only needed in order to set up the garbage collector. If your C function won't perform any OCaml allocations, then you don't need this, and you can tell OCaml to jump directly to your C function like this:

external call_2 : float -> float = "call_2" "noalloc"

Let's compare the generated assembly code for the calls in both cases:

Normal                                    "noalloc"       

pushl   %eax                              pushl   %eax
movl    $call_1, %eax                     call    call_2
call    caml_c_call                       addl    $4, %esp
addl    $4, %esp
   ...
caml_c_call:
movl    (%esp), %edx
movl    %edx, G(caml_last_return_address)
leal    4(%esp), %edx
movl    %edx, G(caml_bottom_of_stack)
jmp     *%eax

As you can see, the "noalloc" version is much shorter.