Last week was “down tools week” at Red Gate. This happens several times a year and consists of a week of time during which developers are allowed to work on any project they like. Some people spend the time learning new tools, some prototype a new product and some just spend the time working on something they find interesting.
I chose to work with Laurie on a project to do zip backup of an SQL database. This was to be based on a current Red Gate product, HyperBac, which uses a device driver (a file system filter) and a user level service to intercept read and write calls on chosen files by chosen processes, replacing the data that the reading/writing process sees with a modification of the data that comes from the disc. The idea of the project was to gain some experience of Clojure, a language that I’ve blogged a fair amount about in the past, particularly in the early 1.1 and 1.2 versions, though haven’t used it for some time.
The device driver communicates with the user process code using messages to and from a HyperBac device. This meant that we needed to use low level Win32 calls and deal with unmanaged memory, so we used the JNA library to do this. Some of the project therefore involved transcribing datatypes implemented as C structs into the JNA equivalent. JNA made this fairly straight forward.
Clojure seems to have moved forward quite a lot since I last used it. It’s now really easy to get started, and if you are going to use Leiningen you simply need to have the Java runtime installed on your machine and then you can this script which will use wget to download the various Clojure libraries when you run it with the self-install command line argument.
We did all of editing inside emacs, using the swank-clojure code to allow us to compile and load editor buffers into the running Clojure session. This mode offers a clojure-jack-in command which searches up the directory structure from the location of the current buffer, finds a Leiningen project, starts Clojure and loads the project into the session, and then links the emacs session into it. This makes it really easy to start a REPL, update it with modifications and other code changes, and then interactively test out the added functionality.
After a couple of blue screens, we started running inside a virtual machune with windbg attached across a virtual serial port as a kernel debugger. This made it fairly easy to debug the JNA calls that we were using, as we could see in the REPL (read-eval-print-loop – the interactive prompt) the data that we were trying to write to the device driver and then we could set breakpoints in the driver code and look at the data structures that were passed to see if they contained the right data. The REPL made the whole experience highly efficient – rather than writing a program to validate a series of messages to and from the driver, we could issue a request to the driver, check the result and then get that working before sending the next message.
Leiningen projects have a straightforward structure. In order to use the swank integration, you just need to add the lein-swank plugin to the project definition. The project file defines the entry point of the code for when you want to generate a standalone application from the project. This works really well – you develop inside the REPL, dynamically adding code as you go along, and then produce a standalone jar that contains the final application and starts it at the defined entry point.
The warn-on-reflection causes Clojure to warn in cases where it needs to use reflection, allowing you at add type annotations to get better performance. These annotations allow the system to add a simple type check to code, after which the code can be specialised to expect the defined type.
The JNA definitions that come with some with the latest version of the library were missing some Win32 functions, so we defined them using the following JNA definition. The JNA library uses reflection to find the methods in the interface, and generates the appropriate interop thunks.
We could use these definitions inside Clojure using code like the following.
(.SetFileInformationByHandle Win32Extras/INSTANCE fh FileEndOfFileInfo size 8)
We used a Java zip library which expects you to provide an OutputStream. Clojure provides a way to implement Java interfaces, allowing you to name the function that is called to construct the new instance, and the implementations of the various methods from the interface.
After the above definition, we defined Clojure methods with the following signatures to provide the implementation.
(defn –init [fh] …)
(defn –close [this] …)
(defn –write-int [this b] …)
The main complication was that the service needed to handle the file being written as a set of non-contiguous blocks which we needed to buffer and then pass through as contiguous blocks to the zip library code. There was also the complication of the device driver only accepting blocks with size a multiple of 512 bytes (apart from the final block). In order to write a smaller segment of memory, the existing driver needs to issue a read to get the full block into memory and then write back this data after the new values have been merged in. Our use of a zip stream allowed us to buffer and hence avoid this complication.
After getting the code working, and testing the entry point from the REPL, the Leiningen uberjar command produces a jar file that contains the entire application plus all of the libraries on which it depends. This jar is easy to copy around and run on another machine.
The whole experience reinforced to me how useful a REPL is for exploring while you develop code. [When programming C# inside Visual Studio I sometimes to do some experimentation from inside the immediate window, but this is far from being a proper REPL windows.] It’s easy to check your understanding by making instances of various objects, calling methods and checking the results. All the benefits of Lisp… while still being able to run and deploy like any old Java application.