quicklisp and clsql-fluid

This is a tech post and the intended audience is the few people who will bash their heads against a piece of software until it gets installed. I’ve just gone through one of these “sessions”, so I thought I’d document it here for the next person.

Zach Beane created quicklisp, which is a great way to load up common lisp libraries without a headache.

To give an example of this, to load my favorite web programming framework, all I need to do is type the following: (ql:quickload 'weblocks)

It loads everything I need and I can instantly get started working. Very cool! Now after a bit of programming I find that I’d like to customise the storage layer to use a mysql database, but that’s when it gets hairy…

First of all, weblocks’ clsql interface depends on clsql-fluid, which as a project doesn’t package anything… so this doesn’t give quicklisp an opportunity to distribute it. I guess someone could make a package, but I assume Zach would like to have the project maintainers do this instead of himself.

Secondly quicklisp already had installed the clsql library for me and I couldn’t find a command to remove it. So after looking through the documentation and the source I went and deleted things under my quicklisp directory (and ~/.cache) to remove quicklisp’s version of clsql. A remove command in quicklisp would be a nice touch. It would also be nice to get some like functionality for quicklisp in managing software beyond installing & searching. Maybe Zach already has plans for these for when the code goes out of beta…

Thirdly, I fetched clsql-fluid as described on the website and linked it in and loaded it up… but got errors:

"The name NIL does not designate any package."

What is happening here is that clsql-uffi tests that it’s properly loaded by looking for a defined symbol (atol64) to have an associated function. If that function doesn’t exist then we havn’t loaded the foreign libraries and therefore haven’t successfully loaded. The problem with this is that the clsql-uffi package doesn’t exist yet, so we get the error when the test code tries to (find-package #:clsql-uffi).

Since this test is a predicate of sorts I decided that instead of erroring it should just return nil, so I wrapped those probing statements in an: (ignore-errors ...) statement. Once I’d done this it allowed me to get to the next level of errors :)

Fourthly, the syntax used in the uffi calls seems different than what the uffi installed via quicklisp expects. After getting beyond the “nil package” problem above, I encountered this error:

caught ERROR:
; (in macroexpansion of (UFFI:CONVERT-FROM-FOREIGN-STRING CHAR-PTR
:LOCALE ...))
; (hint: For more precise location, try *BREAK-ON-SIGNALS*.)
; error while parsing arguments to DEFMACRO CONVERT-FROM-FOREIGN-STRING:
; unknown keyword: :LOCALE; expected one of :NULL-TERMINATED-P,
:LENGTH

After looking around a bit I got the tip that cffi-uffi-compat might be more compatible than quicklisp’s version of uffi, so I edited clsq.asd, clsql-mysql.asd, clsql-uffi.asd, and uffi/clsql-uffi-package.lisp and edited them to use the uffi compatibility layer from cffi. After all that was done I did another (ql:quickload 'clsql-mysql) and the clsql-fluid version loaded. whew

I hope this helps others stuck in a similar situation and here is the result of a git diff in the root of the clsql-fluid source:

diff --git a/clsql-mysql.asd b/clsql-mysql.asd
index a176f85..1263457 100644
--- a/clsql-mysql.asd
+++ b/clsql-mysql.asd
@@ -20,8 +20,8 @@
 (in-package #:clsql-mysql-system)
 
 (eval-when (:compile-toplevel :load-toplevel :execute)
-  (unless (find-package 'uffi)
-    (asdf:operate 'asdf:load-op 'uffi)))
+  (unless (find-package 'cffi-uffi-compat)
+    (asdf:operate 'asdf:load-op 'cffi-uffi-compat)))
 
 (defvar *library-file-dir* (append (pathname-directory *load-truename*)
                                   (list "db-mysql")))
@@ -48,8 +48,8 @@
   t)
 
 (defmethod operation-done-p ((o load-op) (c clsql-mysql-source-file))
-  (and (symbol-function (intern (symbol-name '#:mysql-get-client-info)
-                               (find-package '#:mysql)))
+  (and (ignore-errors (symbol-function (intern (symbol-name '#:mysql-get-client-info)
+                                              (find-package '#:mysql))))
        t)) 
 
 (defmethod perform ((o compile-op) (c clsql-mysql-source-file))
@@ -80,7 +80,7 @@
   :description "Common Lisp SQL MySQL Driver"
   :long-description "cl-sql-mysql package provides a database driver to the MySQL database system."
 
-  :depends-on (uffi clsql clsql-uffi)
+  :depends-on (cffi-uffi-compat clsql clsql-uffi)
   :components
diff --git a/clsql-mysql.asd b/clsql-mysql.asd
index a176f85..1263457 100644
--- a/clsql-mysql.asd
+++ b/clsql-mysql.asd
@@ -20,8 +20,8 @@
 (in-package #:clsql-mysql-system)
 
 (eval-when (:compile-toplevel :load-toplevel :execute)
-  (unless (find-package 'uffi)
-    (asdf:operate 'asdf:load-op 'uffi)))
+  (unless (find-package 'cffi-uffi-compat)
+    (asdf:operate 'asdf:load-op 'cffi-uffi-compat)))
 
 (defvar *library-file-dir* (append (pathname-directory *load-truename*)
                                   (list "db-mysql")))
@@ -48,8 +48,8 @@
   t)
 
 (defmethod operation-done-p ((o load-op) (c clsql-mysql-source-file))
-  (and (symbol-function (intern (symbol-name '#:mysql-get-client-info)
-                               (find-package '#:mysql)))
+  (and (ignore-errors (symbol-function (intern (symbol-name '#:mysql-get-client-info)
+                                              (find-package '#:mysql))))
        t)) 
 
 (defmethod perform ((o compile-op) (c clsql-mysql-source-file))
@@ -80,7 +80,7 @@
   :description "Common Lisp SQL MySQL Driver"
   :long-description "cl-sql-mysql package provides a database driver to the MySQL database system."
 
-  :depends-on (uffi clsql clsql-uffi)
+  :depends-on (cffi-uffi-compat clsql clsql-uffi)
   :components
   ((:module :db-mysql
            :components
diff --git a/clsql-uffi.asd b/clsql-uffi.asd
index e5a75f6..e78f09e 100644
--- a/clsql-uffi.asd
+++ b/clsql-uffi.asd
@@ -50,8 +50,8 @@
   nil) ;;; library will be loaded by a loader file
 
 (defmethod operation-done-p ((o load-op) (c clsql-uffi-source-file))
-  (and (symbol-function (intern (symbol-name '#:atol64)
-                               (find-package '#:clsql-uffi)))
+  (and (ignore-errors (symbol-function (intern (symbol-name '#:atol64)
+                                              (find-package '#:clsql-uffi))))
        t))
 
 (defmethod perform ((o compile-op) (c clsql-uffi-source-file))
@@ -79,7 +79,7 @@
   :description "Common UFFI Helper functions for Common Lisp SQL Interface Library"
   :long-description "cl-sql-uffi package provides common helper functions using the UFFI for the CLSQL package."
 
-  :depends-on (uffi clsql)
+  :depends-on (cffi-uffi-compat clsql)
 
   :components
   ((:module :uffi
diff --git a/clsql.asd b/clsql.asd
index 35e172a..51da309 100644
--- a/clsql.asd
+++ b/clsql.asd
@@ -26,8 +26,8 @@
 
 ;; need to load uffi for below perform :after method
 (eval-when (:compile-toplevel :load-toplevel :execute)
-  (unless (find-package 'uffi)
-    (asdf:operate 'asdf:load-op 'uffi)))
+  (unless (find-package 'cffi-uffi-compat)
+    (asdf:operate 'asdf:load-op 'cffi-uffi-compat)))
 
 (defsystem clsql
     :name "CLSQL"
diff --git a/uffi/clsql-uffi-package.lisp b/uffi/clsql-uffi-package.lisp
index be5abf5..153ca1f 100644
--- a/uffi/clsql-uffi-package.lisp
+++ b/uffi/clsql-uffi-package.lisp
@@ -19,7 +19,7 @@
 (in-package #:cl-user)
 
 (defpackage #:clsql-uffi
-  (:use #:cl #:uffi)
+  (:use #:cl #:cffi-uffi-compat)
   (:export
    #:find-and-load-foreign-library
    #:canonicalize-type-list

Stopping to look at the scenery

In working with old VW’s you end up looking at a lot of… old VW stuff. I assume this is true of most things you do, you end up looking at historical info to get an idea where to take the next step.

Sometimes this leads you to the most boring series of endless searches that yield nothing, but sometimes you luck out and find what your looking for and can even be pleasantly surprised along the way:


471155
stare

right

Interestingly enough I arrived at the reason why I was looking in these old brochures in the first place… the hose layout for my 1963 :)


left
right

It’s nice when you can get period correct info from a sources that are so inspiring. You can check out more at thesamba.com

American vs European Consumer Protection.

I subscribe to the bicycle frame builders mailing list and like most good mailing lists threads can go on forever and get mired in details or tangents. Occasionally things jump out at you as important and this post by Elliott McFadden did and I thought I’d pass it along:

I listened to a product liability attorney talk about the difference
between the American and European systems of consumer protection. In
America, we keep regulations relatively light and underfund enforcement
then let lawsuits do the real heavy lifting. In Europe, regulation is
higher and enforcement well staffed and financed while large lawsuits are
relatively rare.

As a citizen and a business owner, I’d rather have the European model. It
seems more predictable and less expensive in the long run. Plus
governments are usually the only ones with the resources and time to
regularly take on real abusers with deep pockets.

bus updates

It’s only fitting to give an update on today 9/20/10, which happens to be my bus’ 48th birthday.

To celebrate, I worked a bit on the headlight buckets.  This is the driver side bucket, which has some rust in a few areas due to the elements.  The rubber seals were very brittle and may have never been changed.  Now 48 years later it’ll get it’s “eyes” cleaned up and converted back to 6v sealed beams.  I’ll be painting tomorrow after work.

Updates can be followed on my photo stream on flickr.

Twisted spoke patterns

It is totally structurally unsound to do so, but man it looks cool:

I’ve thought of doing a few myself, but for a later time when I’m a more experienced wheel builder and I’d like some wall hangers.  Currently my most unorthodox builds have been a three leading / three trailing pattern on Mehran’s bike and a crow foot pattern on my Rebolledo track bike.

Franz mentioned on Dr Dobbs site

A lot of people ask about my day job at Franz and I usually have a rehearsed answer that goes something like this:

We work on a version of a programming language called Common Lisp and we have built tools on top of it. One of these tools is a database useful for studying complex data sets.

It would be cool to say something like “twitter” or “vimeo” and have instant recognition of my area of work, but with Franz you get a puzzled look and usually 10 minutes of back and forth to properly describe what we do.

With that confusion in mind I jumped to read this Dr. Dobbs interview with our CEO, Jans. I think to the technical people out there, some of this will make sense:

Dr. Dobbs interview with Jans Aasman of Franz Inc

Converting composite tiff files to pdf

I wrote a few posts ago about the joys of converting a bunch of images into a pdf with a single command. Well I’ve been working with a set of tiff files files that Andrew Whitlock gave me for the oacdp project. The tiff files are a composite tiff’s meaning that they themselves are made up of a collection of tiff files.

My goal is to extract the tiff files, convert them to png, and then create a pdf from those pngs. I chose imagemagick because it’s been a great tool and is straightforward to use. I found that wasn’t quite the case for these files.

Firstly the tiff file needs to be broken into it’s component images

convert composite.tiff image-%03d.png

This produces a set of files that are named “image-000.png” and up.

Secondly the tiff files need to be assembled into a pdf file

convert images-*.png final.pdf

As discussed in my previous post, it’s not that easy because there is the issue of imagemagick’s poor memory handling when creating a pdf, so I used the pdfjoin technique from the previous post

pdfjoin --outfile final.pdf images-*.png

All is good.

Now enter an enormous tiff file, with 318 sub images that also brings it’s own set of issues with imagemagick. Often attempting to extract the sub images was causing my machine to completely freeze and kernel panic, which may be more of a kernel issue than imagemagick’s problem, but it was the immensely huge memory usage that brought it there, so I decided to learn another lesson from my previous post and I broke the job into many small calls to imagemagick rather than one large one.

Using my language of choice, I wrote a bit of lisp to do the decomposition of the tiff file, conversion to png, and assemblage into a pdf. Here it is:


;;;; Copyright 2010 Elliott Johnson
;;;; Distributed under the GPL - 3.0
;;;; http://www.gnu.org/licenses/gpl.html

(defpackage :net.elliottjohnson.lisp.convert
  (:use :cl
        :cl-user
        #+sbcl :sb-ext)
  (:nicknames :convert))
(in-package :net.elliottjohnson.lisp.convert)

(defvar *convert-binary*
  "/usr/bin/convert"
  "An acceptable name for the convert binary that's installed on your system.")
(defvar *identify-binary*
  "/usr/bin/identify"
  "An acceptable name for the identify binary that's installed on your system.")
(defvar *pdfjoin-binary*
  "/usr/bin/pdfjoin"
  "An acceptable name for the pdfjoin binary that's installed on your system.")

(defun acceptable-exit-code (process)
  (when (= 0 (process-exit-code process))
    t))

#+sbcl
(defun image-count (multi-image-filename)
  (let ((process (run-program
                  *identify-binary*
                  (list multi-image-filename)
                  :output :stream
                  :error nil)))
    (if (acceptable-exit-code process)
        (let ((file-count 0)
              (stream (process-output process)))
          (if (input-stream-p stream)
              (progn
                (loop for line = (read-line stream nil nil)
                   while line
                   do (incf file-count))
                (1- file-count))
              (error "Bad Outputstream: \"~S ~S\" return ~S~%"
                     *identify-binary*
                     multi-image-filename
                     process)))

        (error "Failed to execute: \"~S ~S\" return ~S~%"
               *identify-binary*
               multi-image-filename
               process))))

#+sbcl
(defun simple-convert (source destination &optional args)
  (let ((process (run-program *convert-binary*
                              (if args
                                  (if (listp args)
                                      `(,source ,@args ,destination)
                                      (list source args destination))
                                  (list source destination)))))
    (unless (acceptable-exit-code process)
      (error "Failed to execute: \"~S ~S ~S\" returned ~S~%"
             *convert-binary*
             source
             destination
             process))))

#+sbcl
(defun join-pdf (pdf-file-pattern dest-pdf-file)
  (let ((process (run-program *pdfjoin-binary*
                              (list "--fitpaper" "false"
                                    "--outfile" dest-pdf-file
                                    pdf-file-pattern))))
    (unless (acceptable-exit-code process)
      (error "Failed to join pdf: \"~A --outfile ~A ~A\" returned ~S~%"
             *pdfjoin-binary*
             dest-pdf-file
             pdf-file-pattern
             process))))

(defun name-source-file (multi-image-filename current-count)
  (format nil "~A\[~D-~D\]" multi-image-filename current-count current-count))

(defun name-dest-file (dest-dir dest-prefix current-count dest-suffix)
  (format nil
          "~A/~A~3,'0D.~A"
          dest-dir
          dest-prefix
          current-count
          dest-suffix))

(defun convert-suffix-to-pdf (source-file)
  (format nil
          "~A/~A.pdf"
          (directory-namestring source-file)
          (pathname-name source-file)
          '("-density" "100%")))

(defun name-dest-files (dest-dir dest-prefix dest-suffix)
  (format nil
          "~A/~A*.~A"
          dest-dir
          dest-prefix
          dest-suffix))

(defun name-dest-pdffile (dest-dir dest-pdf-name)
  (format nil
          "~A/~A"
          dest-dir
          dest-pdf-name))

(defun create-pdf-from-images (dest-dir
                               dest-prefix
                               dest-suffix
                               dest-pdf-name)
  (let ((source-files (directory (name-dest-files dest-dir
                                                  dest-prefix
                                                  dest-suffix))))
    (loop for file in source-files
         do (simple-convert (format nil "~A" file)
                            (convert-suffix-to-pdf file)
                            '("-density" "100%"))))
  (let ((pdf-file-pattern (name-dest-files dest-dir
                                           dest-prefix
                                           "pdf")))
    (join-pdf pdf-file-pattern
              (name-dest-pdffile dest-dir dest-pdf-name))))

(defun convert-multi-image-file (multi-image-file
                                 dest-dir
                                 dest-prefix
                                 dest-suffix
                                 dest-pdf-name)
  (let ((multi-image-count (image-count multi-image-file)))
    (loop for i from 0 to multi-image-count
         do (simple-convert (name-source-file multi-image-file i)
                            (name-dest-file dest-dir
                                            dest-prefix
                                            i
                                            dest-suffix))))
  (create-pdf-from-images dest-dir dest-prefix dest-suffix dest-pdf-name))

Currently it’s only extended to run on sbcl, but it would be trivial to have it work other places. The function to use is the last one and it’s used something like this:

(convert-multi-image-file "/path/to/my/file.tiff" 
                          "/my/dir/" 
                          "image-" 
                          "png" 
                          "final.pdf")

The end result is a pdf file and the png files needed for the project. The code makes use of the square bracket syntax of imagemagick to specify a particular file in a multi image file (such as a video file, pdf, or in this case a tiff). The Imagemagick FAQ is what eventually lead me to the technique.

This method can render everything out with little load in less than one minute for what used to rarely complete in over 45 minutes with massive loads (above 9).

Updating the Canon 5d MII firmware under linux

I just grabbed the 2.0.4 version of the Canon 5d Mark II firmware and these are the steps it took to install it using GNU/Linux.

  1. Fetch the Mac OS X dmg version of the firmware
  2. Install dmg2img
  3. Convert the DMG to an IMG file
    dmg2img eos5d2204.dmg
  4. mount it temporarily
    mount -t hfs -o loop eos5d2204.dmg /mnt/tmp
  5. insert a CF card and note where it’s mounted
  6. Copy it to the card:
    cp /mnt/tmp/5d200204.fir /path/to/CF/card

See linux is easy ;)

Actually, would it be a pain for Canon to just release the firmware in a zip file? I’d think that would work for everybody, but I’m guessing that they are relying on a possible built in checksumming in a DMG and their exe files? I’m not sure about the checksum, but that would make sense.

For those curious, here is the sha256 checksum of the 5d200204.fir file the above process created:

424b1990b52af12748f9675c2085e58949ac3fc682b1e61717c2218f89cdd149  5d200204.fir

The new firmware adds a bunch more features to the video feature set of the camera, including various frame rate options. For more info see canon’s site.