Decided to take leap day to migrate from my hosting service of:
elliott@services ~ $ uptime
23:14:03 up 1138 days, 11:57, 6 users, load average: 1.16, 1.50, 1.34
Decided to take leap day to migrate from my hosting service of:
elliott@services ~ $ uptime
23:14:03 up 1138 days, 11:57, 6 users, load average: 1.16, 1.50, 1.34
This is a tech post and the intended audience is the few people who will bash their heads against a piece of software until it gets installed. I’ve just gone through one of these “sessions”, so I thought I’d document it here for the next person.
Zach Beane created quicklisp, which is a great way to load up common lisp libraries without a headache.
To give an example of this, to load my favorite web programming framework, all I need to do is type the following: (ql:quickload 'weblocks)
It loads everything I need and I can instantly get started working. Very cool! Now after a bit of programming I find that I’d like to customise the storage layer to use a mysql database, but that’s when it gets hairy…
First of all, weblocks’ clsql interface depends on clsql-fluid, which as a project doesn’t package anything… so this doesn’t give quicklisp an opportunity to distribute it. I guess someone could make a package, but I assume Zach would like to have the project maintainers do this instead of himself.
Secondly quicklisp already had installed the clsql library for me and I couldn’t find a command to remove it. So after looking through the documentation and the source I went and deleted things under my quicklisp directory (and ~/.cache) to remove quicklisp’s version of clsql. A remove command in quicklisp would be a nice touch. It would also be nice to get some like functionality for quicklisp in managing software beyond installing & searching. Maybe Zach already has plans for these for when the code goes out of beta…
Thirdly, I fetched clsql-fluid as described on the website and linked it in and loaded it up… but got errors:
"The name NIL does not designate any package."
What is happening here is that clsql-uffi tests that it’s properly loaded by looking for a defined symbol (atol64) to have an associated function. If that function doesn’t exist then we havn’t loaded the foreign libraries and therefore haven’t successfully loaded. The problem with this is that the clsql-uffi package doesn’t exist yet, so we get the error when the test code tries to (find-package #:clsql-uffi)
.
Since this test is a predicate of sorts I decided that instead of erroring it should just return nil, so I wrapped those probing statements in an: (ignore-errors ...)
statement. Once I’d done this it allowed me to get to the next level of errors :)
Fourthly, the syntax used in the uffi calls seems different than what the uffi installed via quicklisp expects. After getting beyond the “nil package” problem above, I encountered this error:
caught ERROR:
; (in macroexpansion of (UFFI:CONVERT-FROM-FOREIGN-STRING CHAR-PTR
:LOCALE ...))
; (hint: For more precise location, try *BREAK-ON-SIGNALS*.)
; error while parsing arguments to DEFMACRO CONVERT-FROM-FOREIGN-STRING:
; unknown keyword: :LOCALE; expected one of :NULL-TERMINATED-P,
:LENGTH
After looking around a bit I got the tip that cffi-uffi-compat might be more compatible than quicklisp’s version of uffi, so I edited clsq.asd, clsql-mysql.asd, clsql-uffi.asd, and uffi/clsql-uffi-package.lisp and edited them to use the uffi compatibility layer from cffi. After all that was done I did another (ql:quickload 'clsql-mysql)
and the clsql-fluid version loaded. whew
I hope this helps others stuck in a similar situation and here is the result of a git diff
in the root of the clsql-fluid source:
diff --git a/clsql-mysql.asd b/clsql-mysql.asd index a176f85..1263457 100644 --- a/clsql-mysql.asd +++ b/clsql-mysql.asd @@ -20,8 +20,8 @@ (in-package #:clsql-mysql-system) (eval-when (:compile-toplevel :load-toplevel :execute) - (unless (find-package 'uffi) - (asdf:operate 'asdf:load-op 'uffi))) + (unless (find-package 'cffi-uffi-compat) + (asdf:operate 'asdf:load-op 'cffi-uffi-compat))) (defvar *library-file-dir* (append (pathname-directory *load-truename*) (list "db-mysql"))) @@ -48,8 +48,8 @@ t) (defmethod operation-done-p ((o load-op) (c clsql-mysql-source-file)) - (and (symbol-function (intern (symbol-name '#:mysql-get-client-info) - (find-package '#:mysql))) + (and (ignore-errors (symbol-function (intern (symbol-name '#:mysql-get-client-info) + (find-package '#:mysql)))) t)) (defmethod perform ((o compile-op) (c clsql-mysql-source-file)) @@ -80,7 +80,7 @@ :description "Common Lisp SQL MySQL Driver" :long-description "cl-sql-mysql package provides a database driver to the MySQL database system." - :depends-on (uffi clsql clsql-uffi) + :depends-on (cffi-uffi-compat clsql clsql-uffi) :components diff --git a/clsql-mysql.asd b/clsql-mysql.asd index a176f85..1263457 100644 --- a/clsql-mysql.asd +++ b/clsql-mysql.asd @@ -20,8 +20,8 @@ (in-package #:clsql-mysql-system) (eval-when (:compile-toplevel :load-toplevel :execute) - (unless (find-package 'uffi) - (asdf:operate 'asdf:load-op 'uffi))) + (unless (find-package 'cffi-uffi-compat) + (asdf:operate 'asdf:load-op 'cffi-uffi-compat))) (defvar *library-file-dir* (append (pathname-directory *load-truename*) (list "db-mysql"))) @@ -48,8 +48,8 @@ t) (defmethod operation-done-p ((o load-op) (c clsql-mysql-source-file)) - (and (symbol-function (intern (symbol-name '#:mysql-get-client-info) - (find-package '#:mysql))) + (and (ignore-errors (symbol-function (intern (symbol-name '#:mysql-get-client-info) + (find-package '#:mysql)))) t)) (defmethod perform ((o compile-op) (c clsql-mysql-source-file)) @@ -80,7 +80,7 @@ :description "Common Lisp SQL MySQL Driver" :long-description "cl-sql-mysql package provides a database driver to the MySQL database system." - :depends-on (uffi clsql clsql-uffi) + :depends-on (cffi-uffi-compat clsql clsql-uffi) :components ((:module :db-mysql :components diff --git a/clsql-uffi.asd b/clsql-uffi.asd index e5a75f6..e78f09e 100644 --- a/clsql-uffi.asd +++ b/clsql-uffi.asd @@ -50,8 +50,8 @@ nil) ;;; library will be loaded by a loader file (defmethod operation-done-p ((o load-op) (c clsql-uffi-source-file)) - (and (symbol-function (intern (symbol-name '#:atol64) - (find-package '#:clsql-uffi))) + (and (ignore-errors (symbol-function (intern (symbol-name '#:atol64) + (find-package '#:clsql-uffi)))) t)) (defmethod perform ((o compile-op) (c clsql-uffi-source-file)) @@ -79,7 +79,7 @@ :description "Common UFFI Helper functions for Common Lisp SQL Interface Library" :long-description "cl-sql-uffi package provides common helper functions using the UFFI for the CLSQL package." - :depends-on (uffi clsql) + :depends-on (cffi-uffi-compat clsql) :components ((:module :uffi diff --git a/clsql.asd b/clsql.asd index 35e172a..51da309 100644 --- a/clsql.asd +++ b/clsql.asd @@ -26,8 +26,8 @@ ;; need to load uffi for below perform :after method (eval-when (:compile-toplevel :load-toplevel :execute) - (unless (find-package 'uffi) - (asdf:operate 'asdf:load-op 'uffi))) + (unless (find-package 'cffi-uffi-compat) + (asdf:operate 'asdf:load-op 'cffi-uffi-compat))) (defsystem clsql :name "CLSQL" diff --git a/uffi/clsql-uffi-package.lisp b/uffi/clsql-uffi-package.lisp index be5abf5..153ca1f 100644 --- a/uffi/clsql-uffi-package.lisp +++ b/uffi/clsql-uffi-package.lisp @@ -19,7 +19,7 @@ (in-package #:cl-user) (defpackage #:clsql-uffi - (:use #:cl #:uffi) + (:use #:cl #:cffi-uffi-compat) (:export #:find-and-load-foreign-library #:canonicalize-type-list
In working with old VW’s you end up looking at a lot of… old VW stuff. I assume this is true of most things you do, you end up looking at historical info to get an idea where to take the next step.
Sometimes this leads you to the most boring series of endless searches that yield nothing, but sometimes you luck out and find what your looking for and can even be pleasantly surprised along the way:
Interestingly enough I arrived at the reason why I was looking in these old brochures in the first place… the hose layout for my 1963 :)
It’s nice when you can get period correct info from a sources that are so inspiring. You can check out more at thesamba.com
I subscribe to the bicycle frame builders mailing list and like most good mailing lists threads can go on forever and get mired in details or tangents. Occasionally things jump out at you as important and this post by Elliott McFadden did and I thought I’d pass it along:
I listened to a product liability attorney talk about the difference
between the American and European systems of consumer protection. In
America, we keep regulations relatively light and underfund enforcement
then let lawsuits do the real heavy lifting. In Europe, regulation is
higher and enforcement well staffed and financed while large lawsuits are
relatively rare.As a citizen and a business owner, I’d rather have the European model. It
seems more predictable and less expensive in the long run. Plus
governments are usually the only ones with the resources and time to
regularly take on real abusers with deep pockets.
Old lens’ on newer cameras, what a cool concept. I love the results as well. It goes to show that as the accuracy of the recording medium goes up (ie. 21Mpixels for the 5dm2) the more the lens quality and character make the difference in the end photo… now to hit up some old camera stores!
It’s only fitting to give an update on today 9/20/10, which happens to be my bus’ 48th birthday.
To celebrate, I worked a bit on the headlight buckets. This is the driver side bucket, which has some rust in a few areas due to the elements. The rubber seals were very brittle and may have never been changed. Now 48 years later it’ll get it’s “eyes” cleaned up and converted back to 6v sealed beams. I’ll be painting tomorrow after work.
Updates can be followed on my photo stream on flickr.
It is totally structurally unsound to do so, but man it looks cool:
I’ve thought of doing a few myself, but for a later time when I’m a more experienced wheel builder and I’d like some wall hangers. Currently my most unorthodox builds have been a three leading / three trailing pattern on Mehran’s bike and a crow foot pattern on my Rebolledo track bike.
A lot of people ask about my day job at Franz and I usually have a rehearsed answer that goes something like this:
We work on a version of a programming language called Common Lisp and we have built tools on top of it. One of these tools is a database useful for studying complex data sets.
It would be cool to say something like “twitter” or “vimeo” and have instant recognition of my area of work, but with Franz you get a puzzled look and usually 10 minutes of back and forth to properly describe what we do.
With that confusion in mind I jumped to read this Dr. Dobbs interview with our CEO, Jans. I think to the technical people out there, some of this will make sense:
I wrote a few posts ago about the joys of converting a bunch of images into a pdf with a single command. Well I’ve been working with a set of tiff files files that Andrew Whitlock gave me for the oacdp project. The tiff files are a composite tiff’s meaning that they themselves are made up of a collection of tiff files.
My goal is to extract the tiff files, convert them to png, and then create a pdf from those pngs. I chose imagemagick because it’s been a great tool and is straightforward to use. I found that wasn’t quite the case for these files.
Firstly the tiff file needs to be broken into it’s component images
convert composite.tiff image-%03d.png
This produces a set of files that are named “image-000.png” and up.
Secondly the tiff files need to be assembled into a pdf file
convert images-*.png final.pdf
As discussed in my previous post, it’s not that easy because there is the issue of imagemagick’s poor memory handling when creating a pdf, so I used the pdfjoin technique from the previous post
pdfjoin --outfile final.pdf images-*.png
All is good.
Now enter an enormous tiff file, with 318 sub images that also brings it’s own set of issues with imagemagick. Often attempting to extract the sub images was causing my machine to completely freeze and kernel panic, which may be more of a kernel issue than imagemagick’s problem, but it was the immensely huge memory usage that brought it there, so I decided to learn another lesson from my previous post and I broke the job into many small calls to imagemagick rather than one large one.
Using my language of choice, I wrote a bit of lisp to do the decomposition of the tiff file, conversion to png, and assemblage into a pdf. Here it is:
;;;; Copyright 2010 Elliott Johnson ;;;; Distributed under the GPL - 3.0 ;;;; http://www.gnu.org/licenses/gpl.html (defpackage :net.elliottjohnson.lisp.convert (:use :cl :cl-user #+sbcl :sb-ext) (:nicknames :convert)) (in-package :net.elliottjohnson.lisp.convert) (defvar *convert-binary* "/usr/bin/convert" "An acceptable name for the convert binary that's installed on your system.") (defvar *identify-binary* "/usr/bin/identify" "An acceptable name for the identify binary that's installed on your system.") (defvar *pdfjoin-binary* "/usr/bin/pdfjoin" "An acceptable name for the pdfjoin binary that's installed on your system.") (defun acceptable-exit-code (process) (when (= 0 (process-exit-code process)) t)) #+sbcl (defun image-count (multi-image-filename) (let ((process (run-program *identify-binary* (list multi-image-filename) :output :stream :error nil))) (if (acceptable-exit-code process) (let ((file-count 0) (stream (process-output process))) (if (input-stream-p stream) (progn (loop for line = (read-line stream nil nil) while line do (incf file-count)) (1- file-count)) (error "Bad Outputstream: \"~S ~S\" return ~S~%" *identify-binary* multi-image-filename process))) (error "Failed to execute: \"~S ~S\" return ~S~%" *identify-binary* multi-image-filename process)))) #+sbcl (defun simple-convert (source destination &optional args) (let ((process (run-program *convert-binary* (if args (if (listp args) `(,source ,@args ,destination) (list source args destination)) (list source destination))))) (unless (acceptable-exit-code process) (error "Failed to execute: \"~S ~S ~S\" returned ~S~%" *convert-binary* source destination process)))) #+sbcl (defun join-pdf (pdf-file-pattern dest-pdf-file) (let ((process (run-program *pdfjoin-binary* (list "--fitpaper" "false" "--outfile" dest-pdf-file pdf-file-pattern)))) (unless (acceptable-exit-code process) (error "Failed to join pdf: \"~A --outfile ~A ~A\" returned ~S~%" *pdfjoin-binary* dest-pdf-file pdf-file-pattern process)))) (defun name-source-file (multi-image-filename current-count) (format nil "~A\[~D-~D\]" multi-image-filename current-count current-count)) (defun name-dest-file (dest-dir dest-prefix current-count dest-suffix) (format nil "~A/~A~3,'0D.~A" dest-dir dest-prefix current-count dest-suffix)) (defun convert-suffix-to-pdf (source-file) (format nil "~A/~A.pdf" (directory-namestring source-file) (pathname-name source-file) '("-density" "100%"))) (defun name-dest-files (dest-dir dest-prefix dest-suffix) (format nil "~A/~A*.~A" dest-dir dest-prefix dest-suffix)) (defun name-dest-pdffile (dest-dir dest-pdf-name) (format nil "~A/~A" dest-dir dest-pdf-name)) (defun create-pdf-from-images (dest-dir dest-prefix dest-suffix dest-pdf-name) (let ((source-files (directory (name-dest-files dest-dir dest-prefix dest-suffix)))) (loop for file in source-files do (simple-convert (format nil "~A" file) (convert-suffix-to-pdf file) '("-density" "100%")))) (let ((pdf-file-pattern (name-dest-files dest-dir dest-prefix "pdf"))) (join-pdf pdf-file-pattern (name-dest-pdffile dest-dir dest-pdf-name)))) (defun convert-multi-image-file (multi-image-file dest-dir dest-prefix dest-suffix dest-pdf-name) (let ((multi-image-count (image-count multi-image-file))) (loop for i from 0 to multi-image-count do (simple-convert (name-source-file multi-image-file i) (name-dest-file dest-dir dest-prefix i dest-suffix)))) (create-pdf-from-images dest-dir dest-prefix dest-suffix dest-pdf-name))
Currently it’s only extended to run on sbcl, but it would be trivial to have it work other places. The function to use is the last one and it’s used something like this:
(convert-multi-image-file "/path/to/my/file.tiff" "/my/dir/" "image-" "png" "final.pdf")
The end result is a pdf file and the png files needed for the project. The code makes use of the square bracket syntax of imagemagick to specify a particular file in a multi image file (such as a video file, pdf, or in this case a tiff). The Imagemagick FAQ is what eventually lead me to the technique.
This method can render everything out with little load in less than one minute for what used to rarely complete in over 45 minutes with massive loads (above 9).
I just grabbed the 2.0.4 version of the Canon 5d Mark II firmware and these are the steps it took to install it using GNU/Linux.
dmg2img eos5d2204.dmg
mount -t hfs -o loop eos5d2204.dmg /mnt/tmp
cp /mnt/tmp/5d200204.fir /path/to/CF/card
See linux is easy ;)
Actually, would it be a pain for Canon to just release the firmware in a zip file? I’d think that would work for everybody, but I’m guessing that they are relying on a possible built in checksumming in a DMG and their exe files? I’m not sure about the checksum, but that would make sense.
For those curious, here is the sha256 checksum of the 5d200204.fir file the above process created:
424b1990b52af12748f9675c2085e58949ac3fc682b1e61717c2218f89cdd149 5d200204.fir
The new firmware adds a bunch more features to the video feature set of the camera, including various frame rate options. For more info see canon’s site.