A Peek at Server Logs

So, I’ll happily admit to anyone that asks that I’m terrible about self-promotion; if anything, this blog is an extension of my books: writing about what matters to me for you, hoping to find a match. I don’t pick topics because they sell, or because I think they have wide appeal; I’ve always been drawn to writing about problems that I’ve encountered that I think others may encounter as well.

That’s why I found this chart from StatPress so interesting tonight — it’s a look at the most commonly read entries on this blog.

Blog Topics By Request
Blog Topics By Request

I love Lisp — no denying it. And I’ve spent some time plinking away at using Lisp on various machines; my time spent coercing CLIM to work on Mac OS X is clear testimony to that fact. Unfortunately, I’ve never been able to turn my affection for Lisp into a paying concern; in fact, I’ve barely been able to integrate it with my day job (this paper is as close as it gets, folks!) because my primary career focus is mobile, not any of the problems Lisp typically targets.

Unfortunately, given the amount of time I have for coding, Qt wins out over Lisp and CLIM, or even plain Lisp; the day job is currently focused on research on mobile UIs at Nokia, and Qt (or Windows Phone 7, which I’m just getting started with on my own time) is where it’s at.

I’ll admit some frustration, though — I’d like to be of more use to the community that finds these pages useful, especially because I think it’d be more fun sometimes than just quickly knocking out a few pages about problems I’ve solved on the day job for other Qt developers to refer to. So, open questions to all you people who got here looking for Lisp on Mac OS X: what else can I help you with? Where are your pain points? And what the heck are you using Lisp on Mac OS X for, anyway?

Yet more Qt tips…

Two more for you!

Shared Data
Like Qt’s copy-on-write shared data semantics? Want to do the same for your own class? It’s easier than you think. Simply inherit from QSharedData, which provides the necessary reference-counting semantics. See here for more info.

Running a Slot on a Different Thread
Sometimes you want to schedule a single operation on a different thread; you can do this using QObject::moveToThread to move an object’s thread affinity to a different thread, and then trigger the slot as a queued connection. See here for a discussion of how Qt’s threads and Qt’s object system cooperate.

More Qt tips…

Following on the heels of last week’s post, here are a few more thoughts gleaned from the experience of attending last year’s Qt Developer Days. These are all old news, especially if you’ve been doing Qt for a while, but putting them here may be useful to you, and clears out a page of thoughts in my engineering notes!

QImage vs. QPixmap
Remember that QImage is the general-purpose class for image manipulation whose data lives in main memory, while QPixmap objects are usually backed by the GPU. This has some ramifications for your application:

  • QImage is optimized for I/O and direct pixel access.
  • QPixmap is optimized for direct screen display.
  • On constrained platforms like mobile devices, there may be relatively little GPU RAM available, so you should consider using QPixmap objects sparingly.

Regardless of which you choose, both classes use Qt’s copy-on-write (COW) semantics, so if you have multiple instances of the same image, you pay a small price penalty—basically the cost of the object overhead, not the memory footprint of the image itself.

QStandardItemModel and Performance
Qt’s support for model-view-controller is great, and one of the handiest things about it is that the QStandardItemModel class provides everything you need in a model to get going. It is, however, not necessarily the most efficient way to do so.

If you have your own model already—say you’re porting an existing application, or working with a library with its own data representation—it really is a good idea to implement your own model class from QAbstractItemModel. I’m not bad-mouthing QStandardItemModel, but it’s based on Qt’s collection classes, and odds are that you know better ways to store your data (especially if there’s a lot of it!) than Qt does.

Personally, my approach is that if I don’t have a data container and I’m only dealing with a handful of data items — say, up to a hundred or so objects with a half-dozen roles or so — I use QStandardItemModel and don’t look back. (Qt’s collection classes are, after all, quite good for general applications). But if I’ve already got a data model, or I know I’m going to be dealing with lots and lots of objects, it’s time for a custom model that leverages my knowledge of the data set I’m wrapping.

Use QtConcurrent for Parallel Computation
I’ve not tried this personally, but QtConcurrent looks really cool for doing any kind of concurrent map/reduce style programming. If you need to write code that leverages multiple cores for concurrent computation, it’s best to look here before trying to roll your own framework using QThread.

A grab bag of Qt tips…

So, last year I went to the Qt Developer conference here in S.F. It’s a great opportunity to meet up with Qt Developers from all over the world. One of the amazing things about Qt is how easy it is to get up and running and be productive; unfortunately, it’s easy to get stuck in a rut where you always do things the same way and don’t necessarily try new ways of doing the same things, or really look at performance in your code, and just generally be a fair-to-middling developer with Qt.

I attended a well-run session on Qt tips and tricks from the trenches, and ended up with two pages of longhand scrawl of good thoughts that I either hadn’t really explored in Qt, or realized I needed to learn more about. These got transcribed to a bullet list of items, some of which I’ve followed up on, and some I haven’t. Here are a few, with some of my observations about them, preserved for posterity and your use!

invokeMethod for Deferred Method Invocation
You often want to run some bit of code just after a method call; deferred initialization is a common case, where your constructor does the bare minimum to set something up, and then once everything’s up and running you want to do a bit more initialization. A single-shot timer is a good way to do this; you can use QTimer::singleShot with a short delay to trigger a slot in your class. You can also do the same thing using QMetaObject::invokeMethod, passing Qt::QueuedConnection as the connection type. When you do this, Qt sends a QEvent and the method is invoked as soon as the application enters the main event loop, which is probably what you were trying to accomplish with the timer anyway.

Implementing << and >> for QVariant Serialization
This is a “well, duh” if you’ve read Qt’s serialization documentation, but you can implement << and >> for your class using QDataStream to support serialization.

I like to take the time and do a human-readable string for QDebug::operator<< at the same time, too, so I can log complex data types to the debug console when doing printf-style debugging.

Use const References with foreach
This should go without saying, but a lot of people forget. If you're iterating across a Qt collection using foreach, don't forget you almost certainly want to use constant references to do so. Otherwise, the implementation makes a copy of each item being passed to you at the beginning of each pass through the loop, and destroys that instance at the end. All of this happens on the stack, of course, but you're still paying the constructor and destructor penalty, and if your object is large, you're thrashing your stack footprint needlessly.

I'll post a few more of these next week!

Qt for the next billion…

I could not argue that Nokia’s announcements in February about the shift in focus to Windows Phone 7 from Symbian and MeeGo has caused some grief for my latest book. While there are a slew of Nokia developers — and from the looks of sales, plenty of people wanting to target Nokia products with Qt on Symbian and MeeGo — Nokia’s announcement certainly rattled some cages. Especially here in the US, the shift has been perceived by some to be a death knell for Qt, at least in the mobile space.

As a result, I’m especially excited by Nokia’s commitment to powering applications for the next billion using Qt. While Nokia’s been clear about their continued support for Qt, it’s hard to imagine mobile development using Qt without the “mobile” part. And while Nokia’s announcement is light on details, the Symbian and MeeGo platforms remain relevant for Qt developers today, while Qt itself promises a path to the future.

I use Qt at the office every day — both as a user on the mobile and fixed devices I use, as well as a developer. Without exception, I remain pleased with its portability and flexibility; I can’t imagine doing the work I need to do on any other platform as easily.

Creating a new class of declarative item…

(It’s been too long since I’ve posted, and I suspect most people think I’ve either abandoned this again or figure I’m no longer doing Qt/QML work. Neither is true; I’ve just been busy, doing QML with Qt at the office!)

The Qt documentation covers this pretty well, but since it feels a little esoteric, it’s worth mentioning that you can create a new declarative item

Calling a QML item method from C++

This is another one of those “it’s all in the documentation”, but for whatever reason, I had a heck of a time finding the info when I needed it. (The Qt documentation is generally great — I’m not beefing about that, but rather my personal inability to remember bits of it and not have enough remembered that I can search on things efficiently).

There’s times where you want to have something in C++ call a method defined by a QML item. In my case, I had a ListView that I wanted to manually reposition when its model (a C++ object derived from QStandardItemModel) changed under certain circumstances. I began with a ListView that looked something like this:

    ListView {
        id: events
        objectName: "events"

        property bool moved: false

        model: appmodel
        delegate: MyItemDelegate {}
        Component.onCompleted: positionViewAtIndex(count - 1, ListView.End)
        onMovementStarted: moved = true
        function positionListAtEnd() {
            if (!moved) events.positionViewAtIndex(events.count - 1, ListView.End)

Here, I’m going to use the moved property to track whether the user’s interacted with the list view at all; if she’s moved the list view, I don’t want to start yanking it around when the model updates, but if it’s just sat unused and the model updates, I want the list to always show the last item in the model. When the list view first loads, I do that manually using the onCompleted signal handler; I also want to do it when the model changes, by having the code that updates the model invoke the ListView‘s positionListAtEnd method.

The trick is actually getting a reference to the QObject that corresponds to my ListView. In QML, items are referenced by their id property; in Qt’s meta-object system, they’re referenced by object name—denoted by the objectName property, a string bearing the object’s name.

Don’t confuse the two! You can’t link QML things using object names, and, as near as I can tell, the Qt meta-object system’s C++ implementation doesn’t care one whit about QML ids.

So I give my ListView an objectName property, which for sanity’s sake is the same as the id I’ve assigned: "events". It’s a string, not a name, so those quotes in the QML are important. Once I’ve done that, I can find this object by name in my QML hierarchy using the following bit of C++:

    QObject *object = mView.rootObject();
    if (!object) return;

    QObject* listView = object->findChild<QObject *>("events");
    QVariant returnedValue;
    QMetaObject::invokeMethod(listView, "positionListAtEnd",
            Q_RETURN_ARG(QVariant, returnedValue));

The mView field is just the QDeclarativeView that renders my QML; you can see how I originally linked my C++ and QML in my previous post, Adding C++ Objects to your QML. It has a root object, a QObject descendant (really, a QGraphicsObject that corresponds to the top level of the QML object hierarchy being displayed). To invoke my QML method, I use Qt’s meta-object protocol to first find the child named "events" and then invoke its positionListAtEnd method, discarding the value it returns.

There’s no magic here—as long as you remember that QML items can have an objectName property, and that property is used internally by Qt’s object hierarchy as the object’s name.

Using an Image Provider to Share Images from a C++-hosted Data Model to QML

In a previous post, I demonstrated how to use an image provider for a relatively simple use case: loading QML from the application’s resources. Another common question I’ve heard is how to do the same from an application model; say you’ve got an existing C++ model that carries images for each item; how do you share those images with QML? You may want to do that if you have existing C++ code for a data model that renders the image data for each item (say, by compositing several different things from a model, such as its description and date).

It’s not hard: again, you need to write a QDeclarativeImageProvider that returns the appropriate images given names that the QML will derive from your model. Of course, you also need to extend your model’s roles to include a role for the image name; your model code will populate this role with a unique name for each item’s image, and the QML will use this role’s value in Image elements’ source property to reference the image. In turn, the declarative runtime will use the resulting source values to request the images from the image provider, which then provides the desired images for rendering.

Here’s what I did for a pixmap provider in a recent project, naming images in my model image://model/---, where — is a unique identifier for the item in the model.

The image provider provides only pixmaps, and is a little more sophisticated than the last one I showed you, because it needs to work with a model, whose data can change underneath us:

class ModelIndexProvider : public QObject, public QDeclarativeImageProvider
    ModelIndexProvider(QAbstractItemModel& model, int pathRole, int pixmapRole, 
        QObject* parent = 0);
    QPixmap requestPixmap(const QString& id, QSize* size, const QSize& requestedSize);

public slots:
    void dataUpdated(const QModelIndex & topLeft, const QModelIndex & bottomRight);
    void dataDeleted(const QModelIndex & parent, int start, int end);
    void dataReset();

    QAbstractItemModel& mModel;
    int mPathRole;
    int mPixmapRole;
    QMap mPixmapIndex;

The class uses a QMap to provide an index from the name of each image in the model to its model index; this is used by requestPixmap when it’s passed an id originally from the model and needs to determine the image’s index into the model. requestPixmap looks like this:

QPixmap ModelIndexProvider::requestPixmap(const QString& id, QSize* size, const QSize& requestedSize)
    QString key = QString("image://model/%1").arg(id);
    QModelIndex index = mPixmapIndex[key];
    QPixmap image = mModel.data(index, mPixmapRole).value<QPixmap>();
    QPixmap result;

    if (requestedSize.isValid()) {
        result = image.scaled(requestedSize, Qt::KeepAspectRatio);
    } else {
        result = image;
    *size = result.size();
    return result;

This code just looks up the model index of the image given the image name, and then scales the image already in the model before returning it to the caller.

Of course, the mapping between name and index must be maintained; the code performs the necessary registration to watch the model at construction time:

ModelIndexProvider::ModelIndexProvider(QAbstractItemModel& model, 
    int pathRole, int pixmapRole, 
    QObject* parent) :
    // For each pixmap already in the model, get a mapping between the name and the index
    for(int row = 0; row < mModel.rowCount(); row++) {
        QModelIndex index = mModel.index(row, 0);
        QString path = mModel.data(index, mPathRole).value<QString>();
        mPixmapIndex[path] = index;
    connect(&mModel, SIGNAL(dataChanged(QModelIndex,QModelIndex)),
            this, SLOT(dataUpdated(QModelIndex,QModelIndex)));
    connect(&mModel, SIGNAL(rowsRemoved(QModelIndex,int,int)),
            this, SLOT(dataDeleted(QModelIndex,int,int)));
    connect(&mModel, SIGNAL(modelReset()),
            this, SLOT(dataReset()));

Notice how the constructor takes the roles for the image path and image itself; this way instances of the ModelIndexProvider can be used in more than one project, or more than once in a project, each for different model roles that contain QPixmap instances. The constructor begins by creating the initial index of image names and model indices, and then connects to the model's signals that indicate when the model has changed.

Handling changes isn't difficult; in my case because the model doesn't carry that many items I just recreate the index any time the data is changed or deleted. If you have a complex model that changes a lot, you might want to be smarter about cache invalidation --- or drop the cache entirely and just sequentially search the model for the bitmap data when it's requested, as the QML viewer's pretty good about caching the resulting bitmaps anyway.

void ModelIndexProvider::dataUpdated(const QModelIndex& topLeft, const QModelIndex& bottomRight)
    // For each pixmap already in the model, get a mapping between the name and the index
    for(int row = 0; row < mModel.rowCount(); row++) {
        QModelIndex index = mModel.index(row, 0);
        QString path = mModel.data(index, mPathRole).value<QString>();
        mPixmapIndex[path] = index;

void ModelIndexProvider::dataDeleted(const QModelIndex&, int start, int end)
    // For each pixmap already in the model, get a mapping between the name and the index
    for(int row = 0; row < mModel.rowCount(); row++) {
        QModelIndex index = mModel.index(row, 0);
        QString path = mModel.data(index, mPathRole).value<QString>();
        mPixmapIndex[path] = index;

void ModelIndexProvider::dataReset()

All that remains is to register the image provider with the Qt declarative runtime, as you saw here!

One important thing to remember is that QML tracks your images by the image source, which is a string, not the bitmap behind the source. Even if your bitmap data is in the model in a role as it is here, changing the bitmap won't update the QML! Instead, you need to change the underlying source name to indicate to the QML that it needs to load a new bitmap. In my code, I do this by appending a timestamp in milliseconds to the source attribute; whenever my bitmap changes I update the model with the same image path and a new timestamp, which triggers a model invalidation in the QML declarative view and a subsequent bitmap fetch and redraw.

Here's the code if you want to try it for yourself.

Using an Image Provider to Supply Images to QML Applications from Qt Resources

Your hybrid QML/C++ application can load its QML from a Qt resource, but what about images? If you load the QML from a Qt resource, then all of your images are loaded from the Qt resource segment as well. But what if you want to supply your QML as files (or deliver them over the network)?

Your C++ application can provide images to the Qt declarative runtime through an image provider, so that in your QML you can write something like

Image {
    source: "image://myprovider/image.png"

and the QML runtime will query your application’s image provider for a provider registered to the name myprovider, asking that provider for the image named image.png. Using the image provider framework, it’s easy to load images from your application’s resource segment: all you need to do is provide a custom image provider that loads images from the resource segment and register it with the Qt declarative runtime.

Qt image providers are subclasses of QDeclarativeImageProvider, a class that has methods to return a QPixmap or a QImage given the image’s name. It’s up to you to decide how to organize and fetch those images when you implement an image provider; an obvious way when fetching images from the resource segment is to use the resource name. Our resource-based image provider looks like this:

class ResourceImageProvider : public QDeclarativeImageProvider
    ResourceImageProvider(QDeclarativeImageProvider::ImageType type);
    QImage requestImage(const QString& id, QSize* size, const QSize& requestedSize);
    QPixmap requestPixmap(const QString& id, QSize* size, const QSize& requestedSize);

The constructor accepts an indicator as to whether the returned images provided are pixmaps or images; while pixmaps are faster to draw, they reside in the graphics subsystem’s memory, which is typically a scarce resource, so you should always be sure to use this judiciously.

The image provider gives images to the Qt declarative runtime through the requestImage and requestPixmap methods, which return a QImage or QPixmap, respectively. Each of these methods take the identifier of the image to return, a desired size for the image, and a pointer where to place the actual size of the image that’s returned. The image provider should try to match the requested size if it’s able, although depending on the QML, can accept a smaller or larger image and reflow the QML layout—that depends, of course, on the QML element’s anchors property, of course.

The actual image provider is simple:

ResourceImageProvider::ResourceImageProvider(QDeclarativeImageProvider::ImageType type) :
    // This space intentionally left blank.

    // This space intentionally left blank.

QImage ResourceImageProvider::requestImage(const QString& id, QSize* size, const QSize& requestedSize)
    QString rsrcid = ":/" + id;
    QImage image(rsrcid);
    QImage result;

    if (requestedSize.isValid()) {
        result = image.scaled(requestedSize, Qt::KeepAspectRatio);
    } else {
        result = image;
    *size = result.size();
    return result;

QPixmap ResourceImageProvider::requestPixmap(const QString& id, QSize* size, const QSize& requestedSize)
    QString rsrcid = ":/" + id;
    QPixmap image(rsrcid);
    QPixmap result;

    if (requestedSize.isValid()) {
        result = image.scaled(requestedSize, Qt::KeepAspectRatio);
    } else {
        result = image;
    *size = result.size();
    return result;

The image and pixmap loaders do essentially the same thing: create a Qt resource identifier from the incoming image id, load the image, scale it to the requested size, and return the requested image (as a pixmap in the requestPixmap case) along with its size.

Once you’ve created an image provider, you need to make it available to the QDeclarativeView displaying your QML. In a previous post, I showed you how to create a QDeclarativeView that displays your QML in a QMainWindow; you can use the same code plus the following snippet to add your image provider to the Qt declarative runtime:

    QDeclarativeView* view = new QDeclarativeView(window);
    view->engine()->addImageProvider(QLatin1String("qrc"), new ResourceImageProvider(QDeclarativeImageProvider::Pixmap));

This code pokes at the QDeclarativeView‘s QDeclarativeEngine, giving it an instance of your image provider that returns pixmaps for images in the namespace beginning with qrc. Once you’ve done this, you can now load images in your QML from your Qt resource segment using things like:

Image {
    source: "image://qrc/image.png"

In a later post, I’ll show you how you can use an image provider for another common C++ integration point: loading images directly from a C++-hosted data model.

Although you probably already know this, it’s worth pointing out an important detail about Qt resources: they’re in your application’s read-only segment. If your application is running on a memory-contrained platform like a mobile device, it’s worth choosing carefully whether you place your resources in a Qt resource segment or as additional files on the file system packaged with your application. If they’re in the data segment, it’s possible they’ll be loaded into memory when your application is launched (for example, when started from a memory card on Symbian, which only performs demand-paging from the internal store). Consequently, putting a lot of images in your application’s resources can lead to slower load times and even out-of-memory errors on some platforms, so it’s best to do this with small resources that simply must be loaded quickly like bitmaps for small user interface controls and the like.

You can download the code, too!.

Lisp on a Nokia Booklet 3G…

At the office, managed to wheedle a Nokia Booklet 3G, last year’s entry by Nokia into the netbook category. I’ll not review the device here; there are plenty of reviews of the product on the Web. Suffice it to say that the reviews are pretty accurate; it’s a pretty sweet piece of kit hobbled by the choice of Windows Starter and a slow hard disk. Addressing the performance challenges by putting a better operating system on it leads to a very nice subnotebook for light browsing, note taking, and such. Perhaps surprisingly, I actually reach to it before the iPad for some casual computing, especially at the office.

The operating system I chose is Jolicloud; I wanted to see the UI, and it has built-in compatibility for the GMA500 graphics controller in the Nokia Booklet and didn’t want to mess with patching Ubuntu once I got started. Reports were correct—installing Jolicloud was a snap and just works on the Booklet. What follows here should work on any Ubuntu derivative, and more broadly, probably any Debian-based Linux distribution.

So what about Lisp? I don’t get paid to do Lisp programming, but like playing in Lisp; it sharpens my thinking about the algorithms and code I write, and I’m thinking about introducing my son to Lisp through the excellent book Land of Lisp by Conrad Barski (which you can read about here). It’s also been a while since I’ve worked any of the exercises in Peter Seibel’s Practical Common Lisp (see the web site).

Because Jolicloud was running a Debian fork of Linux (Jolicloud’s parent is Ubuntu, derived from Debian), getting a Lisp was as easy as running the APT package manager. I chose CLISP, because I’d not used it before but it’s the Lisp Barski uses in his book. Bring up a command prompt (Alt-F1), and type:

kf6gpe@kf6gpe-jolicloud:~$ sudo apt-get install clisp
kf6gpe@kf6gpe-jolicloud:~$ sudo apt-get install clisp-doc
kf6gpe@kf6gpe-jolicloud:~$ sudo apt-get install clisp-dev
kf6gpe@kf6gpe-jolicloud:~$ sudo apt-get install emacs
kf6gpe@kf6gpe-jolicloud:~$ sudo apt-get install slime

This gets you both CLISP and its documentation for playing, and emacs and slime if you want to do serious stuff. You can now bring up CLISP from the terminal by typing clisp at the shell. To finish the configuration, put something like the following in your .emacs file:

;;; ================================================================
;;; SLIME configuration
;;; Modified from http://lispm.dyndns.org/news?ID=NEWS-2008-08-27-1

(require 'slime-autoloads)
(slime-setup `(slime-asdf slime-fancy slime-tramp))

(setq slime-autodoc-use-multiline-p t)
(setq slime-repl-history-size 1000)
(setq slime-startup-animation t)
(setq slime-default-lisp 'clisp)
(setq slime-truncate-lines nil)
(setq slime-lisp-implementations
     `((clisp   ("/usr/bin/clisp")              :coding-system utf-8-unix)
       ;;(ccl   ("/usr/bin/ccl")                :coding-system utf-8-unix)
       ;;(abcl  ("abcl")                        :coding-system utf-8-unix)
       ;;(cmucl ("/usr/bin/cmucl" "-quiet")     :coding-system iso-latin-1-unix)
       ;;(ecl   ("/usr/local/bin/ecl")          :coding-system iso-latin-1-unix)
       ;;(gcl   ("gcl")                         :coding-system iso-latin-1-unix)
       ;;(sbcl  ("/usr/bin/sbcl")               :coding-system utf-8-unix)
; do m-- m-x slime ccl to start Clozure Common Lisp from the list above

(global-set-key "\C-cs" 'slime-selector)

(add-hook 'lisp-mode-hook (lambda () (slime-mode t)))
(add-hook 'inferior-lisp-mode-hook (lambda () (inferior-slime-mode t)))

(defmacro defslime-start (name lisp args coding-system)
  `(defun ,name ()
     (require 'slime)
     (slime-start :program ',lisp :coding-system ',coding-system :program-args ',args)))

(defslime-start clisp   "/usr/local/bin/clisp -K full"  nil         utf-8-unix)

This work is shamelessly borrowed from Rainer Joswig; he shows how to set up SLIME on the Macintosh using Aquaemacs. In our case, we comment out all the other Lisp implementations—although I’d bet if you wanted a different Lisp (say, SBCL), it’s no more than an apt-get away.