Which Qt XML parser are you?

Do you like getting the whole story all at once? Or are you more the kind of person that wants to hear things bit by bit, and think about each piece as it comes, and respond a little at a time?

That’s essentially the question to ask when choosing an XML parser to use in Qt. There’s the QDomDocument, which parses your XML document into a fully-navigable document object model (DOM) in memory that you can query, traverse, and change. And then there’s the QXmlStreamReader parser, which reads from a stream and lets you make incremental queries for entities and attributes as you go along. Finally, there’s also the QXmlSimpleReader parser, a simple SAX-like parser that can read the whole document in one shebang or incrementally and calls callbacks on objects you register when the parser encounters entities and attributes.

Which should you use in your project?

There’s a tradeoff between the simplicity of your code and the amount of memory you’re willing to sacrifice at run-time. DOM-based parsing as provided by QDomDocument is awesome, but comes at a cost—the parser has to churn over the entire document as it creates the DOM, and essentially mirrors the document in RAM in with its DOM. The streaming parser is great for larger documents, but requires that you build more structure into your parser, using that structure to mirror the internal hierarchy in your XML document.

For years, I’ve been an avid proponent of using stream-based parsers—readers of Dan’s and my book will remember us exhorting you to use stream-based parsers for mobile, because you can read the data as it comes over the stream and build your document representation in core as efficiently as possible. In a past life, I actually wrote a streaming XML parser for Qualcomm BREW, again adhering to the doctrine that you should hold as little in memory as possible, and process what comes in over the air (or off the wire) as quickly as possible, rather than buffering and processing. If you’re parsing a very large document, or if you’re working on very small devices (think legacy Symbian or Maemo), streaming is definitely the way to go.

While I still think for the large majority of applications that’s probably still the best approach, I had the opportunity recently to use a DOM parser in a commercial setting on a project, and wow, was it easy! In my case, I was parsing small documents—typically well under fifty kb—walking the DOM quickly and reducing their structure to a compact in-memory representation and discarding the DOM. I could have done the same thing with a streaming parser, but here the time it took me to write the code was of the essence; we wanted to get something up and running as quickly as possible. And this was a fast way to do it.

Someday, I’d like to write a benchmark app that uses all three methods to parse a longish (say, 250 kb) document and report on both performance and heap usage. It would also be interesting to see download-and-parse times for the three approaches over a benchmark network like 3G cellular, because depending on the application and document size, network performance becomes a factor, too. Until I can point at hard numbers, though, I’d advise you to consider both the DOM and streaming approach, and choose the approach that best balances network throughput, memory usage, and the amount of time and effort it takes you to get your project off the ground.

A forest of trees!

I recently needed a spatial index for some data I was working with at work, and decided to rough out both a quad tree and a k-d tree implementation for Qt.

Indexing data spatially is important if you have a lot of objects organized spatially, and need to do queries by an object’s location, rather than a single-dimensional key such as an ID. Qt has a framework for doing this in the context of the QGraphicsFramework, but doesn’t have generalized container classes that index along multiple dimensions — think, for example, of a multi-dimensional QMap.

I implemented the quad tree largely from scratch, looking at wikipedia and a couple implementations such as the one here. My quad tree uses C++ templates to abstract the data type, so you can stick anything in it (as long as it can be put in a QList; the underlying store uses QList for object management). Quad trees come up a lot in collision detection in computer graphics and gaming, where you want to know if an object is close enough to another object to collide, but don’t want to search through lots of objects and do collision detection on each object. It works by recursively subdividing the plane containing the objects, performing additional subdivisions each time a region contains more than a specific number of objects.

I cheated when it came to the k-d tree, because there’s a great C implementation on code.google.com that did precisely what I needed. Because I needed to store QVariants in the k-d tree, I just wrapped the entire C library in a Qt class, making the C code static to my implementation. It’s not quite as sexy as a templated version, because it’ll only store objects you can wrap in a QVariant, but for my application it was enough, and in the future if I have a different kind of object I want to store, it’s easy enough to make it storable as a QVariant, too. k-d trees are more general than quad trees, operating in more dimensions than two as they subdivide the search space.

Neither of these implementations are truly “high performance” — for clarity and ease of debugging, they use Qt’s data structures under the hood, which are fairly performant but not as high performance as managing all the data out of the application’s heap using pointers alone. This is especially true for the find methods, which return copies of the data stored in the trees rather than pointers to the actual data. This is, I believe, correct in my case, where I don’t want to expose the internal structure of my data, and where I’m using Qt classes that use references and copy-on-write semantics anyway. However, if you’re looking to squeeze every last bit of performance from your spatial index, you probably don’t want to start with the code you’ll find here. Nonetheless, for general computational tasks, these should serve you quite well.

You can find both of the implementations and a little standard I/O app for using them here.

Debugging Qt’s signal-slot connections…

What do you do if you think you’ve correctly connected a signal to a slot and yet your slot’s not being fired? I ran into that Friday at 5:30pm, just about the time I hoped I’d call it a day and go home. Fortunately in my case, I was simply passing null to connect, and I got a message in the debugging log about trying to connect a signal from a null object. (Unfortunately for me, I didn’t see the log right away, because of all the other stuff I was dumping to the log.)

If you’re in the same boat, what do you do to debug the situation? Here’s a bunch of things to try:

  • Check the compile log for error messages about undefined signals and slots.
  • Check the run-time log for errors on connection. If need be, ensure that connect succeeds by testing its return value (it should return true.
  • Check to make sure that the connect code is reached, that the emit code is reached, and the slot code. (Odds are one of them isn’t.)
  • Make sure you declare the connection like this:

    connect(sender, SIGNAL(someSignal(type)), receiver, SLOT(received(type)))

    The signal and slot specifications should have the argument types, but not the arguments themselves. Moreover, you can omit const and reference specifications in the arguments; the meta-object compiler strips them anyway.
  • Check the parameters of the signal and slot and ensure that they match precisely
  • Check to be sure that you’ve correctly declared the signals and slots portions of your header correctly.
  • For that matter, be sure that your sender and receiver both inherit from QObject, and that you have Q_OBJECT declared in your class definition. (Remember, you need to do both.)
  • If you’ve forgotten part of the QObject declaration in your header, re-run qmake and rebuild.
  • Make sure you make the connection with connect before you invoke any functions that fire the signal. Signals may fire synchronously.
  • Make sure you’re not disconnecting the signal anywhere with disconnect.

Usually, the problem’s pretty easy to track down, especially if you check the log and the signal and slot declarations closely. A common mistake is to try to wire a signal to a slot that’s not been declared a slot, too, so check your headers closely!

Some musings on the Kindle Fire…

So, I’m flattered to have been asked by the dynamic Conder/Darcey pair to be a tech reviewer on an upcoming book for writing applications for the Kindle Fire, which runs Android 2.3 (Gingerbread, SDK level 10, as I recall). Of course, to do the job correctly, I needed a Kindle Fire.

I’ve been an ebook fan for many, many years, having pressed my first books for the Newton Book format back in ’94, and read ebooks on nearly every handheld computing platform since then. I was a happy user of Sony’s ereader products (as previous blog posts can testify to), although this last fall I gradually switched to using Amazon as my provider-of-choice for new ebooks, because I can read them using either the cloud reader or my Windows Phone and iOS devices. I generally seek out books without DRM where I can (like the excellent offerings from O’Reilly), but will tolerate DRM when it’s thrust upon me. So I was excited to have the opportunity to get my hands on a Kindle, both from an end-user as well as a developer perspective.

The Device and its UI
For the money, it’s definitely a good deal. I’ve been quite happy with it for book reading, casual browsing — it’s replaced the iPad as my go-to device for Twitter and Facebook in the evenings, although I still use the iPad more for checking news sources like NPR, BBC, and such, as well as general browsing. It’s also definitely become the official ebook reader; I’m a little self-conscious about how many ebooks I’ve bought since I got the thing.

A noteworthy UI departure from the plethora-of-icons on most handheld devices is the main screen, which provides a most-recently-used Cover-Flow style view, with a few pinned items below that. I really like that; my usage is typically clustered around two or three different pieces of content (a couple Web pages and a book), and it’s very fast.

As much as I like books on it, magazines suck. The problem is the screen size, as others have observed. You just can’t format an 8 1/2″ x 11″ magazine for a 6″ display and expect it to be legible, and if I wanted to spend an evening pinching and zooming to read something, I’d just go get a smart phone out. I’m going to cancel the one magazine I subscribed to before the trial is out; I think the iPad would do much better.

Developing for the Fire
I’d done some puttering with Android before, of course, so the environment’s not new. Having spent more time in it, I’d have to say that it’s on a par with Qt Creator, except that the Eclipse IDE is far, far more cluttered. I had about the same amount of difficulty getting the Kindle to talk to the debugger as I’ve had with Symbian and MeeGo devices — a bit of grumbling and some Googling as a result — but no real difficulties. It’s certainly a change from the pre-iPhone devices, when most mobile devices required all kinds of magic to connect reliably to a development workstation.

I don’t love Java, and I don’t love the various SDK levels and API confusion of Android, but it’s actually not a bad platform for writing software. The APIs mostly make sense, and when they don’t, there’s usually copious documentation and sample code available. I’m rapidly moving from feeling like a newbie to feeling comfortable, and wondering whether I should invest more in Android for my own skillset or Windows Phone 7 for work skills as I poke away at the thing. It helps that I’m only targeting one device with one specific API level. I don’t envy professional developers targeting Android as a whole.

Anyway, when their book is available I’ll be sure to post a link so people can check it out!

Scheduled downtime to protest SOPA and PIPA…

I don’t usually see my writing and blogging as an avenue for my political beliefs and interests, but like many, I feel a need to speak out against SOPA and PIPA. This blog and the other content at lothlorien.com will be unavailable during tomorrow’s “Strike against SOPA”.

I apologize for any inconvenience this may cause.

Deploying Qt Quick Applications that rely on Qt Mobility on Windows…

Whew! Some things seem to take forever for me to figure out!

Here’s the scenario: we’re working on a Qt Quick (QML and C++) application at the lab to run on Microsoft Windows. We’d like to deploy the app to a bunch of study participants to use for a week or so. It’s obviously not practical for me to go around and set everybody up with Qt, Qt Mobility, and build our app. (Especially not this app; we’re using the Qt 4.8 prerelease at this point!). So we need to build a Windows installer that includes the Qt DLLs, our app and its QML, and the Qt Mobility plugins we’re using. We’re using Qt Mobility in our QML — so we need to include the Qt Mobility plugins for Qt Quick, not just the Qt Mobility libraries.

Packaging a Qt application for Windows is straightforward — just follow these instructions for packaging as a shared library. But what about the Qt Quick plugins for Qt Mobility we need?

Well, Qt Quick supports modules for that, as described here. In fact, if your application required QtWebKit’s Qt Quick plugin, you’d create a QtWebKit directory, stick a qmldir file with the line

plugin QtWebKit

copy qtwebkitplugin.dll into the directory you made and Bob’s your uncle!

We use QtMultimediaKit, so at first I created a QtMultimediaKit directory, created a qmldir file in that directory, and copied QtMultimediaKit1.dll from my Qt Mobility build output directory to the QtMultimediaKit directory I’d created.

No luck — in fact, the app just launches and shows nothing. (To add insult to injury, my app uses Qt’s support for Open GL, so not only did I not get the black screen characteristic of a QML error, but I got junk in the application window and no errors on the console.)

Turns out that QtMultimediaKit1.dll goes in the same directory as the application, and the declarative plugin file is somewhere else! If you look in the plugins directory, you’ll see a declarative directory, and the actual Qt Quick declarative plugin you’re looking for is in there! In my case, I needed to copy plugins/declarative/multimedia/release/declarative_multimedia.dll to a new directory QtMultimediaKit, create a single qmldir file that read

plugin declarative_multimedia

and put that adjacent to my application executable.

So, the resulting files and directories for me look something like:

imageformats/qgif4.dll
imageformats/qjpeg.dll
QtMultimediaKit/declarative_multimedia.dll
QtMultimediaKit/qmldir
msvcr90.dll
QtCore4.dll
QtDeclarative4.dll
QtGui4.dll
QtMultimedia4.dll
QtMultimediaKit1.dll
QtNetwork4.dll
QtOpenGL4.dll
QtScript4.dll
QtSql4,dll
QtXml4.dll
QtXmlPatterns4.dll
myapplication.exe

Now off to learn how to make a Windows installer. I think I’ll try Inno Setup.

Nokia N9 PR 1.1 and new Qt SDK…

The N9 PR 1.1 is out, with lots of great updates. Your N9 should see the update automatically; if like me you’re in a hurry:

  1. Open Settings
  2. Choose Applications
  3. Choose Manage Applications
  4. Press Updates
  5. Press the refresh button at the top of the display.

Budget about a half hour for the update; it downloads the update over WiFi or the cell connection, and then re-flashes your device. While it’s updating, your phone won’t do anything else (although I had a calendar alarm pop up during the update process!) I followed the suggestion and made a back-up first, although it looks like user data preservation works correctly; I didn’t lose anything after the update.

To go along with this, be sure to check out the new Qt SDK! See here for the links and news about what’s in it. I’ve been using Qt Mobility 1.2 on the desktop for a while, so it’s nice to have an integrated build with all the bits and pieces.

Happy hacking!

Enabling WiFi Hotspot on Nokia N9s in the United States…

I’m going to forget this the first time I have to reset my Nokia N9.

As it turns out, the WiFi Hotspot mode on the N9 doesn’t fully meet FCC SAR regulations, so it’s disabled in the United States (launching it generates a brief error and then the app exits). While I appreciate Nokia’s respect for regulations, I don’t plan on holding my N9 close to soft tissues while in hotspot mode, and dang it, I need hotspot mode occasionally.

Fortunately, brighter folks than me have found a hack; here’s a link to a post by salyavin at talk.maemo.org. It involves adding a new startup script that touches a configuration file on boot to enable the application. You’ll need to download the script from the link, install developer mode if you haven’t, and copy the script to the correct location.

Now my iPhone can stay safely locked in my desk drawer on weekends when I go cafe-hopping!

Edit: Another link with the same information is

Are your Qt signals not disconnecting?

This was probably obvious to everyone except me…
disconnect(object)
is not the same as
disconnect(object, 0, 0, 0)

This makes perfect sense when you look at the overloaded function definitions for disconnect, one of which reads:
bool QObject::disconnect ( const QObject * receiver, const char * method = 0 ).

So if you write disconnect(object), you’re really disconnecting all signals received by object, not all signals sent by object.

This tripped me up in a big way today; I was switching the slot a signal was connected to at run time as part of a simple state machine, and the state machine’s various slots were all being invoked! Not a nice situation, but easily solved by using the correct variant of disconnect.

A Peek at Server Logs

So, I’ll happily admit to anyone that asks that I’m terrible about self-promotion; if anything, this blog is an extension of my books: writing about what matters to me for you, hoping to find a match. I don’t pick topics because they sell, or because I think they have wide appeal; I’ve always been drawn to writing about problems that I’ve encountered that I think others may encounter as well.

That’s why I found this chart from StatPress so interesting tonight — it’s a look at the most commonly read entries on this blog.

Blog Topics By Request
Blog Topics By Request

I love Lisp — no denying it. And I’ve spent some time plinking away at using Lisp on various machines; my time spent coercing CLIM to work on Mac OS X is clear testimony to that fact. Unfortunately, I’ve never been able to turn my affection for Lisp into a paying concern; in fact, I’ve barely been able to integrate it with my day job (this paper is as close as it gets, folks!) because my primary career focus is mobile, not any of the problems Lisp typically targets.

Unfortunately, given the amount of time I have for coding, Qt wins out over Lisp and CLIM, or even plain Lisp; the day job is currently focused on research on mobile UIs at Nokia, and Qt (or Windows Phone 7, which I’m just getting started with on my own time) is where it’s at.

I’ll admit some frustration, though — I’d like to be of more use to the community that finds these pages useful, especially because I think it’d be more fun sometimes than just quickly knocking out a few pages about problems I’ve solved on the day job for other Qt developers to refer to. So, open questions to all you people who got here looking for Lisp on Mac OS X: what else can I help you with? Where are your pain points? And what the heck are you using Lisp on Mac OS X for, anyway?