Re-did with Redis

For the last few years I've been writing about transitioning a web service to more efficient technologies. One of the pieces in that puzzle was using a message queue to simplify the software stack immensely. I originally went with MQTT, but I've just switched to using Redis instead.

I recently began using Redis for some Rails software I'm working on, and it impressed me enough that I figured I'd see how well it would work for my needs. What I wrote when I went with MQTT is that something needs to function as a persistence/communication/coordination layer, and Redis seems to fit that role in spades.

Redis supports publish/subscribe functionality that can work pretty much like a drop in replacement for what something like MQTT offers. If that were all I needed, Redis would be overkill, but the persistence support in Redis allows for interactions that required a lot of workaround code in MQTT.

One example is in how I process feed info in calendaRSS. For MQTT, I resolved that by creating a topic hierarchy for each site (e.g., /cal/feed1/title, /cal/feed1/description, /cal/feed2/title, ...) and mass publishing retained messages to each one. With Redis, all that essentially goes away and becomes a single HSET call for each site.

Another example is for individual items in the feed. For MQTT, I avoid all that messy hierarchy manipulation and just sent them as JSON objects, with only the last value being persisted (the very minimum to keep track of where new items start in the next fetch). With Redis, I can just XADD all their hashes to a stream directly, and then sort through as many or as few as I need to deal with to process a new fetch.

One thing Redis isn't is RAM-friendly. It mainly functions as an in-memory database, so the more data I store, the bigger the footprint I have to deal with. MQTT sips a bit over 1MB RAM for me, but Redis with a fresh install was drinking 4MB and I've already managed to get it slurping up 12MB. That's really still not very much in modern terms, and the features Redis offers might even justify running that on my tiny 128MB server, but I sure am happy I did get a bigger box not that long ago.

So while this latest move is another that isn't quite attuned to minimalism, it is furthering simplicity, which I think is an equally worthy goal. It offers a slew of other very useful middleware features that I can see incorporating into other services I have planned in the future. As always, keep an eye out for updates on how things progress.