Queues

Job queues, message queues and other queues. Almost all of them in one place.

About

There are many queueing systems out there. Each one of them is different and was created for solving certain problems. This page tries to collect the libraries that are widely popular and have a successful record of running on (big) production systems.

The goal is to create a quality list of queues with a collection of articles, blog posts, slides, and videos about them. After reading the linked articles, you should have a good idea about: the pros and cons of each queue, a basic understanding of how the queue works, and what each queue is trying to achieve. Basically, you should have all the information you need to decide which queue will best fit your needs.

The whole site is open source and is updated quite frequently. If you find any mistakes or see something missing, feel free to post an issue or send a pull request. All contributions are welcome! Thanks.

Projects

ActiveMQ

http://activemq.apache.org/

java

Apache ActiveMQ ™ is the most popular and powerful open source messaging and Integration Patterns server.

Resources

Introduction to Messaging With Apache ActiveMQ - Bruce Snyder (video) vimeo.com

Apollo

http://activemq.apache.org/apollo/

java
amqp

ActiveMQ's next generation of messaging.

Resources

Apollo Next Generation Message Queuing Posts Some Impressive Benchmarks infoq.com
Introduction to ActiveMQ Apollo slideshare.net
Introducing Apache Apollo: Part I christianposta.com
Introducing Apache Apollo: Part II christianposta.com

Beanstalkd

http://kr.github.io/beanstalkd/

Beanstalkd is a simple, fast work queue.

Created to improve the response time for the Causes on Facebook app.
Written in C is very fast and lightweight.
It provides simple protocol, inspired by memcached
Queues have O(log n) time push and pop operations.
It can be run in persistent mode, which will write all jobs to binlog.
Long list of client libraries for many languages

Resources

#243 Beanstalkd and Stalker - RailsCasts railscasts.com
What are the advantages and disadvantages of Beanstalkd as a work queue? at Quora quora.com
Beanstalk Messaging Queue nubyonrails.com

Celery

http://www.celeryproject.org/

python

Distributed Task Queue

Resources

Using Celery for queuing requests blog.serverdensity.com
An Introduction to Celery by Gilang Chandrasa (slides) speakerdeck.com
Celery - Best Practices denibertovic.com

Crossroads I/O

http://www.crossroads.io/

c
zeromq

Crossroads I/O ('libxs') is a library for building scalable and high performance distributed applications. It fits between classic BSD sockets, JMS/AMQP-style message queues, and enterprise message-oriented middleware.

Resources

Crossroads I/O core library source code github.com
ZeroC ICE vs 0MQ/ZeroMQ vs Crossroads IO vs Open Source DDS stackoverflow.com
ANNOUNCE Crossroads I/O release 1.0.0 lists.zeromq.org
ZeroMQ vs Crossroads I/O stackoverflow.com

Darner

https://github.com/wavii/darner

simple, lightweight message queue

Resources

Darner, Wavii’s new queue server: 10x faster, 10x smaller wavii.wordpress.com
How do Darner and RabbitMQ compare? at Quora quora.com

Delayed::Job

https://github.com/collectiveidea/delayed_job

ruby
mysql

Database based asynchronously priority queue system

Resources

#171 Delayed Job - RailsCasts railscasts.com
Do It Later With Delayed Job. therailsway.com
Delayed Jobs Callbacks and Hooks in Rails salsify.com

Gearman

http://gearman.org

Gearman Job Server

Resources

What Powers Instagram: Hundreds of Instances, Dozens of Technologies instagram-engineering.tumblr.com
Gearman and ZeroMQ, two different animals speakerdeck.com
Gearman: A Job Server made for Scale (Presentation) slideshare.net
Scalable architecture with Gearman job server. cirello.org
Installing, configuring and running Gearman Job Server + PHP on Ubuntu modernfidelity.co.uk
Playing with Gearman toys.lerdorf.com
Notes on using Gearman with Python saltycrane.com

HornetQ

http://www.jboss.org/hornetq

java
amqp
jms

An open source project to build a multi-protocol, embeddable, very high performance, clustered, asynchronous messaging system.

Resources

Building a Distributed Messaging System at Groupon engineering.groupon.com
Leveraging the HornetQ Message Broker at Shopzilla slideshare.net
Getting Started with HornetQ 2.3 java.dzone.com

IronMQ

http://www.iron.io/mq

go
service

IronMQ is an easy-to-use highly available message queuing service. It is available as a cloud service on Amazon and Rackspace as well as on-premise with Iron.io's enterprise offering. Features include a nice dashboard to manage queues, easy to create webhooks, unicast and multicast Push Queues, autoscaling alerts for worker processes, and error queues.

Resources

Push Queues for Reliable Messaging blog.iron.io
Using IronMQ for Delayed Processing and Increasing Scale developer.rackspace.com
Message Queues: Comparing Beanstalkd, IronMQ and Amazon SQS sitepoint.com
Using IronMQ as a Celery Broker celery.readthedocs.org
Laravel 4 + IronMQ Push Queues blog.iron.io
Using IronMQ as a DelayedJob Broker blog.iron.io

Apache Kafka

http://kafka.apache.org/

java

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

Resources

Open-sourcing Kafka, LinkedIn’s distributed message queue blog.linkedin.com
Kafka: The Great Logfile in the Sky by John Pignata (slides) speakerdeck.com
Apache Kafka - distributed publish-subscribe messaging system - Morten Kjetland (video) vimeo.com
Running a Multi-Broker Apache Kafka 0.8 Cluster on a Single Node michael-noll.com
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013 [Presentation] slideshare.net
Intra-cluster Replication in Apache Kafka engineering.linkedin.com

Kestrel

http://robey.github.io/kestrel/

scala

simple, distributed message queue system

Resources

Scarling » Kestrel - Robey robey.lag.net
Kestrel in Production at Papertrail blog.scoutapp.com
Misleading Title About Queueing railstips.org
Don't Forget About the Kestrel Messaging Queue java.dzone.com

nanomsg

http://nanomsg.org/

c
zeromq

nanomsg is a socket library that provides several common communication patterns

Resources

Getting Started with 'nanomsg' tim.dysinger.net
Nanomsg: ZeroMQ done right hguemar.fedorapeople.org

NSQ

https://github.com/bitly/nsq

realtime distributed message processing at scale

Resources

NSQ: realtime distributed message processing at scale at bit.ly blog word.bitly.com
NSQ - NYC Golang Meetup by Matt Reiferson (slides) speakerdeck.com
Stream Processing: Philosophy, Concepts, and Technologies (slides) speakerdeck.com
Go Advent Day 22 - A Journey Into NSQ blog.gopheracademy.com

Apache Qpid

http://qpid.apache.org/

java
amqp
cpp

Apache Qpid™ makes messaging tools that speak AMQP and support many languages and platforms (in C++, Python, Java JMS and .NET)

Resources

Performance comparison between ZeroMQ, RabbitMQ and Apache Qpid stackoverflow.com
RabbitMQ vs Apache ActiveMQ vs Apache qpid bhavin.directi.com
Programming in Apache Qpid access.redhat.com
QpidComponents.org is an open source development community dedicated to plugins and tools for enterprise-grade messaging in Apache Qpid qpidcomponents.org

queue_classic

https://github.com/ryandotsmith/queue_classic

ruby
postgres

Simple, efficient worker queue for Ruby & PostgreSQL.

Resources

queue_classic by Ryan Smith (slides) dl.dropboxusercontent.com
#344 Queue Classic - RailsCasts railscasts.com

RabbitMQ

http://www.rabbitmq.com/

erlang
amqp

Robust messaging for applications

Resources

RabbitMQ tutorials rabbitmq.com
Smalltalk on RabbitMQ and Nanite paperplanes.de
22 Billion Served: Julien Genestoux of Superfeedr blog.gopivotal.com
Lift AMQP with RabbitMQ and Scala – Tutorial and Screencast timperrett.com
High powered messaging with RabbitMQ slideshare.net
RabbitMQ in Action: Distributed Messaging for Everyone (book) amazon.com

Resque

http://resquework.org/

ruby
redis

A rock-solid job queue, written in Ruby, backed by Redis.

Build at GitHub as an alternative to existing solutions. Powers GitHub background jobs ever since.
Heavily relying on Redis as backend
fork(2) is used for spawning workers
Web UI for managing running jobs, active workers etc.
Many plugins (like resque-scheduler, resque-retry)
Big community

Resources

Introducing Resque at GitHub.com blog github.com
Queuing in Ruby with Redis and Resque at Heroku Center devcenter.heroku.com
Do YOU KNOW Resque rubylearning.com
How Queuing With Resque Works girders.org

RestMQ

http://restmq.com/

python
redis

RestMQ is a message queue which uses HTTP as transport, JSON to format a minimalist protocol and is organized as REST resources. It stands on the shoulder of giants, built over Python, Twisted, Cyclone (a Tornado implementation over twisted) and Redis.

Resources

RestMQ - HTTP/Redis based Message Queue (slides) slideshare.net

RQ

http://python-rq.org/

python
redis

RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers

Resources

Why and how Pricing Assistant migrated from Celery to RQ - Paris.py #2 slideshare.net

Sidekiq

http://sidekiq.org/

ruby
redis

Simple, efficient background processing for Ruby.

Based on Redis
Multithreaded, employs Celluloid under the hood
Web UI
Resque Compatibility
Sidekiq Pro - paid version ($750) with support and additional features (batches, notifications, reliability, metrics)
Fast growing community