Fork me on GitHub

Queues

Job queues, message queues and other queues. Almost all of them in one place.

About

There are many queueing systems out there. Each one of them is different and was created for solving certain problems. This page tries to collect the libraries that are widely popular and have a successful record of running on (big) production systems.

The goal is to create a quality list of queues with a collection of articles, blog posts, slides, and videos about them. After reading the linked articles, you should have a good idea about: the pros and cons of each queue, a basic understanding of how the queue works, and what each queue is trying to achieve. Basically, you should have all the information you need to decide which queue will best fit your needs.

The whole site is open source and is edited on a daily basis. If you find any mistakes or see something missing, feel free to post an issue or send a pull request. All contributions are welcome! Thanks.

Projects

Beanstalkd

http://kr.github.io/beanstalkd/

  • c

Beanstalkd is a simple, fast work queue.

Resources

Crossroads I/O

http://www.crossroads.io/

  • c
  • zeromq

Crossroads I/O ('libxs') is a library for building scalable and high performance distributed applications. It fits between classic BSD sockets, JMS/AMQP-style message queues, and enterprise message-oriented middleware.

Resources

Gearman

http://gearman.org

  • c

Gearman Job Server

Resources

IronMQ

http://www.iron.io/mq

  • go
  • service

IronMQ is an easy-to-use highly available message queuing service. It is available as a cloud service on Amazon and Rackspace as well as on-premise with Iron.io's enterprise offering. Features include a nice dashboard to manage queues, easy to create webhooks, unicast and multicast Push Queues, autoscaling alerts for worker processes, and error queues.

Resources

Apache Kafka

http://kafka.apache.org/

  • java

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

Resources

Apache Qpid

http://qpid.apache.org/

  • java
  • amqp
  • cpp

Apache Qpid™ makes messaging tools that speak AMQP and support many languages and platforms (in C++, Python, Java JMS and .NET)

Resources

Resque

http://resquework.org/

  • ruby
  • redis

A rock-solid job queue, written in Ruby, backed by Redis.

Resources

RestMQ

http://restmq.com/

  • python
  • redis

RestMQ is a message queue which uses HTTP as transport, JSON to format a minimalist protocol and is organized as REST resources. It stands on the shoulder of giants, built over Python, Twisted, Cyclone (a Tornado implementation over twisted) and Redis.

Resources

Sidekiq

http://sidekiq.org/

  • ruby
  • redis

Simple, efficient background processing for Ruby.

  • Based on Redis
  • Multithreaded, employs Celluloid under the hood
  • Web UI
  • Resque Compatibility
  • Sidekiq Pro - paid version ($500) with support and additional features (batches, notifications, reliability, metrics)
  • Fast growing community

Resources

ZeroMQ

http://www.zeromq.org/

  • cpp

The Intelligent Transport Layer

Resources