Fork me on GitHub

Queues

Job queues, message queues and other queues. Almost all of them in one place.

About

There are many queueing systems out there. Each one of them is different and was created for solving certain problems. This page tries to collect the libraries that are widely popular and have a successful record of running on (big) production systems.

The goal is to create a quality list of queues with a collection of articles, blog posts, slides, and videos about them. After reading the linked articles, you should have a good idea about: the pros and cons of each queue, a basic understanding of how the queue works, and what each queue is trying to achieve. Basically, you should have all the information you need to decide which queue will best fit your needs.

The whole site is open source and is updated quite frequently. If you find any mistakes or see something missing, feel free to post an issue or send a pull request. All contributions are welcome! Thanks.

Popular Projects

Amazon Simple Queue Service (SQS)

http://aws.amazon.com/sqs/

  • service

Amazon Simple Queue Service (Amazon SQS) is a messaging queue service that handles message or work flows between other components in a system.

Resources

Beanstalkd

http://kr.github.io/beanstalkd/

  • c

Beanstalkd is a simple, fast work queue.

Resources

IronMQ

http://www.iron.io/mq

  • go
  • service

IronMQ is an easy-to-use highly available message queuing service. It is available as a cloud service on Amazon and Rackspace as well as on-premise with Iron.io's enterprise offering. Features include a nice dashboard to manage queues, easy to create webhooks, unicast and multicast Push Queues, autoscaling alerts for worker processes, and error queues.

Resources

Apache Kafka

http://kafka.apache.org/

  • scala

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

Resources

Mappedbus

http://mappedbus.io/

  • java
  • high-throughput
  • low-latency
  • message-passing
  • ipc

A Java based high throughput, low latency message bus, using either a memory mapped file or shared memory as transport.

Resources

NATS

https://nats.io

  • go
  • .NET
  • node
  • nginx
  • java
  • ruby
  • python
  • scala

NATS is an open-source, high-performance, lightweight cloud messaging system.

Resources

Apache Qpid

http://qpid.apache.org/

  • java
  • amqp
  • cpp

Apache Qpid™ makes messaging tools that speak AMQP and support many languages and platforms (in C++, Python, Java JMS and .NET)

Resources

RabbitMQ

http://www.rabbitmq.com/

  • erlang
  • amqp

Robust messaging for applications

Resources

Resque

https://github.com/resque/resque

  • ruby
  • redis

A rock-solid job queue, written in Ruby, backed by Redis.

Resources

RestMQ

http://restmq.com/

  • python
  • redis

RestMQ is a message queue which uses HTTP as transport, JSON to format a minimalist protocol and is organized as REST resources. It stands on the shoulder of giants, built over Python, Twisted, Cyclone (a Tornado implementation over twisted) and Redis.

Resources

Siberite

http://siberite.org/

  • go

Simple, lightweight, leveldb backed message queue

  • Darner rewritten in Go with additional features
  • Single topic can be consumed multiple times using durable cursors
  • Uses Kestrel (memcached) protocol
  • Keeps all messages out of process
  • Small amount of in-resident memory regardless of queue size
  • Two-phase reliable fetch

Resources

Sidekiq

http://sidekiq.org/

  • ruby
  • crystal
  • redis

Simple, efficient background processing for Ruby.

  • Based on Redis
  • Multithreaded, employs Celluloid under the hood
  • Web UI
  • Resque Compatibility
  • Sidekiq Pro - paid version ($750) with support and additional features (batches, notifications, reliability, metrics)
  • Fast growing community

Resources

ZeroMQ

http://www.zeromq.org/

  • cpp
  • java

The Intelligent Transport Layer

Resources