Hiep Doan

Best practice to configure your RabbitMq cluster hosted with GCP's Kubernetes engine

April 13, 2020

  • rabbitmq
  • kubernetes

This article discusses few best practices that we have used with our RabbitMq cluster hosted with GCP’s Kubernetes application. Note that every configuration comes with a certain trade-off. It might work for us but not necessarily for your different requirement and workload.

Set lazy mode for queues

With this mode it basically means that the messages delivered through RabbitMq are persisted to disk, instead of being kept in memory. This comes at a cost of higher latency in our case, however it brings much more stability for us.

This setting helps us to avoid the case when suddenly the number of messages spike and it needs more than the limit set by us for the RabbitMq’s pods. As a result, we do not need to worry if the RabbitMq’s pods might crash in those cases.

To set this mode using policy to all queues, you can run this command:

rabbitmqctl set_policy "lazy-queue" ".*" '{"queue-mode":"lazy"}' --apply-to queues

Set time-to-live (TTL) for messages

Unprocessed messages can stay for a long time in the queue (even for forever) and prevent new tasks from being processed. Thus, we need to configure a maximum time that a message can live in the queue.

rabbitmqctl set_policy TTL ".*" '{"message-ttl":600000}' --apply-to queues

Use dead-letter queue to store unprocessed messages

When a message is failed to be processed by the consumer or it exceeds the time-to-live duration, it should not be discarded by RabbitMq but rather should be kept in a separate dead-letter queue so we can investigate the message’s content and why it failed later.

Here we can configure a dead-letter exchange with RabbitMq so a message can be sent there and in turn, being forwarded to the dead letter queue.

It’s also configure different dead-letter queue for different topics using routing key.

rabbitmqctl set_policy "event-queue-policy" ".*" '{"dead-letter-exchange":"dead-letter-exchange"}' --apply-to queues

Set max length for dead-letter queue

Since all failed messages can be sent to the dead-letter queue, we have the risk that the queue itself can grow indefinitely. Thus it makes sense to only keep a certain amount of failed messages.

rabbitmqctl set_policy "deal-letter-queue-policy" "^dead-letter-queue$" '{"max-length":1000}' --apply-to queues

Hiep Doan

Written by Hiep Doan, a software engineer living in the beautiful Switzerland. I am passionate about building things which matters and sharing my experience along the journey. I also selectively post my articles in Medium. Check it out!