|Posted: Sat Dec 01, 2007 10:27 am
Joined: 06 Sep 2007
Recently we did some load testing of RabbitMQ, working with Intel.
Their press release is reported here:
The use case was a simulated OPRA data feed using combination of:
- Pantor FAST client (essentially a codec) combined with an AMQP
client written in C (yes we are hoping to get this into the community)
- RabbitMQ AMQP broker version 1.2 on the server
Please don't read too much into the latency numbers here: the timed
path included two network hops as well as message processing at the
broker; also, somewhat annoyingly, the numbers are averaged over
multiple scenarios. We wanted to look at throughput because OPRA
feeds are heading to 1 million ticks per second and it's a good load
We shall publish more info soon but the numbers are as follows:
1. Ingress of about 1.3 million OPRA messages per second
2. Replicated out to four clients at once (unicast pub/sub not multicast)
3. So simultaneously, egress of about 5 million OPRA messages per second
The broker cluster was on one multicore box with 16 cores. The network
was a full TCP/IP stack, and a standard 1GigE network (= the bottleneck).
The set up was:
1 Client Box --> 1 Server Box --> 1 Client Box
We used Intel's 16 core Caneland box for the server and the FAST/AMQP
client was delivered by Pantor, working with us.
How come the numbers are so high? Well, one reason is that we used
FAST, which is a codec. Each OPRA message was FAST-compressed
and batched into a block or 'datagram' of 16 compressed OPRA messages.
This is gradually becoming normal practice in the world of market data
feeds because the loads are high and people do not have enough
bandwidth to cope.
So in our test, each datagram contained 16 OPRA messages, and was
sent as one 256 byte AMQP message.
So the throughput can also be seen as:
1. Ingress of 80,000 AMQP messages per second (256b per message)
2. Replicated out to four clients at once (unicast pub/sub)
3. So simultaneously, egress of 320,000 AMQP messages per second (256b
I.e the real load is about 400,000 mps.
There are several ways to get these numbers higher:
- tune RabbitMQ for speed
- use multicast
- use Infiniband
- use faster cores
We just did some more tests using Intel's 45nm cores which look
promising in this regard.
The point is: for most use cases you can get good performance using
COTS hardware. This means you can spend your valuable project
investment dollars on making the user experience better instead of
messing about with deep tech.
We think scalability, stability and ease of use are more important than
raw speed. If you try to run RabbitMQ and do not see what you
expect along any of these metrics, please let us know and we'll help you.
+44 20 7617 7339 (UK)
+44 77 9865 2911 (cell)
+1 650 206 2517 (US)
rabbitmq-discuss mailing list
Post recived from mailinglist
|Back to top
All times are GMT
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum