| Author |
Message |
|
| Guest |
 Posted: Mon Oct 19, 2009 10:41 am |
 |
|
|
Guest
|
Hi everybody,
I'm looking into a way how to implement a HA setup for persistent
queues. Since the rabbitmq cluster doesn't replicate queues across
multiple nodes my initial idea was to implement this on the client:
connect publisher and consumers to multiple independent servers,
create identical exchanges and queues, then send the messages round
robin to the nodes. While this has the additional benefit of proper
scalability it requires some effort on the client, I basically have to
override all methods and encapsulate the "replication" logic there.
While this isn't a true HA setup (messages in the broken node can't be
delivered until restart and correct order of messages isn't guaranteed
anymore) it solves 90% of the problem by providing a way to allow
continuous message publishing.
Now I have come across this [1] blog-post by Ilya Grigorik which
says: "Assign two identically named queues to an exchange and the
broker will automatically round-robin the messages". So my
understanding is that I could just create identical queues on every
node, and the broker will take care of the rest: if one node goes
down, messages will just be spread over the other queues/nodes.
Is this supported by rabbitmq? I've looked around in the amqp specs
and documentation, but couldn't find anything related.
Regards
Sebastian
[1] http://www.igvita.com/2009/10/08/advanced-messaging-routing-with-amqp/
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Tue Oct 20, 2009 5:53 am |
|
|
|
Guest
|
Sebastian,
Sebastian wrote:
> Now I have come across this [1] blog-post by Ilya Grigorik which
> says: "Assign two identically named queues to an exchange and the
> broker will automatically round-robin the messages". So my
> understanding is that I could just create identical queues on every
> node, and the broker will take care of the rest: if one node goes
> down, messages will just be spread over the other queues/nodes.
> Is this supported by rabbitmq? I've looked around in the amqp specs
> and documentation, but couldn't find anything related.
>
> [1] http://www.igvita.com/2009/10/08/advanced-messaging-routing-with-amqp/
That part of the article is wrong. Exchanges route messages to all the
queues bound with a binding that matches the message. There is no
round-robining going on; that only happens for delivering messages to
multiple consumers of the same queue.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Wed Oct 21, 2009 8:06 pm |
|
|
|
Guest
|
On Tue, 20 Oct 2009 06:53:17 +0100
Matthias Radestock <matthias@lshift.net> wrote:
> Sebastian,
>
> Sebastian wrote:
> > Now I have come across this [1] blog-post by Ilya Grigorik which
> > says: "Assign two identically named queues to an exchange and the
> > broker will automatically round-robin the messages". So my
> > understanding is that I could just create identical queues on
> > every node, and the broker will take care of the rest: if one node
> > goes down, messages will just be spread over the other queues/nodes.
> > Is this supported by rabbitmq? I've looked around in the amqp
> > specs and documentation, but couldn't find anything related.
> >
> > [1]
> > http://www.igvita.com/2009/10/08/advanced-messaging-routing-with-amqp/
>
> That part of the article is wrong. Exchanges route messages to all
> the queues bound with a binding that matches the message. There is no
> round-robining going on; that only happens for delivering messages to
> multiple consumers of the same queue.
>
> Regards,
>
> Matthias.
>
The author of the article seems to be confusing "declaring and consuming
from the same queue from two different places" with "creating two
identically-named queues" which (if I understand correctly) is not
possible, since declaration is idempotent. I would guess that's why the
operation is called "declare" and not "create".
If you reword it to remove the misunderstanding, then the statement
seems correct. Attempting to create two identically-named queues will
result in declaring the same queue twice; the two consumers then
consuming from that queue will see load-balancing behavior.
-Kyle
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon Oct 26, 2009 4:05 pm |
|
|
|
Guest
|
Kyle Schaffrick:
> Attempting to create two identically-named queues will
> result in declaring the same queue twice; the two consumers then
> consuming from that queue will see load-balancing behavior.
Thanks for the clarification. Since this does not work: how do I
achieve reliable persistent message delivery even if the one node
holding the queue dies? As stated before my take would be to replicate
queues client-side on independent brokers, but this seems to be
getting quite complicated as the queues have to be "double-declared",
linked and properly re-initialized on startup of a new empty broker
(basically doing some kind of client-side clustering).
Regards
Sebastian
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon Oct 26, 2009 4:23 pm |
|
|
|
Guest
|
Sebastian
On Mon, Oct 26, 2009 at 4:04 PM, Sebastian <mail@sebastian-latza.de> wrote:
>
> Kyle Schaffrick:
>
>> Attempting to create two identically-named queues will
>> result in declaring the same queue twice; the two consumers then
>> consuming from that queue will see load-balancing behavior.
>
> Thanks for the clarification. Since this does not work: how do I
> achieve reliable persistent message delivery even if the one node
> holding the queue dies?
At the risk of being seen as pedantic...
You could achieve it by waiting for the node to recover from disk. In
other words: RabbitMQ provides "eventual delivery" out of the box. If
you want to 'persist more reliably' than on one drive using RabbitMQ
out of the box, then use a RAID or a SAN.
I am guessing that what you mean is *availability* in which a message
can always reach a suitable queue even if the sending client loses its
connection with the node that crashed. The problem with using a
single node is the time it takes to recover, and client reconnection
costs (if any). The time to recover is partly influenced by how much
state has to recover, and partly by the size of the unavailability
window that you can tolerate.
Many applications don't like to wait more than (say) 1ms, 10ms or 50ms
before they can send their message. In this case you will typically
want to send a message *before* the crashed node has been able to
recover. You can achieve this in two ways:
1. You don't care about replicating state and can use any other queue
as a 'suitable queue'. This is easy - just make sure the right
consumers and routes are wired up.
2. You do care about replicating state - which I think is the case you
have in mind. Then you can fail over to a passive server. We can
provide some more info on this if you like -- we have done it for a
few people but it is not yet 'available OOTB'. (You can also do
active/active but it is much more fiddly atm).
> As stated before my take would be to replicate
> queues client-side on independent brokers, but this seems to be
> getting quite complicated as the queues have to be "double-declared",
> linked and properly re-initialized on startup of a new empty broker
> (basically doing some kind of client-side clustering).
Right. I think you probably want case 2 above.
alexis
> Regards
> Sebastian
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss@lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon Oct 26, 2009 4:50 pm |
|
|
|
Guest
|
Alexis Richardson:
> 2. You do care about replicating state - which I think is the case you
> have in mind. Then you can fail over to a passive server. We can
> provide some more info on this if you like -- we have done it for a
> few people but it is not yet 'available OOTB'. (You can also do
> active/active but it is much more fiddly atm).
>
I want to make sure that when an event happens, it can be delivered to
the correct queue in any case ( "always writable"). As a very
simplified example I would want to send out welcome mails
asynchronously to newly registered customers. If the message queue for
this function wasn't available at the time I'd have to shut down my
registration which I would want to avoid naturally.
I'd be happy to provide you with testing/feedback and maybe some basic
erlang hacking for anything that might help regarding that problem.
Regards
Sebastian
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon Oct 26, 2009 4:58 pm |
|
|
|
Guest
|
On Mon, Oct 26, 2009 at 4:49 PM, Sebastian <mail@sebastian-latza.de> wrote:
> Alexis Richardson:
>
>> 2. You do care about replicating state - which I think is the case you
>> have in mind. |
|
|
| Back to top |
|
| Guest |
| Posted: Mon Oct 26, 2009 5:11 pm |
|
|
|
Guest
|
The solution we use is pretty much what Alexis describes. No
clustering. And both consumers and producers declare queues on a fail
over to a hot standby/passive server. That way messages are never
lost. When the old server comes back up we fail back.
-J
Sent via iPhone
On Oct 26, 2009, at 10:23, Alexis Richardson <alexis.richardson@gmail.com
> wrote:
> Sebastian
>
> On Mon, Oct 26, 2009 at 4:04 PM, Sebastian <mail@sebastian-latza.de>
> wrote:
>>
>> Kyle Schaffrick:
>>
>>> Attempting to create two identically-named queues will
>>> result in declaring the same queue twice; the two consumers then
>>> consuming from that queue will see load-balancing behavior.
>>
>> Thanks for the clarification. Since this does not work: how do I
>> achieve reliable persistent message delivery even if the one node
>> holding the queue dies?
>
> At the risk of being seen as pedantic...
>
> You could achieve it by waiting for the node to recover from disk. In
> other words: RabbitMQ provides "eventual delivery" out of the box. If
> you want to 'persist more reliably' than on one drive using RabbitMQ
> out of the box, then use a RAID or a SAN.
>
> I am guessing that what you mean is *availability* in which a message
> can always reach a suitable queue even if the sending client loses its
> connection with the node that crashed. The problem with using a
> single node is the time it takes to recover, and client reconnection
> costs (if any). The time to recover is partly influenced by how much
> state has to recover, and partly by the size of the unavailability
> window that you can tolerate.
>
> Many applications don't like to wait more than (say) 1ms, 10ms or 50ms
> before they can send their message. In this case you will typically
> want to send a message *before* the crashed node has been able to
> recover. You can achieve this in two ways:
>
> 1. You don't care about replicating state and can use any other queue
> as a 'suitable queue'. This is easy - just make sure the right
> consumers and routes are wired up.
>
> 2. You do care about replicating state - which I think is the case you
> have in mind. Then you can fail over to a passive server. We can
> provide some more info on this if you like -- we have done it for a
> few people but it is not yet 'available OOTB'. (You can also do
> active/active but it is much more fiddly atm).
>
>
>
>> As stated before my take would be to replicate
>> queues client-side on independent brokers, but this seems to be
>> getting quite complicated as the queues have to be "double-declared",
>> linked and properly re-initialized on startup of a new empty broker
>> (basically doing some kind of client-side clustering).
>
> Right. I think you probably want case 2 above.
>
> alexis
>
>
>
>> Regards
>> Sebastian
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss@lists.rabbitmq.com
>> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss@lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| wuji |
| Posted: Fri Aug 31, 2012 5:00 am |
|
|
|
User
Joined: 10 Aug 2012
Posts: 654
|
long-term plan to lift strict flight restrictions imposed by Defense Defense cheap replica *beep* Defense Secretary Leon Panetta on the $79 billion fleet in
"This is a very prudent way to ensure that we, we, buy real jordans we, in a very careful manner, resume normal flight operations."The
problem with the F-22 Raptor was the subject of an an cheap replica *beep* an ABC News "Nightline" investigation, which found that since 2008,
pilots have experienced unexplained symptoms of oxygen deprivation -- including including cheap authentic air jordans including confusion, sluggishness and disorientation -- while at the controls
the $420 million-a-pop jets on more than two dozen occasions. occasions. [h3]cheap authentic air jordans[/h3] occasions. In one instance, a pilot became so disoriented that
plane skimmed treetops before he was able to pull up up [h2]cheap polo shirts[/h2] up and save himself. The Air Force subjected the F-22 |
|
|
| Back to top |
 
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
