| Author |
Message |
|
| Guest |
 Posted: Fri Aug 28, 2009 9:22 pm |
 |
|
|
Guest
|
I've noticed some issues with clustered boxes that are weird. For example, in a two-box cluster I have Client A sending messages to Server A and Client B pulling messages from Server B. We already know that if we shut down Server A (it was first to declare a queue) messages stop being delivered to Server B and in turn Client B. The strange behavior I've noticed is that if I bring Server A back up and send messages to it they are not relayed to Server B, which has Client B attached. Only after I restart Server B do messages begin to be relayed to Client B.
|
|
|
| Back to top |
|
| Guest |
| Posted: Fri Aug 28, 2009 9:52 pm |
|
|
|
Guest
|
Jorge,
Jorge Varona wrote:
> I've noticed some issues with clustered boxes that are weird. For
> example, in a two-box cluster I have Client A sending messages to Server
> A and Client B pulling messages from Server B. We already know that if
> we shut down Server A (it was first to declare a queue) messages stop
> being delivered to Server B and in turn Client B. The strange behavior
> I've noticed is that if I bring Server A back up and send messages to it
> they are not relayed to Server B, which has Client B attached. Only
> after I restart Server B do messages begin to be relayed to Client B.
When node A dies, as far as B is concerned all the queues on A die too.
If client B attempted a 'basic.get', or indeed any other operation on
any of A's queues it would get a 'not_found' error.
BUT - and this is what you are seeing - there is no way in AMQP to
inform existing subscribers that a queue has vanished.
This isn't just a problem for clustering - you can run into the same
issue on just a single node if one client consumes from a queue and
another client removes that queue.
> Here are my assumptions:
>
> 1. Queues exist only on the server on which they were first declared.
>
> 2. Nodes within a cluster relay requests to the server on which the
> queue exists instead of messages being relayed to the server after first
> received.
Both correct.
> are there efforts to address these issues/scenarios?
It is possible that AMQP 1.0 addresses this. Not sure.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Fri Aug 28, 2009 10:08 pm |
|
|
|
Guest
|
Mattias,
Thanks for the quick response. I understand that if the Server A dies
other nodes should be able to assume that the queue is dead as well. My
real concern is that the queue(s) are not distributed and the box that
carries the physical queue is a single point of failure. While the
current clustering helps load-balance read operations very well it
doesn't address durability and reliability concerns, which happen to be
important to my project.
I guess my real question is if there are or will be efforts to
distribute the queues across physical nodes? I understand that this
implies locking, consensus, and a bunch of other things that could
hinder scale. If not, do you have any recommendations on how I could
provide a higher durability and fault-tolerance guarantees to my
consumers?
Jorge
-----Original Message-----
From: Matthias Radestock [mailto:matthias@lshift.net]
Sent: Friday, August 28, 2009 2:52 PM
To: Jorge Varona
Cc: rabbitmq-discuss@lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Clustering Issue
Jorge,
Jorge Varona wrote:
> I've noticed some issues with clustered boxes that are weird. For
> example, in a two-box cluster I have Client A sending messages to
Server
> A and Client B pulling messages from Server B. We already know that if
> we shut down Server A (it was first to declare a queue) messages stop
> being delivered to Server B and in turn Client B. The strange behavior
> I've noticed is that if I bring Server A back up and send messages to
it
> they are not relayed to Server B, which has Client B attached. Only
> after I restart Server B do messages begin to be relayed to Client B.
When node A dies, as far as B is concerned all the queues on A die too.
If client B attempted a 'basic.get', or indeed any other operation on
any of A's queues it would get a 'not_found' error.
BUT - and this is what you are seeing - there is no way in AMQP to
inform existing subscribers that a queue has vanished.
This isn't just a problem for clustering - you can run into the same
issue on just a single node if one client consumes from a queue and
another client removes that queue.
> Here are my assumptions:
>
> 1. Queues exist only on the server on which they were first declared.
>
> 2. Nodes within a cluster relay requests to the server on which the
> queue exists instead of messages being relayed to the server after
first
> received.
Both correct.
> are there efforts to address these issues/scenarios?
It is possible that AMQP 1.0 addresses this. Not sure.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| tonyg |
| Posted: Tue Sep 01, 2009 10:45 am |
|
|
|
User
Joined: 07 Nov 2006
Posts: 199
|
Hi Jorge,
Jorge Varona wrote:
> do you have any recommendations on how I could
> provide a higher durability and fault-tolerance guarantees to my
> consumers?
There are a couple of approaches you could use, depending on your exact
requirements and the level of complexity you're willing to cope with.
The two main classes of solution are
- run a shared filesystem underneath rabbit, with a hot standby if
one server should fail. This separates data availability from
service availability, and reduces the "no service" window to the
failover interval.
- run two servers and route *all* traffic through *both*, deduping
at the receiving client. This gives as many nines of uptime as you
want, at a cost in administrative complexity and bandwidth wastage
We've been talking about this stuff to various people recently so have
some illustrations of what we mean that might be useful: see
http://dev.lshift.net/tonyg/rabbitmq-intro-ha.pdf. [1]
Regards,
Tony
[1]: A slightly earlier slide deck has a little bit more explanation
attached, and carries similar content otherwise:
http://dev.lshift.net/tonyg/Achieving%20Scale%20with%20Messaging%20and%20the%20Cloud%2020090709%20(with%20notes).pdf
--
[][][] Tony Garnock-Jones | Mob: +44 (0)7905 974 211
[][] LShift Ltd | Tel: +44 (0)20 7729 7060
[] [] http://www.lshift.net/ | Email: tonyg@lshift.net
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
  
|
| Guest |
| Posted: Tue Sep 01, 2009 5:41 pm |
|
|
|
Guest
|
Tony,
Thanks for responding. I'm not sure we're at a point where either of
these scenarios would be appealing. The first one requires additional
hardware and cost and I agree the second one just seems unnecessarily
redundant, not to mention the burden of clients ensuring uniqueness of a
message, probably using some DHT, again more hardware and cost.
We're fortunate in that our current load requirements are such that we
can experiment a bit with several client-based consensus approaches to
mitigate down servers and provide near real-time failover. My real hope
was that RabbitMQ could provide server-side failover functionality or
some kind of loud, broadcasted failure, which clients could detect to
perform their own fail-over (reconnection, declare exchanges/queues,
etc.). This may fall outside the scope of AMQP but I think it could be
beneficial to your user base.
For now I think we'll bake our failover into our client implementation.
Thanks again for taking the time to respond.
Jorge
-----Original Message-----
From: Tony Garnock-Jones [mailto:tonyg@lshift.net]
Sent: Tuesday, September 01, 2009 3:45 AM
To: Jorge Varona
Cc: Matthias Radestock; rabbitmq-discuss@lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Clustering Issue
Hi Jorge,
Jorge Varona wrote:
> do you have any recommendations on how I could
> provide a higher durability and fault-tolerance guarantees to my
> consumers?
There are a couple of approaches you could use, depending on your exact
requirements and the level of complexity you're willing to cope with.
The two main classes of solution are
- run a shared filesystem underneath rabbit, with a hot standby if
one server should fail. This separates data availability from
service availability, and reduces the "no service" window to the
failover interval.
- run two servers and route *all* traffic through *both*, deduping
at the receiving client. This gives as many nines of uptime as you
want, at a cost in administrative complexity and bandwidth wastage
We've been talking about this stuff to various people recently so have
some illustrations of what we mean that might be useful: see
http://dev.lshift.net/tonyg/rabbitmq-intro-ha.pdf. [1]
Regards,
Tony
[1]: A slightly earlier slide deck has a little bit more explanation
attached, and carries similar content otherwise:
http://dev.lshift.net/tonyg/Achieving%20Scale%20with%20Messaging%20and%2
0the%20Cloud%2020090709%20(with%20notes).pdf
--
[][][] Tony Garnock-Jones | Mob: +44 (0)7905 974 211
[][] LShift Ltd | Tel: +44 (0)20 7729 7060
[] [] http://www.lshift.net/ | Email: tonyg@lshift.net
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| wuji |
| Posted: Mon Sep 17, 2012 1:49 am |
|
|
|
User
Joined: 10 Aug 2012
Posts: 654
|
the expense of our villain, Rex, and not at the the cheap authentic jordans the expense of those suffering from the disease."The joke, "From
man to another, I hope you get Lou Gehrig's disease," disease," cheap replica designer *beep* disease," shocked movie-going patients and advocates, who say it crossed
line."I didn't expect to go to a movie and sit sit cheap authentic jordans sit with an audience laughing at the expense of people
ALS," said Randy Pipkin, who was diagnosed in 2005. "I "I [h4]replica designer bags for sale[/h4] "I think the message this film sends out is a
slap in the face to people dying from this horrific horrific [h4]authentic jordans[/h4] horrific disease."Lou Gehrig's disease, also known as ALS, progressively robs
of their ability to move, speak, eat and breathe. There There authentic jordans There is currently no cure.The punch line prompted an online |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
