Erlang Mailing Lists

Author Message

<  RabbitMQ mailing list  ~  Weird Issue when cluster member restarts

Guest
Posted: Fri Nov 06, 2009 2:28 am Reply with quote
Guest
Hi..

My setup is as follows :
* 2 RabbitMQ (1.7.0 running on R13B02.1) servers in a cluster (say rabbit@m1, rabbit@m2).. both disk nodes.
* 2 Clients (client@m1 connected to rabbit@m1, client@m2 connected to rabbit@m2). I am using the Erlang client (which currently has an issue when connecting to a server which is part of a cluster... this I fixed by setting "insist = true" in the #'connection.open' record)
* both clients have processes that have subscribed to the same durable Queue (Q) using the same key (K). Q is bound to a durable topic exchange (X). X and Q are created on rabbit@m1.

Now this is what i do..
* I bring down rabbit@m1, and client@m1 (They are co hosted... so both are brought down together). I do a "rabbitmqctl list_queues" on rabbit@m2... I do not see Q there..
* I bring rabbit@m1 up again... After which I see Q when i do a "rabbitmqctl list_queues".
* Before I bring up client@m2, I publish a messge to X with a routing key K...
Issue
Guest
Posted: Fri Nov 06, 2009 11:28 am Reply with quote
Guest
Hi Arun,

On Fri, Nov 06, 2009 at 07:58:09AM +0530, Arun Suresh wrote:
> My setup is as follows :
> * 2 RabbitMQ (1.7.0 running on R13B02.1) servers in a cluster (say rabbit@m1,
> rabbit@m2).. both disk nodes.
> * 2 Clients (client@m1 connected to rabbit@m1, client@m2 connected to
> rabbit@m2). I am using the Erlang client (which currently has an issue when
> connecting to a server which is part of a cluster... this I fixed by setting
> "insist = true" in the #'connection.open' record)

Can you elaborate on this issue a bit? Is this redirection coming back
from the server?

> * both clients have processes that have subscribed to the same durable Queue
> (Q) using the same key (K). Q is bound to a durable topic exchange (X). X
> and Q are created on rabbit@m1.

I don't understand what you mean by a queue with a key. I take it you
have one topic exchange, one queue bound to the exchange with one
binding key (is this the K you refer to above?), and two consumers on
the queue. (From reading the rest of your email, yes this is clearly
what you mean).

> Now this is what i do..
> * I bring down rabbit@m1, and client@m1 (They are co hosted... so both are
> brought down together). I do a "rabbitmqctl list_queues" on rabbit@m2... I
> do not see Q there..

Ok, so the queue has been created on m1, not m2.

> * I bring rabbit@m1 up again... After which I see Q when i do a "rabbitmqctl
> list_queues".
> * Before I bring up client@m2, I publish a messge to X with a routing key
> K...
> Issue : Since client@m2 is up and has a process subscribed to Q, I expect
> that process to receive the message... It doesnt..

Indeed it won't.

Clustering makes all exchanges and queues available on all nodes of the
cluster. *However*, a queue is still located on a single node, even
though it is accessible via all nodes. The queue will be created on the
node the client is connected to when it creates the queue. Thus your
queue has been created on m1, thus it's accessible from both m1 and m2,
but when m1 goes down, the queue will go down with it.

From that point on, m2 has no record of the queue, so msgs sent to X
with K to go to Q will not ever reach Q - as far as m2 is concerned, Q
just does not exist.

When m1 comes back up, Q will be recovered.

> Further more... I notice that when i bring client@m2 back up, It restarts
> its subscribers... now all messages published to X with K are sent ONLY to
> client@m2.

Well client@m1 died. This means its subscriptions to Q were lost - there
is no concept of persisting subscriptions, so when rabbit@m1 comes back
up, the Q will have no consumers at all. If you restart client@m2 then
yes, it'll recreate its subscription to Q, and thus receive all the msgs
sent to that Q. If you also restart client@m1 then that too will
recreate its subscription to Q and share the msgs with client@m2 as
before.

Hope this is of help.

Matthew

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist
Guest
Posted: Mon Nov 09, 2009 2:29 am Reply with quote
Guest
Hello Mathew..

Thanks for ur reply.. was out of town so couldn attend to mail earlier.. my comments inline...

On Fri, Nov 6, 2009 at 4:56 PM, Matthew Sackman <matthew@lshift.net (matthew@lshift.net)> wrote:
Quote:
Hi Arun,

On Fri, Nov 06, 2009 at 07:58:09AM +0530, Arun Suresh wrote:
> My setup is as follows :
> * 2 RabbitMQ (1.7.0 running on R13B02.1) servers in a cluster (say rabbit@m1,
> rabbit@m2).. both disk nodes.
> * 2 Clients (client@m1 connected to rabbit@m1, client@m2 connected to
> rabbit@m2). I am using the Erlang client (which currently has an issue when
> connecting to a server which is part of a cluster... this I fixed by setting
> "insist = true" in the #'connection.open' record)


Can you elaborate on this issue a bit? Is this redirection coming back
from the server?
Yes.. I guess its a redirection request coming from the server..
basicaly.. the response to my 'connection.open' request is a 'conncetion.redirect' which on recipt, the client throws an exception and exits..
ive gotten around this issue by setting 'insist = true' in the request record...
Maybe ill look at the Java client and see whats done there ?

Quote:

> * both clients have processes that have subscribed to the same durable Queue
> (Q) using the same key (K). Q is bound to a durable topic exchange (X). X
> and Q are created on rabbit@m1.


I don't understand what you mean by a queue with a key. I take it you
have one topic exchange, one queue bound to the exchange with one
binding key (is this the K you refer to above?), and two consumers on
the queue. (From reading the rest of your email, yes this is clearly
what you mean).

> Now this is what i do..
> * I bring down rabbit@m1, and client@m1 (They are co hosted... so both are
> brought down together). I do a "rabbitmqctl list_queues" on rabbit@m2... I
> do not see Q there..


Ok, so the queue has been created on m1, not m2.

> * I bring rabbit@m1 up again... After which I see Q when i do a "rabbitmqctl
> list_queues".
> * Before I bring up client@m2, I publish a messge to X with a routing key
> K...
> Issue

Display posts from previous:  

All times are GMT
Page 1 of 1
This forum is locked: you cannot post, reply to, or edit topics.

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum