Erlang/OTP Forums
Author Message

<  RabbitMQ mailing list  ~  Queue inspector.
Guest
Posted: Mon Nov 05, 2007 5:59 pm Reply with quote
Guest
First off, I want to say how much we love amqp and rabbitMQ!
Back to top
Guest
Posted: Mon Nov 05, 2007 6:16 pm Reply with quote
Guest
Matt,

matt wrote:
> First off, I want to say how much we love amqp and rabbitMQ!

Thank you. It is always nice to hear from real users.

> We've ( meez.com) been using it in our production environment (when
> it was alpha) for about 6 months now I think.
>
> In brief, we use rabbitMQ as a way of communicating with our
> rendering cluster. We have user facing tomcat servers that provide a
> way of giving the user options to create an avatar. Upon save, we
> send export requests via rabbitMQ and they're picked up by nodes in
> the export cloud. Upon success of the rendered images, a
> notification is sent back and the user is notified that the exports
> were successful.

That is a pretty neat use case, and it is good to hear that RabbitMQ
worked out well for you.

> the big immediate need we're looking for is a way to see how much is
> being queued up on the rabbitMQ cluster. That way, we can
> programatically bring EC2 nodes up or take then down based on need.
>
> I know that this is in the pipeline - but is there *any* way of doing
> this now and/or is there something that could be easily added to
> support this?

queue.declare-ok, the result of a queue.declare operation, in one of its
fields contains the number of messages in the queue. This is part of the
amqp 0-8 spec and implemented by RabbitMQ. In combination with
queue.declare's passive flag it allows you to determine the length of
existing queues.

If you are using the RabbitMQ Java client, check out
http://www.rabbitmq.com/javadoc/com/rabbitmq/client/impl/ChannelN.html#queueDeclare(int,%20java.lang.String,%20boolean,%20boolean,%20boolean,%20boolean,%20java.util.Map),
and
http://www.rabbitmq.com/javadoc/com/rabbitmq/client/impl/AMQImpl.Queue.DeclareOk.html#getMessageCount()

This is only a very limited interface - you need to know the names of
queues and you only get a report on the number of messages, not their
storage size etc - but perhaps that is sufficient for your needs?


Hope this helps, and thank you for using RabbitMQ,


Matthias.

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top
Guest
Posted: Mon Nov 05, 2007 7:26 pm Reply with quote
Guest
Getting the message count only appears to get updated when you declare the queue.
Back to top
Guest
Posted: Mon Nov 05, 2007 11:22 pm Reply with quote
Guest
Matt,


matt wrote:
> Getting the message count only appears to get updated when you declare
> the queue. What if you wanted to poll the queue and see how many
> messages were on it every 15 seconds or so?

queue.declare has a passive mode of operation where it doesn't actually
create a new queue. That's what I was referring to when I wrote that "in
combination with queue.declare's passive flag it allows you to determine
the length of existing queues".


Matthias.

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top
Guest
Posted: Mon Nov 05, 2007 11:24 pm Reply with quote
Guest
On a related note, is there a way of querying the server to get the
names of all the queues?

Thanks

Brian

On 11/5/07, Matthias Radestock <matthias@lshift.net> wrote:
> Matt,
>
>
> matt wrote:
> > Getting the message count only appears to get updated when you declare
> > the queue. What if you wanted to poll the queue and see how many
> > messages were on it every 15 seconds or so?
>
> queue.declare has a passive mode of operation where it doesn't actually
> create a new queue. That's what I was referring to when I wrote that "in
> combination with queue.declare's passive flag it allows you to determine
> the length of existing queues".
>
>
> Matthias.
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss@lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top
Guest
Posted: Tue Nov 06, 2007 12:10 am Reply with quote
Guest
Sorry, I missed that important piece. Smile
Back to top
alexis
Posted: Tue Nov 06, 2007 3:17 pm Reply with quote
User Joined: 06 Sep 2007 Posts: 80 Location: London
Brian

Not yet, I'm afraid Sad

Management features are due in a future version of AMQP. How
important is this for you today?

alexis


On 11/5/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
> On a related note, is there a way of querying the server to get the
> names of all the queues?
>
> Thanks
>
> Brian
>
> On 11/5/07, Matthias Radestock <matthias@lshift.net> wrote:
> > Matt,
> >
> >
> > matt wrote:
> > > Getting the message count only appears to get updated when you declare
> > > the queue. What if you wanted to poll the queue and see how many
> > > messages were on it every 15 seconds or so?
> >
> > queue.declare has a passive mode of operation where it doesn't actually
> > create a new queue. That's what I was referring to when I wrote that "in
> > combination with queue.declare's passive flag it allows you to determine
> > the length of existing queues".
> >
> >
> > Matthias.
> >
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss@lists.rabbitmq.com
> > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss@lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>


--
Alexis Richardson
+44 20 7617 7339 (UK)
+44 77 9865 2911 (cell)
+1 650 206 2517 (US)

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top View user's profile Send private message Yahoo Messenger
Guest
Posted: Tue Nov 06, 2007 6:01 pm Reply with quote
Guest
I am evaluating rabbitmq/amqp to see if it would do what we need it
to. Our system has a worker/master architecture and we would like to
use rabbitmq for the master<-> worker messaging. There are a few
constraints in our system though:

- workers come and go at any time.
- each worker needs a private queue - the master process needs to be
able to see which workers are currently connected. My initial thought
is that this could be accomplished by the master getting a list of
queues.

My current thought is to have a separate queue that workers use to
register/unregister themselves. They could put the name of their
private queue on this registration queue.

But, it would be nice to have more introspection capabilities.

Thanks

Brian

On 11/6/07, Alexis Richardson <alexis.richardson@cohesiveft.com> wrote:
> Brian
>
> Not yet, I'm afraid Sad
>
> Management features are due in a future version of AMQP. How
> important is this for you today?
>
> alexis
>
>
> On 11/5/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
> > On a related note, is there a way of querying the server to get the
> > names of all the queues?
> >
> > Thanks
> >
> > Brian
> >
> > On 11/5/07, Matthias Radestock <matthias@lshift.net> wrote:
> > > Matt,
> > >
> > >
> > > matt wrote:
> > > > Getting the message count only appears to get updated when you declare
> > > > the queue. What if you wanted to poll the queue and see how many
> > > > messages were on it every 15 seconds or so?
> > >
> > > queue.declare has a passive mode of operation where it doesn't actually
> > > create a new queue. That's what I was referring to when I wrote that "in
> > > combination with queue.declare's passive flag it allows you to determine
> > > the length of existing queues".
> > >
> > >
> > > Matthias.
> > >
> > > _______________________________________________
> > > rabbitmq-discuss mailing list
> > > rabbitmq-discuss@lists.rabbitmq.com
> > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > >
> >
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss@lists.rabbitmq.com
> > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >
>
>
> --
> Alexis Richardson
> +44 20 7617 7339 (UK)
> +44 77 9865 2911 (cell)
> +1 650 206 2517 (US)
>

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top
alexis
Posted: Tue Nov 06, 2007 6:13 pm Reply with quote
User Joined: 06 Sep 2007 Posts: 80 Location: London
Brian

Thanks for your note - that helps.

On 11/6/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
> I am evaluating rabbitmq/amqp to see if it would do what we need it
> to. Our system has a worker/master architecture and we would like to
> use rabbitmq for the master<-> worker messaging.

Great Smile

Work queues are a common pattern and we would love to hear how you get on.



> There are a few
> constraints in our system though:
>
> - workers come and go at any time.

Indeed this is also a common situation when you want your system to be
robust - as I assume you do. Some of this will be supported more
'inherently' in future versions of AMQP, eg for federations of
brokers. But I think there is enough in RabbitMQ as it is today, and
as it currently clusters, to get you going.


> - each worker needs a private queue

Is that strictly needed for your use case?

You could have workers in groups, sharing a queue on a round robin
basis. Then they can 'pull' work and no registry is needed.


> - the master process needs to be
> able to see which workers are currently connected.

Yes - if you want to 'push' work and each worker must have a private queue.


> My initial thought
> is that this could be accomplished by the master getting a list of
> queues.

Got it.


> My current thought is to have a separate queue that workers use to
> register/unregister themselves. They could put the name of their
> private queue on this registration queue.

I would certainly do this if you want to 'push' work. In effect you
are making the registry explicit and this would be very useful for
other implementation patterns, eg shared services 'fed' by queues (a
bit like an 'ESB').


> But, it would be nice to have more introspection capabilities.

I wholly agree!

alexis





>
> Thanks
>
> Brian
>
> On 11/6/07, Alexis Richardson <alexis.richardson@cohesiveft.com> wrote:
> > Brian
> >
> > Not yet, I'm afraid Sad
> >
> > Management features are due in a future version of AMQP. How
> > important is this for you today?
> >
> > alexis
> >
> >
> > On 11/5/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
> > > On a related note, is there a way of querying the server to get the
> > > names of all the queues?
> > >
> > > Thanks
> > >
> > > Brian
> > >
> > > On 11/5/07, Matthias Radestock <matthias@lshift.net> wrote:
> > > > Matt,
> > > >
> > > >
> > > > matt wrote:
> > > > > Getting the message count only appears to get updated when you declare
> > > > > the queue. What if you wanted to poll the queue and see how many
> > > > > messages were on it every 15 seconds or so?
> > > >
> > > > queue.declare has a passive mode of operation where it doesn't actually
> > > > create a new queue. That's what I was referring to when I wrote that "in
> > > > combination with queue.declare's passive flag it allows you to determine
> > > > the length of existing queues".
> > > >
> > > >
> > > > Matthias.
> > > >
> > > > _______________________________________________
> > > > rabbitmq-discuss mailing list
> > > > rabbitmq-discuss@lists.rabbitmq.com
> > > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > >
> > >
> > > _______________________________________________
> > > rabbitmq-discuss mailing list
> > > rabbitmq-discuss@lists.rabbitmq.com
> > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > >
> >
> >
> > --
> > Alexis Richardson
> > +44 20 7617 7339 (UK)
> > +44 77 9865 2911 (cell)
> > +1 650 206 2517 (US)
> >
>


--
Alexis Richardson
+44 20 7617 7339 (UK)
+44 77 9865 2911 (cell)
+1 650 206 2517 (US)

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top View user's profile Send private message Yahoo Messenger
Guest
Posted: Tue Nov 06, 2007 8:02 pm Reply with quote
Guest
> Brian
>
> Thanks for your note - that helps.

No problem.

> On 11/6/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
> > I am evaluating rabbitmq/amqp to see if it would do what we need it
> > to. Our system has a worker/master architecture and we would like to
> > use rabbitmq for the master<-> worker messaging.
>
> Great Smile
>
> Work queues are a common pattern and we would love to hear how you get on.

As I move forward I will let you know how it goes.

>
>
> > There are a few
> > constraints in our system though:
> >
> > - workers come and go at any time.
>
> Indeed this is also a common situation when you want your system to be
> robust - as I assume you do. Some of this will be supported more
> 'inherently' in future versions of AMQP, eg for federations of
> brokers. But I think there is enough in RabbitMQ as it is today, and
> as it currently clusters, to get you going.

This is definitely a very common pattern - although our system has a
few unusual properties (more on that below)

>
> > - each worker needs a private queue
>
> Is that strictly needed for your use case?

Yes, each worker does need a private queue so workers can directly
send messages to a given worker. But, it gets more complicated than
that in a number of ways:

- The master needs to be able to push messages to arbitrary sets of
workers. Our general syntax (our master in written in python) looks
like this:

push(list_of_workers, msg)

So you see things like this:

push(getIDs(), msg) that does a broadcast or:

push(range(10), msg) that pushes to workers 0,1,...10

- Obviously, when pushing messages to specific workers, you can't do
dynamic load balancing. To support dynamic load balancing we have
another single shared FIFO queue that sits on top of the private
worker queues. We have a scheduler that pulls things off the shared
queue and puts them on the private queues, based on how many messages
there are in the private queues at any given moment. I am not sure if
rabbitmq/amqp will be able to handle this more complex queue logic as
it would require logic *in* the rabbitmq server......

So, the next question is ..... how easy it is to build custom rabbitmq
servers that have custom queue logic. Basically, I need queues to be
able to act as clients to other queues.

Currently we have implemented this system using twisted (an
asynchronous networking framework in python). It works extremely
well....but it doesn't scale to a large number of workers, which is
why we are looking at erlang/amqp. It might be the case that we need
to implement our server at a lower level in erlang, but I really like
rabbitmq/amqp.

Any ideas you have about how to approach this would be greatly appreciated.

> You could have workers in groups, sharing a queue on a round robin
> basis. Then they can 'pull' work and no registry is needed.

We have thought about things like this, but we would still need to
have the push mode as well.

>
> > - the master process needs to be
> > able to see which workers are currently connected.
>
> Yes - if you want to 'push' work and each worker must have a private queue.
>
>
> > My initial thought
> > is that this could be accomplished by the master getting a list of
> > queues.
>
> Got it.
>
>
> > My current thought is to have a separate queue that workers use to
> > register/unregister themselves. They could put the name of their
> > private queue on this registration queue.
>
> I would certainly do this if you want to 'push' work. In effect you
> are making the registry explicit and this would be very useful for
> other implementation patterns, eg shared services 'fed' by queues (a
> bit like an 'ESB').

OK, it is good to know that I am on the right track.

Thanks

Brian

>
> > But, it would be nice to have more introspection capabilities.
>
> I wholly agree!
>
> alexis
>
>
>
>
>
> >
> > Thanks
> >
> > Brian
> >
> > On 11/6/07, Alexis Richardson <alexis.richardson@cohesiveft.com> wrote:
> > > Brian
> > >
> > > Not yet, I'm afraid Sad
> > >
> > > Management features are due in a future version of AMQP. How
> > > important is this for you today?
> > >
> > > alexis
> > >
> > >
> > > On 11/5/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
> > > > On a related note, is there a way of querying the server to get the
> > > > names of all the queues?
> > > >
> > > > Thanks
> > > >
> > > > Brian
> > > >
> > > > On 11/5/07, Matthias Radestock <matthias@lshift.net> wrote:
> > > > > Matt,
> > > > >
> > > > >
> > > > > matt wrote:
> > > > > > Getting the message count only appears to get updated when you declare
> > > > > > the queue. What if you wanted to poll the queue and see how many
> > > > > > messages were on it every 15 seconds or so?
> > > > >
> > > > > queue.declare has a passive mode of operation where it doesn't actually
> > > > > create a new queue. That's what I was referring to when I wrote that "in
> > > > > combination with queue.declare's passive flag it allows you to determine
> > > > > the length of existing queues".
> > > > >
> > > > >
> > > > > Matthias.
> > > > >
> > > > > _______________________________________________
> > > > > rabbitmq-discuss mailing list
> > > > > rabbitmq-discuss@lists.rabbitmq.com
> > > > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > > >
> > > >
> > > > _______________________________________________
> > > > rabbitmq-discuss mailing list
> > > > rabbitmq-discuss@lists.rabbitmq.com
> > > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > >
> > >
> > >
> > > --
> > > Alexis Richardson
> > > +44 20 7617 7339 (UK)
> > > +44 77 9865 2911 (cell)
> > > +1 650 206 2517 (US)
> > >
> >
>
>
> --
> Alexis Richardson
> +44 20 7617 7339 (UK)
> +44 77 9865 2911 (cell)
> +1 650 206 2517 (US)
>

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top
alexis
Posted: Wed Nov 07, 2007 5:58 pm Reply with quote
User Joined: 06 Sep 2007 Posts: 80 Location: London
Brian


On 11/6/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
>
> our system has a
> few unusual properties (more on that below)
>
> > > - each worker needs a private queue
> >
> > Is that strictly needed for your use case?
>
> Yes, each worker does need a private queue so workers can directly
> send messages to a given worker. But, it gets more complicated than
> that in a number of ways:
>
> - The master needs to be able to push messages to arbitrary sets of
> workers. Our general syntax (our master in written in python) looks
> like this:
>
> push(list_of_workers, msg)
>
> So you see things like this:
>
> push(getIDs(), msg) that does a broadcast or:
>
> push(range(10), msg) that pushes to workers 0,1,...10

OK, this is good. I assume you also have queues for work done
(results) and work in progress (lease-based?).


> - Obviously, when pushing messages to specific workers, you can't do
> dynamic load balancing. To support dynamic load balancing we have
> another single shared FIFO queue that sits on top of the private
> worker queues. We have a scheduler that pulls things off the shared
> queue and puts them on the private queues, based on how many messages
> there are in the private queues at any given moment.

Got it.


> I am not sure if
> rabbitmq/amqp will be able to handle this more complex queue logic as
> it would require logic *in* the rabbitmq server......
>
> So, the next question is ..... how easy it is to build custom rabbitmq
> servers that have custom queue logic. Basically, I need queues to be
> able to act as clients to other queues.

OK, there are at least three answers to this question Smile

1. You can do this today by running a scheduler that is a regular
client and acts both as a producer and a consumer. You may have
latency concerns if your units of work are fine grained (say <10ms
total work time per task). If you do not, then I would recommend
looking at this as a solution. If you do have latency concerns then I
would still start here as a prototype since you will be able to verify
that it scales. By the way, you can also improve latency by
colocating processes (or using a single process - see 3 below).

2. Future versions of AMQP will support 'chaining' of brokers and
federation better. Solutions of type 1 will, most likely, be able to
gracefully adopt the likely new features at low cost.

3. You can mod the broker by running 'in broker consumers' and 'in
broker producers'. This is a very interesting area for future work
but it would require a knowledge of Erlang. We would be happy to help
on list. However if this is your goal then I would still start at 1
(which also lets you try out multiple brokers and remain
interoperability-compliant).



> Currently we have implemented this system using twisted (an
> asynchronous networking framework in python). It works extremely
> well....but it doesn't scale to a large number of workers, which is
> why we are looking at erlang/amqp.

Interesting stuff - how many workers? (If that's not an impertinent
question ..)


> It might be the case that we need
> to implement our server at a lower level in erlang, but I really like
> rabbitmq/amqp.

Thank-you very much. I hope we can help you avoid having to care
about the lower levels.


> > I would certainly do this if you want to 'push' work. In effect you
> > are making the registry explicit and this would be very useful for
> > other implementation patterns, eg shared services 'fed' by queues (a
> > bit like an 'ESB').
>
> OK, it is good to know that I am on the right track.

Cool. Please let us know if you need help, eg with the deregistration methods.

alexis














>
> Thanks
>
> Brian
>
> >
> > > But, it would be nice to have more introspection capabilities.
> >
> > I wholly agree!
> >
> > alexis
> >
> >
> >
> >
> >
> > >
> > > Thanks
> > >
> > > Brian
> > >
> > > On 11/6/07, Alexis Richardson <alexis.richardson@cohesiveft.com> wrote:
> > > > Brian
> > > >
> > > > Not yet, I'm afraid Sad
> > > >
> > > > Management features are due in a future version of AMQP. How
> > > > important is this for you today?
> > > >
> > > > alexis
> > > >
> > > >
> > > > On 11/5/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
> > > > > On a related note, is there a way of querying the server to get the
> > > > > names of all the queues?
> > > > >
> > > > > Thanks
> > > > >
> > > > > Brian
> > > > >
> > > > > On 11/5/07, Matthias Radestock <matthias@lshift.net> wrote:
> > > > > > Matt,
> > > > > >
> > > > > >
> > > > > > matt wrote:
> > > > > > > Getting the message count only appears to get updated when you declare
> > > > > > > the queue. What if you wanted to poll the queue and see how many
> > > > > > > messages were on it every 15 seconds or so?
> > > > > >
> > > > > > queue.declare has a passive mode of operation where it doesn't actually
> > > > > > create a new queue. That's what I was referring to when I wrote that "in
> > > > > > combination with queue.declare's passive flag it allows you to determine
> > > > > > the length of existing queues".
> > > > > >
> > > > > >
> > > > > > Matthias.
> > > > > >
> > > > > > _______________________________________________
> > > > > > rabbitmq-discuss mailing list
> > > > > > rabbitmq-discuss@lists.rabbitmq.com
> > > > > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > rabbitmq-discuss mailing list
> > > > > rabbitmq-discuss@lists.rabbitmq.com
> > > > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > > >
> > > >
> > > >
> > > > --
> > > > Alexis Richardson
> > > > +44 20 7617 7339 (UK)
> > > > +44 77 9865 2911 (cell)
> > > > +1 650 206 2517 (US)
> > > >
> > >
> >
> >
> > --
> > Alexis Richardson
> > +44 20 7617 7339 (UK)
> > +44 77 9865 2911 (cell)
> > +1 650 206 2517 (US)
> >
>


--
Alexis Richardson
+44 20 7617 7339 (UK)
+44 77 9865 2911 (cell)
+1 650 206 2517 (US)

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top View user's profile Send private message Yahoo Messenger
Guest
Posted: Wed Nov 07, 2007 8:42 pm Reply with quote
Guest
Alexis,

On 11/7/07, Alexis Richardson <alexis.richardson@cohesiveft.com> wrote:
> Brian
>
>
> On 11/6/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
> >
> > our system has a
> > few unusual properties (more on that below)
> >
> > > > - each worker needs a private queue
> > >
> > > Is that strictly needed for your use case?
> >
> > Yes, each worker does need a private queue so workers can directly
> > send messages to a given worker. But, it gets more complicated than
> > that in a number of ways:
> >
> > - The master needs to be able to push messages to arbitrary sets of
> > workers. Our general syntax (our master in written in python) looks
> > like this:
> >
> > push(list_of_workers, msg)
> >
> > So you see things like this:
> >
> > push(getIDs(), msg) that does a broadcast or:
> >
> > push(range(10), msg) that pushes to workers 0,1,...10
>
> OK, this is good. I assume you also have queues for work done
> (results) and work in progress (lease-based?).

Yep, these queues would also have to be there.


> > So, the next question is ..... how easy it is to build custom rabbitmq
> > servers that have custom queue logic. Basically, I need queues to be
> > able to act as clients to other queues.
>
> OK, there are at least three answers to this question Smile
>
> 1. You can do this today by running a scheduler that is a regular
> client and acts both as a producer and a consumer. You may have
> latency concerns if your units of work are fine grained (say <10ms
> total work time per task). If you do not, then I would recommend
> looking at this as a solution. If you do have latency concerns then I
> would still start here as a prototype since you will be able to verify
> that it scales. By the way, you can also improve latency by
> colocating processes (or using a single process - see 3 below).

The other option that is related to this one would be to have each
worker pull from both its private queue and the shared one. I would
just need to make some decisions about which has priority in given
situations. Unfortunately, latency is a big issue for us - the lower
the better.

> 2. Future versions of AMQP will support 'chaining' of brokers and
> federation better. Solutions of type 1 will, most likely, be able to
> gracefully adopt the likely new features at low cost.

I see. That would be fantastic, as long as the chaining was
relatively flexible.

> 3. You can mod the broker by running 'in broker consumers' and 'in
> broker producers'. This is a very interesting area for future work
> but it would require a knowledge of Erlang. We would be happy to help
> on list. However if this is your goal then I would still start at 1
> (which also lets you try out multiple brokers and remain
> interoperability-compliant).

I don't mind learning Erlang if it is needed. My other option is to
just write a custom solution from scratch in Erlang. But rabbitmq
seems to have many of the pieces already, so I would really like to
use it if possible. Thanks for the offer of help - I will keep you
posted.

> > Currently we have implemented this system using twisted (an
> > asynchronous networking framework in python). It works extremely
> > well....but it doesn't scale to a large number of workers, which is
> > why we are looking at erlang/amqp.
>
> Interesting stuff - how many workers? (If that's not an impertinent
> question ..)

We are using this architecture for parallel computing in Python. Our
workers are basically Python interpreters exposed to the network. Our
"clients" are also python scripts that submit python code/objects to
the workers to be performed. As you can imagine I am looking at the
qpid python amqp client. We have run already with up to 256 workers
and we already see scaling problems. Our stuff works just fine on
supercomputers, so we would like to scale up to.....2**10, 2**11,
2**12, etc. We hit scaling problems in a number of ways:

1) # of file descriptors per process.
2) broadcasting/multicasting to all (or large subsets) of workers.
3) Latency - the latency determines the grain size of work, which we
would like to be as small as possible.
4) Large messages. In some cases, the messages can be very large
(files or large data sets). It is not uncommon for people to want to
distribute 100 MB messages to subsets of workers. Our current
solution generates a short lived temporary copy of the large objects
for each worker they are sent to. This means if we have 128 workers,
we use 128*msg_size memory for a short period of time. We know how to
fix this in Python, but it still doesn't solve the other scalability
problems. All that to say, any solution that we come up with must
also be able to handle large messages well. Does rabbitmq have any
streaming support?

>
> > It might be the case that we need
> > to implement our server at a lower level in erlang, but I really like
> > rabbitmq/amqp.
>
> Thank-you very much. I hope we can help you avoid having to care
> about the lower levels.

I would love to avoid the lower level stuff, but I also realize we
have a somewhat unusual set of design requirements (scalability,
low-latency && ability to handle large messages). In various
prototypes we even implemented custom protocols on top of TCP. Once
you start doing that Erlang becomes a "high-level" solution Smile

>
> > > I would certainly do this if you want to 'push' work. In effect you
> > > are making the registry explicit and this would be very useful for
> > > other implementation patterns, eg shared services 'fed' by queues (a
> > > bit like an 'ESB').
> >
> > OK, it is good to know that I am on the right track.
>
> Cool. Please let us know if you need help, eg with the deregistration methods.

Ahh yes, the deregistration methods could be subtle. The main concern
there is if a worker dies without notifying the server that they are
going to die. What facilities does rabbitmq/amqp have to handle this
type of stuff?

Thanks much for your help!

Brian


> alexis
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> >
> > Thanks
> >
> > Brian
> >
> > >
> > > > But, it would be nice to have more introspection capabilities.
> > >
> > > I wholly agree!
> > >
> > > alexis
> > >
> > >
> > >
> > >
> > >
> > > >
> > > > Thanks
> > > >
> > > > Brian
> > > >
> > > > On 11/6/07, Alexis Richardson <alexis.richardson@cohesiveft.com> wrote:
> > > > > Brian
> > > > >
> > > > > Not yet, I'm afraid Sad
> > > > >
> > > > > Management features are due in a future version of AMQP. How
> > > > > important is this for you today?
> > > > >
> > > > > alexis
> > > > >
> > > > >
> > > > > On 11/5/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
> > > > > > On a related note, is there a way of querying the server to get the
> > > > > > names of all the queues?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Brian
> > > > > >
> > > > > > On 11/5/07, Matthias Radestock <matthias@lshift.net> wrote:
> > > > > > > Matt,
> > > > > > >
> > > > > > >
> > > > > > > matt wrote:
> > > > > > > > Getting the message count only appears to get updated when you declare
> > > > > > > > the queue. What if you wanted to poll the queue and see how many
> > > > > > > > messages were on it every 15 seconds or so?
> > > > > > >
> > > > > > > queue.declare has a passive mode of operation where it doesn't actually
> > > > > > > create a new queue. That's what I was referring to when I wrote that "in
> > > > > > > combination with queue.declare's passive flag it allows you to determine
> > > > > > > the length of existing queues".
> > > > > > >
> > > > > > >
> > > > > > > Matthias.
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > rabbitmq-discuss mailing list
> > > > > > > rabbitmq-discuss@lists.rabbitmq.com
> > > > > > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > rabbitmq-discuss mailing list
> > > > > > rabbitmq-discuss@lists.rabbitmq.com
> > > > > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Alexis Richardson
> > > > > +44 20 7617 7339 (UK)
> > > > > +44 77 9865 2911 (cell)
> > > > > +1 650 206 2517 (US)
> > > > >
> > > >
> > >
> > >
> > > --
> > > Alexis Richardson
> > > +44 20 7617 7339 (UK)
> > > +44 77 9865 2911 (cell)
> > > +1 650 206 2517 (US)
> > >
> >
>
>
> --
> Alexis Richardson
> +44 20 7617 7339 (UK)
> +44 77 9865 2911 (cell)
> +1 650 206 2517 (US)
>

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top
0x6e6562
Posted: Wed Nov 07, 2007 11:21 pm Reply with quote
User Joined: 12 Jul 2007 Posts: 250
Brian,

On 7 Nov 2007, at 20:42, Brian Granger wrote:
>>>
>>> Yes, each worker does need a private queue so workers can directly
>>> send messages to a given worker. But, it gets more complicated than
>>> that in a number of ways:
>>>
>>> - The master needs to be able to push messages to arbitrary sets of
>>> workers. Our general syntax (our master in written in python) looks
>>> like this:
>>>
>>> push(list_of_workers, msg)
>>>
>>> So you see things like this:
>>>
>>> push(getIDs(), msg) that does a broadcast or:
>>>
>>> push(range(10), msg) that pushes to workers 0,1,...10

I've been following this thread but I don't think I've understood
everything completely.

Can you clarify what it is about your system that requires the
workflow to be producer driven and consumer driven? Are the
interactions stateful in any way? Or am I missing the point?

Ben

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top View user's profile Send private message
Guest
Posted: Wed Nov 07, 2007 11:59 pm Reply with quote
Guest
> Can you clarify what it is about your system that requires the
> workflow to be producer driven and consumer driven? Are the
> interactions stateful in any way? Or am I missing the point?

We have different types of queues and there needs to be somewhat
complex logic that moves (consumes) messages from certain shared
queues into (publish) other private ones. In pseudo code it looks
something like:

if private_queue = move_message():
msg = shared_queue.consume()
private_queue.publish(msg)

The important point is that the result of move_message() is very
dynamic. That is, it depends on things like 1) the number of private
queues 2) the number of messages in those queues currently 3) the
particular resources and characteristics of the consumers of the
private queues (how much RAM they have, available libraries, etc.)

Now you may wonder why we would want this. We have two types of
messages: 1) messages that need to be sent to a particular client and
2) messages that can be sent to any one client. In this second case,
we typically want to send the message to the least loaded client. Our
messages are packets of work that the clients (they are workers)
perform. We use the shared queue to perform dynamical load balancing
for work that can be performed by any qualified worker. But that work
still has to go through the private queues to preserve ordering.

Does this make a bit more sense?

Thanks

Brian

> Ben
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss@lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top
0x6e6562
Posted: Thu Nov 08, 2007 12:09 am Reply with quote
User Joined: 12 Jul 2007 Posts: 250
>
>> Now you may wonder why we would want this. We have two types of
>> messages: 1) messages that need to be sent to a particular client and
>> 2) messages that can be sent to any one client. In this second case,
>> we typically want to send the message to the least loaded client.
>> Our
>> messages are packets of work that the clients (they are workers)
>> perform. We use the shared queue to perform dynamical load balancing
>> for work that can be performed by any qualified worker. But that
>> work
>> still has to go through the private queues to preserve ordering.
>>
>> Does this make a bit more sense?
>
> Do your clients do anything contextually specific to the producer or
> is it just that the work needs to get load balanced dynamically
> according to the resources available to any given consumer?
>
> Ben


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top View user's profile Send private message

Display posts from previous:  

All times are GMT
View next topic
View previous topic
Page 1 of 2
Goto page 1, 2  Next
Erlang Forum - Trap Exit Forum Index  ~  RabbitMQ mailing list
This forum is locked: you cannot post, reply to, or edit topics.

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
Hosted by Erlang Solutions    |    Powered by phpBB and NoseBleed 1.14