Erlang/OTP Forums
Author Message

<  RabbitMQ mailing list  ~  Scalable multithreaded consumer
Guest
Posted: Thu Jan 17, 2008 11:57 pm Reply with quote
Guest
Hi everyone,
Back to top
Guest
Posted: Fri Jan 18, 2008 1:02 am Reply with quote
Guest
Hi everyone,
Back to top
Guest
Posted: Mon Jan 21, 2008 12:00 pm Reply with quote
Guest
Karel,

Karel Alfonso wrote:

> I
Back to top
tonyg
Posted: Mon Jan 21, 2008 12:33 pm Reply with quote
User Joined: 07 Nov 2006 Posts: 199
Hi Karel,

Karel Alfonso wrote:
> I haven
Back to top View user's profile Send private message MSN Messenger
Guest
Posted: Mon Jan 21, 2008 11:36 pm Reply with quote
Guest
Hi everyone,

Thanks so much for your reply.

I'm working on a system that requires to download via POP3 every X amount
of time, the email of potentially 10 000 users (each user can have their
POP3 account on the internet).

There is a Java component in the system that is activated via a scheduled
task that sends 10 000 events (messages) to start the mail download via POP3
for each user.

In Java we have the problems with the scalability using multithreading,
that is, I wished I could have 10 000 threads running at the same time on
the system. I'm also limited by the number of File descriptors the OS will
allow me to open at once and the number of threads per process, etc.

I'm considering Erlang because of the scalability and concurrency
features. I know I can have thousands of Erlang process connected to POP3
accounts on the internet, provided I have enough file descriptors, without
doing many fancy stuff.

When I came across RabbitMQ, I thought that was a good way to interface
Java with Erlang ... But first I wanted to know if I could process or
consume the messages on the Java side with an efficient thread pool, then in
a second phase, move the message consumption to Erlang.

I hope that gives a better explanation of what I'm after. It might be the
case the RabbitMQ is not the right technology for what I'm trying to do. I
appreciate your time and suggestions

Regards,

Karel Alfonso
Tech Lead
Inventri
Level 4, 365 Little Collin St.
Melbourne, VIC 3000

Phone: 61 3 86760374
Mobile: 0402663985
-----Original Message-----
From: Matthias Radestock [mailto:matthias@lshift.net]
Sent: Monday, 21 January 2008 7:00 AM
To: Karel Alfonso
Cc: rabbitmq-discuss@lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Scalable multithreaded consumer

Karel,

Karel Alfonso wrote:

> I'm planning to use RabbitMQ in a system that will process events in
> the order of the thousands. I haven't found in RabbitMQ a mechanism to
> process concurrently the messages received. Is there any suggestion or
> guideline I could follow?

It is certainly possible to perform high-volume concurrent event
processing with RabbitMQ, but coming up with the best setup requires a
better understanding of your specific requirements.

Can you elaborate on what you are trying to accomplish? How many event
producers are there? How many event consumers are there? What routing do
you need between them? What delivery guarantees do you require? What
ordering guarantees do you require? What AMQP client libraries are you
using (e.g. RabbitMQ's Java client, the Erlang client etc)?


Regards,

Matthias.


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top
tonyg
Posted: Wed Jan 30, 2008 1:31 pm Reply with quote
User Joined: 07 Nov 2006 Posts: 199
Hi Karel,

Karel Alfonso wrote:
> There is a Java component in the system that is activated via a scheduled
> task that sends 10 000 events (messages) to start the mail download via POP3
> for each user.

Each message is basically "please poll the POP3 mailbox for user XYZ", then?

> In Java [...] I wished I could have 10 000 threads running at the same time

Oh yes, wouldn't that be nice Smile

> I'm considering Erlang because of the scalability and concurrency
> features. I know I can have thousands of Erlang process connected to POP3
> accounts on the internet, provided I have enough file descriptors, without
> doing many fancy stuff.

Absolutely. Because of the file-descriptor limit, you'd probably still
want some kind of queueing system in place, though. (Even if it was just
an ad-hoc one on the erlang side.)

> When I came across RabbitMQ, I thought that was a good way to interface
> Java with Erlang ... But first I wanted to know if I could process or
> consume the messages on the Java side with an efficient thread pool, then in
> a second phase, move the message consumption to Erlang.

Yes, that is a sensible thing to do. A queue containing all the
poll-requests could have several consumers, in one or several JVMs,
consuming from it, making a simple load-balancing worker pool.

How are you getting on? If you're using RabbitMQ, and have any questions
or comments, we'd love to hear about it.

Thanks,
Tony
--
[][][] Tony Garnock-Jones | Mob: +44 (0)7905 974 211
[][] LShift Ltd | Tel: +44 (0)20 7729 7060
[] [] http://www.lshift.net/ | Email: tonyg@lshift.net

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top View user's profile Send private message MSN Messenger
Guest
Posted: Wed Jan 30, 2008 1:46 pm Reply with quote
Guest
Tony Garnock-Jones wrote:
> Each message is basically "please poll the POP3 mailbox for user XYZ", then?
>
>> In Java [...] I wished I could have 10 000 threads running at the same time
>
> Oh yes, wouldn't that be nice Smile

Not to steal any Erlang steam here but those of you who need at least some
kind of Actor-ish lightweight behaviour in their Java should check out the
ForkJoin work going on in the jsr166y project
(http://gee.cs.oswego.edu/dl/concurrency-interest/). Tim Peierls set up a
Wiki at http://artisans-serverintellect-com.si-eioswww6.com/default.asp?W1
which provides some overview and example code.

Holger
(ZOMG! I can't believe I'm advocating Java..)

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top
Guest
Posted: Wed Jan 30, 2008 2:27 pm Reply with quote
Guest
On 1/30/08, Holger Hoffst
Back to top
Guest
Posted: Wed Jan 30, 2008 4:02 pm Reply with quote
Guest
On Wed, Jan 30, 2008 at 06:26:29AM -0800, David Pollak wrote:
> Or you could look at Scala (http://scala-lang.org ) which has Erlang-style
> Actors built right in.

Yes, are they preempting yet?

Matthew

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top
Guest
Posted: Wed Jan 30, 2008 9:56 pm Reply with quote
Guest
Thanks Tony for your reply.

I have to say I decided to use ActiveMQ with clustered queue consumers where
each of them has a thread pool managed by Jencks, an MDB solution that
integrates nicely with ActiveMQ.

I would have loved to have RabbitMQ and Erlang consumers but I have to wait
till we have Erlang client implemented in RabbitMQ, then we can move away
from consuming messages in Java.

In my solution each message means "poll the pop3 account of user XYZ for new
mails", that's right. So this is a solution where I need thousands of
clients/consumers to poll a POP3 account for a specific user.

I looked into an Erlang solution without RabbitMQ, but I don't think
enterprise APIs in Erlang are as mature and proven as those for Java. For
example, the POP3 client in Erlang, epop, dates back since 1998, and we
don't have yet a JavaMail-like package in Erlang, so, things become more
difficult.

Scala offeres the Acto model as well, but using the threading capability
offered by Java.

For the moment I'll use ActiveMQ with clustered consumers, in the future
I'll move the implementation to use RabbitMQ with Erlang consumers.

Thanks for your support,

Cheers,

Karel


-----Original Message-----
From: Tony Garnock-Jones [mailto:tonyg@lshift.net]
Sent: Thursday, 31 January 2008 12:31 AM
To: Karel Alfonso
Cc: rabbitmq-discuss@lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Scalable multithreaded consumer

Hi Karel,

Karel Alfonso wrote:
> There is a Java component in the system that is activated via a
scheduled
> task that sends 10 000 events (messages) to start the mail download via
POP3
> for each user.

Each message is basically "please poll the POP3 mailbox for user XYZ", then?

> In Java [...] I wished I could have 10 000 threads running at the same
time

Oh yes, wouldn't that be nice Smile

> I'm considering Erlang because of the scalability and concurrency
> features. I know I can have thousands of Erlang process connected to POP3
> accounts on the internet, provided I have enough file descriptors, without
> doing many fancy stuff.

Absolutely. Because of the file-descriptor limit, you'd probably still
want some kind of queueing system in place, though. (Even if it was just
an ad-hoc one on the erlang side.)

> When I came across RabbitMQ, I thought that was a good way to interface
> Java with Erlang ... But first I wanted to know if I could process or
> consume the messages on the Java side with an efficient thread pool, then
in
> a second phase, move the message consumption to Erlang.

Yes, that is a sensible thing to do. A queue containing all the
poll-requests could have several consumers, in one or several JVMs,
consuming from it, making a simple load-balancing worker pool.

How are you getting on? If you're using RabbitMQ, and have any questions
or comments, we'd love to hear about it.

Thanks,
Tony
--
[][][] Tony Garnock-Jones | Mob: +44 (0)7905 974 211
[][] LShift Ltd | Tel: +44 (0)20 7729 7060
[] [] http://www.lshift.net/ | Email: tonyg@lshift.net


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top
0x6e6562
Posted: Thu Jan 31, 2008 6:37 am Reply with quote
User Joined: 12 Jul 2007 Posts: 250
Karel,

On 30 Jan 2008, at 21:55, Karel Alfonso wrote:
>
> I would have loved to have RabbitMQ and Erlang consumers but I have
> to wait
> till we have Erlang client implemented in RabbitMQ, then we can move
> away
> from consuming messages in Java.

Were you looking for an AMQP client for Erlang?

Ben

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top View user's profile Send private message
alexis
Posted: Thu Jan 31, 2008 2:57 pm Reply with quote
User Joined: 06 Sep 2007 Posts: 80 Location: London
Karel,

If you look at the bottom of the downloads page
(http://www.rabbitmq.com/download.html) you will see an Erlang client
which works with RabbitMQ. Do please give that a go, if that is
indeed what you needed. I hope you will then find RabbitMQ's
clustering features to be adequate with respect to your needs.

It would be great to hear more about how you are getting on with your
use case involving both email and messaging. We have seen similar
cases, unfortunately still 'off list' which we hope to be able to
share with the community.

alexis


On Jan 31, 2008 12:36 AM, Ben Hood <0x6e6562@gmail.com> wrote:
> Karel,
>
> On 30 Jan 2008, at 21:55, Karel Alfonso wrote:
> >
> > I would have loved to have RabbitMQ and Erlang consumers but I have
> > to wait
> > till we have Erlang client implemented in RabbitMQ, then we can move
> > away
> > from consuming messages in Java.
>
> Were you looking for an AMQP client for Erlang?
>
> Ben
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss@lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>



--
Alexis Richardson
+44 20 7617 7339 (UK)
+44 77 9865 2911 (cell)
+1 650 206 2517 (US)

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top View user's profile Send private message Yahoo Messenger
tonyg
Posted: Thu Jan 31, 2008 3:12 pm Reply with quote
User Joined: 07 Nov 2006 Posts: 199
Hi Karel,

Karel Alfonso wrote:
> In my solution each message means "poll the pop3 account of user XYZ for new
> mails", that's right. So this is a solution where I need thousands of
> clients/consumers to poll a POP3 account for a specific user.

RabbitMQ's Java library and Erlang broker should have no trouble with
that kind of solution - unless there was some showstopper you've found
that I haven't thought of?

Can you tell us more about the surrounding environment - for instance,
is Spring an important part of your total application?

Regards,
Tony
--
[][][] Tony Garnock-Jones | Mob: +44 (0)7905 974 211
[][] LShift Ltd | Tel: +44 (0)20 7729 7060
[] [] http://www.lshift.net/ | Email: tonyg@lshift.net

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top View user's profile Send private message MSN Messenger
Guest
Posted: Thu Jan 31, 2008 4:00 pm Reply with quote
Guest
And FWIW, you might want to look at Scala.
Back to top
Guest
Posted: Fri Feb 01, 2008 4:02 am Reply with quote
Guest
Hi everyone,

First, thanks so much for the support and quick replies.

The problem I have at hand is to "periodically poll the POP3 accounts of 5
thousand users or more and download the mails that match certain criteria".

As the user base will increase in the future, I'm thinking of having a
clustered solution for scalability purpose.

Right now I'm using ActiveMQ with clustered queue consumers. The master
distributes the messages in a round-robin manner to each of the consumers.
The consumers handle the message through Jencks, an MDB solution which has a
thread pool. To scale this architecture I have to add more consumer nodes
(with Jenck).

I thought about Erlang cause in only one box I can have thousands of
processes running, provided that I have enough file descriptors in the OS.
Then if on top of that I can have a clustered solution with RabbitMQ, that
would be the perfect solution for me.

Using RabbitMQ's Java libraries to consume the messages using a thread
pool does not offer any advantage to me, given that that is the solution I
have at the moment.

The other question might be out of the subject of this list, but once you
have a RabbitMQ cluster, which technologies would you suggest to share the
file system amongst the cluster?

Thanks in advance for all your help.

Regards,

Karel Alfonso
Tech Lead
Inventri
Level 4, 365 Little Collin St.
Melbourne, VIC 3000

Phone: 61 3 86760374
Mobile: 0402663985

-----Original Message-----
From: Tony Garnock-Jones [mailto:tonyg@lshift.net]
Sent: Friday, 1 February 2008 2:12 AM
To: Karel Alfonso
Cc: rabbitmq-discuss@lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Scalable multithreaded consumer

Hi Karel,

Karel Alfonso wrote:
> In my solution each message means "poll the pop3 account of user XYZ for
new
> mails", that's right. So this is a solution where I need thousands of
> clients/consumers to poll a POP3 account for a specific user.

RabbitMQ's Java library and Erlang broker should have no trouble with
that kind of solution - unless there was some showstopper you've found
that I haven't thought of?

Can you tell us more about the surrounding environment - for instance,
is Spring an important part of your total application?

Regards,
Tony
--
[][][] Tony Garnock-Jones | Mob: +44 (0)7905 974 211
[][] LShift Ltd | Tel: +44 (0)20 7729 7060
[] [] http://www.lshift.net/ | Email: tonyg@lshift.net


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist
Back to top

Display posts from previous:  

All times are GMT
View next topic
View previous topic
Page 1 of 1
Erlang Forum - Trap Exit Forum Index  ~  RabbitMQ mailing list
This forum is locked: you cannot post, reply to, or edit topics.

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
Hosted by Erlang Solutions    |    Powered by phpBB and NoseBleed 1.14