| Author |
Message |
< RabbitMQ mailing list ~ General approaches for tracking unreclaimed memory |
| gar1t |
 Posted: Mon Oct 26, 2009 6:42 pm |
 |
|
|
User
Joined: 11 Aug 2009
Posts: 55
|
Apart from using rabbitmqctrl to check some of the more obvious
sources for lost memory (e.g. connections, queues, etc.) are there
other techniques for tracking down where the broker might be
allocating resources that aren't cleaned up?
As this is Erlang, I'd assume one point of focus is process creation
and using whatever techniques are used in Erlang -- e.g. tracing, etc.
One thing I see routinely in the rabbit logs that I'd like to cleanup is:
exception on TCP connection <0.31875.6> from XXX.XXX.XXX.XXX:XXXXX
connection_closed_abruptly
However, using pidstat, I see memory allocation in the beam process
occurring independently of those log messages.
Garrett
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
 
|
| Guest |
| Posted: Mon Oct 26, 2009 10:31 pm |
|
|
|
Guest
|
Garrett,
Garrett Smith wrote:
> Apart from using rabbitmqctl to check some of the more obvious
> sources for lost memory (e.g. connections, queues, etc.) are there
> other techniques for tracking down where the broker might be
> allocating resources that aren't cleaned up?
The output of the various rabbitmqctl list_* commands is definitely the
first port of call. Do the "row counts" of these increase? Or do the
message counts in the queues increase?
> As this is Erlang, I'd assume one point of focus is process creation
> and using whatever techniques are used in Erlang -- e.g. tracing, etc.
See the recent thread at
http://www.nabble.com/Re%3A-RabbitMQ-crashes-hard-when-it-runs-out-of-memory-td26014566.html
for some further avenues of investigation.
> One thing I see routinely in the rabbit logs that I'd like to cleanup is:
>
> exception on TCP connection <0.31875.6> from XXX.XXX.XXX.XXX:XXXXX
> connection_closed_abruptly
That is caused by clients disconnecting without following the proper
AMQP shutdown protocol, e.g. they simply close the socket instead of
sending a "connection.close" command and waiting for "connection.close_ok".
With correctly implemented client libraries and applications you should
only see that in the event of an application crash or network problems.
However, there are no ill effects from this. In particular, no resource
leakage results from abruptly closed connections.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| gar1t |
| Posted: Tue Oct 27, 2009 11:43 pm |
|
|
|
User
Joined: 11 Aug 2009
Posts: 55
|
(repost to list)
On Mon, Oct 26, 2009 at 5:30 PM, Matthias Radestock <matthias@lshift.net> wrote:
> Garrett Smith wrote:
>>
>> Apart from using rabbitmqctl to check some of the more obvious
>> sources for lost memory (e.g. connections, queues, etc.) are there
>> other techniques for tracking down where the broker might be
>> allocating resources that aren't cleaned up?
>
> The output of the various rabbitmqctl list_* commands is definitely the
> first port of call. Do the "row counts" of these increase? Or do the message
> counts in the queues increase?
Hah, I missed something obvious here. Thanks for reminding me to start
at the beginning 
So, here's the problem, which everyone reasonably faces: when a
consumer goes away, their queues pile up, which is basically a "leak"
at the application/system level (not with rabbit).
I'm being a little lazy here by not looking into the docs or the AMQP
spec, but is there a way to guard against an eventual crash in this
case (I believe this has manifested itself for me once by running out
of Erlang processes, but memory is certainly an issue as well)?
Having worked with qpid a bit, their broker will respect the
time-to-live/expires info associated with a message. I know, at least
in 1.6, rabbit didn't do that. That would be one solution.
A fallback would be a limit to the number of messages a queue could handle.
Short of that, I can run a daemon to routinely check rabbit for queues
that pile up, but that's a moving part I'd prefer to avoid if
possible.
Garrett
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Wed Oct 28, 2009 1:01 am |
|
|
|
Guest
|
Garrett,
Garrett Smith wrote:
> Hah, I missed something obvious here. Thanks for reminding me to start
> at the beginning
>
> So, here's the problem, which everyone reasonably faces: when a
> consumer goes away, their queues pile up, which is basically a "leak"
> at the application/system level (not with rabbit).
>
> I'm being a little lazy here by not looking into the docs or the AMQP
> spec, but is there a way to guard against an eventual crash in this
> case (I believe this has manifested itself for me once by running out
> of Erlang processes, but memory is certainly an issue as well)?
I'd definitely recommend reading the spec. If you did then you'd know
about the exclusive and auto-delete flags to queue.declare  Queues
declared with the former get automatically deleted when their creating
connection disappears, and the latter when their last consumer disappears.
HTH,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| gar1t |
| Posted: Wed Oct 28, 2009 3:50 pm |
|
|
|
User
Joined: 11 Aug 2009
Posts: 55
|
On Tue, Oct 27, 2009 at 8:00 PM, Matthias Radestock <matthias@lshift.net> wrote:
>> So, here's the problem, which everyone reasonably faces: when a
>> consumer goes away, their queues pile up, which is basically a "leak"
>> at the application/system level (not with rabbit).
>>
>> I'm being a little lazy here by not looking into the docs or the AMQP
>> spec, but is there a way to guard against an eventual crash in this
>> case (I believe this has manifested itself for me once by running out
>> of Erlang processes, but memory is certainly an issue as well)?
>
> I'd definitely recommend reading the spec. If you did then you'd know about
> the exclusive and auto-delete flags to queue.declare Queues declared with
> the former get automatically deleted when their creating connection
> disappears, and the latter when their last consumer disappears.
Yup, I'm familiar with those flags (and actually have read the spec, a
few times even
I need both non-exclusive/non-auto-delete so that the queue is
preserved when either the creating connection or the last consumer
goes away. Various agents come and go (restarting is one example, but
there are other cases). Not to preach to the choir, but one of the
benefits of async messaging is you can insulate from dropped
connections.
It sounds like I'm left with the option of running a cleanup/check
process occasionally to ensure that we don't run into queues piling
up. No other options, given exclusive=false and autodelete=false?
I know that the expires/TTL is ignored in rabbit, and is perhaps not
entirely fleshed out in the 0.8 spec, but it seems an important
feature wrt resource resource management.
Garrett
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Wed Oct 28, 2009 4:00 pm |
|
|
|
Guest
|
On Wed, Oct 28, 2009 at 3:48 PM, Garrett Smith <g@rre.tt (g@rre.tt)> wrote:
Quote: Yup, I'm familiar with those flags (and actually have read the spec, a
few times even
I need both non-exclusive/non-auto-delete so that the queue is
preserved when either the creating connection or the last consumer
goes away. Various agents come and go (restarting is one example, but
there are other cases). Not to preach to the choir, but one of the
benefits of async messaging is you can insulate from dropped
connections.
It sounds like I'm left with the option of running a cleanup/check
process occasionally to ensure that we don't run into queues piling
up. No other options, given exclusive=false and autodelete=false?
I know that the expires/TTL is ignored in rabbit, and is perhaps not
entirely fleshed out in the 0.8 spec, but it seems an important
feature wrt resource resource management.
Unless you've also got persitant messages, why not have auto-delete queues that every producer and consumer just declares up front? If no-one's around using that queue, it goes away, once someone needs it, the declare makes it appear.
Or am I missing something?
Robby
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Wed Oct 28, 2009 4:01 pm |
|
|
|
Guest
|
Garrett,
Garrett Smith wrote:
> I need both non-exclusive/non-auto-delete so that the queue is
> preserved when either the creating connection or the last consumer
> goes away.
Why do you want the queue to be preserved when you do not want any of
its messages preserved?
> I know that the expires/TTL is ignored in rabbit, and is perhaps not
> entirely fleshed out in the 0.8 spec, but it seems an important
> feature wrt resource resource management.
Discarding messages from a queue based on some expiry property is but
one of many different conceivable ways in which one may wish to manage
resources. Yes, it happens to be one that is half-defined by the spec,
but I doubt it's the most useful one. We will get round to implementing
it eventually, but we are considering others too, such as limiting
queues by msg count or size, eviction policies based on message
properties other than expiry, etc.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| gar1t |
| Posted: Wed Oct 28, 2009 5:22 pm |
|
|
|
User
Joined: 11 Aug 2009
Posts: 55
|
On Wed, Oct 28, 2009 at 11:01 AM, Matthias Radestock
<matthias@lshift.net> wrote:
> Garrett,
>
> Garrett Smith wrote:
>> I need both non-exclusive/non-auto-delete so that the queue is
>> preserved when either the creating connection or the last consumer
>> goes away.
>
> Why do you want the queue to be preserved when you do not want any of
> its messages preserved?
Let's say I restart a process and terminate a connection on an
exclusive queue. I'm interested in the messages -- I'm gonna come back
online after the restart. I don't want to delete the queue. Trust me
Let's say, on the other hand, that I decommission the queue owner and
(admittedly sloppily) don't delete the queue. Messages pile up and I
run out of memory/processes.
I understand this is entirely application specific. It's like running
out of disk space when you keep writing to a database. I understand
that and will deal with it easily enough, though I do think that
protections against resource exhaustion are important.
>> I know that the expires/TTL is ignored in rabbit, and is perhaps not
>> entirely fleshed out in the 0.8 spec, but it seems an important
>> feature wrt resource resource management.
>
> Discarding messages from a queue based on some expiry property is but
> one of many different conceivable ways in which one may wish to manage
> resources. Yes, it happens to be one that is half-defined by the spec,
> but I doubt it's the most useful one. We will get round to implementing
> it eventually, but we are considering others too, such as limiting
> queues by msg count or size, eviction policies based on message
> properties other than expiry, etc.
In my experience (again, working with qpid), time-to-live/expires is a
flexible and effective way to deal with this problem. Not trying to
spark a debate, but it's not obvious to me how that approach is
inferior to the others you listed. I understand if this is topic for
future consideration though.
Garrett
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Wed Oct 28, 2009 7:12 pm |
|
|
|
Guest
|
Garrett,
Garrett Smith wrote:
> Let's say I restart a process and terminate a connection on an
> exclusive queue. I'm interested in the messages -- I'm gonna come back
> online after the restart. I don't want to delete the queue. Trust me
>
>
> Let's say, on the other hand, that I decommission the queue owner and
> (admittedly sloppily) don't delete the queue. Messages pile up and I
> run out of memory/processes.
>
> I understand this is entirely application specific. It's like running
> out of disk space when you keep writing to a database. I understand
> that and will deal with it easily enough, though I do think that
> protections against resource exhaustion are important.
Understood. However, message expiry is entirely in the control of the
client, so it's not something a server can rely on for keeping resources
bounded unless it can trust the clients and clients have been correctly
coded to supply a reasonable expiry time.
Also, you'd still want a mechanism to delete the queue in your scenario,
because otherwise the system will waste resources on publishing and then
deleting messages from it.
It seems like what you are really after is a server-controlled "time to
live when unused" setting for a queue, i.e. the server would
automatically delete the queue after it hasn't been used for a while.
And you'd probably want to put limits on queue sizes too. etc, etc.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
