| Author |
Message |
|
| Guest |
 Posted: Thu Dec 20, 2007 3:49 pm |
 |
|
|
Guest
|
I've been experimenting with RabbitMQ 1.2.0 on a FreeBSD 6.2 box,
installed using the Generic Unix instructions from
http://www.rabbitmq.com/install.html - (not doing any clustering), and
am seeing my memory and disk usage growing pretty steadily as I'm
starting to make use of persistent messages and durable queues. I
should also mention I'm using my alternate Python client library.
In my mnesia/rabbit directory, rabbit_persister.LOG has grown to 23Mb,
and "top" shows the SIZE of the process to be 113M, up from about 18M
when the daemon first starts.
I found some earlier messages in Google about rabbit memory usage, for
example:
http://www.nabble.com/beam-instances-and-growing-memory-usage-td13805341.html
and used
rabbit_amqqueue:stat_all().
To look at the queues, and just see
[{ok,{resource,<<"/">>,queue,<<"syslog_interpret">>},0,1}]
Which I take to mean just one queue named 'syslog_interpret' with zero
messages and 1 consumer?
Does having a large and growing rabbit_persister.LOG file indicate
messages are piling up somewhere? Are there any other Erlang commands
for examining the system to see where any logjams may be?
Would publishing persistent messages to a durable exchange with no
queues bound to it at all (yet) cause the messages to stick around?
Do persistent messages require some kind of acknowledgment I'm perhaps
missing out on?
Thanks for any suggestions.
Barry
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Fri Dec 21, 2007 8:28 pm |
|
|
|
Guest
|
Barry,
Barry Pederson wrote:
> I've been experimenting with RabbitMQ 1.2.0 on a FreeBSD 6.2 box,
> installed using the Generic Unix instructions from
> http://www.rabbitmq.com/install.html - (not doing any clustering), and
> am seeing my memory and disk usage growing pretty steadily as I'm
> starting to make use of persistent messages and durable queues. I
> should also mention I'm using my alternate Python client library.
>
> In my mnesia/rabbit directory, rabbit_persister.LOG has grown to 23Mb,
> and "top" shows the SIZE of the process to be 113M, up from about 18M
> when the daemon first starts.
How are you consuming messages from the queues? Delivered messages
(persistent or not) need to be acknowledged explicitly with basic.ack
unless the noAck flag for basic.get/basic.consume was set to true.
Also, what happens when you do one of the following:
- close and re-open the channel on which messages have been consumed
- close and re-open the connection on which messages have been consumed
- restart the server
and then start consuming messages from the same queues? Do you get
messages back that you have received before? If so that would be an
indication that you are not sending acknowledgments.
> I found some earlier messages in Google about rabbit memory usage, for
> example:
>
> http://www.nabble.com/beam-instances-and-growing-memory-usage-td13805341.html
>
> and used
>
> rabbit_amqqueue:stat_all().
>
> To look at the queues, and just see
>
> [{ok,{resource,<<"/">>,queue,<<"syslog_interpret">>},0,1}]
>
> Which I take to mean just one queue named 'syslog_interpret' with zero
> messages and 1 consumer?
Correct, though the message count does not include unacknowledged
delivered messages.
> Does having a large and growing rabbit_persister.LOG file indicate
> messages are piling up somewhere?
Yes.
> Are there any other Erlang commands for examining the system to see
> where any logjams may be?
Try
[process_info(P) || P <- processes(),
process_info(P, memory) > {memory, 100000}].
which will list the details of all processes with a memory consumption
greater than 100000 bytes.
> Would publishing persistent messages to a durable exchange with no
> queues bound to it at all (yet) cause the messages to stick around?
You might be on to something there. My own testing shows a surprising
growth in memory and the persister log in this scenario. I will do some
more digging.
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Dec 27, 2007 9:29 pm |
|
|
|
Guest
|
Matthias Radestock wrote:
> How are you consuming messages from the queues? Delivered messages
> (persistent or not) need to be acknowledged explicitly with basic.ack
> unless the noAck flag for basic.get/basic.consume was set to true.
Yes, I've been trying both basic.ack and the noAck flag - so that part
sounds OK.
> Also, what happens when you do one of the following:
> - close and re-open the channel on which messages have been consumed
> - close and re-open the connection on which messages have been consumed
> - restart the server
> and then start consuming messages from the same queues? Do you get
> messages back that you have received before? If so that would be an
> indication that you are not sending acknowledgments.
Haven't seen any repeated messages, so I guess that messages that were
consumed were acked. For the most part though, I don't have any queues
at all bound to the exchange to which the persistent messages are being
published.
After restarting the server just now, memory usage was back down, and
the rabbit_persister.LOG file was fairly small (62k), where the
rabbit_persister.LOG.previous is about 2.1mb
I'm fairly certain though that on another occasion a week or so ago, a
restarted server came back up with large memory usage and a big
rabbit_persister.LOG - I ended up wiping out the mnesia directory to get
things back to square one.
>> Are there any other Erlang commands for examining the system to see
>> where any logjams may be?
>
> Try
>
> [process_info(P) || P <- processes(),
> process_info(P, memory) > {memory, 100000}].
>
> which will list the details of all processes with a memory consumption
> greater than 100000 bytes.
On first startup, the above bit just returns []
trying again after things grow for a while results in
------------------
[[{registered_name,rabbit_gc_persist},
{current_function,{gen_server,loop,6}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,0},
{messages,[]},
{links,[<0.168.0>]},
{dictionary,[{'$ancestors',[<0.168.0>,rabbit_sup,<0.103.0>]},
{'$initial_call',{gen,init_it,
[gen_server,
<0.168.0>,
<0.168.0>,
{local,rabbit_gc_persist},
rabbit_gc_persist,
[],
[]]}}]},
{trap_exit,false},
{error_handler,error_handler},
{priority,normal},
{group_leader,<0.102.0>},
{heap_size,121393},
{stack_size,12},
{reductions,74294},
{garbage_collection,[{fullsweep_after,65535}]}]]
-------------------
And somewhat later on
--------------------
[[{registered_name,rabbit_gc_persist},
{current_function,{gen_server,loop,6}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,0},
{messages,[]},
{links,[<0.168.0>]},
{dictionary,[{'$ancestors',[<0.168.0>,rabbit_sup,<0.103.0>]},
{'$initial_call',{gen,init_it,
[gen_server,
<0.168.0>,
<0.168.0>,
{local,rabbit_gc_persist},
rabbit_gc_persist,
[],
[]]}}]},
{trap_exit,false},
{error_handler,error_handler},
{priority,normal},
{group_leader,<0.102.0>},
{heap_size,514229},
{stack_size,12},
{reductions,118305},
{garbage_collection,[{fullsweep_after,65535}]}]]
----------------------
Looks like "heap_size" and "reductions" are growing.
>> Would publishing persistent messages to a durable exchange with no
>> queues bound to it at all (yet) cause the messages to stick around?
>
> You might be on to something there. My own testing shows a surprising
> growth in memory and the persister log in this scenario. I will do some
> more digging.
Ah good, I'm glad it's not just me. Not knowing much Erlang I'm at a
bit of a loss when it comes to troubleshooting this.
Barry
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Fri Dec 28, 2007 2:21 pm |
|
|
|
Guest
|
Barry,
Barry Pederson wrote:
> After restarting the server just now, memory usage was back down, and
> the rabbit_persister.LOG file was fairly small (62k), where the
> rabbit_persister.LOG.previous is about 2.1mb
Good. That's what I expected see.
> I'm fairly certain though that on another occasion a week or so ago, a
> restarted server came back up with large memory usage and a big
> rabbit_persister.LOG - I ended up wiping out the mnesia directory to get
> things back to square one.
When RabbitMQ restarts it goes through the persister log and figures out
which messages need (re)delivering. When the persister log is large this
can take some time and can also eat up a lot of memory. Once that's done
though, and assuming there aren't lots of undelivered messages, the
memory usage should drop significantly and a new, shorter persister log
should be written.
>> Try
>>
>> [process_info(P) || P <- processes(),
>> process_info(P, memory) > {memory, 100000}].
>>
>> which will list the details of all processes with a memory consumption
>> greater than 100000 bytes.
> [...]
> Looks like "heap_size" and "reductions" are growing.
...of the rabbit_gc_persist process, which keeps track of undelivered
persistent messages. I get the same results on my system, so I I am
pretty confident that I have reproduced the problem you are seeing.
>>> Would publishing persistent messages to a durable exchange with no
>>> queues bound to it at all (yet) cause the messages to stick around?
>>
>> You might be on to something there. My own testing shows a surprising
>> growth in memory and the persister log in this scenario. I will do
>> some more digging.
>
> Ah good, I'm glad it's not just me.
The problem turns out to be that undeliverable persistent messages stick
around in the rabbit_gc_persist process memory and the persister log
until the node restart.
The fix for this will be in the next RabbitMQ release. If you don't want
to wait that long, and are feeling adventurous, you can try the most
recent snapshot from http://dev.rabbitmq.com/snapshots/rabbitmq/
Thanks for reporting and helping me track down this problem.
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon Dec 31, 2007 3:28 am |
|
|
|
Guest
|
Matthias Radestock wrote:
>
> The problem turns out to be that undeliverable persistent messages stick
> around in the rabbit_gc_persist process memory and the persister log
> until the node restart.
>
> The fix for this will be in the next RabbitMQ release. If you don't want
> to wait that long, and are feeling adventurous, you can try the most
> recent snapshot from http://dev.rabbitmq.com/snapshots/rabbitmq/
I've been running the rabbitmq-200712231455.tar.gz snapshot for the last
couple days, and the memory usage has barely budged at all during that
time - so I think it's good now.
Thanks for working on that.
Barry
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist |
|
|
| Back to top |
|
| wuji |
| Posted: Wed Aug 29, 2012 6:31 am |
|
|
|
User
Joined: 10 Aug 2012
Posts: 654
|
first ordered its pilots to stop wearing the vests last last [h4]cheap Ralph Lauren[/h4] last month, but Air Force spokesperson Lt. Col. Tadd Scholtis
ABC News at the time that while the vests were were [h1]imitation designer *beep*[/h1] were believed to have contributed to the problem, they were
believed to be the root cause of the prior incidents."When incidents."When knockoff designer *beep* incidents."When asked by a reporter if the new solution could
account for the at least five instances in which the the cheap polo shirts the Air Force said ground crews working on the F-22s
their own hypoxia-like symptoms, Little said he "did not have have knockoff designer *beep* have specifics" on those incidents.Still, Gen. Charles Lyon, the head
the team investigating the F-22 problem, made his case in in cheap authentic jordans in the Pentagon against the so-called G-suit and its valve
t |
|
|
| Back to top |
 
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
