| Author |
Message |
|
| Guest |
 Posted: Wed Jul 28, 2010 8:31 pm |
 |
|
|
Guest
|
With R13B04 running on Montavista Linux, I've seen a few cases
recently where the Erlang VM simply exits without any log messages,
crash dumps, or coredumps. It appears to happen only after days of
running under load, making it hard to reproduce and investigate. It
may be related to memory consumption, but I'm not sure. External
programs like heart and memsup simply report "Erlang has closed,"
which according to the source code means they each got a return value
of 0 from read() on their connection to the VM, which in turn would
seem to indicate that the VM side of the connection was simply closed.
This would all seem to indicate that the VM was either killed by
another process or that it called exit() itself. The Linux "OOM
killer" is disabled on this system, and I don't know of any other
process that would be killing the VM. There are no alarms in the logs
about hitting memory high watermarks or anything like that, and we
aren't using any options to change allocators or anything like that.
Anybody ever seen anything like this?
I've found a few places in the VM C source code where exit() is called
without logging anything. Some of these are normal exits, like when
you exit an Erlang shell, where no logging is needed. But others seem
to be error conditions, and there should be logging for those. I think
I'll probably have to patch my system to add logging to those cases to
try to track down this problem -- is there still time to get a patch
like this into R14B? If this issue is memory-related, I suppose it's
possible that a sudden increase in memory consumption could cause the
VM to exit between alarm checks, explaining why things like memsup
don't seem to notice, so it would seem to be fairly critical that
something is logged by the VM itself for such cases.
thanks,
--steve
________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
Post received from mailinglist |
|
|
| Back to top |
|
| vladdu |
| Posted: Wed Jul 28, 2010 8:44 pm |
|
|
|
User
Joined: 28 Feb 2005
Posts: 397
Location: Gothenburg, Sweden
|
On Wed, Jul 28, 2010 at 22:29, Steve Vinoski <vinoski@ieee.org> wrote:
> With R13B04 running on Montavista Linux, I've seen a few cases
> recently where the Erlang VM simply exits without any log messages,
> crash dumps, or coredumps.
Hi,
I got something similar a while ago, with SuSe, but the lack of a
coredump was caused by some weird setting that I don't remember right
now (it worked for some programs but not all, notably not erlang). Are
you sure that core dumps are enabled properly on your system? You can
try to kill the VM manually, to see if there is a core dump or not.
(BTW, in my case it was a memory problem).
regards,
Vlad
________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
Post received from mailinglist |
|
|
| Back to top |
 
|
| Guest |
| Posted: Wed Jul 28, 2010 8:51 pm |
|
|
|
Guest
|
On Wed, Jul 28, 2010 at 4:42 PM, Vlad Dumitrescu <vladdu55@gmail.com> wrote:
> On Wed, Jul 28, 2010 at 22:29, Steve Vinoski <vinoski@ieee.org> wrote:
>> With R13B04 running on Montavista Linux, I've seen a few cases
>> recently where the Erlang VM simply exits without any log messages,
>> crash dumps, or coredumps.
>
> Hi,
>
> I got something similar a while ago, with SuSe, but the lack of a
> coredump was caused by some weird setting that I don't remember right
> now (it worked for some programs but not all, notably not erlang). Are
> you sure that core dumps are enabled properly on your system? You can
> try to kill the VM manually, to see if there is a core dump or not.
> (BTW, in my case it was a memory problem).
Hi Vlad, yes, core dumping is definitely enabled and working properly.
thanks,
--steve
________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Wed Jul 28, 2010 10:56 pm |
|
|
|
Guest
|
On Wed, Jul 28, 2010 at 1:29 PM, Steve Vinoski <vinoski@ieee.org> wrote:
> With R13B04 running on Montavista Linux, I've seen a few cases
> recently where the Erlang VM simply exits without any log messages,
> crash dumps, or coredumps. It appears to happen only after days of
> running under load, making it hard to reproduce and investigate.
We've seen something similar once when we hit the atom table limit.
The fact that the exit occurs after some time of happy running might
be indicative of a hard limit being reached, and as far as I remember,
the nature of that kind of exit could be somewhat ungallant.
Andrei
________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Jul 29, 2010 2:07 am |
|
|
|
Guest
|
> With R13B04 running on Montavista Linux, I've seen a few cases
> recently where the Erlang VM simply exits without any log messages,
> crash dumps, or coredumps. It appears to happen only after days of
> running under load, making it hard to reproduce and investigate. It
> may be related to memory consumption, but I'm not sure. External
> programs like heart and memsup simply report "Erlang has closed,"
> which according to the source code means they each got a return value
> of 0 from read() on their connection to the VM, which in turn would
> seem to indicate that the VM side of the connection was simply closed.
> This would all seem to indicate that the VM was either killed by
> another process or that it called exit() itself. The Linux "OOM
> killer" is disabled on this system, and I don't know of any other
> process that would be killing the VM. There are no alarms in the logs
> about hitting memory high watermarks or anything like that, and we
> aren't using any options to change allocators or anything like that.
>
> Anybody ever seen anything like this?
>
> Hi Steve,
Yes, we've noticed this silent exit behaviour of Erlang VM (R13B04) on
Fedora 12. In our case, about ~12K YAWS/HTTP processes were in contention to
get lock for compiling same YAWS page. We changed YAWS cache refresh
interval to some very high figure and got rid of this problem.
Regards,
Vinayak
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Jul 29, 2010 3:41 am |
|
|
|
Guest
|
Steve,
I've seen something like this that was memory related. Adjusting
'fullsweep_after' option or setting ERL_FULLSWEEP_AFTER environment
variable (http://www.erlang.org/doc/man/erlang.html#system_flag-2)
helped in our case.
Perhaps you should set up a separate watchdog OS process that would
monitor memory consumption of VM by reading the /proc/PID/stat and
logging it every N seconds.
Serge
On 7/28/2010 4:29 PM, Steve Vinoski wrote:
> With R13B04 running on Montavista Linux, I've seen a few cases
> recently where the Erlang VM simply exits without any log messages,
> crash dumps, or coredumps. It appears to happen only after days of
> running under load, making it hard to reproduce and investigate. It
> may be related to memory consumption, but I'm not sure. External
> programs like heart and memsup simply report "Erlang has closed,"
> which according to the source code means they each got a return value
> of 0 from read() on their connection to the VM, which in turn would
> seem to indicate that the VM side of the connection was simply closed.
> This would all seem to indicate that the VM was either killed by
> another process or that it called exit() itself. The Linux "OOM
> killer" is disabled on this system, and I don't know of any other
> process that would be killing the VM. There are no alarms in the logs
> about hitting memory high watermarks or anything like that, and we
> aren't using any options to change allocators or anything like that.
>
> Anybody ever seen anything like this?
>
> I've found a few places in the VM C source code where exit() is called
> without logging anything. Some of these are normal exits, like when
> you exit an Erlang shell, where no logging is needed. But others seem
> to be error conditions, and there should be logging for those. I think
> I'll probably have to patch my system to add logging to those cases to
> try to track down this problem -- is there still time to get a patch
> like this into R14B? If this issue is memory-related, I suppose it's
> possible that a sudden increase in memory consumption could cause the
> VM to exit between alarm checks, explaining why things like memsup
> don't seem to notice, so it would seem to be fairly critical that
> something is logged by the VM itself for such cases.
>
> thanks,
> --steve
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
>
________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Jul 29, 2010 4:44 am |
|
|
|
Guest
|
On Wed, Jul 28, 2010 at 6:54 PM, Andrei Soroker <soroker@gmail.com> wrote:
> On Wed, Jul 28, 2010 at 1:29 PM, Steve Vinoski <vinoski@ieee.org> wrote:
>> With R13B04 running on Montavista Linux, I've seen a few cases
>> recently where the Erlang VM simply exits without any log messages,
>> crash dumps, or coredumps. It appears to happen only after days of
>> running under load, making it hard to reproduce and investigate.
>
> We've seen something similar once when we hit the atom table limit.
> The fact that the exit occurs after some time of happy running might
> be indicative of a hard limit being reached, and as far as I remember,
> the nature of that kind of exit could be somewhat ungallant.
Good idea, I'll try to check for this sort of thing.
thanks,
--steve
________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Jul 29, 2010 4:51 am |
|
|
|
Guest
|
On Wed, Jul 28, 2010 at 10:04 PM, Vinayak Pawar <vinayakapawar@gmail.com> wrote:
>
>> With R13B04 running on Montavista Linux, I've seen a few cases
>> recently where the Erlang VM simply exits without any log messages,
>> crash dumps, or coredumps. It appears to happen only after days of
>> running under load, making it hard to reproduce and investigate. It
>> may be related to memory consumption, but I'm not sure. External
>> programs like heart and memsup simply report "Erlang has closed,"
>> which according to the source code means they each got a return value
>> of 0 from read() on their connection to the VM, which in turn would
>> seem to indicate that the VM side of the connection was simply closed.
>> This would all seem to indicate that the VM was either killed by
>> another process or that it called exit() itself. The Linux "OOM
>> killer" is disabled on this system, and I don't know of any other
>> process that would be killing the VM. There are no alarms in the logs
>> about hitting memory high watermarks or anything like that, and we
>> aren't using any options to change allocators or anything like that.
>>
>> Anybody ever seen anything like this?
>>
> Hi Steve,
>
> Yes, we've noticed this silent exit behaviour of Erlang VM (R13B04) on
> Fedora 12. In our case, about ~12K YAWS/HTTP processes were in contention to
> get lock for compiling same YAWS page. We changed YAWS cache refresh
> interval to some very high figure and got rid of this problem.
Yaws isn't doing that sort of thing in my case, but generally, this is
good info since it's certainly possible some process queue is getting
backed up which of course just worsens over time. We're now running a
periodic check for such things but we haven't caught anything yet
(we're between failures at the moment).
thanks,
--steve
________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Jul 29, 2010 4:55 am |
|
|
|
Guest
|
On Wed, Jul 28, 2010 at 11:38 PM, Serge Aleynikov <serge@aleynikov.org> wrote:
> Steve,
>
> I've seen something like this that was memory related. |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Jul 29, 2010 9:46 am |
|
|
|
Guest
|
On Wed, Jul 28, 2010 at 10:29 PM, Steve Vinoski <vinoski@ieee.org> wrote:
> I've found a few places in the VM C source code where exit() is called
> without logging anything. Some of these are normal exits, like when
> you exit an Erlang shell, where no logging is needed. But others seem
> to be error conditions, and there should be logging for those. I think
> I'll probably have to patch my system to add logging to those cases to
> try to track down this problem -- is there still time to get a patch
> like this into R14B?
Yes, there should still be time. The issue seems important enough
to fix in R14B.
--
Björn Gustavsson, Erlang/OTP, Ericsson AB
________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
Post received from mailinglist |
|
|
| Back to top |
|
| wuji |
| Posted: Fri Aug 24, 2012 8:35 am |
|
|
|
User
Joined: 10 Aug 2012
Posts: 654
|
a small number of cases -- just 11 -- the the cheap authentic air jordans the parents insisted on continuing intensive care while they prayed
divine intervention and a complete cure, even after being told told [h3]cheap real jordans[/h3] told there was no hope for recovery.Overriding Religious BeliefsSuch scenarios
up all sorts of ethical and legal dilemmas for medical medical cheap Ralph Lauren medical caregivers who must try to balance a parent's wishes
what they think is best for their patient. Caplan says says [h4]knockoff designer *beep*[/h4] says in most cases, they ultimately advocate for the patient."You
to take beliefs into account but you can't let any any [h2]cheap replica *beep*[/h2] any parent for any reason hijack what you as a |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
