|
|
| Author |
Message |
< Erlang bugs mailing list ~ infinite loop when beam.smp compiled with -O2 on debian lenn |
| Guest |
 Posted: Mon May 03, 2010 8:29 pm |
 |
|
|
Guest
|
Hi,
We hit a bug while running rabbitmq where the beam.smp process was stuck
in a tight loop in the erts_poll_info method.
The process was eating up 100% of exactly one core (on a multi core box) and
rabbitmq was dysfunctional. Unfortunately
I could not create a small test case to reproduce this condition but it
would happen quite frequently while rabbitmq was in
operation.
The C code for the function didn't provide any hints on what would have been
spinning in that function
(first time looking at this codebase though). Finally looking through the
disassembly in gdb, (at the point of where our process was spinning) I saw
the following lines in the
erts_poll_info_kp method:
0x00000000004f0fe9 <erts_poll_info_kp+185>: nopl 0x0(%rax)
0x00000000004f0ff0 <erts_poll_info_kp+192>: jmp 0x4f0fe9
<erts_poll_info_kp+185>
(Similar assembly code can be seen when the KERNEL_POLL option is
disabled.)
Clearly the above will trivially spin forever anytime we get into that
codepath. The above
looks suspiciously like some code got optimized out by the compiler leaving
the crazy
loop code.
So I compiled with -O1 and then with no optimization at all. Withe -O1, I
saw a
a weird jmp insruction jumping to it's own address:
0x0000000000517102 <erts_poll_info_kp+60>: jmp 0x517102
<erts_poll_info_kp+60>
With no optimization, any of those trivial spins did not exist but I
didn't analyze the unoptimized
code enough to say whether it can be proven to have an infinite loop (i.e.,
whether the optimizing
compiler is simply doing it's job vs. this being a compiler bug).
Anyway, this problem exists at least since erlang-base_12.b.3-dfsg debian
package version and has been
verified to exists in the github version as of today.
Her'es the gcc and debian version info:
$ gcc --version
gcc-4.3.real (Debian 4.3.2-1.1) 4.3.2
Copyright (C) 2008 Free Software Foundation, Inc.
$ cat /etc/debian_version
5.0
I'd be happy to provide any other info as needed.
Thanks
Chetan Ahuja
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon May 03, 2010 9:55 pm |
|
|
|
Guest
|
Chetan Ahuja writes:
> Hi,
>
> We hit a bug while running rabbitmq where the beam.smp process was stuck
> in a tight loop in the erts_poll_info method.
> The process was eating up 100% of exactly one core (on a multi core box) and
> rabbitmq was dysfunctional. Unfortunately
> I could not create a small test case to reproduce this condition but it
> would happen quite frequently while rabbitmq was in
> operation.
>
> The C code for the function didn't provide any hints on what would have been
> spinning in that function
> (first time looking at this codebase though). Finally looking through the
> disassembly in gdb, (at the point of where our process was spinning) I saw
> the following lines in the
> erts_poll_info_kp method:
>
>
> 0x00000000004f0fe9 <erts_poll_info_kp+185>: nopl 0x0(%rax)
> 0x00000000004f0ff0 <erts_poll_info_kp+192>: jmp 0x4f0fe9
> <erts_poll_info_kp+185>
>
> (Similar assembly code can be seen when the KERNEL_POLL option is
> disabled.)
>
> Clearly the above will trivially spin forever anytime we get into that
> codepath. The above
> looks suspiciously like some code got optimized out by the compiler leaving
> the crazy
> loop code.
>
> So I compiled with -O1 and then with no optimization at all. Withe -O1, I
> saw a
> a weird jmp insruction jumping to it's own address:
>
> 0x0000000000517102 <erts_poll_info_kp+60>: jmp 0x517102
> <erts_poll_info_kp+60>
>
> With no optimization, any of those trivial spins did not exist but I
> didn't analyze the unoptimized
> code enough to say whether it can be proven to have an infinite loop (i.e.,
> whether the optimizing
> compiler is simply doing it's job vs. this being a compiler bug).
>
> Anyway, this problem exists at least since erlang-base_12.b.3-dfsg debian
> package version and has been
> verified to exists in the github version as of today.
>
>
> Her'es the gcc and debian version info:
> $ gcc --version
> gcc-4.3.real (Debian 4.3.2-1.1) 4.3.2
> Copyright (C) 2008 Free Software Foundation, Inc.
I looked at the procedure in question (not so easy to locate due to
some "creative" C preprocessor abuse), and noticed an obvious bug:
there's a loop over a linked list that forgets to actually advance
the node pointer to the next element. When optimizing, gcc will notice
that the loop doesn't terminate, omit the body of the loop (the
calculations are dead), which will result in the type of object code
shown above. Thus, it's an Erlang VM bug not a gcc miscompilation.
Try the patch below and let us know if it solves your problem.
/Mikael
--- otp_src_R13B03/erts/emulator/sys/common/erl_poll.c.~1~ 2009-03-12 13:16:29.000000000 +0100
+++ otp_src_R13B03/erts/emulator/sys/common/erl_poll.c 2010-05-03 23:41:32.000000000 +0200
@@ -2404,6 +2404,7 @@ ERTS_POLL_EXPORT(erts_poll_info)(ErtsPol
while (urqbp) {
size += sizeof(ErtsPollSetUpdateRequestsBlock);
pending_updates += urqbp->len;
+ urqbp = urqbp->next;
}
}
#endif
________________________________________________________________
erlang-bugs (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-bugs-unsubscribe@erlang.org
Post received from mailinglist |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
