| Author |
Message |
|
| Guest |
 Posted: Mon Aug 24, 2009 5:46 pm |
 |
|
|
Guest
|
Hi list,
I've got some enormous system memory usage whenever I run erlang (32bit linux
system) :
> $ erl
> Erlang R13B01 (erts-5.7.2) [source] [rq:1] [async-threads:0]
> [kernel-poll:false]
>
> Eshell V5.7.2 (abort with ^G)
> 1> erlang:memory().
> [{total,302822080},
> {processes,489036},
> {processes_used,483292},
> {system,302333044},
> {atom,255645},
> {atom_used,254795},
> {binary,306280},
> {code,1775473},
> {ets,104668}]
That's close to 300MB for a bare shell. Too much, if you ask me :p
I've traced it back to the maximum number of open files allowed by the system.
I use a big number here, to allow that many tcp connections to my server. If
I reduce this limit, erlang memory usage returns to manageable levels:
> $ for i in 999999 900000 800000 700000 600000 500000 400000 300000 200000
> 100000 50000 10000 5000 1000 500 100 50; do ulimit -n $i; echo -n $i;
> erl -noinput -eval 'io:format(" ~p~n", [erlang:memory(system)])' -s erlang
> halt; done
> 999999 301852400
> 900000 296652448
> 800000 291452448
> 700000 286252464
> 600000 281052464
> 500000 152120480
> 400000 146920480
> 300000 141720496
> 200000 74654496
> 100000 38521504
> 50000 20455024
> 10000 6775136
> 5000 4581840
> 1000 2682176
> 500 2656176
> 100 2634088
> 50 2629488
I understand that opening more files uses more memory, but I haven't opened
anything yet. I shouldn't be paying the price for a system limit I haven't
reached yet.
Also, note that the memory usage in the above example is not linear : there
are jumps in the graph.
I am not 100% sure, but I think this is a regression : I tested my software
with a big max-open-file limit before and hadn't noticed the memory usage.
For the immediate future, I'll size the system limit more conservatively. But
I'm interested in a fix or workaround on the erlang side.
Thanks.
--
Vincent de Phily
Mobile Devices
+33 (0) 666 301 306
+33 (0) 142 119 325
Warning
This message (and any associated files) is intended only for the use of its
intended recipient and may contain information that is confidential, subject
to copyright or constitutes a trade secret. If you are not the intended
recipient you are hereby notified that any dissemination, copying or
distribution of this message, or files associated with this message, is
strictly prohibited. If you have received this message in error, please
notify us immediately by replying to the message and deleting it from your
computer. Any views or opinions presented are solely those of the author
vincent.dephily@mobile-devices.fr and do not necessarily represent those of
the
company. Although the company has taken reasonable precautions to ensure no
viruses are present in this email, the company cannot accept responsibility
for any loss or damage arising from the use of this email or attachments.
________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon Aug 24, 2009 9:11 pm |
|
|
|
Guest
|
Vincent de Phily <vincent.dephily@mobile-devices.fr> wrote:
>
>I understand that opening more files uses more memory, but I haven't opened
>anything yet. I shouldn't be paying the price for a system limit I haven't
>reached yet.
>Also, note that the memory usage in the above example is not linear : there
>are jumps in the graph.
>
>I am not 100% sure, but I think this is a regression : I tested my software
>with a big max-open-file limit before and hadn't noticed the memory usage.
The runtime creates an array of port structures (100+ bytes IIRC) sized
as the next power of two at or above your "max-open-files" (a.k.a.
sysconf(_SC_OPEN_MAX)) - this has been true since at least R10, probably
much earlier. I.e. the following from erlang(3) is only correct
regarding the default if you have a "max-open-files" of 1024 (or less) -
which happens to be the default on Linux I believe.
The maximum number of ports that can be open at the same time
is 1024 by default, but can be configured by the environment
variable ERL_MAX_PORTS.
The ERL_MAX_PORTS part *is* correct though - and you can use this to
*reduce* the size of that port array. Oh, there is actually an upper
limit - 2^28 as far as I can see. And 1024 is the *lower* limit. See
init_io() in erts/emulator/beam/io.c.
--Per Hedeland
________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Tue Aug 25, 2009 12:00 pm |
|
|
|
Guest
|
On Monday 24 August 2009 23:09:25 Per Hedeland wrote:
> The runtime creates an array of port structures (100+ bytes IIRC) sized
> as the next power of two at or above your "max-open-files" (a.k.a.
> sysconf(_SC_OPEN_MAX)) - this has been true since at least R10, probably
> much earlier. I.e. the following from erlang(3) is only correct
> regarding the default if you have a "max-open-files" of 1024 (or less) -
> which happens to be the default on Linux I believe.
>
> The maximum number of ports that can be open at the same time
> is 1024 by default, but can be configured by the environment
> variable ERL_MAX_PORTS.
>
> The ERL_MAX_PORTS part *is* correct though - and you can use this to
> *reduce* the size of that port array. Oh, there is actually an upper
> limit - 2^28 as far as I can see. And 1024 is the *lower* limit. See
> init_io() in erts/emulator/beam/io.c.
Thanks a lot for the pointers. I'll use ERL_MAX_PORTS to tweak memory usage on
a per-instance basis.
I still wish the memory wasn't allocated untill needed, but I won't try to
argue against beam developpers regarding performance-related design
decisions 
--
Vincent de Phily
Mobile Devices
+33 (0) 666 301 306
+33 (0) 142 119 325
Warning
This message (and any associated files) is intended only for the use of its
intended recipient and may contain information that is confidential, subject
to copyright or constitutes a trade secret. If you are not the intended
recipient you are hereby notified that any dissemination, copying or
distribution of this message, or files associated with this message, is
strictly prohibited. If you have received this message in error, please
notify us immediately by replying to the message and deleting it from your
computer. Any views or opinions presented are solely those of the author
vincent.dephily@mobile-devices.fr and do not necessarily represent those of
the
company. Although the company has taken reasonable precautions to ensure no
viruses are present in this email, the company cannot accept responsibility
for any loss or damage arising from the use of this email or attachments.
________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org
Post received from mailinglist |
|
|
| Back to top |
|
| kagato |
| Posted: Tue Aug 25, 2009 11:35 pm |
|
|
|
User
Joined: 30 Dec 2007
Posts: 85
|
I'm unclear on the design decisions that went into this, but having
done some embedded work myself, these structures are almost always
preallocated. The prevailing wisdom is that embedded systems must be
able to make acceptable progress in the face of memory exhaustion.
Since Erlang is designed to be highly fault-tolerant, I would imagine
that this is the primary reason that the port-table (and, indeed, the
atom table as well) are pre-allocated. In the embedded systems world,
how something fails is often more important than how it succeeds.
As an example of the behavior to be avoided, look no further than
loading code. If the code-server can't load a file due to memory
exhaustion, it may be unable to upgrade the system to fix the bug
causing the memory exhaustion. Basically, no new code can be loaded,
no configuration files read, etc. All because we don't have the
memory to allocate "on-demand". This causes memory exhaustion to
cause cascading failures in any system that didn't set aside some
memory. This is exceptionally bad, because (under Erlang) it's quite
likely that whatever is consuming memory will get killed and the
memory pressure will go away. If we wedge the system in the meantime,
that's just no good.
From a failure perspective, this may increase the frequency of
failures by creating artificial exhaustion of a resource (namely
ports); but this is worthwhile because it makes those failures less
severe (and more deterministic) by decoupling memory exhaustion
failures from port allocation failures. Designing away "cascading
failures" such as this creates a big design win--especially when you
may end up building an embedded system where memory exhaustion must be
tolerable (and is more often transient).
On Aug 25, 2009, at 4:57 AM, Vincent de Phily wrote:
> On Monday 24 August 2009 23:09:25 Per Hedeland wrote:
>> The runtime creates an array of port structures (100+ bytes IIRC)
>> sized
>> as the next power of two at or above your "max-open-files" (a.k.a.
>> sysconf(_SC_OPEN_MAX)) - this has been true since at least R10,
>> probably
>> much earlier. I.e. the following from erlang(3) is only correct
>> regarding the default if you have a "max-open-files" of 1024 (or
>> less) -
>> which happens to be the default on Linux I believe.
>>
>> The maximum number of ports that can be open at the same time
>> is 1024 by default, but can be configured by the environment
>> variable ERL_MAX_PORTS.
>>
>> The ERL_MAX_PORTS part *is* correct though - and you can use this to
>> *reduce* the size of that port array. Oh, there is actually an upper
>> limit - 2^28 as far as I can see. And 1024 is the *lower* limit. See
>> init_io() in erts/emulator/beam/io.c.
>
> Thanks a lot for the pointers. I'll use ERL_MAX_PORTS to tweak
> memory usage on
> a per-instance basis.
>
> I still wish the memory wasn't allocated untill needed, but I won't
> try to
> argue against beam developpers regarding performance-related design
> decisions
>
>
> --
> Vincent de Phily
> Mobile Devices
> +33 (0) 666 301 306
> +33 (0) 142 119 325
>
> Warning
> This message (and any associated files) is intended only for the use
> of its
> intended recipient and may contain information that is confidential,
> subject
> to copyright or constitutes a trade secret. If you are not the
> intended
> recipient you are hereby notified that any dissemination, copying or
> distribution of this message, or files associated with this message,
> is
> strictly prohibited. If you have received this message in error,
> please
> notify us immediately by replying to the message and deleting it
> from your
> computer. Any views or opinions presented are solely those of the
> author
> vincent.dephily@mobile-devices.fr and do not necessarily represent
> those of
> the
> company. Although the company has taken reasonable precautions to
> ensure no
> viruses are present in this email, the company cannot accept
> responsibility
> for any loss or damage arising from the use of this email or
> attachments.
>
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>
--
Jayson Vantuyl
kagato@souja.net
________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org
Post received from mailinglist |
|
|
| Back to top |
 
|
| wuji |
| Posted: Wed Aug 15, 2012 6:47 am |
|
|
|
User
Joined: 10 Aug 2012
Posts: 654
|
nurse, started taking Adderall after reading a book that told told cheap Ralph Lauren Polo told her how to lie to her doctor to get
drug."Your life becomes a squirrel, just looking for that nut, nut, [h4]Cheap Ralph Lauren Shirts[/h4] nut, looking for that Adderall," she said.Some women start on
to keep up with the demands of career and home, home, [h3]Cheap Ralph Lauren Shirts[/h3] home, while others start looking for a quick weight loss
doctors say the situation is getting out of control."This is is replica designer bags for sale is a significant problem," said Dr. Marvin Seppala, chief medical
at Hazelden, an addiction treatment facility. "We've got an increase increase [h2]replica designer bags for sale[/h2] increase in women using drugs like Adderall ending up in |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
