| Author |
Message |
< Erlang questions mailing list ~ Obsolete exported functions file:raw_{read, write}_file_info |
| Guest |
 Posted: Wed Sep 14, 2011 9:21 am |
 |
|
|
Guest
|
I'm curious if someone happens to know why the file:raw_read_file_info/1 and file:raw_read_file_info/2 methods were made obsolete?
Joseph Norton
norton@alum.mit.edu (norton@alum.mit.edu)
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Wed Sep 14, 2011 11:09 am |
|
|
|
Guest
|
|
| Back to top |
|
| Guest |
| Posted: Wed Sep 14, 2011 4:42 pm |
|
|
|
Guest
|
Joseph Norton <norton@lovely.email.ne.jp> wrote:
jn> I'm curious if someone happens to know why the
jn> file:raw_read_file_info/1 and file:raw_read_file_info/2 methods were
jn> made obsolete?
Joe, I stumbled across those functions while putting DTrace probes into
the efile_drv driver. I thought, hey, those would be quite useful.
I recommend reading the file.erl source. It's quite instructive to see
how many file I/O functions are redirected to the 'file_server_2'
process. For file I/O-intensive applications (e.g. Hibari and Riak I
know, CouchDB and RabbitMQ I'd guess), having all calls to(*)
file:read_file_info/1 serialized by the file server process is a source
of latency that we (DB authors) may desire to live without.
Having planted bugs in prim_file.erl accidentally, it is also quite
instructive to see how many different ways the VM's bootstrap process
can be broken. So I now understand reluctance to deprecate functions
that may be necessary at some weird situation when booting. However,
there may come a time when some of us submit a patch to expose *more*
raw file I/O functions in file.erl, not fewer.
-Scott
(*) Or file:delete/1 or file:rename/2 or ...
_______________________________________________
erlang-questions mailing list
erlang-questions@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Wed Sep 14, 2011 4:58 pm |
|
|
|
Guest
|
> I recommend reading the file.erl source. It's quite instructive to see
> how many file I/O functions are redirected to the 'file_server_2'
> process. For file I/O-intensive applications (e.g. Hibari and Riak I
> know, CouchDB and RabbitMQ I'd guess), having all calls to(*)
> file:read_file_info/1 serialized by the file server process is a source
> of latency that we (DB authors) may desire to live without.
How did you proceed to avoid these calls and reduce latency Scott?
Any hints?
_______________________________________________
erlang-questions mailing list
erlang-questions@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Wed Sep 14, 2011 5:23 pm |
|
|
|
Guest
|
Agreed. For us, file_server_2 represented a significant bottleneck on our 8 core nodes (e.g. delete/2, rename/2, etc.) We created a module that used the low-level prim_file interface directly to get around the bottlenecks. We have a bounded number of processes that perform file operations, so we ended up with the following basic approach in this module:
-spec delete(Name :: file:name()) -> 'ok' | {'error', file:posix()}.
delete(Name) ->
Port = get_port(),
prim_file:delete( Port, Name ).
|
|
|
| Back to top |
|
| Guest |
| Posted: Wed Sep 14, 2011 5:37 pm |
|
|
|
Guest
|
2011/9/14, Scott Lystig Fritchie <fritchie@snookles.com>:
> Joseph Norton <norton@lovely.email.ne.jp> wrote:
>
> jn> I'm curious if someone happens to know why the
> jn> file:raw_read_file_info/1 and file:raw_read_file_info/2 methods were
> jn> made obsolete?
>
> Joe, I stumbled across those functions while putting DTrace probes into
> the efile_drv driver. I thought, hey, those would be quite useful.
>
> I recommend reading the file.erl source. It's quite instructive to see
> how many file I/O functions are redirected to the 'file_server_2'
> process. For file I/O-intensive applications (e.g. Hibari and Riak I
> know, CouchDB and RabbitMQ I'd guess), having all calls to(*)
> file:read_file_info/1 serialized by the file server process is a source
> of latency that we (DB authors) may desire to live without.
That's interesting. I was also chasing a performance problem a couple
of weeks ago (the server crawled to a halt for a minute or two, the
load of the Linux OS went over 50, then everything went back to
normal) and noticed that the file_server process used a lot of CPU. My
solution(?) was to randomize the jobs that would write to the disk, so
the 30 processes tried not to write to the disk at the same time.
_______________________________________________
erlang-questions mailing list
erlang-questions@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Wed Sep 14, 2011 9:17 pm |
|
|
|
Guest
|
Attila Rajmund Nohl <attila.r.nohl@gmail.com> wrote:
arn> That's interesting. I was also chasing a performance problem a
arn> couple of weeks ago (the server crawled to a halt for a minute or
arn> two, the load of the Linux OS went over 50, then everything went
arn> back to normal) and noticed that the file_server process used a lot
arn> of CPU. My solution(?) was to randomize the jobs that would write
arn> to the disk, so the 30 processes tried not to write to the disk at
arn> the same time.
Hrm, you didn't mention how you're writing to disk. If you're using
file:writefile/2, that's another func that ends up going through the
file_server_2.
If you're opening file descriptors for each file (and opening them in
'raw' mode), then you probably should also be using the +A flag when
starting the VM so that all computation by the VM won't be blocked by
slow file I/O.
And if you're already using separate file descriptors and the +A flag,
simply doing too much I/O at once can be a Bad Idea. If your disk is
overloaded (measure using "iostat -x 1" or equivalent), then all you can
really do is wait(*). If you believe that your disk(s) are not yet
saturated and that you believe that your bottleneck is the Erlang VM, it
may be possible that you've got too much parallel I/O relative to the
size of the +A I/O worker pool.
For example, say that you use "+A 4" and (as mentioned above) have 50
file writes happening simultaneously. The code in efile_drv.c assigns a
worker Pthread based on the Erlang port number(**). So you could have
many ports' worth of I/O assigned to each worker Pthread, 12-13 on
average if everything is sync'ed perfectly. In a pathological worst
case, where you opened 4 ports and discarded every 3, you could have all
50 ports assigned to the same worker Pthread.(***)
-Scott
(*) Or fundamentally change the data you're writing and how you do it
and when.
(**) The assignment of port -> worker thread is *not* done by "first
idle Pthread in the pool".
(***) The lack of visibility into this part of the efile_drv.c driver is
a motivating reason that got me active in patching the VM to add DTrace
probes. See https://github.com/slfritchie/otp/tree/dtrace-experiment
for source. Also, Dustin Stallings is hacking on DTrace, starting from
a different direction, see https://github.com/dustin/otp/tree/dtrace.
_______________________________________________
erlang-questions mailing list
erlang-questions@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Sep 15, 2011 9:25 am |
|
|
|
Guest
|
Hi.
I'm working on a patch for the file.erl module itself. In the meantime, see the attached module as a working example. We have been using this approach for benchmarking purposes since early summer. The performance difference is dramatically better than the default file implementation.
thanks,
Joseph Norton
norton@alum.mit.edu
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Sep 15, 2011 9:37 am |
|
|
|
Guest
|
Thanks for sharing Joseph.
One more question:
Is the call prim_file:read_file/1 a better alternative to file:read_file/1?
Regards,
Zabrane
On Sep 15, 2011, at 11:25 AM, Joseph Norton wrote:
>
> Hi.
>
> I'm working on a patch for the file.erl module itself. In the meantime, see the attached module as a working example. We have been using this approach for benchmarking purposes since early summer. The performance difference is dramatically better than the default file implementation.
>
> thanks,
>
> Joseph Norton
> norton@alum.mit.edu
>
> <basho_bench_erlang_file_alternative.erl>
> On Sep 15, 2011, at 1:57 AM, erlang wrote:
>
>>> I recommend reading the file.erl source. It's quite instructive to see
>>> how many file I/O functions are redirected to the 'file_server_2'
>>> process. For file I/O-intensive applications (e.g. Hibari and Riak I
>>> know, CouchDB and RabbitMQ I'd guess), having all calls to(*)
>>> file:read_file_info/1 serialized by the file server process is a source
>>> of latency that we (DB authors) may desire to live without.
>>
>> How did you proceed to avoid these calls and reduce latency Scott?
>> Any hints?
>
_______________________________________________
erlang-questions mailing list
erlang-questions@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Sep 15, 2011 10:26 am |
|
|
|
Guest
|
Zabrane -
Based on benchmarking results, I believe prim_file:read_file is a better alternative for performance. The other reason the prim_file approach is better is it should prevent callers with file i/o to different disks and/or different disk controllers from impacting each other. We have seen a case in production where failure of one disk crashed erlang processes doing file i/o to an unrelated disk. The root cause (not yet repeated and not yet proven though) is the singleton file i/o server process.
Nevertheless, I was hoping to learn from my original post why the file module hasn't been written (or re-written) to provide raw support for other operations beyond just open. My only guess is that file i/o hasn't been a bottleneck for most Erlang applications.
thanks,
Joseph Norton
norton@alum.mit.edu
On Sep 15, 2011, at 6:36 PM, Zabrane Mickael wrote:
> Thanks for sharing Joseph.
>
> One more question:
> Is the call prim_file:read_file/1 a better alternative to file:read_file/1?
>
> Regards,
> Zabrane
>
> On Sep 15, 2011, at 11:25 AM, Joseph Norton wrote:
>
>>
>> Hi.
>>
>> I'm working on a patch for the file.erl module itself. In the meantime, see the attached module as a working example. We have been using this approach for benchmarking purposes since early summer. The performance difference is dramatically better than the default file implementation.
>>
>> thanks,
>>
>> Joseph Norton
>> norton@alum.mit.edu
>>
>> <basho_bench_erlang_file_alternative.erl>
>> On Sep 15, 2011, at 1:57 AM, erlang wrote:
>>
>>>> I recommend reading the file.erl source. It's quite instructive to see
>>>> how many file I/O functions are redirected to the 'file_server_2'
>>>> process. For file I/O-intensive applications (e.g. Hibari and Riak I
>>>> know, CouchDB and RabbitMQ I'd guess), having all calls to(*)
>>>> file:read_file_info/1 serialized by the file server process is a source
>>>> of latency that we (DB authors) may desire to live without.
>>>
>>> How did you proceed to avoid these calls and reduce latency Scott?
>>> Any hints?
>>
>
>
_______________________________________________
erlang-questions mailing list
erlang-questions@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Sep 15, 2011 10:34 am |
|
|
|
Guest
|
Thanks a lot.
On Sep 15, 2011, at 12:25 PM, Joseph Norton wrote:
>
> Zabrane -
>
> Based on benchmarking results, I believe prim_file:read_file is a better alternative for performance. The other reason the prim_file approach is better is it should prevent callers with file i/o to different disks and/or different disk controllers from impacting each other. We have seen a case in production where failure of one disk crashed erlang processes doing file i/o to an unrelated disk. The root cause (not yet repeated and not yet proven though) is the singleton file i/o server process.
>
> Nevertheless, I was hoping to learn from my original post why the file module hasn't been written (or re-written) to provide raw support for other operations beyond just open. My only guess is that file i/o hasn't been a bottleneck for most Erlang applications.
>
> thanks,
>
> Joseph Norton
> norton@alum.mit.edu
_______________________________________________
erlang-questions mailing list
erlang-questions@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Sep 15, 2011 5:23 pm |
|
|
|
Guest
|
As a followup to my previous e-mail, here is one possible patch for the file.erl module itself. I have not tested this patch so please consider it as pseudo-code only.
https://github.com/norton/otp/commit/6541f14800ab687dc6aa262e464f5ff9d72c2deb
thanks,
On 2011/09/15, at 18:25, Joseph Norton wrote:
>
> Hi.
>
> I'm working on a patch for the file.erl module itself. In the meantime, see the attached module as a working example. We have been using this approach for benchmarking purposes since early summer. The performance difference is dramatically better than the default file implementation.
>
> thanks,
>
> Joseph Norton
> norton@alum.mit.edu
>
> <basho_bench_erlang_file_alternative.erl>
> On Sep 15, 2011, at 1:57 AM, erlang wrote:
>
>>> I recommend reading the file.erl source. It's quite instructive to see
>>> how many file I/O functions are redirected to the 'file_server_2'
>>> process. For file I/O-intensive applications (e.g. Hibari and Riak I
>>> know, CouchDB and RabbitMQ I'd guess), having all calls to(*)
>>> file:read_file_info/1 serialized by the file server process is a source
>>> of latency that we (DB authors) may desire to live without.
>>
>> How did you proceed to avoid these calls and reduce latency Scott?
>> Any hints?
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions
Joseph Wayne Norton
norton@alum.mit.edu
_______________________________________________
erlang-questions mailing list
erlang-questions@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Post received from mailinglist |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
