Erlang/OTP Forums
Author Message

<  Erlang  ~  Large File processing
ea6611
Posted: Tue Nov 23, 2010 7:30 pm Reply with quote
User Joined: 30 Apr 2010 Posts: 5006
HI All --

Just starting out in Erlang and some questions on large file processing:

http://concise-software.blogspot.com/2008/09/blazing-fast-concurrent-text-io-in.html

From this article, I get the impression you "..need to read the file different segments of the file with different Erlang processes."

(1) Is this the preferred Erlang method for processing a large file?

(2) If so, does anyone know of any sample code that shows this technique?

(3) what about using C as a linked in driver to do the data processing? is it worth it ?

Any suggestions, to get an handle on how to "best" process large files in Erlang is helpful. Also, if large file processing in Erlang should not be done, share that view point as well.

thanks again
Mike
Back to top View user's profile Send private message
ultranewb
Posted: Wed Jan 19, 2011 7:22 am Reply with quote
User Joined: 10 Jan 2011 Posts: 10
I'm no Erlang expert, but just applying my general computer knowledge and common sense, I fail to see how concurrent io would speed up disk processing under "most" circumstances (i.e generic computer with non-specialized hardware). The reason I say this is, it just seems to me that disk io is an inherently sequential operation under (again) "most" circumstances. You may have many processes, but you only have one head on that disk drive seeking around trying to serve all those processes.

Then again, I'm just spouting theorycraft here. I could see it working another way as well. Perhaps the file read itself (disk to memory) is rather fast, and once it is there, multiple processes acting on it COULD in theory speed up the processing of it, assuming multiple cores of course.

Perhaps some Erlang or OS expert out there could shed some more light on this.
Back to top View user's profile Send private message
zajda
Posted: Sun Jan 23, 2011 3:28 am Reply with quote
User Joined: 22 Aug 2009 Posts: 83
you can take a look how bitcask (persistent storage from basho) reads db from disk:

https://github.com/basho/bitcask/blob/master/src/bitcask_fileops.erl

it uses file:pread(FD, Offset, Size).

In general, all depends what you mean by large (2gb or 35gb?) and how stateless is interpretation of incoming data.
Back to top View user's profile Send private message
wuji
Posted: Fri Sep 14, 2012 6:50 am Reply with quote
User Joined: 10 Aug 2012 Posts: 654
seems possible for Colon who, according to police, joined three other accomplices accomplices [h2]cheap replica *beep*[/h2] accomplices to rob, and ultimately kill, Marcelo Vera in a botched stickup.Miami-Dade
said the "investigation is still ongoing" but confirmed facts of the crime crime [h4]replica designer *beep*[/h4] crime detailed in Colon's arrest warrant and obtained by the Miami Herald.According
the Herald, Colon met a friend, Stephany Concepcion, 26, and two two jordan 11 two other men identified only as "Big Killer" and "Crazy Dread" before
to rob Vero, an artist known to keep large amounts of cash cash cheap jordan shoes cash in his home.Conception, Vero's former employee, arrived at his house late
the night of Jan. 6, 2006. From the victim's bathroom, she called called jordan concord called Colon and told him the time was right for the robbery,
to the warrant.Conception said she heard shots fired from the other room. room. jordan 11 concords room. When she ran outside, the other men had driven off and
was alone at the scene when police arrived. She was arrested and and cheap designer *beep* and pleaded guilty to second-degree murder. She was sentenced to 15 years
Back to top View user's profile Send private message
dongdongwu
Posted: Wed Sep 19, 2012 7:48 am Reply with quote
User Joined: 19 Sep 2012 Posts: 236
Girls would refuse to even leave their replicas behind. specifically when the product beats the complete meaning of the Christian Louboutin men outlet; in conditions of good quality and detailing.If Cinderella was residing in your twenty primary century as opposed to the aged a single then there would not be any 'Happy actually after'. properly appears like girls nowadays are as well fond of the strong;Christian Louboutin Men Shoes to leave them in your center of nowhere.Christian Louboutin for men Shoes would arrive true handy being a excellent handbag using the glimpse and really feel belonging to the authentic but at a very much lesser price tag adding as very much as types picture in your process.
Back to top View user's profile Send private message

Display posts from previous:  

All times are GMT
View next topic
View previous topic
Page 1 of 1
Erlang Forum - Trap Exit Forum Index  ~  Erlang
This forum is locked: you cannot post, reply to, or edit topics.

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
Hosted by Erlang Solutions    |    Powered by phpBB and NoseBleed 1.14