|
User
Joined: 07 Mar 2008
Posts: 15
|
My program has a large number of parallel operating processes, each process has its own ETS table (set,private). each process does a bit of work on it's own table, and then sends a message to another process.... But even though each process has its own ETS table, when moving the program from a single core to a quad core (going from smp disabled to smp enabled), the execution time increases twofold, (when retaining the same cpu speed, ram fsb...). so it goes from 200us, to 400us per single traversing of the entire net of processes (the processing are traversed thousands of times...). I reprogrammed the entire thing using only a single public ets table (just to see the difference), the single cpu program didn't change its speed, but the quad core increased execution time even further. Rewriting the program in dict, does on the other hand speed up the execution when moving from single core to quad core. Though as you guys know running benchmarks, dict though having a very small (<<1us) fetch time, has a huge store time, becoming larger and larger with increased number of elements stored (10us-100us......), and in my case, each process needs to scale up to deal with millions of elements, hence the using of ETS. Using SET on the other hand is even worse than dict, in both insertion and random fetching.
The question is: Why does it slow down with smp activated when each of the processes has its own ets table? is it somehow all stored on the same thing? Any ideas?
So far I've got this far in the problem:
I think that it is not the case that there is a bottle necking due to mail boxes. For one, each process does not have more than 100 other processes connected to it (during a standard test), and in a smaller test where each process is connected to only 2 or 4 other processes, same thing occurs. I use ETS table so that I won't have the building up of messages in the mail box, as soon as a message arrives at a process, he right away enters it into the table with the key of the Pid of the process that sent it a message, and the value the sender had send with its Pid (exp: {self(), prediction, Value}). With the insertion time of ~2 microseconds and only having lets say 4 other processes connected to another processes, there is no bottle necking due to mail box. (that's why I'm using ETS, it's essential for the speed of the network, and to have the ability to efficiently and quickly access any value, any input, any time... on the process at random)
I've tested this setup without most of the calculations done in each process to see what happens with just message passing(second order derivatives...and other calculation), disabled) same problem occurs. I've now tried the code on a single CPU laptop, very peculiar thing happens. Without smp enabled it runs at ~300us per pass, with smp enabled (and it still only has 1 cpu, I simply: erl -smp), it goes up to ~450us still. Something funny is going on with the smp and ets.
I've ran fprof, side by side on both when smp is enabled and when it is disabled, the analysis reports are identical (there was a single difference in one of the values of 1 microsecond or so, but that's due to variation.... fprof does that sometimes).
Anyone ever came across a similar problem with ets tables?
Huge thanks,
-Gene |
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
