| Author |
Message |
< Erlang ~ How do processes handle message overload? |
| deakster |
 Posted: Wed Oct 08, 2008 12:03 pm |
 |
|
|
Joined: 08 Oct 2008
Posts: 2
|
In a real time system, if there is a process that receives a large number of messages per second (heartbeats, network updates etc), how does it handle situations where the frequency of messages gets greater than its ability to process them? What happens if a process gets sent its next message before it could complete the previous message's handler in the receive block?
I am presuming there is some kind of queue/buffer in place (I hope so anyway, surely messages aren't discarded)? How can we effectively manage these situations, as it certainly doesn't seem like an unlikely scenario to me.
(I am relatively new to erlang, although have written a basic server etc, but have not came across documentation regarding these issues). |
|
|
| Back to top |
 
|
| klaar |
| Posted: Wed Oct 08, 2008 5:33 pm |
|
|
|
User
Joined: 06 Oct 2008
Posts: 11
Location: Göteborg/Sweden
|
I am not very experienced with the erlang runtime either, but as far as I know:
- You can send a message to a process at any time.
- Received messages are pushed into a FIFO queue
- You pop messages from the message queue using receive
- If the popped message is not matched by a clause in your receive. It will be saved in a buffer until a match is found, previously unmatched messages are then put back into the message queue.
- A process handles one message at the time, and then it handles one message, etc.
In Armstrong's thesis he discusses the problem with messages that does not, and never will, match, the request handler in a server, and which will therefore continue to clog up the message queue.
His solution is to use a clause at the end which matches anything, in that clause you can handle the invalid message, ie. discard it and log the error. |
|
|
| Back to top |
|
| deakster |
| Posted: Wed Oct 08, 2008 6:00 pm |
|
|
|
Joined: 08 Oct 2008
Posts: 2
|
Thanks, that clears up a some things I wanted to know.
But assuming messages always have a successful match, but the process simply receives too many messages to process in time, I guess the queue will build up.
There must be some way to control/respond to that, or an overloaded process's queue will continue building indefinitely. |
|
|
| Back to top |
|
| klaar |
| Posted: Wed Oct 08, 2008 8:31 pm |
|
|
|
User
Joined: 06 Oct 2008
Posts: 11
Location: Göteborg/Sweden
|
Erlang nodes are contrained by the resources of the hardware and operating system upon which they run.
- If you can't handle the load, your processor is not fast enough/you don't have enough of them.
- If you can't buffer enough work to survive a load peak, you don't have enough memory.
I did some testing in the shell, and you can put 10^7 messages to a process which does not process them, the only consequence is that lot of memory is allocated. And when beam runs out of memory it simply crashes. (?)
Depending on how you prioritize the non-functional requirements of your program, you should put much to little attention to this. 
In the tcp/ip-world we ignore this problem by dropping packages like drunk postal office workers during the evenings 
--- Realizing that I didn't really say anything at all
I am assuming that a process does not have direct access to its message queue, does it? Sort of blows the cover, doesn't it? 
First of all, in order to get rid of a growing pile of unfinished work, you need to be able to say no. Implement this in your protocol. It could just be by answering:
{ok, Response} | {error, busy}
Let the client decide how to proceed from there.
You would want to use a proxy (http://en.wikipedia.org/wiki/Proxy_pattern) to actually enforce some kind of policy regarding what is too much. It has to somehow be notified of how much is actually processed by the server/servers. |
|
|
| Back to top |
|
| Mazen |
| Posted: Thu Oct 09, 2008 7:23 am |
|
|
|
User
Joined: 20 Jul 2006
Posts: 164
Location: London
|
If you get too long queues your problem is more likely in your design. Look at how you can avoid doing a lot of time consuming work in the process loop that does the recieve (Perhaps spawn a process to handle a request e.g.). It's all about how you get back to receiving the next message...
Although I would say... don't solve this problem unless you encounter it.
/M |
|
|
| Back to top |
|
| Mazen |
| Posted: Thu Oct 09, 2008 7:39 am |
|
|
|
User
Joined: 20 Jul 2006
Posts: 164
Location: London
|
If you get too long queues your problem is more likely in your design. Look at how you can avoid doing a lot of time consuming work in the process loop that does the recieve (Perhaps spawn a process to handle a request e.g.). It's all about how you get back to receiving the next message...
Although I would say... don't solve this problem unless you encounter it.
/M |
|
|
| Back to top |
|
| dsmith |
| Posted: Fri Oct 10, 2008 6:31 am |
|
|
|
User
Joined: 08 Aug 2007
Posts: 41
Location: Toronto
|
There are many strategies. Some have been mentioned by others, such as discarding messages or starting more handling processes.
Clearly your scenario involve a message cast (i.e. asynchronous messaging) in which you just want to send a message without waiting for a message to be fully processed.
With this in mind, another option is to implement a blocking cast using rpc calls to an intermediate process. This should be possible, although I wouldn't use it often. |
|
|
| Back to top |
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
