| Author |
Message |
|
| johnm at artisan.com |
 Posted: Tue Apr 13, 1999 6:35 pm |
 |
|
|
Guest
|
Hi,
I hope this is the right form for a beginner's question.
Concurrent Programming and Erlang are new to me. I have read through
the book and implemented some trivial programs. Now I would like to
implement something useful. I am not sure of how to best architect
these applications using Erlang and am looking for suggestions. Also,
I am not yet familiar with all of the available libraries and so I may
be missing some big picture. I am looking for are some productive
hints. I want to code the applications myself.
The first application I could use would limit the number of
simultaneous jobs to run. In particular, we have limited number of
software licenses. I'd like to start up a server and give it a list
of jobs queues and their simultaneous limits. Throughout the network
whenever someone wants to run job which uses one of those licenses
they communicate with the server. If a license is not available then
the job blocks until the license is available.
I already have a program that does this written in ocaml but it uses
threads. Although the threads solution is simple I would prefer to
not use them as threads do not allow "introspection". I would prefer
to maintain an explicit queue. This way the state of the queue could
be printed and all of the running and pending jobs could be printed.
As an off-shoot of the first application I would then like to write a
batch queue system. Pretty much the only difference is that a general
batch queuing system would have a smarter server. Furthermore that
server would invoke the job to run and not just count jobs. I think
if i do a good job of writing the job limiter the batch queue would
not be much more complex but just more complicated.
Sorry for bothering but if anyone has any architectural ideas I'd be
very interested in reading them and experimenting with them.
PS - for more information of ocaml and threads see
http://pauillac.inria.fr/ocaml/htmlman/node22.html
Post generated using Mail2Forum (http://m2f.sourceforge.net) |
|
|
| Back to top |
|
| klacke at erix.ericsson.s |
| Posted: Tue Apr 13, 1999 10:34 pm |
|
|
|
Guest
|
Hello,
> be missing some big picture. I am looking for are some productive
> hints. I want to code the applications myself.
>
Ok, I'll just indicate the direction where you should look.
> The first application I could use would limit the number of
> simultaneous jobs to run. In particular, we have limited number of
> software licenses. I'd like to start up a server and give it a list
> of jobs queues and their simultaneous limits.
This means creating sockets, this is done with gen_tcp.erl
> Throughout the network
> whenever someone wants to run job which uses one of those licenses
> they communicate with the server. If a license is not available then
> the job blocks until the license is available.
Typically you would want a concurrent/threaded server here, i.e one
erlang process per opened socket. There are examples of
concurrent tcp/ip servers in the Examples section at
http://www.erlang.org/examples/klacke_examples/index.html
The toplevel process, the one that creates the socket processes could
typically also be responsible for your queues
>
> I already have a program that does this written in ocaml but it uses
> threads. Although the threads solution is simple I would prefer to
> not use them as threads do not allow "introspection".
Erlang process can't be introspected either, you need to send
a message to the process and get a reply, but that is all natural
to the erlang programmer.
> I would prefer
> to maintain an explicit queue. This way the state of the queue could
> be printed and all of the running and pending jobs could be printed.
This would be done by sending a message to the toplevel
process and have it report the situation.
The queue is best implemented by the queue.erl module
>
> As an off-shoot of the first application I would then like to write a
> batch queue system. Pretty much the only difference is that a general
> batch queuing system would have a smarter server. Furthermore that
> server would invoke the job to run and not just count jobs. I think
> if i do a good job of writing the job limiter the batch queue would
> not be much more complex but just more complicated.
No,
Last but not least, try
% erl -man queue
or
% erl -man gen_tcp
Which at least on a unix system is a convenient way to view the man pages.
Cheers
/klacke
Post generated using Mail2Forum (http://m2f.sourceforge.net) |
|
|
| Back to top |
|
| joe at erix.ericsson.se |
| Posted: Wed Apr 14, 1999 6:49 am |
|
|
|
Guest
|
On Wed, 14 Apr 1999, Claes Wikstrom wrote:
>
> Hello,
>
> > be missing some big picture. I am looking for are some productive
> > hints. I want to code the applications myself.
> >
>
> Ok, I'll just indicate the direction where you should look.
>
> > The first application I could use would limit the number of
> > simultaneous jobs to run. In particular, we have limited number of
> > software licenses. I'd like to start up a server and give it a list
> > of jobs queues and their simultaneous limits.
>
> This means creating sockets, this is done with gen_tcp.erl
Hang on, this implies a design decision that is not yet made.
You have to decide:
1) Server in Erlang + clients in Erlang + Distributed Erlang for
communication, or,
2) Server in Erlang + clients in Erlang + socket communication,
or,
3) Server in Erlang + clients in (C, C++, ...) + socket communication
+ API for (C, C++, ..) client side applications
These are in order of complexity (simplest first).
1) would be OK for a quick prototype to get the protocols ok
3) would be for a commercial product that you could sell
2) is a half way house to implementing 3)
>
> > Throughout the network
> > whenever someone wants to run job which uses one of those licenses
> > they communicate with the server. If a license is not available then
> > the job blocks until the license is available.
I think the key architectural/design problem is one of deciding
what you want to do in the event of failure. If the server crashes,
then all the clients are blocked. If there is a communication failure
then you may loose licenses etc.
Here you can virtually design whatever you want.
In a small (closed) system - then maybe there should be no server -
the clients could alternatively take on the roles of client or server and
use a broadcast/lock strategy to negotiate licenses.
In a commercial system all nodes might not be equal. One server
(placed on a reliable node) might service hundreds of clients, etc. -
you still might want to have some hot-standby/fail-over behaviour for
the server,...
This kind of stuff soon gets complicated (but that's OK - our
telecomms stuff is like this :-)
Firstly I'd, like to see a simple "ball park" analysis of the
problem, in terms of;
a) How many clients
b) How many servers
c) Holding times (how long does a client use a license)
d) Reliability requirements.
- Is it acceptable that clients block if the server crashes?
- Do you want hot standby for server crashes?
e) is this a LAN/WAN application?
f) required response times for obtaining/freeing a license
g) security levels (none, ... full) How much effort do you
want to put into making sure that you cannot forge a license
(you can have anything up to a full public/private key system)
h) Maintaince levels (do you want a remote management system)
If so what ...
Once you have some idea of the answers to questions like these you
can *begin* to think about an architecture.
It may be that you have a very specific set of answers - fine then
we can talk architectures. Or, you might want to "grow" a solution for
a very simple idealized system.
/Joe
--
Joe Armstrong Computer Science Laboratory +46 8 719 9452
AT2/ETX/DN/SU Ericsson Telecom SE-126 25 Stockholm Sweden
joe_at_cslab.ericsson.se http://www.ericsson.se/cslab/~joe
Post generated using Mail2Forum (http://m2f.sourceforge.net) |
|
|
| Back to top |
|
| johnm at artisan.com |
| Posted: Wed Apr 14, 1999 3:21 pm |
|
|
|
Guest
|
Hi Joe,
Thanks for the response. It looks like there does not exist a
gen_batch_queue or gen_software_license module which would immediately
solve my problem. :)
Getting on to more specifications of the applications let me first
state the following. I have no intention of selling the programs or
running them outside of the company where i work. If they turn out to
be generically useful I'd gratefully make them available to the public
community.
I'm not sure how much work is involved in developing a robust (crash
resistant) and reasonably efficient application. I am curious about
your remarks
> You have to decide:
>
> 1) Server in Erlang + clients in Erlang + Distributed Erlang for
> communication, or,
>
> 2) Server in Erlang + clients in Erlang + socket communication,
> or,
>
> 3) Server in Erlang + clients in (C, C++, ...) + socket communication
> + API for (C, C++, ..) client side applications
>
> These are in order of complexity (simplest first).
>
> 1) would be OK for a quick prototype to get the protocols ok
> 3) would be for a commercial product that you could sell
> 2) is a half way house to implementing 3)
I figured that implementation #1 would minimize the amount of my work.
Why is option #2 "better" than #1? I presume you are saying that #3
is good for a commercial product as the end-user could interface to it
using C and not Erlang. When you mention option #2 are you suggesting
that there is some good reason for using socket communication in
preference to Distributed Erlang communication?
> I think the key architectural/design problem is one of deciding
> what you want to do in the event of failure. If the server crashes,
> then all the clients are blocked. If there is a communication failure
> then you may loose licenses etc.
I agree. This may sound selfish but I would like to implement a
robust distributed application because it sounds like fun and i have
never written one before.
> Here you can virtually design whatever you want.
Darn, this is where i was hoping for a gen_software_license module :)
> In a small (closed) system - then maybe there should be no server -
> the clients could alternatively take on the roles of client or server and
> use a broadcast/lock strategy to negotiate licenses.
Interesting.
> In a commercial system all nodes might not be equal. One server
> (placed on a reliable node) might service hundreds of clients, etc. -
> you still might want to have some hot-standby/fail-over behaviour for
> the server,...
>
> This kind of stuff soon gets complicated (but that's OK - our
> telecomms stuff is like this :-)
Since this is not a commercial system i have a lot of flexibility
here. I am not familiar with the trade-offs. I presume that the more
robust the system the more work it will take to get it written.
> Firstly I'd, like to see a simple "ball park" analysis of the
> problem, in terms of;
>
> a) How many clients
We have around 100 workstations at our company. This sounds like a
nice number.
> b) How many servers
Other than for reliability or performance i see no reason why the
answer could not be 1. Furthermore, I see no reason why the server
could not run on all of the machines -- all 100 of them.
> c) Holding times (how long does a client use a license)
Rarely less than 10 seconds. Normally a minute or two. There may
also be interactive use where the license could be held for hours at a
time. There may also be large batch jobs where the license is held
for some hours.
> d) Reliability requirements.
> - Is it acceptable that clients block if the server crashes?
> - Do you want hot standby for server crashes?
Yes and yes. I'm not familiar with the coding and performance impact
of having a hot standby. I have never implemented anything like this
before.
> e) is this a LAN/WAN application?
LAN only.
> f) required response times for obtaining/freeing a license
Less than 1 second.
> g) security levels (none, ... full) How much effort do you
> want to put into making sure that you cannot forge a license
> (you can have anything up to a full public/private key system)
None. Environment is completely secure. (Ha!)
> h) Maintaince levels (do you want a remote management system)
> If so what ...
I'm not sure what remote means but yes i do want the ability to
dynamically increase or decrease license counts, add new queues, shut
down queues, etc. This is one of the reasons Erlang intrigued me as
it seems that you guys have familiarity with these kinds of
applications.
> Once you have some idea of the answers to questions like these you
> can *begin* to think about an architecture.
>
> It may be that you have a very specific set of answers - fine then
> we can talk architectures. Or, you might want to "grow" a solution for
> a very simple idealized system.
I'm in no rush. I like the idea of having an architecture in mind,
coding a pilot version, getting feel for how well it will work, making
changes and then when happy adding all the bells and whistles.
By the way, here is a more detailed description of how my license
client and server work. This works well enough in the environment
which i have described above.
My present implementation of the license client in ocaml is quite
small. When the user wants to execute a command on a named license
queue, the user enters
license_client name command
The license_client program communicates with the server by sending it
the license name. The license_client program then does a read and
blocks. When the server decides that the named license is available
to run it writes to the client. The pending license_client read now
succeeds and then the license_client executes the desired user
command.
Once the user command finishes the license_client closes the
connection with the server. The server now knows that the license is
available. This approach works well as licenses are rarely lost even
if the client is killed or crashes. If the server goes down i force
the clients to wait. If the server goes down in the middle of a
transaction then there is a problem. This is an area for improvement.
Since the present implementation of the server uses threads it is
convenient to only remember the currently running processes. It
messes up the code quite a bit to create a queue of pending processes.
This is however a desirable feature and so i would like to add this in
the next implementation.
Post generated using Mail2Forum (http://m2f.sourceforge.net) |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
