| Author |
Message |
|
| thel |
 Posted: Sun Sep 04, 2011 4:06 pm |
 |
|
|
Joined: 08 Aug 2011
Posts: 2
|
Hi
I am looking into using erlang for a FSM (finite state machine) and i will need to implement a statefull failover mechanism between two physical servers, which approch would you recommend for this? Any Best practice advices?
Thomas
Post received from mailinglist |
|
|
| Back to top |
 
|
| Guest |
| Posted: Sun Sep 04, 2011 4:24 pm |
|
|
|
Guest
|
|
| Back to top |
|
| Guest |
| Posted: Tue Sep 06, 2011 9:09 pm |
|
|
|
Guest
|
Stateful, as in the fail-over needs to be "hot" and "online" and replicating the state of the first application faithfully?
The danger with such approaches is that, if the state becomes corrupt through some chain of events, then the replicated copy may also be corrupt, and the "slave" crashes when the "master" crashes. It still works great in case of hardware failure on the master instance, of course.
Sincerely,
jw
--
Americans might object: there is no way we would sacrifice our living standards for the benefit of people in the rest of the world. Nevertheless, whether we get there willingly or not, we shall soon have lower consumption rates, because our present rates are unsustainable.
On Sun, Sep 4, 2011 at 9:06 AM, Thomas Elsgaard <thomas.elsgaard@gmail.com (thomas.elsgaard@gmail.com)> wrote:
Quote: Hi
I am looking into using erlang for a FSM (finite state machine) and i will need to implement a statefull failover mechanism between two physical servers, which approch would you recommend for this? Any Best practice advices?
Thomas
_______________________________________________
erlang-questions mailing list
erlang-questions@erlang.org (erlang-questions@erlang.org)
http://erlang.org/mailman/listinfo/erlang-questions
Post received from mailinglist |
|
|
| Back to top |
|
| uwiger |
| Posted: Wed Sep 07, 2011 7:51 am |
|
|
|
User
Joined: 03 Jul 2006
Posts: 604
Location: Sweden
|
On 6 Sep 2011, at 23:09, Jon Watte wrote:
> Stateful, as in the fail-over needs to be "hot" and "online" and replicating the state of the first application faithfully?
>
> The danger with such approaches is that, if the state becomes corrupt through some chain of events, then the replicated copy may also be corrupt, and the "slave" crashes when the "master" crashes. It still works great in case of hardware failure on the master instance, of course.
You are right. One way to mitigate this is to put some effort into designing a replication format, which is not just mirroring the internal state. Not only will this reduce the likelihood of propagating corrupted state; it will also simplify potential future upgrades and extensions, and make it easier to analyse the traffic flowing between nodes.
One should also think through at which points it is at all meaningful to replicate. I like to refer to "stable-state replication", which doesn't really say anything about the frequency of updates, but rather highlights that there are usually discrete points where recovery from error is meaningful. The transition states between these points tend to be volatile, and replicating them may serve little purpose.
BR,
Ulf W
Ulf Wiger, CTO, Erlang Solutions, Ltd.
http://erlang-solutions.com
_______________________________________________
erlang-questions mailing list
erlang-questions@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Post received from mailinglist |
|
|
| Back to top |
|
| thel |
| Posted: Wed Sep 07, 2011 2:27 pm |
|
|
|
Joined: 08 Aug 2011
Posts: 2
|
On Wed, Sep 7, 2011 at 9:51 AM, Ulf Wiger
<ulf.wiger@erlang-solutions.com> wrote:
>
> On 6 Sep 2011, at 23:09, Jon Watte wrote:
>
>> Stateful, as in the fail-over needs to be "hot" and "online" and replicating the state of the first application faithfully?
>>
>> The danger with such approaches is that, if the state becomes corrupt through some chain of events, then the replicated copy may also be corrupt, and the "slave" crashes when the "master" crashes. It still works great in case of hardware failure on the master instance, of course.
>
> You are right. One way to mitigate this is to put some effort into designing a replication format, which is not just mirroring the internal state. Not only will this reduce the likelihood of propagating corrupted state; it will also simplify potential future upgrades and extensions, and make it easier to analyse the traffic flowing between nodes.
>
> One should also think through at which points it is at all meaningful to replicate. I like to refer to "stable-state replication", which doesn't really say anything about the frequency of updates, but rather highlights that there are usually discrete points where recovery from error is meaningful. The transition states between these points tend to be volatile, and replicating them may serve little purpose.
>
> BR,
> Ulf W
>
> Ulf Wiger, CTO, Erlang Solutions, Ltd.
> http://erlang-solutions.com
>
>
>
>
Hi All
Thanks for the input, there is no easy way  But i will take it into
my considerations
///Thomas
_______________________________________________
erlang-questions mailing list
erlang-questions@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Post received from mailinglist |
|
|
| Back to top |
|
| uwiger |
| Posted: Wed Sep 07, 2011 2:43 pm |
|
|
|
User
Joined: 03 Jul 2006
Posts: 604
Location: Sweden
|
I should also say that some protocols allow for a "recovery window", which further simplifies stable-state replication. I know that some people will overlook this and try for hottest possible redundancy, but in several of the products I've been involved in, this property has been crucial.
If you have resources allocated in hardware (e.g. forwarding engines, DSP:s, etc.), you will need to audit the resource management situation after a failure. If session state information is missing at some level, this may force you to release the entire session, even if most of the data is still there. Traditional Telecoms protocols expect sessions to have a data plane component and a control plane component, and will tolerate temporary loss of the control signaling. Normally, you will have 10-15 seconds to get your house in order and respond sanely to status inquiries.
This is actually part of the secret behind the high availability figures achieved in Telecoms. Even the interaction with end users is designed to be fault-tolerant, giving the serving infrastructure some margin for recovery.
The separation of the rather sensitive data path and the complex but fault-tolerant control plane is also important. Data processing units are kept as simple as possible, and are fault-isolated from the control plane. This model is becoming blurred by the trend towards tighter integration, but as a mental model, it is good to recall what the benefits of separation were.
BR,
Ulf W
On 7 Sep 2011, at 16:27, Thomas Elsgaard wrote:
> On Wed, Sep 7, 2011 at 9:51 AM, Ulf Wiger
> <ulf.wiger@erlang-solutions.com> wrote:
>>
>> On 6 Sep 2011, at 23:09, Jon Watte wrote:
>>
>>> Stateful, as in the fail-over needs to be "hot" and "online" and replicating the state of the first application faithfully?
>>>
>>> The danger with such approaches is that, if the state becomes corrupt through some chain of events, then the replicated copy may also be corrupt, and the "slave" crashes when the "master" crashes. It still works great in case of hardware failure on the master instance, of course.
>>
>> You are right. One way to mitigate this is to put some effort into designing a replication format, which is not just mirroring the internal state. Not only will this reduce the likelihood of propagating corrupted state; it will also simplify potential future upgrades and extensions, and make it easier to analyse the traffic flowing between nodes.
>>
>> One should also think through at which points it is at all meaningful to replicate. I like to refer to "stable-state replication", which doesn't really say anything about the frequency of updates, but rather highlights that there are usually discrete points where recovery from error is meaningful. The transition states between these points tend to be volatile, and replicating them may serve little purpose.
>>
>> BR,
>> Ulf W
>>
>> Ulf Wiger, CTO, Erlang Solutions, Ltd.
>> http://erlang-solutions.com
>>
>>
>>
>>
>
> Hi All
>
> Thanks for the input, there is no easy way But i will take it into
> my considerations
>
> ///Thomas
Ulf Wiger, CTO, Erlang Solutions, Ltd.
http://erlang-solutions.com
_______________________________________________
erlang-questions mailing list
erlang-questions@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Wed Sep 07, 2011 9:21 pm |
|
|
|
Guest
|
On Wed, Sep 7, 2011 at 16:42, Ulf Wiger <ulf.wiger@erlang-solutions.com> wrote:
>
> I should also say that some protocols allow for a "recovery window", which further simplifies stable-state replication. I know that some people will overlook this and try for hottest possible redundancy, but in several of the products I've been involved in, this property has been crucial.
>
On another plane, having a smaller stable-state often makes it easier
to verify for correctness. If you have some way to make sure that the
state is stable, then by all means check it! You can often export this
state to another process for periodic verification as well. While it
does not _mend_ the error, it _detects_.
Think about a system managing money. Like a certain Friar back in the
day, Luca Pacioli, your system can do double-entry bookkeeping by
having two processes, each playing the role of a ledger account. This
means that you can check the system for a stable state by cross query
on the ledgers. The stable state is small, namely the end balance of
the account and it is enough for verification.
The basic idea is that your process has an internal state s(), and
your system defines several projections of the form -spec projectionX(
s() ) -> t()., where t() is the type of the projection image. You use
these projections for verification, for state marshalling, for system
inspection and so on. It may be you need several different projections
for different purposes, but often they can be coalesced into a few. As
Ulf mentioned, simple t()'s means easier upgrade paths as well.
Notice that state does not come equal. A lot of the internal state of
a process is not valid if something goes wrong anyway. So there is
little reason to keep it around. Sometimes, you keep state which acts
as a scratch pad for your calculations, most often on the stack. This
is not important either when things crash. Most crashes are due to
state inconsistencies anyway, so keeping a leash on the scratchpad
will definitely make your life worse.
Sometimes, you are lucky and data are self-verifying. Sometimes not.
But thinking about what properties your data will have is good for
several reasons - It also makes it easier to write QuickCheck/ProPer
tests. This is also why there is no easy way to do this. If you want
to make a system fault tolerant, you need two machines as a start. But
then you need to make sure that the right kind of information flows
between the two machines. Erlang will make it easy to transfer
information. But no language can verify the properties of the
information that flows. At least not easily.
--
J.
_______________________________________________
erlang-questions mailing list
erlang-questions@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Post received from mailinglist |
|
|
| Back to top |
|
| wuji |
| Posted: Tue Sep 18, 2012 7:59 am |
|
|
|
User
Joined: 10 Aug 2012
Posts: 654
|
way society perceives unhealthy behaviors, like smoking, drinking and drug drug designer replica *beep* drug use, which used to fall under the general perception
a rebellious "cool."Still, what gives? Why is Slickster Danny Zuko Zuko replica designer *beep* Zuko from "Grease" out and Justin Bieber, the popstar with
boy-next-door charm in?"[Being] cool by definition requires a reference point-- point-- cheap real jordans point-- what is boring, normal, or even uncool," said Lewis.
day culture stops changing is the day our notions of of cheap designer *beep* of coolness will also be frozen in time."Unemployment Unchanged at
Pct, 80,000 Jobs AddedAnother So-So Jobs ReportBy SUSANNA KIM and and cheap designer *beep* and BILL McGUIREJuly 6, 2012— Employers added 80,000 jobs in
and the unemployment rate remained unchanged at 8.2 percent, the the [h4]discount designer *beep*[/h4] the Labor Department announced Friday, in another so-so report on
U.S. economy that promises to frame the debate for the the cheap authentic jordans the fall presidential election.Economists had expected that employers added around
jobs in June, higher than the revised 77,000 jobs added added [h2]discount designer *beep*[/h2] added in May, but lower than what is needed for |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
