Recently our client FADU
bakery shop decided to limit the caching to minimal to cut the cost
in software system. Uhh! The web application is no more responsive now :(. In our web app several of pages are populated with the
data retrieved from services deployed on external server. With the
minimal caching, most of the hits to pages will causes round-trip to
the services. This connection to services is configured with timeout
period of 60 sec and if the service doe not respond in this duration
web page will assume either the service in unavailable or some
exception.
Moreover, if service call
fails, user could be forced to wait for 60 sec before an exception
occur and resources such as memory, connections and threads could
exhausted preventing other user to connect to system.
Adding more web servers and
load balancing may delay the point at which resource become
exhausted but it will not solve the problem.
Is not it be dreamy if there
is something that handle the failure more elegantly? This is where
the circuit breaker comes in play where we wrap the logic to connect
to service and retrieve data.
Circuit breaker is a pattern
that wraps the failure prone resource call and monitor for error.
Circuit breaker comes with three states as the implementation is
stateful. The states are close, Open and Half-open. Initially the
circuit breaker is in closed state and passed all request to wrapped
object. Many request to object may fails and once failure reaches the threshold count in fixed interval the circuit will move to open state
where it returns default response or error to caller without
actually calling to wrapped object. This prevents from overloading
the already failing resource. While after some time the failure may
recover. The recovery is handled externally, possibly by restoring or
restarting the failure component or repair the network or other ways.
Now, we need a mechanism to detect if the failure is recovered.
This is where the third state called half-open state comes in play.
This is reached after some duration of failure. At this time again
the request is sent to wrapped object but the result of call is
important. If the call is successful the wrapped object is recovered
and the circuit breaker moved to closed state again! If the call is
fails the timeout is reset and circuit move to open state.
There are some standard open source implementation like Apache Camel, Microsoft API for circuit breaker or Netflix implementation are available. If you are accessing the
resources that likely to fail wrap the resource to circuit breaker
and enjoy the responsiveness even in case of failure. Is not it
magical?
