A managed service has a clear contract for success, failure and recovery. Write down who owns each operation, what it depends on and how long the caller can wait. Then test those decisions by deliberately failing a dependency in a controlled environment.
Inventory one complete operation
Choose an operation such as “show order status” and follow it through the edge service, application, database and any remote systems. For each boundary record owner, protocol, authentication method, timeout, retry owner, expected traffic and a useful correlation identifier. This applies to HTTP APIs and to older SOAP services; SOAP also requires attention to its message and fault contract.
In a mixed estate, use the integration contract to separate Java, .NET and legacy implementation details. A shared transport alone does not standardize error meanings, identity or transaction behavior. Start with one representative successful response and one failure response that both sides can interpret.
Separate being alive from being ready
On narrow screens, scroll this table horizontally. Keyboard users can focus it and use the arrow keys.
| Check | Purpose | Design question |
|---|---|---|
| Liveness | Can this process continue functioning? | Would restarting actually address the failed condition? |
| Readiness | Should new traffic be routed here? | Which dependencies are essential to this instance serving requests? |
| Operation result | Did the requested business action succeed? | Can the caller distinguish rejection, temporary failure and an unknown outcome? |
| Recovery signal | Has useful service returned? | What evidence permits traffic or retries to resume? |
Kubernetes probe guidance explains distinct liveness, readiness and startup checks in that environment. The distinction is useful even with another supervisor. A shared database outage should not automatically cause every application process to restart in a loop. Keep health endpoints inexpensive and avoid exposing secrets or internal topology in public responses.
Allocate waiting time explicitly
Suppose an invented internal lookup has a 1,000-millisecond dependency budget. Allow at most two attempts, each capped at 400 milliseconds, with one 100-millisecond delay between them. The worst planned allocation is 400 + 100 + 400 = 900 milliseconds, leaving 100 milliseconds within that budget. Your end-to-end request still needs time for its own processing, queues and response handling.
Before each attempt, recalculate the remaining budget from a monotonic clock. Use the smaller of the attempt cap and remaining time, and propagate cancellation. A client read timeout alone may not bound every stage of a multi-step operation. The network programming guide explains connection and read boundaries.
Download the source, checks and README (ZIP)
javac RetryLab.java
java RetryLabsuccess after 2 attempt(s), 300 ms
timeout at 580 ms
stop on HTTP 401The first scenario spends 80 milliseconds on a failed response, waits 100, then succeeds in 120: 300 total. The second attempt in the next scenario needs 600 milliseconds but gets 400, so the model stops at 580. An authorization failure stops immediately. The model advances simulated time; it tests policy arithmetic and does not measure a network client.
Decide when repeating an operation is safe
Retry only selected transient outcomes and only where repeating the operation has defined semantics. HTTP’s idempotency rules concern the intended effect of repeated requests. A timed-out write can already have committed; blindly resending it can create a second action.
For a write API that supports idempotency, retain an operation identifier and bind it to the authenticated caller and request payload. The server must record the effect and deduplication outcome consistently. Keep a status lookup for uncertain results. For overload, respect the service’s retry guidance and budget, add jitter in a real deployment, and bound total attempts. If every layer retries three times, nested retry policies can multiply work dramatically.
A circuit breaker can stop repeatedly calling a failing dependency and later permit limited recovery probes. Define fallback semantics: showing a clearly labeled cached status may be acceptable, while silently reporting an unconfirmed transaction as successful is not.
Test failure and recovery before relying on them
In a controlled environment, inject connection refusal, a delayed response, HTTP 503, authentication failure and a response lost after a committed write. For each, record elapsed time, attempts, caller-visible result and any duplicate effects. Restore the dependency and verify recovery without a retry surge.
Link alerts to a runbook with an owner, first inspection step and rollback decision. Preserve trace context where supported and compare failures with service-level measurements. A successful health response demonstrates that check’s condition; a representative operation test establishes whether the user’s task is working.