When we do a connection to a backend which fails, it is typically desirable to not retry immediately (to avoid flooding the network or the server with requests) and instead do some form of exponential backoff.
We have several parameters:
Exponentially back off the start time of connection attempts up to a limit of MAX_BACKOFF.
ConnectWithBackoff() current_backoff = INITIAL_BACKOFF current_deadline = now() + INITIAL_BACKOFF while (TryConnect(Max(current_deadline, MIN_CONNECT_TIMEOUT)) != SUCCESS) SleepUntil(current_deadline) current_backoff = Min(current_backoff * MULTIPLIER, MAX_BACKOFF) current_deadline = now() + current_backoff
Exponentially increase up to a limit of MAX_BACKOFF the intervals between connection attempts. This is what stubby 2 uses, and is equivalent if TryConnect() fails instantly.
LegacyConnectWithBackoff() current_backoff = INITIAL_BACKOFF while (TryConnect(MIN_CONNECT_TIMEOUT) != SUCCESS) SleepFor(current_backoff) current_backoff = Min(current_backoff * MULTIPLIER, MAX_BACKOFF)
The grpc C implementation currently uses this approach with an initial backoff of 1 second, multiplier of 2, and maximum backoff of 120 seconds. (This will change)
Stubby, or at least rpc2, uses exactly this algorithm with an initial backoff of 1 second, multiplier of 1.2, and a maximum backoff of 120 seconds.