Question

I was going through the rpc semantics, at-least-once and at-most-once semantics, how does they work?

Couldn’t understand the concept of their implementation.

Answer 1

In both cases, the goal is to invoke the function once. However, the difference is in their failure modes. In “at-least-once”, the system will retry on failure until it knows that the function was successfully invoked, while “at-most-once” will not attempt a retry (or will ensure that there is a negative acknowledgement of the invocation before retrying).

As to how these are implemented, this can vary, but the pseudo-code might look like this:

At least once:
    request_received = false
    while not request_received:
       send RPC
       wait for acknowledgement with timeout
       if acknowledgment received and acknowledgement.is_successful:
          request_received = true


At most once:
   request_sent = false
   while not request_sent:
      send RPC
      request_sent = true
      wait for acknowledgement with timeout
      if acknowledgment received and not acknowledgement.is_successful:
         request_sent = false

An example case where you want to do “at-most-once” would be something like payments (you wouldn’t want to accidentally bill someone’s credit card twice), where an example case of “at-least-once” would be something like updating a database with a particular value (if you happen to write the same value to the database twice in a row, that really isn’t going to have any effect on anything). You almost always want to use “at-least-once” for non-mutating (a.k.a. idempotent) operations; by contrast, most mutating operations (or at least ones that incrementally mutate the state and are thus dependent on the current/prior state when applying the mutation) would need “at-most-once”.

I should add that it is fairly common to implement “at most once” semantics on top of an “at least once” system by including an identifier in the body of the RPC that uniquely identifies it and by ensuring on the server that each ID seen by the system is processed only once. You can think of the sequence numbers in TCP packets (ensuring the packets are delivered once and in order) as a special case of this pattern. This approach, however, can be somewhat challenging to implement correctly on distributed systems where retries of the same RPC could arrive at two separate computers running the same server software. (One technique for dealing with this is to record the transaction where the RPC is received, but then to aggregate and deduplicate these records using a centralized system before redistributing the requests inside the system for further processing; another technique is to opportunistically process the RPC, but to reconcile/restore/rollback state when synchronization between the servers eventually detects this duplication… this approach would probably not fly for payments, but it can be useful in other situations like forum posts).