.%*.                                                        .-.
               .%@@@+.                                                .--=%@@@-
               =@@@@@@-                                         :--+@@@@@@@@@*
               *@%@@@@@%:                               :--=#@@@@@@@@@@@@#@@+
               @@::%@@@@@@-:                   :::-=%@@@@@@@@@@@@@@%#*-  :@*
              .@@   =@@@@@@@@@+-:::::::-=*@@@@@@@@@@@@@@@@@@@%##-        @@:
              #@#     +%@@@@@@@@@@@@@@@@@@@@@@@@@@@@@%%#*:              -@@
              #@-        *%@@@@@@@@@@@@@@@@@%%%#=                       %@=
              @@:             -=+++==     ..                            @@:
              @@        .               -@@@                           +@@
             #@%     =@@@@-            +@@@@#              *+          %@#
             #@-   -@@@@@@@.          +@@@@@%           .%@@@@=        @@:
             @@:  +@@@@@@@@@         +@+ +@@@         %@@@@@@@@@      -@@
             @@  *@%@*  :@@@        *@+  -@@@       %@@@@@@@@@@@#     @@@
            #@% %@ @%    %@@       *@*   .@@@     =@@@*@@   :@@@@     @@:
            %@:## *@+    :@@      :@@     @@@    %@@: @#     .@@@    .@@
            @@:.  @@:    .@@      @@      @@@   %@:  @@       @@%    @@%
            @@   -@@.    =@@     @@.      @@@  %@.  #@-       @@#    @@.
           %@%   +@@.    @@.    %@-       @@@ #@    @@.      :@@    -@@
           %@:   +@@+   .@%    #@=        @@@.@    *@@       +@+    @@*
           @@:   +@@%. .%%    *@+         @@@      %@@      -@#     @@
           @@    :@@@@@@%    :@@          @@%      %@@      @#     *@@
          %@@     #@@@@*     @@           @@%      %@@.    %@.     @@:
          @@:      ===:     @@.           @@%      %@@:  =@%.     .@@
          @@.              #@=            @@%      %@@%+%@#.      @@%
          @@              +@*             @@#      -@@@@@*       .@@
         %@@             :@@              @@#       %@@#:        =@@
         @@:            .@@               @@#        :           @@=
         @@.            @@.               @@#                   :@@
         @@            #@=                @@%                   %@%
        #@@           +@+                 #@%                  .@@
        @@:          =@#                  +@%                  =@%
        @@          :@%                   -@@                  @@+
        @@         .@@                    -@:                 -@@
       -@@         @%                     .:                  *@#
       @@*         #                                          @@.
       @@                                                    =@@
      .@@                                                    @@*
      .@@                                                   .@@
      *@@                                                   =@#
      @@*                                                   @@*
      @@                                                   -@@
     .@@                                                   +@*
     .@@                                                   @@-
     -@@                                                  =@@
     %@%                                                  #@*
     @@                                                   @@:
     @@                                                  +@@
    .@@                                                  %@+
    :@%                                                  @@.
    :@%                                                 *@%
    -@.                                                 %@=
    %@                                                  @@.
    @@                                                 *@%
    @@                                                 %@=
    @%                                                 @@:
   :@%                                                +@@
   :@*                =+-                             #@+
   :@:              .%@@@.                            @@-
   -@              =@@@@@#                           :@@
   =@             *@#.=@@%                           #@%
   *@            #@-  -@@%                           %@-
   @@           #@:   :@@%                           @@:
   @@          #%     .@@%               .#@@*      -@@
   @%         #%      .@@%              +@@@@@-     %@%
   @#        %@       :@@%            .@@@@@@@@     %@:
   @#       %@        :@@%           +@@% .@@@@     @@:
   @#      %@         :@@%          #@@:   :@@@    -@@
   @#     @@.         .@@@        .@@%     .@@@=   %@%
   @#    @@.          .@@@:      %@@-       @@@%   %@:
   @#   @@             @@@@    %@@@         *@@@   @@:
   @# .@@              @@@@@@@@@@=          :@@@   @@
   * @@@               :@@@@@@@@.           :@@@  *@@
.@@@@@%                 -@@@@@.             .@@@  @@=
@@@@@:                    ..                 @@@  @@.
-@%:                                         @@@  @@.
                                             @@@  @@
                                             -@@ =@@
                                             .@@ @@+
                                             .@@:@@.
                                              @@#@-
                                              @@@.
                                              +@-
                                               -
      
mediocregopher's lil web corner

Home  //  Posts  /  Follow  /  RSS  //  Source  /  License

RESP as a General Purpose Serialization Protocol


For the past six thousand years I have been the maintainer of a medium-popularity driver for the Redis database.

Radix: the best Redis driver for Go, objectively

But this post isn't going to be about Redis, really, or even about Go. It's going to be about Redis's serialization protocol: RESP.

RESP3 specification

Side note on the versioning: the RESP protocol is currently on its 3rd version, but the 2nd version is still widely used and is the default protocol that the Redis database will use unless you specify otherwise. Version 3 is more of an extension to 2, there's only one mildly incompatible change.

Why do I want to talk about Redis's serialization protocol? Well... cause it's actually a pretty good, general purpose serialization protocol, both for the transfer of data over the wire and storage at rest. There is nothing Redis-specific in its design, and I've taken to using it in place of JSON in some of my personal projects.

If you're interested in the protocol itself and what it looks like I recommend looking through the spec, linked above. It's quite easy to understand. If you've used Redis at all then you're probably already somewhat familiar with the types and datastructures involved, as they are referenced directly in the documentation for each command and probably by your redis driver itself (unless it supports generic marshaling/unmarshaling, like mine ;) ).

High Level Description

At a high-level RESP has the following properties:

And you know that RESP3 is going to be speedy, because it's used by one of the most speed conscious databases out there. My implementation in Go manages to avoid allocations in almost all cases, which I credit to RESP's design.

A Quick Dive

There's a specific property of the protocol which is worth mentioning, as I think it's indicative of the design of the protocol at large. That property is this: integers are encoded simply as their ASCII-encoded form, with a predefined prefix and suffix to bound them. For example, the integer 1234 would be:

:1234\r\n

This may seem counter-intuitive, given that it's a binary-safe protocol. It's even more counter-intuitive when you note that ASCII-encoded integers are also used as the length prefix of blob (binary) strings:

$5\r\nHello\r\n

(The 5 in there is the length of the string Hello).

Here's the thing: how many bytes does it take to represent the string 5? Answer: 1. How about the string 8000? Now you're up to 4. If we were to encode the string length as a 64-bit integer that would be 8 bytes. So until your string is 10000000 bytes long you're actually saving bytes by just using the ASCII-encoding. There's no significant loss in speed either, since it's quite trivial to turn an ASCII string into an integer via some bit-twiddling. You get the benefits of a traditional varint, but with a huge savings in simplicity.

The Rough Edges

There's no real gotchas or deal-breakers, but just some things of note which I could point out.

There are a few types and datastructures (besides the ones I noted) that are more specific to a client/server communication protocol, like errors and attributes. Whether or not these are useful is up to your use-case, but you certainly don't have to use them.

While the representation is human-readable for debugging purposes, I wouldn't describe it as human-writeable. Therefore I wouldn't recommend RESP for something like a configuration file which needs to be interacted with directly by a human. I would instead consider it in places where you might otherwise consider using BSON, protobuf, or MessagePack.

Final Thoughts

My love for Redis comes from something beyond its surface level features and benchmarks. There's a certain attitude of design used which I really admire, a willingness to do something different and odd, and to then find the hidden benefits of that approach and build on them. This attitude permeates the entire design of Redis, its serialization protocol included. You can feel it in decisions like using an ASCII-encoding of an integer as its wire representation. It seems like the wrong move, but in practice it's actually pretty reasonable, and comes with some hidden benefits.

I've learned a lot from antirez, Redis's founder and lead maintainer until relatively recently. His blog is worth perusing through if you want more of his odd perspective:

antirez.com

And I hope that with this you'll appreciate this odd-but-great serialization protocol, and perhaps consider it for your uses as well.

Published 2023-05-29


This post is part of a series.
Previously: Re: The Mindless Tyranny of "What If It Changes?"
Next: How to Errors Good


This site can also be accessed via the gemini protocol: gemini://mediocregopher.com/

What is gemini?