Wire protocol
Definitions
The following terms are used in this document:
-
server: A dqlite node.
-
client: Either application code (typically wanting to issue database queries) or a dqlite node (typically pushing replicational data).
-
connection: A TCP or Unix socket connection established by the client against a server.
-
word: A sequence of 8 bytes.
-
protocol version: A positive number stored in a word using little endian representation.
-
message: A sequence of bytes sent either by the client to the server or by the server to the client. It consists of a header and a body. The header consists of a single word with the following layout:
-
byte 0 to 3: Size of the message body, expressed in words and stored using little endian representation. For example a value of [2 1 0 0] means that the message body consists of 258 bytes.
-
byte 4: Type code identifiying the schema of the message body.
-
byte 5: Revision number of the message body schema. Within the same protocol version, a new schema revision can only add fields to the previous revision. So a client capable of understanding up to revision N of a certain message schema can still handle messages with revision >N by simply ignoring the extra bytes.
-
byte 6 to 7: Currently unused.
The message body is a sequence of fields as described by the associated schema. Message types and their schemas are listed below.
-
Setup
As soon as a connection is established, the client must send to the server a single word containing the protocol version it wishes to use.
Conversation
After the setup, communication between client and server happens by message exchange. Typically the client will send to the server a message containing a request and the server will send to the client a message containing a response.
Data types
Each field in a message body has a specific data type, as described in the message schema. Available data types are:
uint64
A single word containing an unsigned integer in little endian representation.
int64
A single word containing a two-complement signed integer in little endian representation.
uint32
Four bytes containing an unsigned integer in little endian representation.
text
A sequence of one or more words containing a UTF-8 encoded zero-terminated string. All bytes past the terminating zero byte are zeroed as well.
tuple
A tuple is composed by a header and a body.
The format of the header changes depending on whether the tuple is a sequence of parameters to bind to a statement, or a sequence of values of a single row yielded by a query.
For a tuple of parameters the format of the header is:
| Size | Content |
|---|---|
| 8 bits | Number of values in the tuple |
| 8 bits | Type code of the 1st value of the tuple |
| 8 bits | Type code of the 2nd value of the tuple, or 0 |
| 8 bits | Type code of the 3rd value of the tuple, or 0 |
| … |
This repeats until reaching a full 64-bit word. If there are more than 7 parameters to bind, the header will grow additional 64-bit words as needed, following the same pattern: a sequence of 8-bit slots with type codes of the parameters followed by a sequence of zero bits, until word boundary is reached.
For a tuple of row values the format of the header is:
| Size | Content |
|---|---|
| 4 bits | Type code of the 1st value of the tuple |
| 4 bits | Type code of the 2nd value of the tuple, or 0 |
| 4 bits | Type code of the 3rd value of the tuple, or 0 |
| … |
This repeats until reaching a full 64-bit word. If there are more than 16 values, the header will grow additional 64-bit words as needed, following the same pattern: a sequence of 4-bit slots with type codes of the values followed by a sequence of zero bits, until word boundary is reached.
After the header the body follows immediately, which contains all parameters or values in sequence, encoded using type-specific rules.
The codes of available types for tuple values and the associated encoding formats are:
| Code | Value |
|---|---|
| 1 | Integer value stored using the int64 encoding |
| 2 | An IEEE 754 floating point number stored in a single word (little endian) |
| 3 | A string value using the text encoding |
| 4 | A binary blob: the first word of the value is the length of the blob (little endian) |
| 5 | A SQL NULL value encoded as a zeroed word |
| 10 | An ISO-8601 date value using the text encoding |
| 11 | A boolean value using uint64 encoding (0 for false and 1 for true) |
node-info
Information about a node in the cluster. It consists of the node ID (in uint64 encoding) followed by the node address (in text encoding).
file
A single database file. It consists of the file name (in text encoding), followed by the file size (in uint64 encoding) and finally a blob with the file content.
Client messages
The client can send to the server messages with the following type codes and associated schemas:
0 - Get current leader
| Type | Value |
|---|---|
| uint64 | Unused field |
1 - Client registration
| Type | Value |
|---|---|
| uint64 | ID of the client |
3 - Open a database
| Type | Value |
|---|---|
| text | The name of the database |
| uint64 | Currently unused |
| text | Currently unused |
0 - Get current leader
| Type | Value |
|---|---|
| uint64 | Unused field |
1 - Client registration
| Type | Value |
|---|---|
| uint64 | ID of the client |
3 - Open a database
| Type | Value |
|---|---|
| text | The name of the database |
| uint64 | Currently unused |
| text | Currently unused |
4 - Prepare a statement
| Type | Value |
|---|---|
| uint64 | ID of the open database to use |
| text | SQL text of the statement |
5 - Execute a prepared statement
| Type | Value |
|---|---|
| uint32 | ID of the open database to use |
| uint32 | ID of the prepared statement to execute |
| tuple | A tuple of parameters to bind to the prepared statement |
6 - Execute a prepared statement yielding rows
| Type | Value |
|---|---|
| uint32 | ID of the open database to use |
| uint32 | ID of the prepared statement to execute |
| tuple | A tuple of parameters to bind to the prepared statement |
7 - Finalize a prepared statement
| Type | Value |
|---|---|
| uint32 | ID of the open database to use |
| uint32 | ID of the prepared statement to finalize |
8 - Execute a SQL text
| Type | Value |
|---|---|
| uint64 | ID of the open database to use |
| text | SQL text to execute |
| tuple | A tuple of parameters to bind |
9 - Execute a SQL text yielding rows
| Type | Value |
|---|---|
| uint64 | ID of the open database to use |
| text | SQL text to execute |
| tuple | A tuple of parameters to bind |
10 - Interrupt the execution of a statement yielding rows
| Type | Value |
|---|---|
| uint64 | ID of the open database currently executing the query |
11 - Start pushing replication data
| Type | Value |
|---|---|
| node-info | ID and address of the node pushing the data |
12 - Add a non-voting node to the cluster
| Type | Value |
|---|---|
| node-info | ID and address of the node to add |
13 - Promote a non-voting node to voting
| Type | Value |
|---|---|
| uint64 | ID of the node to promote |
14 - Remove a node from the cluster
| Type | Value |
|---|---|
| uint64 | ID of the node to remove |
15 - Dump the content of a database
| Type | Value |
|---|---|
| text | Name of the database to dump |
16 - List all nodes of the cluster
| Type | Value |
|---|---|
| uint64 | Currently unused |
Server messages
The server can send to the client messages with the following type codes and associated schemas:
0 - Failure response
| Type | Value |
|---|---|
| uint64 | Code identifying the failure type |
| text | Human-readable failure message |
1 - Node information
| Type | Value |
|---|---|
| node-info | Information about a single node |
2 - Welcome
| Type | Value |
|---|---|
| uint64 | Currently unused |
3 - Cluster information
| Type | Value |
|---|---|
| node-info | First node |
| node-info | Second node (if any) |
| … |
4 - Database information
| Type | Value |
|---|---|
| uint32 | Database ID |
| uint32 | Unused |
5 - Prepared statement information
| Type | Value |
|---|---|
| uint32 | Database ID |
| uint32 | Statement ID |
| uint64 | Number of parameters |
6 - Statement execution result
| Type | Value |
|---|---|
| uint64 | ID of last row inserted, or 0 |
| uint64 | Number of rows affected or 0 |
7 - Batch of table rows
| Type | Value |
|---|---|
| tuple | Column values of the first row in the batch |
| tuple | Column values of the second row in the batch (if any) |
| … | |
| uint64 | End marker |
The end marker is the value 0xffffffffffffffff if the statement currently
yielding rows has completed and there are no more rows, or otherwise
0xeeeeeeeeeeeeeeee if there are more rows and another batch will be sent.
8 - Aknowledgment
| Type | Value |
|---|---|
| uint64 | Unused |
9 - Database files
| Type | Value |
|---|---|
| file | Main database file |
| file | Write-ahead log file |