Protofish3 is long-lived server-to-server protocol that allows servers to transfer multiple data simultaneously. By leveraging QUIC, thousands of simultaneous data transfer can be done efficiently. Protofish supports selecting one of Unreliable, Reliable, Dual mode for each transfer. The unique feature of Protofish3 is Dual Mode Transfer. By retransmitting only lossed packet, receiving side can access a pair of reliable and unreliable stream. In Zako3, the unreliable side is used for realtime playback to user, and the reliable side is used for audio caching. Aditionally, as the primarily-intended use of Protofish is realtime audio transfer, Protofish3 has credit-based backpressure handling system. Thus the max transfer rate can be capped naturally to the processing capability of the receiver.
Protofish3 is designed to work over QUIC. QUIC natively supports basic unreliable/reliable streams with multiplexing. QUIC also contains TLS, which Protofish isn't get involved.
Conn, Chan, Xfer - meaning Connection, Channel, Transfer - is three fundamental concepts in Protofish. Basically, they are in order of subset. Conn can contain multiple Chans, and Chan can contain multiple Xfers.
Conn is the largest block of the protocol, which exactly follows lifetime of underlying QUIC connection.
Chan is a single stream that data can be transferred through. Chan can run two Xfers: outbound and inbound. XferStart packet from any side initiates Xfer from the side who sent XferStart to the peer. Only one or zero Xfer can exist in one direction. i.e. Max two Xfers can exist in a single Chan.
Additionally, Msg can be sent through Chan. Msg is arbitrary binary data that can be sent in both direction, any time. It can be sent while there is ongoing Xfer.
Xfer is one-shot representation of data moving through Chan. Xfer is separated from Chan to cleanly separate state machine and behaviors of each concepts.
Xfer has three modes: Unrel, Rel, Dual. Unrel does unreliable transfer, Rel does reliable transfer, and Dual does the both. All of them is frame-based, and supports backpressure controlling.
- All of three modes MUST support backpressure controller.
- Rel MUST guarantee integrity of full stream until the end.
- Unrel MUST guarantee the final packet arrives. This is called Final Packet Validation(FPV). FPV actively prevents the very end of the media getting cut off.
Xfer has Idle, Transfering, Finalizing state. It defaults to Idle. When one side triggers XferStart, Xfer becomes Transfering state. And when XferEnd is triggered, It becomes Finalizing state. Finalizing state allows all three modes to do retransmission. And when XferEndAck is triggered by the peer, the Xfer becomes Idle. Additionally, XferRecvEnd can be sent from the receiver side to halt the transfer in progress.
stateDiagram-v2
[*] --> Transfering : XferStart
Transfering --> Finalizing : XferEnd
Transfering --> [*] : XferRecvEnd
Finalizing --> [*] : XferEndAck
In Rel mode, all Xfer is done by XferData packet in the Chan.
In Unrel or Dual mode, basically all packets are transfered by Chan-ID-tagged QUIC datagram(UnrelPacket) by default. UnrelPacket contains Seq Number. Sequence Number is u32, and starts with 1. When the sender sends XferEnd, it MUST include the final Seq Number in the XferEnd packet. This is for Final Packet Validation(FPV). Receiver can validate the final packet is received by checking if the final Seq Number is received.
In Dual mode or Finalizing state, when two Unrel packet with missing Seq Numbers(e.g. 3 4 5 8: 6, 7 are missing) are received, NACK is sent through the Chan with missing Seq Numbers. Then, sender MUST send Retrans packet through the Chan with the content.
ACK packet, which marks the maximum value in a set of continuous received sequence number, MAY sent from receiver to sender, allowing sender to drain retransmission buffer.
Sender MUST maintain retransmission buffer per Xfer. It is RECOMMENDED to explicitly set a maximum capacity of the retransmission buffer to prevent overflow. Packets that guaranteed it's received by ACK can be drained from the retransmission buffer.
Protofish uses credit-based backpressure system to control backpressure in per-Xfer manner. Receiver MUST send XferCreditUpdate through the Chan to allow the sender to send. XferCreditUpdate can be batched. Sender MAY periodically send XferCreditRequest to avoid state drift and hang.
When a client connects to a server, client initiates a Chan. Handshake procedure is done in the very first Chan.
Protofish3 has several frame types. Frame type carries the basic block of the protocol.
u8,u16,u32,u64- unsigned integer with 8, 16, 32, 64 bits.bytes- Arbitrary binary data with length prefix. Length prefix isVarInt.array(T)- Array of type T with length prefix. Length prefix isVarInt.seq_number- Sequence Number. Is u32.VarInt- QUIC-style Variable-length integer. (c.f. https://datatracker.ietf.org/doc/html/rfc9000#name-variable-length-integer-enc)
In QUIC stream, each frame is prefixed with a u8 Stream Packet Type.
0x00:Msg-[ bytes(VarInt) ]0x01:XferStart Unrel0x02:XferStart Rel0x03:XferStart Dual0x04:XferEnd-[ final_seq_number: seq_number ]0x05:XferEndAck- No payload0x06:XferRecvEnd- No payload0x07:XferData-[ seq_number: seq_number, data: bytes(VarInt) ]0x08:XferCreditUpdate-[ credit: VarInt ]0x09:XferCreditRequest- No payload0x0A:ACK-[ latest_continuous_seq_number: seq_number ]0x0B:NACK-[ array(seq_number) ]0x0C:Retrans-[ array( (seq_number, data: bytes) ) ]
- Since Protofish usually has long-running connection, it's RECOMMENDED to provide automatic backoff reconnection logic in the implementation.