RFC: Extend STORE protocol to give meta information about capabilities in-protocol

[oskarth]

Problem

STORE protocol is symmetric, and just knowing a peer is using STORE doesn't tell you much about what they are storing, if anything. The node can primarily be a "server" node or a "client" node.

Proposed solution

There are a few different dimensions of information that are desirable:

• Are they persisting messages at all? For which topics? (e.g. default pubsub topic to start with)
• For how long are they store? (7d retention policy e.g.)
• For how long have they been online (e.g. online window in epochs, assuming honest => extend with checks)

This informs a peer with enough data to know if:

• they should ask some other peer
• they should ask more than one peer

Rought sketch for protobufs:

RPC {
  HistoryQuery
  HistoryResponse
  MetaRequest
  MetaResponse

Acceptance criteria

• Rough consensus
• Raw spec

Notes

Adaptive node discussion context: #87

Additionally do similar for FILTER.

cc @staheri14 @jm-clius

also fyi @cammellos re core pov

[D4nte]

Alice needs to do one history request to Bob to know whether Bob stores the pubsubTopic she is interested in. How would this initial request look like?

As /vac/waku/store/2.0.0-beta2 is meant to retrieve historical data, it may make more sense to have a separate protocol to exchange information about storage capabilities such as /vac/waku/store/2.0.0-beta2/info or /vac/waku/store/2.0.0-beta2/identify.

Also, you make a good point in the problem section:

STORE protocol is symmetric, and just knowing a peer is using STORE doesn't tell you much about what they are storing, if anything. The node can primarily be a "server" node or a "client" node.

peer is using STORE

What does this mean in practical term? I assume that if Bob is using STORE it means that he includes /vac/waku/store/2.0.0-beta2 as part of his identify response to Alice.

IMHO, if Alice does not store any data but only supports the STORE protocol as a querying node then I don't believe she should advertise that she supports /vac/waku/store/2.0.0-beta2.

When Alice connects to Bob:

• Alice does not sends /vac/waku/store/2.0.0-beta2 as part of identify
• Bob sends /vac/waku/store/2.0.0-beta2 as part of identify.

From this point, it is clear to Alice and Bob who can query the other one for historical data.
I do not believe (but I'd need to dig deeper in the identify protocol to confirm) that Bob needs to know whether Alice supports the store protocol at any point in time.

Finally, if down the line we use pubsubTopics as a way to shard the network, then it may be critical for Alice to know upfront if Bob is part of her shard.
In this case, then the pubsubTopic could be appended to the protocol string advertised by Bob, e.g. /vac/waku/store/2.0.0-beta2/topic/waku/default so that from the moment the identify protocol is done, Alice know whether she should use Bob as a store server or drop the connection.

[staheri14]

The solution looks good to me @oskarth! such info re nodes capabilities is also necessary for the FT store protocol where nodes want to retrieve message history from a peer who has been online for a certain time window.

I have some suggestions:
The current solution assumes that two nodes are randomly connected and have no prior information about each others' resources and capabilities. This somewhat implies that a querying node has do a linear search (linear in the total number of nodes) and run the meta request/response protocol to find out a suitable candidate to query historical messages from.
Aa an alternative and perhaps a more efficient solution we may want to consider an approach where nodes periodically advertise their capabilities over a designated pubsub topic . This way, a requestor node connects to another node with the knowledge that the queried node is capable of responding to that request. Moreover, the advertised capability messages can also be persisted in the store db.
The capability messages can contain the same fields as in the meta request i.e.,

Are they persisting messages at all? For which topics? (e.g. default pubsub topic to start with)
For how long are they store? (7d retention policy e.g.)
For how long have they been online (e.g. online window in epochs, assuming honest => extend with checks)

[oskarth]

Great comments! Thanks both. Sounds like we have to do a bit more thinking and investigation on this one to come up with a good synthesis.

[jm-clius]

I do not believe (but I'd need to dig deeper in the identify protocol to confirm) that Bob needs to know whether Alice supports the store protocol at any point in time.

This may fall outside the domain that Waku should have influence over - from libp2p's perspective, if a peer has a certain protocol mounted that protocol should be advertised in the identify message. I think, however, we can use a similar pattern to what you suggested for topics, by appending the server/client/both information to the mounted protocol string.

This is not yet clear in my mind, but I'd propose something that will keep "protocols" and "capabilities" semantically separated. A protocol is a type of capability, but "capabilities" is a broader concept. We could even distinguish between "configured" capabilities, e.g. if a node is configured as a store client or server or both, and more "transient" capabilities, e.g. how much history the node is actually keeping, history topics, etc. "Configured" capabilities could be seen as protocol extensions and appended to the protocol string, while "transient" requires a capability negotiation or further discovery process as suggested.

[D4nte]

from libp2p's perspective, if a peer has a certain protocol mounted that protocol should be advertised in the identify message

The specs says This is a list of protocols supported by the peer. Please note I got some clarification from libp2p folks: libp2p/specs#324 (comment)