VIP: Unbounded Sequence Types

[Sporarum]

Unbounded Sequence Types

Introduction

This section introduces relevant terminology

Parameters and arguments

Parameters are the inputs as defined by the function, arguments are the values or expressions effectively passed to the input:

#       v parameter
def foo(x: uint256)...

#   v argument
foo(4)

Bounded sequence types

We define bounded sequence types to be the types taking as type parameter their maximum length:

1. Bytes[n]
2. String[n]
3. DynArray[T, n]

Notably this list does not include fixed-size lists (T[n] where T is any type except Bytes[n], String[n], flag types), since there the n is not an upper bound, but the actual length of the sequence.

For bounded sequence types, variables with a smaller bound can be assigned to variables with a bigger bound:

def foo() -> None:
	s1: String[10] = "Hi" # valid since "Hi" has 2 characters, so fits into "at most 10"
	s2: String[20] = s1 # valid since "at most 10" fits into "at most 20"
	s3: String[2]  = s1 # invalid since "at most 10" does not fit into "at most 2"
	return

In other words, for all n1 <= n2, String[n1] is a subtype of String[n2]: String[n1] <: String[n2].

Data locations

The EVM contains essentially 4 locations where data can be located:

1. Calldata
2. Memory
3. Transient storage
4. Storage

(There are also others like the stack and the contract bytecode itself, but those are not relevant for this discussion.)

Calldata contains the parameters to an external function, and is immutable.
Memory contains parameters to internal functions, as well as temporary variables (ones defined inside functions).
Transient storage and storage store persistent data, for example contract variables (which can be accessed with self).

Memory, while cheaper than storage, is not free.
The total cost depends on the highest index allocated, and is quadratic.
This means allocating more memory than needed can be extremely expensive.

Motivation

Currently, before a function's arguments are passed, memory is allocated to fit the expected type into memory.
If a function has a String[1000] as parameter, even if the argument is String[10], we allocate the full 1000 chars worth of memory !

And of course you cannot pass a String[11] as an argument for a String[10] parameter, as that could break assumptions on the other end.

This leads to a dilemma: How big do I make that sequence type ?

1. If you make it too small, this heavily contrains users in what they can do
2. If you make it too big, the cost will get astronomical, even if the user calls the method with a small instance

This is a big limitation that Vyper has that Solidity doesn't, and stops us from implementing a lot of useful patterns, for example:

1. Multicall: A contract which calls the contracts as defined in its parameter, and aggregates the results.
2. Virtual Machine: Like a multicall, but the output of calling another contract's method can be used to decide which contract to call next, and with what arguments.
3. Forwarding parameters from one contract's method to another contract's method.

The goal of this proposal is thus to remove this dilemma, in the simplest and clearest manner.

Overview

The idea is to add new types which represent sequences of any lengths, for example String[INF]:

@external
def size(s: String[INF]) -> uint256:
	self.compute_size(s)

def compute_size(s: String[INF]) -> uint256:
    return len(s)

As you can see, this avoids the dilemma completely: the size of s will depend on what it is called with.

Spec

Front-end

Unbounded versions of bounded sequence types are added:

1. String[INF]
2. Bytes[INF]
3. DynArray[T, INF]

(optional) Structs can contain fields of unbounded types, this makes them unbounded structs.

(optional) Unbounded structs can be elements of fixed-size lists, the generated type is unbounded.

Unbounded types are unbounded sequence types and unbounded structs.

For the following examples, assume:

struct MyStruct: # Not an unbounded type
    i: uint256

Examples of unbounded types:

# Unbounded because no bound

Bytes[INF]
String[INF]
DynArray[MyStruct, INF]

# If unbounded structs are allowed:
struct MyUnboundedStruct:
    bytes: Bytes[INF]

# Unbounded because contains unbounded struct

DynArray[MyUnboundedStruct, 4]
DynArray[MyUnboundedStruct, INF]

# If unbounded structs can are valid for fixed-size lists:
MyUnboundedStruct[4]

Unbounded types are only valid as:

1. method parameters types
2. method return types
3. local variable types (for variables inside methods)
4. (optional) field types for structs

struct NameBox:
    name: String[INF] # valid

name1: String[INF]    # invalid, not a local variable
name_box1: NameBox # invalid, not a local variable

@external # following also valid for internal methods
def foo(
    name2: String[INF]       # valid, parameter
    name_box2: NameBox    # valid, parameter
) -> (String[INF], NameBox): # valid, return type
  
  name3: String[INF] = name2        # valid, local variable
  name_box3: NameBox = name_box2 # valid, local variable
  
  return (name3, name_box2)

Unbounded sequence types are super-types of their bounded counterparts (String[n] <: String[INF]), for example a String[4] can be assigned to a variable of type String[INF].
This is not true in reverse, a String[INF] cannot be assigned to a variable of type String[4], as we cannot be sure it is of length <= 4.

convert-ing from an unbounded type to a bounded counterpart succeeds if the length fits, and reverts otherwise.
convert-ing from an unbounded type to another unbounded type follows the same rules as if they were bounded, for example padding is adjusted accordingly.
convert-ing from an unbounded type to anything else first converts to the most appropriate unbounded type (if required), and then converts again to the destination type. Example: convert(s: String[INF], Bytes[5]) is equivalent to convert(convert(s: String[INF], Bytes[INF]), Bytes[5]).

There exists one additional way to convert some unbounded sequence types to bounded ones: slice, see below.

Built-ins

Changes to built-ins, both to make them work better with the new features, as well as to standardize notation.

Semantically different:

• slice
  • b: Bytes | bytes32 | String to b: Bytes[INF] | bytes32 | String[INF]
  • return Bytes | String to
    • Bytes[length] | String[length] if length is known at compile time
    • Bytes[32] if type of b is bytes32
    • <type of b> otherwise
    • Note: this is the current behavior, but is undocumented
• raw_call
  • data: Bytes to data: Bytes[INF]
  • (optional)
    • deprecate max_outsize
    • make it return Bytes[INF]
    • (optional) or like slice: if max_outsize is know at compile time, returns Bytes[max_outsize], else returns Bytes[INF]
• msg.data, self.code, and <address>.code
  • now of type Bytes[INF]
  • Should remove special handling of these types around slicing

Cleanup notation:

• raw_create
  • initcode: Bytes[...] to initcode: Bytes[INF]
• raw_log
  • data: Bytes | bytes32 to data: Bytes[INF] | bytes32
• raw_revert
  • data: Bytes to data: Bytes[INF]
• extract32
  • b: Bytes to b: Bytes[INF]
• as_wei_value
  • unit: str to unit: String[INF]
    or to unit: String[n] where n is the greatest number of letters in the valid formats
• len
  • b: Bytes | String | DynArray[_Type, _Integer] to b: Bytes[INF] | String[INF] | DynArray[_Type, INF]

Simplifies the compiler

Things like Bytes.any() can be replaced by the representation for Bytes[INF], since String[n] <: String[INF] for any n.

Back-end

TODO

Backwards compatibility

TODO

Alternatives considered

Optimize calls so that the cost depends on the argument value or type, and not on the parameter type

While beneficial, it could still lead to issues:

1. The library or contract designer assumes Bytes[1000] is plenty, but end-user needs to pass Bytes[1024].
2. Many built-ins already use types which are unbounded, this would not allow us to create true forwarders/wrappers for these methods, and makes these built-ins harder to document.

Different syntax for the same idea

• String:
  Forward compat: If we ever want fixed-size lists of strings this will lead to ambiguity: String[5] vs (String)[5].
  Does not generalize to dynamic arrays, since we can't remove their subscript.
• String[] (Solidity's solution):
  Issues with dynamic arrays: DynArray[uint256, ] or DynArray[uint256].
• String[...]:
  A bit verbose, but does work well with dynamic arrays: DynArray[uint256, ...]

The syntaxes above are also less clear about sub-typing.
Since String[4] <: String[5], it follows that String[4] <: String[INF] and not String[INF] <: String[4].
That is less clear with the other syntaxes:

1. String[4] <: String and not String <: String[4]
2. String[4] <: String[] and not String[] <: String[4]
3. String[4] <: String[...] and not String[...] <: String[4]

INF could also be INFTY, INFINITY, etc, or an one of their lower-case versions.
All uppercase was chosen because this value is like a constant.
And INF in particular was chosen because it is short to type while remaining intelligible.

[Sporarum]

Draft implementation: Sporarum#1

[Sporarum]

One subject which was not mentioned in this VIP is the subject of json abi types.

Introduction

The ABI is the standard way through which contracts declare their interfaces and interact with each-other.
There are many ABI types, but we are interested here in the following:

• bytes: Bytestrings of arbitrary length
• string: UTF-8 strings of arbitrary length
• <type>[]: Arrays of <type> of arbitrary length

To learn more, see https://docs.soliditylang.org/en/latest/abi-spec.html#types

Motivation

<type>[]

Currently the compiler does not translate <type>[].
This is due to DynArray requiring a length parameter in vyper, but that length not being recorded in the ABI.

As a consequence, the only way to call methods with <type>[] in their abi signature is to either:

1. Create a vyper interface representing the method, and calling that
2. raw_call-ing that method

This is rather unwieldy, and adds friction when interacting with non-vyper contracts.

bytes and string

Currently the compiler translates bytes and string to Bytes[n] and String[n], where the n is inferred from usage.
In particular it is inferred from the first occurrence of the abi-encoded method in the source file.

Example:

vyper/tests/functional/codegen/test_interfaces.py

Line 533 in 6a53488

def test_json_abi_bytes_clampers(get_contract, tx_failed, assert_compile_failed, make_input_bundle):

@external
def test_fail1() -> Bytes[2]:
    # should compile, but raise runtime exception
    return extcall self.foo.returns_Bytes3() # First usage, expects `Bytes[2]`

[...]

@external
def test_fail3() -> Bytes[3]:
    # should revert - returns_Bytes3 is inferred to have return type Bytes[2]
    # (because test_fail3 comes after test_fail1)
    return extcall self.foo.returns_Bytes3()

The type of returns_bytes3 is inferred to be Bytes[2] because that is what the first call to it expects (test_fail1).
This assertion is then baked into the bytecode, and this makes test_fail3 revert.
But you'll notice test_fail3 is compatible with the actual return value of return_bytes3.
Furthermore reordering the methods such that test_fail3 is the first call to returns_bytes3 makes it work!

This behavior is needlessly restrictive, and the fact the ordering of methods has such side-effects is potentially confusing, especially given no warning or error is emitted at compile-time.

Overview

The translation of abi types is changed as follows:

• bytes -> Bytes[INF]
• string -> String[INF]
• <abi type>[] -> DynArray[<vyper type>, INF] where <vyper type> is the abi translation of <abi type>

Example:

[
    {
        "name": "name",
        "type": "function",
        "inputs": [],
        "outputs": [
            {
                "name": "",
                "type": "string"
            }
        ],
        "payable": false,
        "stateMutability": "nonpayable"
    },
    {
        "name": "decode",
        "type": "function",
        "inputs": [
            {
                "name": "xs",
                "type": "bytes"
            }
        ],
        "outputs": [
            {
                "name": "",
                "type": "uint256[]"
            }
        ],
        "payable": false,
        "stateMutability": "nonpayable"
    }
]

Would be equivalent to the following interface:

@external
def name() -> String[INF]:
    ...

@external
def decode(xs: Bytes[INF]) -> DynArray[uint256, INF]:
    ...

And can be implemented:

import abi_interface_above as above_i

implements: above_i

@external
def name() -> String[INF]:
    return "Potato"

@external
def decode(xs: Bytes[INF]) -> DynArray[uint256, INF]:
    return convert(xs, DynArray[uint256, INF])

And called:

import abi_interface_above as above_i

implements: above_i

def my_method():
    other = above_i(<some address>)
    name: String[INF] = extcall other.name()
    decoded: DynArray[uint256, INF] = extcall other.decode(b"\x00\x00\x07")

Spec

The translation of abi types is changed as follows:

• bytes -> Bytes[INF]
• string -> String[INF]
• <abi type>[] -> DynArray[<vyper type>, INF] where <vyper type> is the abi translation of <abi type>

As a consequence calling and implementing abi interfaces is safer:

• Calling a method which returns bytes requires explicitly handling the length
• Implementing a method with a bytes parameter requires having a Bytes[INF] parameter, and therefore handling the length
  (resp. with string and String[INF])
  This moves vyper closer to the ideal of "everything which can revert should be explicit"

Backward Compatibility

<type>[] was not allowed previously, so no backward compatibility issues there.

Any call to a method returning abi bytes or string will now fail to compile with an error like "String[64] is not a subtype of String[INF]"

Any implements of an abi interface containing at least one method with at least one abi bytes or string parameter will now fail to compile with an error like "method not implemented decode"
(or if implements error messages are improved: "method implemented incorrectly decode: signature expects Bytes[INF] parameter, but parameter has type Bytes[32]")

To fix offending contracts, the following should be done:
Calls:

def bar():
    name: String[64] = extcall other.name() # TypeMismatch: Given reference has type String[INF], expected String[64]

# rewritten to
def bar():
    tmp: String[INF] = extcall other.name()
    name: String[64] = convert(tmp, String[64])

# or
def bar():
    name: String[64] = convert(extcall other.name(), String[64])

Implements:

@external
def decode(xs: Bytes[32]): # InterfaceViolation: function decode does not implement other.decode: parameter `xs: Bytes[32]` should be `xs: Bytes[INF]`
    <body>

# rewritten to
@external
def decode(_xs: Bytes[INF]):
    xs: Bytes[32] = convert(_xs, Bytes[32])
    <body>

Optional: Add type assertions

By adding some helper syntax, we can smooth the above overhead:
name: assert String[64] = <rhs> means "assert rhs has length <= 64, and store that in name: String[64]"

The rewrites above thus become:

def bar():
    name: String[64] = extcall other.name() # TypeMismatch: Given reference has type String[INF], expected String[64]

# rewritten to
def bar():
    name: assert String[64] = extcall other.name()



@external
def decode(xs: Bytes[32]): # InterfaceViolation: function decode does not implement other.decode: parameter `xs: Bytes[32]` should be `xs: Bytes[INF]`
    <body>

# rewritten to
@external
def decode(xs: assert Bytes[32]):
    <body>

Alternatives considered

Remove boilerplate at the cost of safety

Instead of forcing the user to explicitly handle the length, we can omit these checks when the type comes from somewhere abi-encoded:

implements: other

@external
def decode(xs: Bytes[32]): # Compiles, but will revert if called with data longer than 32 bytes
    <body>

def bar():
    name: String[64] = extcall other.name() # Compiles, but will revert if name() returns a string longer than 64 bytes

This can be done by adding a length wildcard ..., and using this different translation scheme:

• abi bytes -> vyper Bytes[...]
• abi string -> vyper String[...]
• abi <abi type>[] -> vyper DynArray[<vyper type>, ...]

Types with a length of ... can stand in for any length (they are both sub- and super-types of any length, including INF).
As they undermine compile-time safety, they can only appear when translating abi types, and in particular:

• Cannot be expressed in source code, writing Bytes[...] causes an error
• Will never be the inferred type of an expression

With ..., this VIP is source backwards-compatible.
At the cost of no longer forcing the user to be explicit in where the code can revert.

Only remove boilerplate when implementing abi methods

Skip the length check, but only for implements:

implements: other

@external
def decode(xs: Bytes[32]): # Compiles, but will revert if called with data longer than 32 bytes
    <body>

def bar():
    name: String[64] = extcall other.name() # TypeMismatch: Given reference has type String[INF], expected String[64]

So the user can implement simply, but must handle calls explicitly:

implements: other

@external
def decode(xs: Bytes[32]): # Compiles, but will revert if called with data longer than 32 bytes
    <body>

def bar():
    tmp: String[INF] = extcall other.name()
    name: String[64] = convert(tmp, String[64]) # Is explicit about potentially reverting

Only remove boilerplate when calling abi methods

The opposite of the above, skip the length check, but only for calls:

implements: other

@external
def decode(xs: Bytes[32]): # InterfaceViolation: function decode does not implement other.decode: parameter `xs: Bytes[32]` should be `xs: Bytes[INF]`
    <body>

def bar():
    name: String[64] = extcall other.name() # Compiles, but will revert if name() returns a string longer than 64 bytes

So the user must implements explicitly, but can call simply:

implements: other

@external
def decode(_xs: Bytes[INF]):
    xs: Bytes[32] = convert(_xs, Bytes[32]) # Compiles, but explicit about reverting if _xs is longer than 32 bytes
    <body>

def bar():
    name: String[64] = extcall other.name() # Compiles, but will revert if name() returns a string longer than 64 bytes

[Sporarum]

On further consideration, my recommendation is now to go with the "Remove boilerplate at the cost of safety" alternative