diff --git a/README.md b/README.md index 206a9d6..dca0f4b 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,7 @@ See also: - [Example](#example) - [Usage](#usage) +- [Size Limits](#size-limits) - [API](#api) - [License](#license) @@ -213,6 +214,46 @@ API to convert the `out` `AsyncIterable` to a standard Node.js stream, or it can be directly fed to a [`stream.pipeline()`](https://nodejs.org/api/stream.html#stream_stream_pipeline_source_transforms_destination_callback). +## Size Limits + +Every decode and encode entry point accepts an optional `CarCodecOptions` with two +caps, both **on by default** and matching [go-car](https://github.com/ipld/go-car): + +| Option | Bounds | Default | +| --- | --- | --- | +| `maxAllowedHeaderSize` | the CAR header (the dag-cbor header block) | `32 << 20` (32 MiB) | +| `maxAllowedSectionSize` | each section: the CID plus its block body, combined | `8 << 20` (8 MiB) | + +Both count the varint-declared length, excluding the varint prefix itself, which is +the quantity go-car's parser compares. A length strictly greater than a cap throws a +`RangeError`, on decode from the length prefix before the body is read, skipped, or +pulled from the source, and on encode before any bytes for that item are written. + +Because `maxAllowedSectionSize` bounds the CID and body together, the effective +maximum block body is the cap minus its CID (about 8388572 bytes for a normal +sha2-256 CIDv1, not 8 MiB exactly). + +Passing `undefined` for a cap uses its default. Set a cap to `0` to reject every +section, or to `Number.MAX_SAFE_INTEGER` to disable it (the `varint` package refuses +to decode a length above 2^53-1, so a cap there can never fire). A provided value +that is not a non-negative safe integer throws a `TypeError`. + +The caps apply to every entry point that parses or emits the CAR format: + +- decode, streaming: `CarReader.fromIterable`, `CarBlockIterator.fromIterable`, + `CarCIDIterator.fromIterable`, `CarIndexer.fromIterable`, `CarIndexedReader.fromFile` +- decode, in-memory: `CarReader.fromBytes`, `CarBlockIterator.fromBytes`, + `CarCIDIterator.fromBytes`, `CarIndexer.fromBytes`, `CarBufferReader.fromBytes` +- encode: `CarWriter.create`, `CarWriter.createAppender`, `CarBufferWriter.createWriter` + +Setting the same `CarCodecOptions` on both ends gives one size profile that holds on +read and write, so the library never emits a CAR it would refuse to read back. +`CarWriter.updateRootsInBytes` and `CarWriter.updateRootsInFile` take no options; they +overwrite a header in place under the default header cap. + +Separately, the decoder caps a single multihash **digest** at a hardcoded 32 MiB +(matching go-cid's unexported `maxDigestAlloc`). + ## API ### Contents @@ -223,8 +264,8 @@ be directly fed to a - [`async CarReader#get(key)`](#async-carreadergetkey) - [`async * CarReader#blocks()`](#async--carreaderblocks) - [`async * CarReader#cids()`](#async--carreadercids) -- [`async CarReader.fromBytes(bytes)`](#async-carreaderfrombytesbytes) -- [`async CarReader.fromIterable(asyncIterable)`](#async-carreaderfromiterableasynciterable) +- [`async CarReader.fromBytes(bytes, options)`](#async-carreaderfrombytesbytes-options) +- [`async CarReader.fromIterable(asyncIterable, options)`](#async-carreaderfromiterableasynciterable-options) - [`async CarReader.readRaw(fd, blockIndex)`](#async-carreaderreadrawfd-blockindex) - [`class CarIndexedReader`](#class-carindexedreader) - [`async CarIndexedReader#getRoots()`](#async-carindexedreadergetroots) @@ -233,24 +274,24 @@ be directly fed to a - [`async * CarIndexedReader#blocks()`](#async--carindexedreaderblocks) - [`async * CarIndexedReader#cids()`](#async--carindexedreadercids) - [`async CarIndexedReader#close()`](#async-carindexedreaderclose) -- [`async CarIndexedReader.fromFile(path)`](#async-carindexedreaderfromfilepath) +- [`async CarIndexedReader.fromFile(path, options)`](#async-carindexedreaderfromfilepath-options) - [`class CarBlockIterator`](#class-carblockiterator) - [`async CarBlockIterator#getRoots()`](#async-carblockiteratorgetroots) -- [`async CarBlockIterator.fromBytes(bytes)`](#async-carblockiteratorfrombytesbytes) -- [`async CarBlockIterator.fromIterable(asyncIterable)`](#async-carblockiteratorfromiterableasynciterable) +- [`async CarBlockIterator.fromBytes(bytes, options)`](#async-carblockiteratorfrombytesbytes-options) +- [`async CarBlockIterator.fromIterable(asyncIterable, options)`](#async-carblockiteratorfromiterableasynciterable-options) - [`class CarCIDIterator`](#class-carciditerator) - [`async CarCIDIterator#getRoots()`](#async-carciditeratorgetroots) -- [`async CarCIDIterator.fromBytes(bytes)`](#async-carciditeratorfrombytesbytes) -- [`async CarCIDIterator.fromIterable(asyncIterable)`](#async-carciditeratorfromiterableasynciterable) +- [`async CarCIDIterator.fromBytes(bytes, options)`](#async-carciditeratorfrombytesbytes-options) +- [`async CarCIDIterator.fromIterable(asyncIterable, options)`](#async-carciditeratorfromiterableasynciterable-options) - [`class CarIndexer`](#class-carindexer) - [`async CarIndexer#getRoots()`](#async-carindexergetroots) -- [`async CarIndexer.fromBytes(bytes)`](#async-carindexerfrombytesbytes) -- [`async CarIndexer.fromIterable(asyncIterable)`](#async-carindexerfromiterableasynciterable) +- [`async CarIndexer.fromBytes(bytes, options)`](#async-carindexerfrombytesbytes-options) +- [`async CarIndexer.fromIterable(asyncIterable, options)`](#async-carindexerfromiterableasynciterable-options) - [`class CarWriter`](#class-carwriter) - [`async CarWriter#put(block)`](#async-carwriterputblock) - [`async CarWriter#close()`](#async-carwriterclose) -- [`async CarWriter.create(roots)`](#async-carwritercreateroots) -- [`async CarWriter.createAppender()`](#async-carwritercreateappender) +- [`async CarWriter.create(roots, options)`](#async-carwritercreateroots-options) +- [`async CarWriter.createAppender(options)`](#async-carwritercreateappenderoptions) - [`async CarWriter.updateRootsInBytes(bytes, roots)`](#async-carwriterupdaterootsinbytesbytes-roots) - [`async CarWriter.updateRootsInFile(fd, roots)`](#async-carwriterupdaterootsinfilefd-roots) - [`class CarBufferWriter`](#class-carbufferwriter) @@ -274,7 +315,7 @@ be directly fed to a - [`CarBufferReader#get(key)`](#carbufferreadergetkey) - [`CarBufferReader#blocks()`](#carbufferreaderblocks) - [`CarBufferReader#cids()`](#carbufferreadercids) -- [`CarBufferReader.fromBytes(bytes)`](#carbufferreaderfrombytesbytes) +- [`CarBufferReader.fromBytes(bytes, options)`](#carbufferreaderfrombytesbytes-options) - [`CarBufferReader.readRaw(fd, blockIndex)`](#carbufferreaderreadrawfd-blockindex) @@ -354,9 +395,10 @@ the `CID`s contained within the CAR referenced by this reader. -### `async CarReader.fromBytes(bytes)` +### `async CarReader.fromBytes(bytes, options)` - `bytes` `(Uint8Array)` +- `options` `(object, optional)`: Optional size limits; see [Size Limits](#size-limits). - Returns: `Promise` @@ -366,9 +408,10 @@ access to the data via the `CarReader` API. -### `async CarReader.fromIterable(asyncIterable)` +### `async CarReader.fromIterable(asyncIterable, options)` - `asyncIterable` `(AsyncIterable)` +- `options` `(object, optional)`: Optional size limits; see [Size Limits](#size-limits). - Returns: `Promise` @@ -489,9 +532,10 @@ This must be called for proper resource clean-up to occur. -### `async CarIndexedReader.fromFile(path)` +### `async CarIndexedReader.fromFile(path, options)` - `path` `(string)` +- `options` `(object, optional)`: Optional size limits; see [Size Limits](#size-limits). - Returns: `Promise` @@ -544,9 +588,10 @@ zero or more `CID`s. -### `async CarBlockIterator.fromBytes(bytes)` +### `async CarBlockIterator.fromBytes(bytes, options)` - `bytes` `(Uint8Array)` +- `options` `(object, optional)`: Optional size limits; see [Size Limits](#size-limits). - Returns: `Promise` @@ -557,9 +602,10 @@ of the CAR is parsed as the `Block`s as yielded. -### `async CarBlockIterator.fromIterable(asyncIterable)` +### `async CarBlockIterator.fromIterable(asyncIterable, options)` - `asyncIterable` `(AsyncIterable)` +- `options` `(object, optional)`: Optional size limits; see [Size Limits](#size-limits). - Returns: `Promise` @@ -609,9 +655,10 @@ zero or more `CID`s. -### `async CarCIDIterator.fromBytes(bytes)` +### `async CarCIDIterator.fromBytes(bytes, options)` - `bytes` `(Uint8Array)` +- `options` `(object, optional)`: Optional size limits; see [Size Limits](#size-limits). - Returns: `Promise` @@ -622,9 +669,10 @@ of the CAR is parsed as the `CID`s as yielded. -### `async CarCIDIterator.fromIterable(asyncIterable)` +### `async CarCIDIterator.fromIterable(asyncIterable, options)` - `asyncIterable` `(AsyncIterable)` +- `options` `(object, optional)`: Optional size limits; see [Size Limits](#size-limits). - Returns: `Promise` @@ -676,9 +724,10 @@ zero or more `CID`s. -### `async CarIndexer.fromBytes(bytes)` +### `async CarIndexer.fromBytes(bytes, options)` - `bytes` `(Uint8Array)` +- `options` `(object, optional)`: Optional size limits; see [Size Limits](#size-limits). - Returns: `Promise` @@ -688,9 +737,10 @@ iterator as it is consumed. -### `async CarIndexer.fromIterable(asyncIterable)` +### `async CarIndexer.fromIterable(asyncIterable, options)` - `asyncIterable` `(AsyncIterable)` +- `options` `(object, optional)`: Optional size limits; see [Size Limits](#size-limits). - Returns: `Promise` @@ -756,9 +806,10 @@ any remaining bytes are written. -### `async CarWriter.create(roots)` +### `async CarWriter.create(roots, options)` - `roots` `(CID[]|CID|void)` +- `options` `(object, optional)`: Optional size limits; see [Size Limits](#size-limits). - Returns: `WriterChannel`: The channel takes the form of `{ writer:CarWriter, out:AsyncIterable }`. @@ -768,7 +819,9 @@ Create a new CAR writer "channel" which consists of a -### `async CarWriter.createAppender()` +### `async CarWriter.createAppender(options)` + +- `options` `(object, optional)`: Optional size limits; see [Size Limits](#size-limits). - Returns: `WriterChannel`: The channel takes the form of `{ writer:CarWriter, out:AsyncIterable }`. @@ -930,6 +983,8 @@ single-byte multihash code, such as SHA2-256 (i.e. the most common CIDv1). - `options.byteOffset` `(number, optional)` - `options.byteLength` `(number, optional)` - `options.headerSize` `(number, optional)` + - `options.maxAllowedHeaderSize` `(number, optional)`: Optional size limit; see [Size Limits](#size-limits). + - `options.maxAllowedSectionSize` `(number, optional)`: Optional size limit; see [Size Limits](#size-limits). - Returns: `CarBufferWriter` @@ -1090,9 +1145,10 @@ Returns a `CID[]` of the `CID`s contained within the CAR referenced by this read -### `CarBufferReader.fromBytes(bytes)` +### `CarBufferReader.fromBytes(bytes, options)` - `bytes` `(Uint8Array)` +- `options` `(object, optional)`: Optional size limits; see [Size Limits](#size-limits). - Returns: `CarBufferReader` diff --git a/src/api.ts b/src/api.ts index a314482..f109a83 100644 --- a/src/api.ts +++ b/src/api.ts @@ -59,12 +59,14 @@ export interface BlockWriter { } export interface CarBufferWriter { + readonly maxAllowedHeaderSize: number + readonly maxAllowedSectionSize: number addRoot(root: CID, options?: { resize?: boolean }): CarBufferWriter write(block: Block): CarBufferWriter close(options?: { resize?: boolean }): Uint8Array } -export interface CarBufferWriterOptions { +export interface CarBufferWriterOptions extends CarCodecOptions { roots?: CID[] // defaults to [] byteOffset?: number // defaults to 0 byteLength?: number // defaults to buffer.byteLength @@ -72,6 +74,11 @@ export interface CarBufferWriterOptions { headerSize?: number // defaults to size needed for provided roots } +export interface CarCodecOptions { + maxAllowedHeaderSize?: number // default 32 << 20 (32MiB) + maxAllowedSectionSize?: number // default 8 << 20 (8MiB) +} + export interface WriterChannel { writer: BlockWriter out: AsyncIterable diff --git a/src/buffer-decoder.js b/src/buffer-decoder.js index b399dda..e07e387 100644 --- a/src/buffer-decoder.js +++ b/src/buffer-decoder.js @@ -3,6 +3,7 @@ import { CID } from 'multiformats/cid' import * as Digest from 'multiformats/hashes/digest' import { CIDV0_BYTES, decodeV2Header, decodeVarint, getMultihashLength, V2_HEADER_LENGTH } from './decoder-common.js' import { CarV1HeaderOrV2Pragma } from './header-validator.js' +import { resolveLimits } from './limits.js' /** * @typedef {import('./api.js').Block} Block @@ -12,6 +13,8 @@ import { CarV1HeaderOrV2Pragma } from './header-validator.js' * @typedef {import('./coding.js').CarHeader} CarHeader * @typedef {import('./coding.js').CarV2Header} CarV2Header * @typedef {import('./coding.js').CarV2FixedHeader} CarV2FixedHeader + * @typedef {import('./api.js').CarCodecOptions} CarCodecOptions + * @typedef {import('./limits.js').CarLimits} CarLimits */ /** @@ -21,13 +24,17 @@ import { CarV1HeaderOrV2Pragma } from './header-validator.js' * @name decoder.readHeader(reader) * @param {BytesBufferReader} reader * @param {number} [strictVersion] + * @param {CarLimits} [limits] * @returns {CarHeader | CarV2Header} */ -export function readHeader (reader, strictVersion) { +export function readHeader (reader, strictVersion, limits = resolveLimits()) { const length = decodeVarint(reader.upTo(8), reader) if (length === 0) { throw new Error('Invalid CAR header (zero length)') } + if (length > limits.maxAllowedHeaderSize) { + throw new RangeError(`CAR header of length ${length} exceeds maxAllowedHeaderSize of ${limits.maxAllowedHeaderSize}`) + } const header = reader.exactly(length, true) const block = decodeDagCbor(header) if (CarV1HeaderOrV2Pragma.toTyped(block) === undefined) { @@ -49,7 +56,7 @@ export function readHeader (reader, strictVersion) { } const v2Header = decodeV2Header(reader.exactly(V2_HEADER_LENGTH, true)) reader.seek(v2Header.dataOffset - reader.pos) - const v1Header = readHeader(reader, 1) + const v1Header = readHeader(reader, 1, limits) return Object.assign(v1Header, v2Header) } @@ -57,15 +64,20 @@ export function readHeader (reader, strictVersion) { * Reads CID sync * * @param {BytesBufferReader} reader - * @returns {CID} + * @param {number} sectionLength + * @returns {{ cid: CID, cidLength: number }} */ -function readCid (reader) { +function readCid (reader, sectionLength) { + const cidStart = reader.pos const first = reader.exactly(2, false) if (first[0] === CIDV0_BYTES.SHA2_256 && first[1] === CIDV0_BYTES.LENGTH) { - // cidv0 32-byte sha2-256 - const bytes = reader.exactly(34, true) - const multihash = Digest.decode(bytes) - return CID.create(0, CIDV0_BYTES.DAG_PB, multihash) + // cidv0: 0x12 code byte + 0x20 length byte + 32-byte sha2-256 digest + const cidLength = 34 + if (cidLength > sectionLength) { + throw new Error(`Invalid CAR section (CID of length ${cidLength} exceeds section length ${sectionLength})`) + } + const bytes = reader.exactly(cidLength, true) + return { cid: CID.create(0, CIDV0_BYTES.DAG_PB, Digest.decode(bytes)), cidLength } } const version = decodeVarint(reader.upTo(8), reader) @@ -73,9 +85,13 @@ function readCid (reader) { throw new Error(`Unexpected CID version (${version})`) } const codec = decodeVarint(reader.upTo(8), reader) - const bytes = reader.exactly(getMultihashLength(reader.upTo(8)), true) - const multihash = Digest.decode(bytes) - return CID.create(version, codec, multihash) + const mhLength = getMultihashLength(reader.upTo(8)) + const cidLength = Number(reader.pos - cidStart) + mhLength + if (cidLength > sectionLength) { + throw new Error(`Invalid CAR section (CID of length ${cidLength} exceeds section length ${sectionLength})`) + } + const bytes = reader.exactly(mhLength, true) + return { cid: CID.create(version, codec, Digest.decode(bytes)), cidLength } } /** @@ -84,34 +100,38 @@ function readCid (reader) { * `{ cid, length, blockLength }` which can be used to either index the block * or read the block binary data. * - * @name async decoder.readBlockHead(reader) + * @name decoder.readBlockHead(reader) * @param {BytesBufferReader} reader + * @param {CarLimits} limits * @returns {BlockHeader} */ -export function readBlockHead (reader) { +export function readBlockHead (reader, limits) { // length includes a CID + Binary, where CID has a variable length // we have to deal with const start = reader.pos - let length = decodeVarint(reader.upTo(8), reader) - if (length === 0) { + const sectionLength = decodeVarint(reader.upTo(8), reader) + if (sectionLength === 0) { throw new Error('Invalid CAR section (zero length)') } - length += (reader.pos - start) - const cid = readCid(reader) - const blockLength = length - Number(reader.pos - start) // subtract CID length - - return { cid, length, blockLength } + if (sectionLength > limits.maxAllowedSectionSize) { + throw new RangeError(`CAR section of length ${sectionLength} exceeds maxAllowedSectionSize of ${limits.maxAllowedSectionSize}`) + } + const length = Number(reader.pos - start) + sectionLength + const { cid, cidLength } = readCid(reader, sectionLength) + return { cid, length, blockLength: sectionLength - cidLength } } /** * Returns Car header and blocks from a Uint8Array * * @param {Uint8Array} bytes + * @param {CarCodecOptions} [options] * @returns {{ header : CarHeader | CarV2Header , blocks: Block[]}} */ -export function fromBytes (bytes) { +export function fromBytes (bytes, options) { + const limits = resolveLimits(options) let reader = bytesReader(bytes) - const header = readHeader(reader) + const header = readHeader(reader, undefined, limits) if (header.version === 2) { const v1length = reader.pos - header.dataOffset reader = limitReader(reader, header.dataSize - v1length) @@ -119,7 +139,7 @@ export function fromBytes (bytes) { const blocks = [] while (reader.upTo(8).length > 0) { - const { cid, blockLength } = readBlockHead(reader) + const { cid, blockLength } = readBlockHead(reader, limits) blocks.push({ cid, bytes: reader.exactly(blockLength, true) }) } diff --git a/src/buffer-reader-browser.js b/src/buffer-reader-browser.js index 61e8971..5e09ba7 100644 --- a/src/buffer-reader-browser.js +++ b/src/buffer-reader-browser.js @@ -129,14 +129,15 @@ export class CarBufferReader { * @static * @memberof CarBufferReader * @param {Uint8Array} bytes + * @param {import('./api.js').CarCodecOptions} [options] - Optional size limits; see [Size Limits](#size-limits). * @returns {CarBufferReader} */ - static fromBytes (bytes) { + static fromBytes (bytes, options) { if (!(bytes instanceof Uint8Array)) { throw new TypeError('fromBytes() requires a Uint8Array') } - const { header, blocks } = BufferDecoder.fromBytes(bytes) + const { header, blocks } = BufferDecoder.fromBytes(bytes, options) return new CarBufferReader(header, blocks) } } diff --git a/src/buffer-writer.js b/src/buffer-writer.js index f4345d8..f8ef8be 100644 --- a/src/buffer-writer.js +++ b/src/buffer-writer.js @@ -2,6 +2,7 @@ import * as CBOR from '@ipld/dag-cbor' import { Token, Type } from 'cborg' import { tokensToLength } from 'cborg/length' import varint from 'varint' +import { resolveLimits } from './limits.js' /** * @typedef {import('./api.js').CID} CID @@ -22,8 +23,9 @@ class CarBufferWriter { /** * @param {Uint8Array} bytes * @param {number} headerSize + * @param {import('./limits.js').CarLimits} limits */ - constructor (bytes, headerSize) { + constructor (bytes, headerSize, limits) { /** @readonly */ this.bytes = bytes this.byteOffset = headerSize @@ -34,6 +36,11 @@ class CarBufferWriter { */ this.roots = [] this.headerSize = headerSize + + /** @readonly */ + this.maxAllowedHeaderSize = limits.maxAllowedHeaderSize + /** @readonly */ + this.maxAllowedSectionSize = limits.maxAllowedSectionSize } /** @@ -123,6 +130,9 @@ export const blockLength = ({ cid, bytes }) => { */ export const addBlock = (writer, { cid, bytes }) => { const byteLength = cid.bytes.byteLength + bytes.byteLength + if (byteLength > writer.maxAllowedSectionSize) { + throw new RangeError(`CAR section of length ${byteLength} exceeds maxAllowedSectionSize of ${writer.maxAllowedSectionSize}`) + } const size = varint.encode(byteLength) if (writer.byteOffset + size.length + byteLength > writer.bytes.byteLength) { throw new RangeError('Buffer has no capacity for this block') @@ -143,6 +153,9 @@ export const close = (writer, options = {}) => { const { roots, bytes, byteOffset, headerSize } = writer const headerBytes = CBOR.encode({ version: 1, roots }) + if (headerBytes.length > writer.maxAllowedHeaderSize) { + throw new RangeError(`CAR header of length ${headerBytes.length} exceeds maxAllowedHeaderSize of ${writer.maxAllowedHeaderSize}`) + } const varintBytes = varint.encode(headerBytes.length) const size = varintBytes.length + headerBytes.byteLength @@ -266,6 +279,8 @@ export const estimateHeaderLength = (rootCount, rootByteLength = 36) => * @param {number} [options.byteOffset] * @param {number} [options.byteLength] * @param {number} [options.headerSize] + * @param {number} [options.maxAllowedHeaderSize] + * @param {number} [options.maxAllowedSectionSize] * @returns {CarBufferWriter} */ export const createWriter = (buffer, options = {}) => { @@ -277,7 +292,7 @@ export const createWriter = (buffer, options = {}) => { } = options const bytes = new Uint8Array(buffer, byteOffset, byteLength) - const writer = new CarBufferWriter(bytes, headerSize) + const writer = new CarBufferWriter(bytes, headerSize, resolveLimits(options)) for (const root of roots) { writer.addRoot(root) } diff --git a/src/decoder-common.js b/src/decoder-common.js index 338e36e..3c6c682 100644 --- a/src/decoder-common.js +++ b/src/decoder-common.js @@ -57,8 +57,13 @@ export function decodeV2Header (bytes) { return header } +// Matches go-cid's hardcoded, unexported maxDigestAlloc (go-cid/cid.go). Bounds +// the multihash digest length, not the whole CID. +const MAX_DIGEST_ALLOC = 32 << 20 // 32MiB + /** - * Checks the length of the multihash to be read afterwards + * Returns the total byte length of the multihash to be read afterwards, throwing + * if its declared digest length exceeds the allocation bound. * * ```js * // needs bytes to be read first @@ -66,6 +71,7 @@ export function decodeV2Header (bytes) { * ``` * * @param {Uint8Array} bytes + * @returns {number} */ export function getMultihashLength (bytes) { // | code | length | .... | @@ -74,9 +80,10 @@ export function getMultihashLength (bytes) { varint.decode(bytes) // code const codeLength = /** @type {number} */(varint.decode.bytes) - const length = varint.decode(bytes.subarray(varint.decode.bytes)) + const digestLength = varint.decode(bytes.subarray(codeLength)) const lengthLength = /** @type {number} */(varint.decode.bytes) - const mhLength = codeLength + lengthLength + length - - return mhLength + if (digestLength > MAX_DIGEST_ALLOC) { + throw new RangeError(`CID digest of length ${digestLength} exceeds maximum of ${MAX_DIGEST_ALLOC}`) + } + return codeLength + lengthLength + digestLength } diff --git a/src/decoder.js b/src/decoder.js index e09fa88..a698077 100644 --- a/src/decoder.js +++ b/src/decoder.js @@ -3,6 +3,7 @@ import { CID } from 'multiformats/cid' import * as Digest from 'multiformats/hashes/digest' import { CIDV0_BYTES, decodeV2Header, decodeVarint, getMultihashLength, V2_HEADER_LENGTH } from './decoder-common.js' import { CarV1HeaderOrV2Pragma } from './header-validator.js' +import { resolveLimits } from './limits.js' /** * @typedef {import('./api.js').Block} Block @@ -13,6 +14,8 @@ import { CarV1HeaderOrV2Pragma } from './header-validator.js' * @typedef {import('./coding.js').CarV2Header} CarV2Header * @typedef {import('./coding.js').CarV2FixedHeader} CarV2FixedHeader * @typedef {import('./coding.js').CarDecoder} CarDecoder + * @typedef {import('./api.js').CarCodecOptions} CarCodecOptions + * @typedef {import('./limits.js').CarLimits} CarLimits */ /** @@ -22,13 +25,17 @@ import { CarV1HeaderOrV2Pragma } from './header-validator.js' * @name async decoder.readHeader(reader) * @param {BytesReader} reader * @param {number} [strictVersion] + * @param {CarLimits} [limits] * @returns {Promise} */ -export async function readHeader (reader, strictVersion) { +export async function readHeader (reader, strictVersion, limits = resolveLimits()) { const length = decodeVarint(await reader.upTo(8), reader) if (length === 0) { throw new Error('Invalid CAR header (zero length)') } + if (length > limits.maxAllowedHeaderSize) { + throw new RangeError(`CAR header of length ${length} exceeds maxAllowedHeaderSize of ${limits.maxAllowedHeaderSize}`) + } const header = await reader.exactly(length, true) const block = decodeDagCbor(header) if (CarV1HeaderOrV2Pragma.toTyped(block) === undefined) { @@ -50,21 +57,26 @@ export async function readHeader (reader, strictVersion) { } const v2Header = decodeV2Header(await reader.exactly(V2_HEADER_LENGTH, true)) reader.seek(v2Header.dataOffset - reader.pos) - const v1Header = await readHeader(reader, 1) + const v1Header = await readHeader(reader, 1, limits) return Object.assign(v1Header, v2Header) } /** * @param {BytesReader} reader - * @returns {Promise} + * @param {number} sectionLength + * @returns {Promise<{ cid: CID, cidLength: number }>} */ -async function readCid (reader) { +async function readCid (reader, sectionLength) { + const cidStart = reader.pos const first = await reader.exactly(2, false) if (first[0] === CIDV0_BYTES.SHA2_256 && first[1] === CIDV0_BYTES.LENGTH) { - // cidv0 32-byte sha2-256 - const bytes = await reader.exactly(34, true) - const multihash = Digest.decode(bytes) - return CID.create(0, CIDV0_BYTES.DAG_PB, multihash) + // cidv0: 0x12 code byte + 0x20 length byte + 32-byte sha2-256 digest + const cidLength = 34 + if (cidLength > sectionLength) { + throw new Error(`Invalid CAR section (CID of length ${cidLength} exceeds section length ${sectionLength})`) + } + const bytes = await reader.exactly(cidLength, true) + return { cid: CID.create(0, CIDV0_BYTES.DAG_PB, Digest.decode(bytes)), cidLength } } const version = decodeVarint(await reader.upTo(8), reader) @@ -72,9 +84,13 @@ async function readCid (reader) { throw new Error(`Unexpected CID version (${version})`) } const codec = decodeVarint(await reader.upTo(8), reader) - const bytes = await reader.exactly(getMultihashLength(await reader.upTo(8)), true) - const multihash = Digest.decode(bytes) - return CID.create(version, codec, multihash) + const mhLength = getMultihashLength(await reader.upTo(8)) + const cidLength = Number(reader.pos - cidStart) + mhLength + if (cidLength > sectionLength) { + throw new Error(`Invalid CAR section (CID of length ${cidLength} exceeds section length ${sectionLength})`) + } + const bytes = await reader.exactly(mhLength, true) + return { cid: CID.create(version, codec, Digest.decode(bytes)), cidLength } } /** @@ -85,40 +101,44 @@ async function readCid (reader) { * * @name async decoder.readBlockHead(reader) * @param {BytesReader} reader + * @param {CarLimits} limits * @returns {Promise} */ -export async function readBlockHead (reader) { +export async function readBlockHead (reader, limits) { // length includes a CID + Binary, where CID has a variable length // we have to deal with const start = reader.pos - let length = decodeVarint(await reader.upTo(8), reader) - if (length === 0) { + const sectionLength = decodeVarint(await reader.upTo(8), reader) + if (sectionLength === 0) { throw new Error('Invalid CAR section (zero length)') } - length += (reader.pos - start) - const cid = await readCid(reader) - const blockLength = length - Number(reader.pos - start) // subtract CID length - - return { cid, length, blockLength } + if (sectionLength > limits.maxAllowedSectionSize) { + throw new RangeError(`CAR section of length ${sectionLength} exceeds maxAllowedSectionSize of ${limits.maxAllowedSectionSize}`) + } + const length = Number(reader.pos - start) + sectionLength + const { cid, cidLength } = await readCid(reader, sectionLength) + return { cid, length, blockLength: sectionLength - cidLength } } /** * @param {BytesReader} reader + * @param {CarLimits} limits * @returns {Promise} */ -async function readBlock (reader) { - const { cid, blockLength } = await readBlockHead(reader) +async function readBlock (reader, limits) { + const { cid, blockLength } = await readBlockHead(reader, limits) const bytes = await reader.exactly(blockLength, true) return { bytes, cid } } /** * @param {BytesReader} reader + * @param {CarLimits} limits * @returns {Promise} */ -async function readBlockIndex (reader) { +async function readBlockIndex (reader, limits) { const offset = reader.pos - const { cid, length, blockLength } = await readBlockHead(reader) + const { cid, length, blockLength } = await readBlockHead(reader, limits) const index = { cid, length, blockLength, offset, blockOffset: reader.pos } reader.seek(index.blockLength) return index @@ -131,11 +151,13 @@ async function readBlockIndex (reader) { * * @name decoder.createDecoder(reader) * @param {BytesReader} reader + * @param {CarCodecOptions} [options] * @returns {CarDecoder} */ -export function createDecoder (reader) { +export function createDecoder (reader, options) { + const limits = resolveLimits(options) const headerPromise = (async () => { - const header = await readHeader(reader) + const header = await readHeader(reader, undefined, limits) if (header.version === 2) { const v1length = reader.pos - header.dataOffset reader = limitReader(reader, header.dataSize - v1length) @@ -149,14 +171,14 @@ export function createDecoder (reader) { async * blocks () { await headerPromise while ((await reader.upTo(8)).length > 0) { - yield await readBlock(reader) + yield await readBlock(reader, limits) } }, async * blocksIndex () { await headerPromise while ((await reader.upTo(8)).length > 0) { - yield await readBlockIndex(reader) + yield await readBlockIndex(reader, limits) } } } diff --git a/src/encoder.js b/src/encoder.js index c305f8f..734d41e 100644 --- a/src/encoder.js +++ b/src/encoder.js @@ -1,11 +1,13 @@ import { encode as dagCborEncode } from '@ipld/dag-cbor' import varint from 'varint' +import { resolveLimits } from './limits.js' /** * @typedef {import('multiformats').CID} CID * @typedef {import('./api.js').Block} Block * @typedef {import('./coding.js').CarEncoder} CarEncoder * @typedef {import('./coding.js').IteratorChannel_Writer} IteratorChannel_Writer + * @typedef {import('./limits.js').CarLimits} CarLimits */ const CAR_V1_VERSION = 1 @@ -14,10 +16,14 @@ const CAR_V1_VERSION = 1 * Create a header from an array of roots. * * @param {CID[]} roots + * @param {CarLimits} [limits] * @returns {Uint8Array} */ -export function createHeader (roots) { +export function createHeader (roots, limits = resolveLimits()) { const headerBytes = dagCborEncode({ version: CAR_V1_VERSION, roots }) + if (headerBytes.length > limits.maxAllowedHeaderSize) { + throw new RangeError(`CAR header of length ${headerBytes.length} exceeds maxAllowedHeaderSize of ${limits.maxAllowedHeaderSize}`) + } const varintBytes = varint.encode(headerBytes.length) const header = new Uint8Array(varintBytes.length + headerBytes.length) header.set(varintBytes, 0) @@ -27,9 +33,10 @@ export function createHeader (roots) { /** * @param {IteratorChannel_Writer} writer + * @param {CarLimits} limits * @returns {CarEncoder} */ -function createEncoder (writer) { +function createEncoder (writer, limits) { // none of this is wrapped in a mutex, that needs to happen above this to // avoid overwrites @@ -39,7 +46,7 @@ function createEncoder (writer) { * @returns {Promise} */ async setRoots (roots) { - const bytes = createHeader(roots) + const bytes = createHeader(roots, limits) await writer.write(bytes) }, @@ -49,7 +56,11 @@ function createEncoder (writer) { */ async writeBlock (block) { const { cid, bytes } = block - await writer.write(new Uint8Array(varint.encode(cid.bytes.length + bytes.length))) + const sectionLength = cid.bytes.length + bytes.length + if (sectionLength > limits.maxAllowedSectionSize) { + throw new RangeError(`CAR section of length ${sectionLength} exceeds maxAllowedSectionSize of ${limits.maxAllowedSectionSize}`) + } + await writer.write(new Uint8Array(varint.encode(sectionLength))) await writer.write(cid.bytes) if (bytes.length) { // zero-length blocks are valid, but it'd be safer if we didn't write them diff --git a/src/indexed-reader.js b/src/indexed-reader.js index c15eada..e76021a 100644 --- a/src/indexed-reader.js +++ b/src/indexed-reader.js @@ -187,14 +187,15 @@ export class CarIndexedReader { * @static * @memberof CarIndexedReader * @param {string} path + * @param {import('./api.js').CarCodecOptions} [options] - Optional size limits; see [Size Limits](#size-limits). * @returns {Promise} */ - static async fromFile (path) { + static async fromFile (path, options) { if (typeof path !== 'string') { throw new TypeError('fromFile() requires a file path string') } - const iterable = await CarIndexer.fromIterable(Readable.from(fs.createReadStream(path))) + const iterable = await CarIndexer.fromIterable(Readable.from(fs.createReadStream(path)), options) /** @type {Map} */ const index = new Map() /** @type {string[]} */ diff --git a/src/indexer.js b/src/indexer.js index 03c6b70..2fbf374 100644 --- a/src/indexer.js +++ b/src/indexer.js @@ -88,13 +88,14 @@ export class CarIndexer { * @static * @memberof CarIndexer * @param {Uint8Array} bytes + * @param {import('./api.js').CarCodecOptions} [options] - Optional size limits; see [Size Limits](#size-limits). * @returns {Promise} */ - static async fromBytes (bytes) { + static async fromBytes (bytes, options) { if (!(bytes instanceof Uint8Array)) { throw new TypeError('fromBytes() requires a Uint8Array') } - return decodeIndexerComplete(bytesReader(bytes)) + return decodeIndexerComplete(bytesReader(bytes), options) } /** @@ -107,23 +108,25 @@ export class CarIndexer { * @static * @memberof CarIndexer * @param {AsyncIterable} asyncIterable + * @param {import('./api.js').CarCodecOptions} [options] - Optional size limits; see [Size Limits](#size-limits). * @returns {Promise} */ - static async fromIterable (asyncIterable) { + static async fromIterable (asyncIterable, options) { if (!asyncIterable || !(typeof asyncIterable[Symbol.asyncIterator] === 'function')) { throw new TypeError('fromIterable() requires an async iterable') } - return decodeIndexerComplete(asyncIterableReader(asyncIterable)) + return decodeIndexerComplete(asyncIterableReader(asyncIterable), options) } } /** * @private * @param {BytesReader} reader + * @param {import('./api.js').CarCodecOptions} [options] * @returns {Promise} */ -async function decodeIndexerComplete (reader) { - const decoder = createDecoder(reader) +async function decodeIndexerComplete (reader, options) { + const decoder = createDecoder(reader, options) const { version, roots } = await decoder.header() return new CarIndexer(version, roots, decoder.blocksIndex()) diff --git a/src/iterator.js b/src/iterator.js index 70bd94b..e07f1ee 100644 --- a/src/iterator.js +++ b/src/iterator.js @@ -107,10 +107,11 @@ export class CarBlockIterator extends CarIteratorBase { * @static * @memberof CarBlockIterator * @param {Uint8Array} bytes + * @param {import('./api.js').CarCodecOptions} [options] - Optional size limits; see [Size Limits](#size-limits). * @returns {Promise} */ - static async fromBytes (bytes) { - const { version, roots, iterator } = await fromBytes(bytes) + static async fromBytes (bytes, options) { + const { version, roots, iterator } = await fromBytes(bytes, options) return new CarBlockIterator(version, roots, iterator) } @@ -124,10 +125,11 @@ export class CarBlockIterator extends CarIteratorBase { * @async * @static * @param {AsyncIterable} asyncIterable + * @param {import('./api.js').CarCodecOptions} [options] - Optional size limits; see [Size Limits](#size-limits). * @returns {Promise} */ - static async fromIterable (asyncIterable) { - const { version, roots, iterator } = await fromIterable(asyncIterable) + static async fromIterable (asyncIterable, options) { + const { version, roots, iterator } = await fromIterable(asyncIterable, options) return new CarBlockIterator(version, roots, iterator) } } @@ -207,10 +209,11 @@ export class CarCIDIterator extends CarIteratorBase { * @static * @memberof CarCIDIterator * @param {Uint8Array} bytes + * @param {import('./api.js').CarCodecOptions} [options] - Optional size limits; see [Size Limits](#size-limits). * @returns {Promise} */ - static async fromBytes (bytes) { - const { version, roots, iterator } = await fromBytes(bytes) + static async fromBytes (bytes, options) { + const { version, roots, iterator } = await fromBytes(bytes, options) return new CarCIDIterator(version, roots, iterator) } @@ -225,43 +228,47 @@ export class CarCIDIterator extends CarIteratorBase { * @static * @memberof CarCIDIterator * @param {AsyncIterable} asyncIterable + * @param {import('./api.js').CarCodecOptions} [options] - Optional size limits; see [Size Limits](#size-limits). * @returns {Promise} */ - static async fromIterable (asyncIterable) { - const { version, roots, iterator } = await fromIterable(asyncIterable) + static async fromIterable (asyncIterable, options) { + const { version, roots, iterator } = await fromIterable(asyncIterable, options) return new CarCIDIterator(version, roots, iterator) } } /** * @param {Uint8Array} bytes + * @param {import('./api.js').CarCodecOptions} [options] * @returns {Promise<{ version:number, roots:CID[], iterator:AsyncIterable}>} */ -async function fromBytes (bytes) { +async function fromBytes (bytes, options) { if (!(bytes instanceof Uint8Array)) { throw new TypeError('fromBytes() requires a Uint8Array') } - return decodeIterator(bytesReader(bytes)) + return decodeIterator(bytesReader(bytes), options) } /** * @param {AsyncIterable} asyncIterable + * @param {import('./api.js').CarCodecOptions} [options] * @returns {Promise<{ version:number, roots:CID[], iterator:AsyncIterable}>} */ -async function fromIterable (asyncIterable) { +async function fromIterable (asyncIterable, options) { if (!asyncIterable || !(typeof asyncIterable[Symbol.asyncIterator] === 'function')) { throw new TypeError('fromIterable() requires an async iterable') } - return decodeIterator(asyncIterableReader(asyncIterable)) + return decodeIterator(asyncIterableReader(asyncIterable), options) } /** * @private * @param {BytesReader} reader + * @param {import('./api.js').CarCodecOptions} [options] * @returns {Promise<{ version:number, roots:CID[], iterator:AsyncIterable}>} */ -async function decodeIterator (reader) { - const decoder = createDecoder(reader) +async function decodeIterator (reader, options) { + const decoder = createDecoder(reader, options) const { version, roots } = await decoder.header() return { version, roots, iterator: decoder.blocks() } } diff --git a/src/limits.js b/src/limits.js new file mode 100644 index 0000000..06f50ab --- /dev/null +++ b/src/limits.js @@ -0,0 +1,43 @@ +/** + * @typedef {import('./api.js').CarCodecOptions} CarCodecOptions + * @typedef {Required} CarLimits + */ + +// Matches go-car's DefaultMaxAllowedHeaderSize / DefaultMaxAllowedSectionSize +// (v2/internal/carv1/car.go). +export const DEFAULT_MAX_ALLOWED_HEADER_SIZE = 32 << 20 // 32MiB +export const DEFAULT_MAX_ALLOWED_SECTION_SIZE = 8 << 20 // 8MiB + +/** + * Validate a single provided cap. `undefined` is not "provided" and is left for + * the caller to default. Rejects negatives, non-integers, NaN, Infinity and + * non-numbers with a single predicate. + * + * @param {string} name + * @param {number|undefined} value + * @param {number} fallback + * @returns {number} + */ +function resolve (name, value, fallback) { + if (value === undefined) { + return fallback + } + if (!Number.isSafeInteger(value) || value < 0) { + throw new TypeError(`${name} must be a non-negative safe integer, got ${value}`) + } + return value +} + +/** + * Resolve a `CarCodecOptions` into a complete set of caps, applying the default + * for each option left unset and validating any value provided. + * + * @param {CarCodecOptions} [options] + * @returns {CarLimits} + */ +export function resolveLimits (options) { + return { + maxAllowedHeaderSize: resolve('maxAllowedHeaderSize', options?.maxAllowedHeaderSize, DEFAULT_MAX_ALLOWED_HEADER_SIZE), + maxAllowedSectionSize: resolve('maxAllowedSectionSize', options?.maxAllowedSectionSize, DEFAULT_MAX_ALLOWED_SECTION_SIZE) + } +} diff --git a/src/reader-browser.js b/src/reader-browser.js index b226207..56d23a1 100644 --- a/src/reader-browser.js +++ b/src/reader-browser.js @@ -142,13 +142,14 @@ export class CarReader { * @static * @memberof CarReader * @param {Uint8Array} bytes + * @param {import('./api.js').CarCodecOptions} [options] - Optional size limits; see [Size Limits](#size-limits). * @returns {Promise} */ - static async fromBytes (bytes) { + static async fromBytes (bytes, options) { if (!(bytes instanceof Uint8Array)) { throw new TypeError('fromBytes() requires a Uint8Array') } - return decodeReaderComplete(bytesReader(bytes)) + return decodeReaderComplete(bytesReader(bytes), options) } /** @@ -165,23 +166,25 @@ export class CarReader { * @static * @memberof CarReader * @param {AsyncIterable} asyncIterable + * @param {import('./api.js').CarCodecOptions} [options] - Optional size limits; see [Size Limits](#size-limits). * @returns {Promise} */ - static async fromIterable (asyncIterable) { + static async fromIterable (asyncIterable, options) { if (!asyncIterable || !(typeof asyncIterable[Symbol.asyncIterator] === 'function')) { throw new TypeError('fromIterable() requires an async iterable') } - return decodeReaderComplete(asyncIterableReader(asyncIterable)) + return decodeReaderComplete(asyncIterableReader(asyncIterable), options) } } /** * @private * @param {BytesReader} reader + * @param {import('./api.js').CarCodecOptions} [options] * @returns {Promise} */ -export async function decodeReaderComplete (reader) { - const decoder = createDecoder(reader) +export async function decodeReaderComplete (reader, options) { + const decoder = createDecoder(reader, options) const header = await decoder.header() const blocks = [] for await (const block of decoder.blocks()) { diff --git a/src/writer-browser.js b/src/writer-browser.js index 90f43a3..3dc50c3 100644 --- a/src/writer-browser.js +++ b/src/writer-browser.js @@ -2,6 +2,7 @@ import { CID } from 'multiformats/cid' import { bytesReader, readHeader } from './decoder.js' import { createEncoder, createHeader } from './encoder.js' import { create as iteratorChannel } from './iterator-channel.js' +import { resolveLimits } from './limits.js' /** * @typedef {import('./api.js').Block} Block @@ -121,12 +122,13 @@ export class CarWriter { * @static * @memberof CarWriter * @param {CID[] | CID | void} roots + * @param {import('./api.js').CarCodecOptions} [options] - Optional size limits; see [Size Limits](#size-limits). * @returns {WriterChannel} The channel takes the form of * `{ writer:CarWriter, out:AsyncIterable }`. */ - static create (roots) { + static create (roots, options) { roots = toRoots(roots) - const { encoder, iterator } = encodeWriter() + const { encoder, iterator } = encodeWriter(resolveLimits(options)) const writer = new CarWriter(roots, encoder) const out = new CarWriterOut(iterator) return { writer, out } @@ -143,11 +145,12 @@ export class CarWriter { * @async * @static * @memberof CarWriter + * @param {import('./api.js').CarCodecOptions} [options] - Optional size limits; see [Size Limits](#size-limits). * @returns {WriterChannel} The channel takes the form of * `{ writer:CarWriter, out:AsyncIterable }`. */ - static createAppender () { - const { encoder, iterator } = encodeWriter() + static createAppender (options) { + const { encoder, iterator } = encodeWriter(resolveLimits(options)) encoder.setRoots = () => Promise.resolve() const writer = new CarWriter([], encoder) const out = new CarWriterOut(iterator) @@ -210,11 +213,14 @@ export class CarWriterOut { } } -function encodeWriter () { +/** + * @param {import('./limits.js').CarLimits} limits + */ +function encodeWriter (limits) { /** @type {IteratorChannel} */ const iw = iteratorChannel() const { writer, iterator } = iw - const encoder = createEncoder(writer) + const encoder = createEncoder(writer, limits) return { encoder, iterator } } diff --git a/test/node-test-indexed-reader.js b/test/node-test-indexed-reader.js index 4021a66..6fb22c9 100644 --- a/test/node-test-indexed-reader.js +++ b/test/node-test-indexed-reader.js @@ -1,9 +1,13 @@ /* eslint-env mocha */ +import { promises as fs } from 'fs' +import os from 'os' import path from 'path' import { fileURLToPath } from 'url' +import { encode as cbEncode } from '@ipld/dag-cbor' +import { encode as vEncode } from 'varint' import { CarIndexedReader } from '../src/indexed-reader.js' -import { assert, goCarIndex } from './common.js' +import { assert, carBytes, goCarIndex, rndCid } from './common.js' import { verifyRoots, verifyHas, @@ -43,3 +47,51 @@ describe('CarIndexedReader fromFile()', () => { } }) }) + +describe('CarIndexedReader fromFile() size limits', () => { + /** @param {Uint8Array[]} chunks */ + const concatBytes = (chunks) => { + const length = chunks.reduce((p, c) => p + c.length, 0) + const bytes = new Uint8Array(length) + let off = 0 + for (const chunk of chunks) { + bytes.set(chunk, off) + off += chunk.length + } + return bytes + } + const emptyHeader = cbEncode({ version: 1, roots: [] }) + const validV1Header = concatBytes([Uint8Array.from(vEncode(emptyHeader.length)), emptyHeader]) + + /** + * @param {Uint8Array} data + * @returns {Promise} + */ + const writeTmp = async (data) => { + const p = path.join(os.tmpdir(), `js-car-limits-${data.length}-${data[data.length - 1]}.car`) + await fs.writeFile(p, data) + return p + } + + // Regression guard: fromFile() forwards no options here, so this only + // passes because CarIndexer applies the default cap on its own. + it('rejects an oversized section from the length prefix (default cap)', async () => { + const cid = rndCid.bytes + const section = concatBytes([Uint8Array.from(vEncode(1_000_000_000 + cid.length)), cid]) + const p = await writeTmp(concatBytes([validV1Header, section])) + try { + await assert.isRejected(CarIndexedReader.fromFile(p), RangeError, 'maxAllowedSectionSize') + } finally { + await fs.unlink(p) + } + }) + + it('applies an explicit section-cap override through fromFile', async () => { + const p = await writeTmp(carBytes) + try { + await assert.isRejected(CarIndexedReader.fromFile(p, { maxAllowedSectionSize: 0 }), RangeError, 'maxAllowedSectionSize') + } finally { + await fs.unlink(p) + } + }) +}) diff --git a/test/test-decode-limits.spec.js b/test/test-decode-limits.spec.js new file mode 100644 index 0000000..27d6eae --- /dev/null +++ b/test/test-decode-limits.spec.js @@ -0,0 +1,245 @@ +/* eslint-env mocha */ + +import { encode as cbEncode } from '@ipld/dag-cbor' +import { encode as vEncode } from 'varint' +import { CarBufferReader } from '../src/buffer-reader.js' +import { CarIndexer } from '../src/indexer.js' +import { CarBlockIterator } from '../src/iterator.js' +import { DEFAULT_MAX_ALLOWED_SECTION_SIZE } from '../src/limits.js' +import { CarReader } from '../src/reader.js' +import { assert, carBytes, goCarV2Bytes, makeIterable, rndCid } from './common.js' + +// Async entry points, each with a fromBytes and a fromIterable form. CarReader +// fully decodes during construction and exposes blocks() after; the others +// decode lazily and yield via iteration. +const ENTRIES = [ + { name: 'CarBlockIterator', cls: CarBlockIterator }, + { name: 'CarReader', cls: CarReader }, + { name: 'CarIndexer', cls: CarIndexer } +] +const [blockIter] = ENTRIES + +/** @param {{ name: string }} entry */ +const isReader = (entry) => entry.name === 'CarReader' + +/** + * Drive a decode to completion (header + every block) and return the block + * count. Rejects if decoding rejects at any point. + * + * @param {{ name: string, cls: any }} entry + * @param {Uint8Array} data + * @param {import('../src/api.js').CarCodecOptions} [options] + * @param {'fromBytes'|'fromIterable'} [via] + * @param {number} [chunkSize] + */ +async function decodeAll (entry, data, options, via = 'fromIterable', chunkSize = 64) { + const input = via === 'fromBytes' ? data : makeIterable(data, chunkSize) + const result = await entry.cls[via](input, options) + const iterable = isReader(entry) ? result.blocks() : result + let count = 0 + for await (const item of iterable) { + if (item) { + count++ + } + } + return count +} + +/** @param {Uint8Array[]} chunks */ +function concatBytes (chunks) { + const length = chunks.reduce((p, c) => p + c.length, 0) + const bytes = new Uint8Array(length) + let off = 0 + for (const chunk of chunks) { + bytes.set(chunk, off) + off += chunk.length + } + return bytes +} + +/** @param {Uint8Array} payload */ +function lengthPrefixed (payload) { + return concatBytes([Uint8Array.from(vEncode(payload.length)), payload]) +} + +const validV1Header = lengthPrefixed(cbEncode({ version: 1, roots: [] })) + +describe('decode size limits', () => { + describe('defaults are on with no options', () => { + for (const via of /** @type {const} */(['fromBytes', 'fromIterable'])) { + for (const entry of ENTRIES) { + it(`${entry.name}.${via}: normal CAR decodes under the defaults`, async () => { + const count = await decodeAll(entry, carBytes, undefined, via) + assert.ok(count > 0) + }) + } + } + + it('v2 CAR decodes under the defaults', async () => { + assert.ok(await decodeAll(blockIter, goCarV2Bytes, undefined) > 0) + }) + }) + + describe('rejects before buffering the body', () => { + // A length prefix declaring a huge section, followed by only a CID and no + // body. Under the default section cap the decoder must reject from the + // prefix (RangeError), not stream-and-buffer and fail with + // "Unexpected end of data". + const huge = 1_000_000_000 + + it('section over the default cap rejects with RangeError', async () => { + const cid = rndCid.bytes + const section = concatBytes([Uint8Array.from(vEncode(huge + cid.length)), cid]) + const data = concatBytes([validV1Header, section]) + await assert.isRejected(decodeAll(blockIter, data), RangeError, 'maxAllowedSectionSize') + }) + + it('lifting the cap to MAX_SAFE_INTEGER changes the failure to end-of-data', async () => { + const cid = rndCid.bytes + const section = concatBytes([Uint8Array.from(vEncode(huge + cid.length)), cid]) + const data = concatBytes([validV1Header, section]) + await assert.isRejected( + decodeAll(blockIter, data, { maxAllowedSectionSize: Number.MAX_SAFE_INTEGER }), + Error, + 'Unexpected end of data' + ) + }) + }) + + describe('section cap boundaries', () => { + it('equal to the cap passes, cap+1 rejects', async () => { + // largest section (cid + body) in carBytes, learned from an uncapped decode + const iter = await CarBlockIterator.fromIterable(makeIterable(carBytes, 64)) + let maxSection = 0 + for await (const { cid, bytes } of iter) { + maxSection = Math.max(maxSection, cid.bytes.length + bytes.length) + } + assert.ok(maxSection > 0) + assert.ok(await decodeAll(blockIter, carBytes, { maxAllowedSectionSize: maxSection }) > 0) + // one below the largest section: rejected + await assert.isRejected( + decodeAll(blockIter, carBytes, { maxAllowedSectionSize: maxSection - 1 }), + RangeError, + 'maxAllowedSectionSize' + ) + }) + + it('0 rejects every section', async () => { + await assert.isRejected(decodeAll(blockIter, carBytes, { maxAllowedSectionSize: 0 }), RangeError, 'maxAllowedSectionSize') + }) + }) + + describe('header cap', () => { + it('rejects a v1 header over the cap', async () => { + // carBytes' header is 99 bytes; cap well below + await assert.isRejected(decodeAll(blockIter, carBytes, { maxAllowedHeaderSize: 10 }), RangeError, 'maxAllowedHeaderSize') + }) + + it('fires on the inner v1 header of a v2 CAR (recursion passes the cap)', async () => { + // Set the cap to exactly the v2 pragma length: the pragma passes (equal), + // the larger inner v1 header fails, proving the recursive readHeader gets + // the same cap. + const pragmaLen = cbEncode({ version: 2 }).length + await assert.isRejected(decodeAll(blockIter, goCarV2Bytes, { maxAllowedHeaderSize: pragmaLen }), RangeError, 'maxAllowedHeaderSize') + }) + }) + + describe('CID bounded by its section', () => { + it('rejects a CIDv1 declaring a multihash past the section end', async () => { + // CIDv1 raw sha2-256 declaring a 1000-byte digest, inside a section only + // large enough for the CID prefix. Section passes the section cap; the CID + // does not fit the section. + const cidPrefix = concatBytes([Uint8Array.from([0x01, 0x55, 0x12]), Uint8Array.from(vEncode(1000))]) + const section = concatBytes([Uint8Array.from(vEncode(cidPrefix.length + 1)), cidPrefix]) + const data = concatBytes([validV1Header, section]) + await assert.isRejected(decodeAll(blockIter, data), Error, 'exceeds section length') + }) + + it('rejects a sub-34-byte section opening with the CIDv0 prefix', async () => { + // 0x12 0x20 triggers the CIDv0 branch; a 20-byte section cannot hold a + // 34-byte CIDv0, so it must throw rather than read into the next section. + const section = concatBytes([Uint8Array.from(vEncode(20)), Uint8Array.from([0x12, 0x20]), new Uint8Array(18)]) + const data = concatBytes([validV1Header, section]) + await assert.isRejected(decodeAll(blockIter, data), Error, 'exceeds section length') + }) + }) + + describe('digest cap (MAX_DIGEST_ALLOC)', () => { + it('rejects a CIDv1 declaring a digest over 32 MiB, with the section cap lifted', async () => { + const hugeDigest = (32 << 20) + 1 + const cidPrefix = concatBytes([Uint8Array.from([0x01, 0x55, 0x12]), Uint8Array.from(vEncode(hugeDigest))]) + const section = concatBytes([Uint8Array.from(vEncode(cidPrefix.length + 1)), cidPrefix]) + const data = concatBytes([validV1Header, section]) + // Section cap lifted so the section bound passes; the digest cap must fire + // first (before any digest is read). + await assert.isRejected( + decodeAll(blockIter, data, { maxAllowedSectionSize: Number.MAX_SAFE_INTEGER }), + RangeError, + 'CID digest' + ) + }) + }) + + describe('CarIndexer rejects from the prefix instead of stream-and-discarding', () => { + it('rejects an oversized section without pulling its body', async () => { + const cid = rndCid.bytes + const section = concatBytes([Uint8Array.from(vEncode(1_000_000_000 + cid.length)), cid]) + const data = concatBytes([validV1Header, section]) + await assert.isRejected(decodeAll({ name: 'CarIndexer', cls: CarIndexer }, data), RangeError, 'maxAllowedSectionSize') + }) + }) + + describe('option validation', () => { + for (const bad of [-1, 1.5, NaN, Infinity, '8']) { + it(`rejects maxAllowedSectionSize=${String(bad)} with TypeError`, async () => { + await assert.isRejected( + // @ts-expect-error deliberately bad input + decodeAll(blockIter, carBytes, { maxAllowedSectionSize: bad }), + TypeError, + 'must be a non-negative safe integer' + ) + }) + } + }) + + it('uses the documented default section size constant', () => { + assert.strictEqual(DEFAULT_MAX_ALLOWED_SECTION_SIZE, 8 << 20) + }) + + describe('CarBufferReader (in-memory sync)', () => { + /** + * @param {Uint8Array} bytes + * @param {import('../src/api.js').CarCodecOptions} [options] + */ + const count = (bytes, options) => CarBufferReader.fromBytes(bytes, options).blocks().length + + it('decodes a normal CAR under the defaults', () => { + assert.ok(count(carBytes) > 0) + }) + + it('rejects a section over the default cap', () => { + const cid = rndCid.bytes + const section = concatBytes([Uint8Array.from(vEncode(1_000_000_000 + cid.length)), cid]) + const data = concatBytes([validV1Header, section]) + assert.throws(() => count(data), RangeError, 'maxAllowedSectionSize') + }) + + it('rejects a header over an explicit cap', () => { + assert.throws(() => count(carBytes, { maxAllowedHeaderSize: 10 }), RangeError, 'maxAllowedHeaderSize') + }) + + it('a section exactly at an explicit cap passes, cap+1 rejects', () => { + const cid = rndCid.bytes + const body = new Uint8Array(8) + const cap = cid.length + body.length + const section = concatBytes([Uint8Array.from(vEncode(cap)), cid, body]) + const data = concatBytes([validV1Header, section]) + assert.ok(count(data, { maxAllowedSectionSize: cap }) > 0) + assert.throws(() => count(data, { maxAllowedSectionSize: cap - 1 }), RangeError, 'maxAllowedSectionSize') + }) + + it('rejects a bad option value with TypeError', () => { + assert.throws(() => count(carBytes, { maxAllowedSectionSize: -1 }), TypeError, 'must be a non-negative safe integer') + }) + }) +}) diff --git a/test/test-encode-limits.spec.js b/test/test-encode-limits.spec.js new file mode 100644 index 0000000..23c41f5 --- /dev/null +++ b/test/test-encode-limits.spec.js @@ -0,0 +1,228 @@ +/* eslint-env mocha */ + +import { encode as cbEncode } from '@ipld/dag-cbor' +import { expect } from 'aegir/chai' +import { CarBufferReader } from '../src/buffer-reader.js' +import * as CarBufferWriter from '../src/buffer-writer.js' +import { DEFAULT_MAX_ALLOWED_SECTION_SIZE } from '../src/limits.js' +import { CarReader } from '../src/reader.js' +import { CarWriter } from '../src/writer.js' +import { assert, makeData, makeIterable } from './common.js' + +/** @param {Uint8Array[]} chunks */ +function concatBytes (chunks) { + const length = chunks.reduce((p, c) => p + c.length, 0) + const bytes = new Uint8Array(length) + let off = 0 + for (const chunk of chunks) { + bytes.set(chunk, off) + off += chunk.length + } + return bytes +} + +/** @param {AsyncIterable} iterable */ +function collector (iterable) { + const chunks = [] + return (async () => { + for await (const chunk of iterable) { + chunks.push(chunk) + } + return concatBytes(chunks) + })() +} + +/** + * Write roots + blocks through CarWriter and return the full CAR bytes. + * Rejects if any put/close rejects. + * + * @param {import('../src/api.js').CID[]} roots + * @param {import('../src/api.js').Block[]} blocks + * @param {import('../src/api.js').CarCodecOptions} [options] + */ +async function writeAll (roots, blocks, options) { + const { writer, out } = CarWriter.create(roots, options) + const collection = collector(out) + const writes = blocks.map((b) => writer.put(b)) + writes.push(writer.close()) + await Promise.all(writes) + return collection +} + +/** + * Build a synthetic block whose section (cid + body) is `sectionSize` bytes. + * + * @param {import('../src/api.js').CID} cid + * @param {number} sectionSize + */ +function blockOfSection (cid, sectionSize) { + return { cid, bytes: new Uint8Array(sectionSize - cid.bytes.length) } +} + +describe('encode size limits', () => { + /** @type {import('../src/api.js').Block[]} */ + let rawBlocks + /** @type {import('../src/api.js').CID[]} */ + let roots + + before(async () => { + const data = await makeData() + rawBlocks = data.rawBlocks // CIDv1 raw, 4-byte bodies + roots = [data.cborBlocks[0].cid, data.cborBlocks[1].cid] + }) + + describe('CarWriter defaults', () => { + it('writes a normal CAR with no options', async () => { + const bytes = await writeAll(roots, [rawBlocks[0]], undefined) + assert.ok(bytes.length > 0) + }) + + it('rejects a section over the default 8 MiB cap with no options', async () => { + const big = blockOfSection(rawBlocks[0].cid, DEFAULT_MAX_ALLOWED_SECTION_SIZE + 1) + const { writer, out } = CarWriter.create(roots) + await out[Symbol.asyncIterator]().next() // drain the header so writeBlock runs + await expect(writer.put(big)).to.eventually.be.rejectedWith(RangeError, /maxAllowedSectionSize/) + }) + }) + + describe('section cap boundaries', () => { + it('a section exactly at the cap is allowed, one over rejects', async () => { + const cap = rawBlocks[0].cid.bytes.length + 8 // small explicit cap + // exactly at the cap: allowed + await writeAll(roots, [blockOfSection(rawBlocks[0].cid, cap)], { maxAllowedSectionSize: cap, maxAllowedHeaderSize: 1 << 20 }) + // one over: rejected + const { writer, out } = CarWriter.create(roots, { maxAllowedSectionSize: cap, maxAllowedHeaderSize: 1 << 20 }) + await out[Symbol.asyncIterator]().next() + await expect(writer.put(blockOfSection(rawBlocks[0].cid, cap + 1))).to.eventually.be.rejectedWith(RangeError, /maxAllowedSectionSize/) + }) + }) + + describe('header cap surfaces via the mutex', () => { + it('rejects an over-cap header from the first put', async () => { + const { writer } = CarWriter.create(roots, { maxAllowedHeaderSize: 10 }) + await expect(writer.put(rawBlocks[0])).to.eventually.be.rejectedWith(RangeError, /maxAllowedHeaderSize/) + }) + + it('allows a header exactly equal to the cap', async () => { + const headerLen = cbEncode({ version: 1, roots }).length + const bytes = await writeAll(roots, [rawBlocks[0]], { maxAllowedHeaderSize: headerLen }) + assert.ok(bytes.length > 0) + }) + }) + + describe('createAppender', () => { + it('enforces the section cap on appended blocks (no header cap applies)', async () => { + const big = blockOfSection(rawBlocks[0].cid, DEFAULT_MAX_ALLOWED_SECTION_SIZE + 1) + const { writer } = CarWriter.createAppender() + await expect(writer.put(big)).to.eventually.be.rejectedWith(RangeError, /maxAllowedSectionSize/) + }) + + it('writes normally within the caps', async () => { + const { writer, out } = CarWriter.createAppender() + const collection = collector(out) + await writer.put(rawBlocks[0]) + await writer.close() + assert.ok((await collection).length > 0) + }) + }) + + describe('round-trip guarantee', () => { + it('a CAR written with a raised cap decodes under the same profile', async () => { + const data = await makeData() + const profile = { maxAllowedSectionSize: Number.MAX_SAFE_INTEGER, maxAllowedHeaderSize: Number.MAX_SAFE_INTEGER } + const bytes = await writeAll(roots, data.allBlocksFlattened, profile) + const reader = await CarReader.fromIterable(makeIterable(bytes, 64), profile) + const decoded = [] + for await (const block of reader.blocks()) { + decoded.push(block) + } + assert.strictEqual(decoded.length, data.allBlocksFlattened.length) + }) + + it('default-written bytes are identical to explicitly-default-capped bytes', async () => { + const data = await makeData() + const baseline = await writeAll(roots, data.allBlocksFlattened, undefined) + const capped = await writeAll(roots, data.allBlocksFlattened, { + maxAllowedHeaderSize: 32 << 20, + maxAllowedSectionSize: 8 << 20 + }) + assert.deepEqual(capped, baseline) + }) + }) + + describe('option validation', () => { + for (const bad of [-1, 1.5, NaN, Infinity, '8']) { + it(`rejects maxAllowedSectionSize=${String(bad)} with TypeError`, async () => { + // toRoots(roots) runs first inside create() but only validates shape, + // so any valid roots value passes through unchanged before + // resolveLimits(options) throws synchronously here. + // @ts-expect-error deliberately bad input + assert.throws(() => CarWriter.create([], { maxAllowedSectionSize: bad }), TypeError, 'must be a non-negative safe integer') + }) + } + }) + + describe('CarBufferWriter', () => { + it('rejects an over-cap block even when the buffer has room (cap before capacity)', () => { + const cid = rawBlocks[0].cid + const cap = cid.bytes.length + 8 + // buffer is large enough for the block; the cap must still reject it + const writer = CarBufferWriter.createWriter(new ArrayBuffer(1 << 16), { + roots: [], + maxAllowedSectionSize: cap + }) + const overCap = { cid, bytes: new Uint8Array((cap + 1) - cid.bytes.length) } + assert.throws(() => writer.write(overCap), RangeError, 'maxAllowedSectionSize') + }) + + it('the section-cap error precedes the capacity error', () => { + const cid = rawBlocks[0].cid + const cap = cid.bytes.length + 8 + // tiny buffer AND over cap: must report the cap, not capacity + const writer = CarBufferWriter.createWriter(new ArrayBuffer(64), { + roots: [], + maxAllowedSectionSize: cap + }) + const big = { cid, bytes: new Uint8Array(1 << 20) } + assert.throws(() => writer.write(big), RangeError, 'maxAllowedSectionSize') + }) + + it('writes a block whose section is exactly at the cap without throwing', () => { + const cid = rawBlocks[0].cid + const cap = cid.bytes.length + 8 + const writer = CarBufferWriter.createWriter(new ArrayBuffer(1 << 16), { + roots: [], + maxAllowedSectionSize: cap + }) + const atCap = { cid, bytes: new Uint8Array(cap - cid.bytes.length) } + writer.write(atCap) + const bytes = writer.close() + const reader = CarBufferReader.fromBytes(bytes) + assert.strictEqual(reader.blocks().length, 1) + }) + + it('defaults allow a normal block and the result round-trips', () => { + const writer = CarBufferWriter.createWriter(new ArrayBuffer(1 << 16), { roots }) + writer.write(rawBlocks[0]) + const bytes = writer.close() + const reader = CarBufferReader.fromBytes(bytes) + assert.strictEqual(reader.blocks().length, 1) + }) + + it('close rejects a header over an explicit cap', () => { + const writer = CarBufferWriter.createWriter(new ArrayBuffer(1 << 16), { + roots, + maxAllowedHeaderSize: 10 + }) + assert.throws(() => writer.close(), RangeError, 'maxAllowedHeaderSize') + }) + + it('rejects a bad option value with TypeError', () => { + assert.throws( + () => CarBufferWriter.createWriter(new ArrayBuffer(1 << 16), { roots: [], maxAllowedSectionSize: -1 }), + TypeError, + 'must be a non-negative safe integer' + ) + }) + }) +}) diff --git a/test/test-limits.spec.js b/test/test-limits.spec.js new file mode 100644 index 0000000..c01160b --- /dev/null +++ b/test/test-limits.spec.js @@ -0,0 +1,63 @@ +/* eslint-env mocha */ + +import { + DEFAULT_MAX_ALLOWED_HEADER_SIZE, + DEFAULT_MAX_ALLOWED_SECTION_SIZE, + resolveLimits +} from '../src/limits.js' +import { assert } from './common.js' + +describe('resolveLimits', () => { + it('applies both defaults when given undefined', () => { + assert.deepStrictEqual(resolveLimits(), { + maxAllowedHeaderSize: DEFAULT_MAX_ALLOWED_HEADER_SIZE, + maxAllowedSectionSize: DEFAULT_MAX_ALLOWED_SECTION_SIZE + }) + }) + + it('applies both defaults when given an empty object', () => { + assert.deepStrictEqual(resolveLimits({}), { + maxAllowedHeaderSize: DEFAULT_MAX_ALLOWED_HEADER_SIZE, + maxAllowedSectionSize: DEFAULT_MAX_ALLOWED_SECTION_SIZE + }) + }) + + it('overrides only the provided cap, defaults the other', () => { + assert.deepStrictEqual(resolveLimits({ maxAllowedSectionSize: 1024 }), { + maxAllowedHeaderSize: DEFAULT_MAX_ALLOWED_HEADER_SIZE, + maxAllowedSectionSize: 1024 + }) + }) + + it('accepts 0 as a real value (reject every section)', () => { + assert.strictEqual(resolveLimits({ maxAllowedSectionSize: 0 }).maxAllowedSectionSize, 0) + }) + + it('accepts Number.MAX_SAFE_INTEGER as the opt-out', () => { + assert.strictEqual( + resolveLimits({ maxAllowedHeaderSize: Number.MAX_SAFE_INTEGER }).maxAllowedHeaderSize, + Number.MAX_SAFE_INTEGER + ) + }) + + for (const bad of [-1, 1.5, NaN, Infinity, '8']) { + it(`throws TypeError for ${String(bad)}`, () => { + assert.throws( + // @ts-expect-error deliberately bad input + () => resolveLimits({ maxAllowedSectionSize: bad }), + TypeError, + 'maxAllowedSectionSize must be a non-negative safe integer' + ) + }) + } + + // The loop above only exercises the section-cap side of the validation; this + // proves the same predicate is applied to the header cap too. + it('throws TypeError for a negative maxAllowedHeaderSize', () => { + assert.throws( + () => resolveLimits({ maxAllowedHeaderSize: -1 }), + TypeError, + 'maxAllowedHeaderSize must be a non-negative safe integer' + ) + }) +})