Skip to content

Commit 8fe745a

Browse files
committed
chore: editorial fixes
1 parent eea310a commit 8fe745a

2 files changed

Lines changed: 50 additions & 27 deletions

File tree

Lines changed: 41 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# IPIP 0000: TAR Response Format on Web Gateways
1+
# IPIP-288: TAR Response Format on HTTP Gateways
22

33
- Start Date: 2022-06-10
44
- Related Issues:
@@ -48,15 +48,20 @@ directories inside.
4848
However, there are certain behaviors, detailed in the [security section](#security)
4949
that should be handled. To test such behaviors, the following fixtures can be used:
5050

51-
- [`bafybeibfevfxlvxp5vxobr5oapczpf7resxnleb7tkqmdorc4gl5cdva3y`][inside-dag] is a UnixFS
52-
DAG that contains a file with a relative path that points inside the root directory.
53-
Downloading it as a TAR must work.
54-
- [`bafkreict7qp5aqs52445bk4o7iuymf3davw67tpqqiscglujx3w6r7hwoq`][inside-dag-tar] is an
55-
example TAR file that corresponds to the aforementioned UnixFS DAG. Its structure can be
56-
inspected in order to check if new implementations conform to the specification.
57-
- [`bafybeicaj7kvxpcv4neaqzwhrqqmdstu4dhrwfpknrgebq6nzcecfucvyu`][outside-dag] is a UnixFS
58-
DAG that contains a file with a relative path that points outside the root directory.
59-
Downloading it as a TAR must error.
51+
- [`bafybeibfevfxlvxp5vxobr5oapczpf7resxnleb7tkqmdorc4gl5cdva3y`][inside-dag]
52+
is a UnixFS DAG that contains a file with a name that looks like a relative
53+
path that points inside the root directory. Downloading it as a TAR must
54+
work.
55+
56+
- [`bafkreict7qp5aqs52445bk4o7iuymf3davw67tpqqiscglujx3w6r7hwoq`][inside-dag-tar]
57+
is an example TAR file that corresponds to the aforementioned UnixFS DAG. Its
58+
structure can be inspected in order to check if new implementations conform
59+
to the specification.
60+
61+
- [`bafybeicaj7kvxpcv4neaqzwhrqqmdstu4dhrwfpknrgebq6nzcecfucvyu`][outside-dag]
62+
is a UnixFS DAG that contains a file with a name that looks like a relative
63+
path that points outside the root directory. Downloading it as a TAR must
64+
error.
6065

6166
## Design rationale
6267

@@ -78,13 +83,28 @@ downloading it.
7883
CLI users will be able to download a directory with existing tools like `curl` and `tar` without
7984
having to talk to implementation-specific RPC APIs like `/api/v0/get` from Kubo.
8085

86+
Fetching a directory from a local gateway will be as simple as:
87+
88+
```console
89+
$ export DIR_CID=bafybeigccimv3zqm5g4jt363faybagywkvqbrismoquogimy7kvz2sj7sq
90+
$ curl "http://127.0.0.1:8080/ipfs/$DIR_CID?format=tar" | tar xv
91+
bafybeigccimv3zqm5g4jt363faybagywkvqbrismoquogimy7kvz2sj7sq
92+
bafybeigccimv3zqm5g4jt363faybagywkvqbrismoquogimy7kvz2sj7sq/1 - Barrel - Part 1 - alt.txt
93+
bafybeigccimv3zqm5g4jt363faybagywkvqbrismoquogimy7kvz2sj7sq/1 - Barrel - Part 1 - transcript.txt
94+
bafybeigccimv3zqm5g4jt363faybagywkvqbrismoquogimy7kvz2sj7sq/1 - Barrel - Part 1.png
95+
```
96+
8197
### Compatibility
8298

8399
This IPIP is backwards compatible: adds a new opt-in response type, does not
84100
modify preexisting behaviors.
85101

102+
Existing content type `application/x-tar` is used when request is made with an `Accept` header.
103+
86104
### Security
87105

106+
Third-party UnixFS file names may include unexpected values, such as `../`.
107+
88108
Manually created UnixFS DAGs can be turned into malicious TAR files. For example,
89109
if a UnixFS directory contains a file that points at a relative path outside
90110
its root, the unpacking of the TAR file may overwrite local files outside the expected
@@ -102,18 +122,20 @@ suggested to use a CAR file if they want to download the raw files.
102122

103123
### Alternatives
104124

105-
One discussed alternative would be to support uncompressed ZIP files. However, TAR and
106-
TAR-related libraries are already supported and implemented for UnixFS files. Therefore,
107-
the addition of a TAR response format is facilitated, while introduction of ZIP would increase
108-
implementation complexity.
125+
One discussed alternative would be to support uncompressed ZIP files. However,
126+
TAR and TAR-related libraries are already supported by some IPFS
127+
implementations, and are easier to work with in CLI. TAR provides simpler
128+
abstraction, and layering compression on top of TAR stream allows for greater
129+
flexibility than alternative options that come with own, opinionated approaches
130+
to compression.
109131

110-
In addition, we considered supporting [Gzipped TAR](https://github.com/ipfs/go-ipfs/pull/9034).
111-
However, there it may be a vector for DOS attacks since compression requires high CPU power.
132+
In addition, we considered supporting [Gzipped TAR](https://github.com/ipfs/go-ipfs/pull/9034) out of the box,
133+
but decided against it as gzip or alternative compression may be introduced on the HTTP transport layer.
112134

113135
### Copyright
114136

115137
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
116138

117-
[inside-dag]: https://dweb.link/ipfs/bafybeibfevfxlvxp5vxobr5oapczpf7resxnleb7tkqmdorc4gl5cdva3y
118-
[inside-dag-tar]: https://dweb.link/ipfs/bafkreict7qp5aqs52445bk4o7iuymf3davw67tpqqiscglujx3w6r7hwoq
119-
[outside-dag]: https://dweb.link/ipfs/bafybeicaj7kvxpcv4neaqzwhrqqmdstu4dhrwfpknrgebq6nzcecfucvyu
139+
[inside-dag]: https://dweb.link/ipfs/bafybeibfevfxlvxp5vxobr5oapczpf7resxnleb7tkqmdorc4gl5cdva3y?format=car
140+
[inside-dag-tar]: https://dweb.link/ipfs/bafkreict7qp5aqs52445bk4o7iuymf3davw67tpqqiscglujx3w6r7hwoq?format=car
141+
[outside-dag]: https://dweb.link/ipfs/bafybeicaj7kvxpcv4neaqzwhrqqmdstu4dhrwfpknrgebq6nzcecfucvyu?format=car

http-gateways/PATH_GATEWAY.md

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Path Gateway Specification
22

3-
![Status: Work In Progress](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square)
3+
![reliable](https://img.shields.io/badge/status-reliable-green.svg?style=flat-square)
44

55
**Authors**:
66

@@ -181,7 +181,7 @@ For example:
181181

182182
- [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable raw [block](https://docs.ipfs.io/concepts/glossary/#block) to be returned
183183
- [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable [CAR](https://docs.ipfs.io/concepts/glossary/#car) stream to be returned
184-
- [application/x-tar](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types) – returns UnixFS file or a directory as a [TAR](https://en.wikipedia.org/wiki/Tar_(computing)) stream. At the root of the TAR archive, a file or directory, with the CID of the content, is present. Produces 400 Bad Request for content that is not UnixFS.
184+
- [application/x-tar](https://en.wikipedia.org/wiki/Tar_(computing)) – returns UnixFS tree (files and directories) as a [TAR](https://en.wikipedia.org/wiki/Tar_(computing)) stream. Returned tree starts at a root item which name is the same as the requested CID. Produces 400 Bad Request for content that is not UnixFS.
185185
<!-- TODO: https://github.com/ipfs/go-ipfs/issues/8823
186186
- application/vnd.ipld.dag-json OR application/json – requests IPLD Data Model representation serialized into [DAG-JSON format](https://ipld.io/docs/codecs/known/dag-json/)
187187
- application/vnd.ipld.dag-cbor OR application/cbor - requests IPLD Data Model representation serialized into [DAG-CBOR format](https://ipld.io/docs/codecs/known/dag-cbor/)
@@ -366,10 +366,11 @@ and CDNs, implementations should base it on both CID and response type:
366366
- Example: `Etag: "DirIndex-2B423AF_CID-bafy…foo"`
367367

368368
- When a gateway can’t guarantee byte-for-byte identical responses, a “weak”
369-
etag should be used. For example, if CAR is streamed, and blocks arrive in
370-
non-deterministic order, the response should have `Etag: W/"bafy…foo.car"`.
371-
If TAR is generated by traversing an UnixFS directory in non-deterministic
372-
order, the response should have `Etag: W/"bafy…foo.tar"`.
369+
etag should be used.
370+
- Example: If CAR is streamed, and blocks arrive in non-deterministic order,
371+
the response should have `Etag: W/"bafy…foo.car"`.
372+
- Example: If TAR stream is generated by traversing an UnixFS directory in non-deterministic
373+
order, the response should have `Etag: W/"bafy…foo.x-tar"`.
373374

374375
- When responding to [`Range`](#range-request-header) request, a strong `Etag`
375376
should be based on requested range in addition to CID and response format:
@@ -594,9 +595,9 @@ Data sent with HTTP response depends on the type of requested IPFS resource:
594595
- Raw block
595596
- Opaque bytes, see [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw)
596597
- CAR
597-
- CAR file or stream, see [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car)
598+
- Arbitrary DAG as a verifiable CAR file or a stream, see [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car)
598599
- TAR
599-
- TAR file or stream, see [application/x-tar](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types)
600+
- Deserialized UnixFS files and directories as a TAR file or a stream, see [application/x-tar](https://en.wikipedia.org/wiki/Tar_(computing))
600601
<!-- TODO: https://github.com/ipfs/go-ipfs/issues/8823
601602
- dag-json / dag-cbor
602603
- See [https://github.com/ipfs/go-ipfs/issues/8823](https://github.com/ipfs/go-ipfs/issues/8823)

0 commit comments

Comments
 (0)