diff --git a/AGENTS.md b/AGENTS.md index 920881a0..f571837e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -35,7 +35,7 @@ Apache-2.0 · Default branch: `master`. | `internal/client` | Protocol client (calls, notifications, events, handshake) and starting a local server | | `internal/tui` | The TUI: tree, tabs/splits/scoping, keys, mouse, views (changes, sessions, setup), dialogs, settings, brain bar, updates | | `internal/remote` | How a machine is reached (`Transport`: ssh, a sandbox through its provider's ssh gateway with a fresh token, or a local command), ssh config and commands, remote install, bridging, machine catalog, release downloads, cross builds | -| `internal/sandbox` | Hosted sandbox providers (Daytona, boat.dev, Devin Cloud, Tensorlake): create, start, stop, delete, list conch's own, ssh access — a fresh token, or a key of your own authorized in the sandbox. And providers conch has no code for, described by a file (`custom.go`, `providerfile.md` the format): `$CONCH_HOME/providers/NAME.toml` says which call does what and how ssh gets in, read by one provider of conch's own — a description, never generated code, so a wrong file fails rather than does something else. `conch sandbox adapt` (`cmd/conch/adapt.go`) has an agent draft one from the provider's API docs in a pane of its own; the key stays conch's. Format 2 (`format = 2`: a wrapped reply's `[get] item`, `{public_key}` in a create body via `sandbox.PublicKey`, which `internal/remote` sets, and `list[key=value]` selectors in paths) is refused by older conches as an unknown key, which is the point. `registry.go` is the provider registry (github.com/Amitgb14/conch-sandbox-providers, its index on Pages, `CONCH_PROVIDERS_URL` to point elsewhere): install checks the file's hash against the index and records it in `providers/registry.json`, so update can leave an edited file alone and asks again when a file's hosts change — a provider file sends the person's key to its `api` host. Publishing (`cmd/conch/registry.go`) is a pull request through `gh api`, after a live check with the key; tests never reach the real registry (`TestMain` points it at a dead address) | +| `internal/sandbox` | Hosted sandbox providers (Daytona, boat.dev, Devin Cloud, Tensorlake): create, start, stop, delete, list conch's own, ssh access — a fresh token, or a key of your own authorized in the sandbox. And providers conch has no code for, described by a file (`custom.go`, `providerfile.md` the format): `$CONCH_HOME/providers/NAME.toml` says which call does what and how ssh gets in, read by one provider of conch's own — a description, never generated code, so a wrong file fails rather than does something else. `conch sandbox adapt` (`cmd/conch/adapt.go`) has an agent draft one from the provider's API docs in a pane of its own; the key stays conch's. Format 2 (`format = 2`: a wrapped reply's `[get] item`, `{public_key}` in a create body via `sandbox.PublicKey`, which `internal/remote` sets, and `list[key=value]` selectors in paths) and format 3 (`headers`, top-level and per call, with `{uuid}` new for each request — an `Idempotency-Key`, a delete's confirmation; never the key, never a header conch sets) and `[ssh] port = "{reply.PATH}"` (`SSHPort`, a gateway that names its port per connection) are refused by older conches as an unknown key, which is the point. `registry.go` is the provider registry (github.com/Amitgb14/conch-sandbox-providers, its index on Pages, `CONCH_PROVIDERS_URL` to point elsewhere): install checks the file's hash against the index and records it in `providers/registry.json`, so update can leave an edited file alone and asks again when a file's hosts change — a provider file sends the person's key to its `api` host. Publishing (`cmd/conch/registry.go`) is a pull request through `gh api`, after a live check with the key; tests never reach the real registry (`TestMain` points it at a dead address) | | `internal/phone` | The gateway `conch web` runs for a phone: HTTP and a WebSocket in front of every machine's server (a client of each, as the TUI is — `machines.go` keeps one per machine, reached in the background so a phone never waits on ssh, and a pane is addressed `machine:pane`; capabilities are asked of *each* machine, since every one runs its own conch — `phoneNeeds` against that machine's handshake, said in its `Behind` before anything is tried, and refused by `lacks` with that machine named and `conch machine upgrade` rather than "reload the server", which is the wrong advice about a machine nobody is sitting at), pairing codes, per-device tokens kept hashed in `phone.json`, the socket deflated with the window kept across messages (`CompressionContextTakeover`), since a frame is the screen and the next frame is that screen slightly changed — measured 14.9× fewer bytes end to end and *less* processor time a frame than compressing each alone, for a 32 KB window and a flate writer a socket; `view`/`reply`/`full` checked on every route and socket message — and *where*, since the gateway reaches every machine with the person's credentials: a device reaches this computer and the machines named for it (`Device.Machines`, `Reaches`), which is why `machinesFor`, `machineFor`, `paneOn`, `everyAgent`, `everyPane` and the socket's own `paneOn` all take the device, a machine it was not given is left out of every list rather than merely refused, pushes about one are not sent, and an older `phone.json` loads as this computer alone, reading a waiting agent's choices off its screen — each socket holds its *own* connection per machine, since frames come from its own subscriptions and one phone's subscribing would otherwise decide another's, so a message naming a machine this socket has not joined dials it and waits (`machineReady`) rather than refusing: refusing meant the agent a phone had just been shown would not open for the first seconds after the app started, which is how it was found — and the phone's app (`ui/`: plain HTML, CSS and ES modules embedded as they are, no build step — each file deflated once at first use and served that way to a phone that takes it (`uiDeflated`, 102 KB of shell as 34 KB), since nothing in `ui/` can change while conch runs; its logic in `ui/lib.mjs`, tested with node from `uitest/` through `TestUILogic`, which skips without node). Its shapes are a contract (`api.go`) a test pins | | `internal/relay` | How a phone reaches conch web without Tailscale: `conch relay` (`relay.go`, the server — rooms keyed by a hash of a secret only the gateway holds, a phone's WebSocket spliced to one the gateway opens for it, nothing stored), the gateway's link out to it (`link.go`), and the tunnel inside (`tunnel.go`): a Noise NK session (`Noise_NK_25519_AESGCM_SHA256`, flynn/noise) keyed by the gateway's key, which the pairing QR code carries in its fragment past the relay; the phone's calls and its socket carried as length-prefixed JSON frames and served by the gateway's own handler through an in-memory listener, the gateway's direction deflated before it is encrypted. The phone's end is `internal/phone/ui/tunnel.mjs` in WebCrypto, run against this Go end by `TestTunnelJSAgainstGo`; `phone.go` is the same end in Go, for tests. A test taps the relay's bytes and fails if any plaintext is among them. **The relay is trusted all the same**: it serves the phone's app — the code that makes the session and keeps the device's token in `localStorage` — so it could serve one that reads everything; the docs say to run one of your own, and nothing here may claim more. A device paired through the tunnel is marked (`Device.Relay`, from `relay.Through`, which only the in-memory connection can set) so `conch web relay reset` revokes it with the key | | `internal/preview` | The phones a mobile app is tried on, on this computer: iOS simulators through `xcrun simctl`, Android emulators and phones through `adb` and `emulator` (found in the SDK where Android Studio puts it, not only on `PATH`), and AXe (else `idb`) for touching an iOS simulator, which simctl cannot — in points, the screenshot's pixels divided by the device type's `mainScreenScale`. Xcode is found in `/Applications` when `xcode-select` points at the Command Line Tools (`DEVELOPER_DIR` is set for every tool, AXe included, which asks `xcode-select -p` itself); `[preview]` in config.toml names it otherwise. Lists, boots, screenshots, taps, types, presses home or back, and says what kind of mobile app a folder is. `conch preview` (`cmd/conch/preview.go`) is the command over it, asking no server; Conch.app's **Mobile Preview** tab (`MobilePreviewView.swift`, the testable part in `ConchKit/MobilePreview.swift`) runs that command a few times a second while on screen, and an agent looks at its own work with `conch preview shot`. Tests answer from a table of command lines and never reach a real simulator | diff --git a/internal/sandbox/custom.go b/internal/sandbox/custom.go index 6da09df1..8ac25234 100644 --- a/internal/sandbox/custom.go +++ b/internal/sandbox/custom.go @@ -47,7 +47,9 @@ var ProviderFileGuide string // that needs what format 1 lacked says format = 2, which a conch from // before formats calls an unknown key and refuses: refusing is right, // since it would otherwise send {public_key} as those very letters. -const ProviderFormat = 2 +// Format 3 adds headers, which a format-2 conch would leave out and so +// send requests the provider refuses. +const ProviderFormat = 3 // PublicKey is the public key a sandbox is made with when a provider file // puts {public_key} in its create body. internal/remote sets it (the @@ -74,6 +76,9 @@ type ProviderFile struct { Adapted string `toml:"adapted"` // Note is said before a sandbox is made with it. Note string `toml:"note"` + // Headers are sent with every call; a call's own headers are laid + // over them. + Headers map[string]string `toml:"headers"` Create CreateCall `toml:"create"` Get GetCall `toml:"get"` @@ -85,11 +90,13 @@ type ProviderFile struct { SSH SSHWay `toml:"ssh"` } -// Call is one request: "METHOD /path", the path's {id} filled in, and a -// body whose string values may be placeholders. +// Call is one request: "METHOD /path", the path's {id} filled in, a body +// whose string values may be placeholders, and headers of its own — an +// Idempotency-Key = "{uuid}", say, or a delete's confirmation. type Call struct { - Call string `toml:"call"` - Body map[string]any `toml:"body"` + Call string `toml:"call"` + Body map[string]any `toml:"body"` + Headers map[string]string `toml:"headers"` } // CreateCall makes a sandbox; ID is where its id is in the reply. @@ -131,10 +138,10 @@ type ListCall struct { // registered with the person's account there; "authorize", one conch // hands over through Authorize; "none", a credential in User from Access. type SSHWay struct { - Host string `toml:"host"` - User string `toml:"user"` - Port int `toml:"port"` - Key string `toml:"key"` + Host string `toml:"host"` + User string `toml:"user"` + Port SSHPort `toml:"port"` + Key string `toml:"key"` // Authorize lets a public key in ({public_key} in its body). Authorize Call `toml:"authorize"` // Access is asked before each connection when the way in is a @@ -143,6 +150,42 @@ type SSHWay struct { Access Call `toml:"access"` } +// SSHPort is ssh's port: a number, or (format 3) "{reply.PATH}", read +// from [ssh.access]'s reply for a gateway that says its port per +// connection, as it says its host. +type SSHPort struct { + N int + Reply string // the PATH of "{reply.PATH}" +} + +var replyRef = regexp.MustCompile(`\{reply\.([^{}]+)\}`) + +func (p *SSHPort) UnmarshalTOML(v any) error { + switch x := v.(type) { + case int64: + p.N = int(x) + if int64(p.N) != x { + p.N = -1 // out of range, said by Problems + } + case string: + m := replyRef.FindStringSubmatch(x) + if m == nil || m[0] != x { + return fmt.Errorf(`[ssh] port %q: a number, or "{reply.PATH}" alone`, x) + } + p.Reply = m[1] + default: + return fmt.Errorf(`[ssh] port: a number, or "{reply.PATH}"`) + } + return nil +} + +func (p SSHPort) String() string { + if p.Reply != "" { + return "{reply." + p.Reply + "}" + } + return strconv.Itoa(p.N) +} + // conch's own states a provider file may map to. var customStates = map[string]State{ "starting": StateStarting, "started": StateStarted, "stopping": StateStopping, @@ -203,8 +246,13 @@ func (f *ProviderFile) Problems() []string { add := func(format string, a ...any) { p = append(p, fmt.Sprintf(format, a...)) } if f.Format < 0 || f.Format > ProviderFormat { add("format %d: this conch reads up to format %d; update conch to use it", f.Format, ProviderFormat) - } else if need := f.needs2(); need != "" && f.Format < 2 { - add("%s needs format = 2", need) + } else { + if need := f.needs2(); need != "" && f.Format < 2 { + add("%s needs format = 2", need) + } + if need := f.needs3(); need != "" && f.Format < 3 { + add("%s needs format = 3", need) + } } switch { case !providerName.MatchString(f.Name): @@ -238,6 +286,11 @@ func (f *ProviderFile) Problems() []string { add("[%s] body: %v", c.what, err) } } + for _, c := range f.headerSets() { + for _, e := range f.checkHeaders(c.headers) { + add("%s: %s", c.what, e) + } + } if f.Create.ID == "" { add("[create] id: where the new sandbox's id is in the reply") } @@ -268,8 +321,11 @@ func (f *ProviderFile) Problems() []string { if f.SSH.Host == "" || f.SSH.User == "" { add("[ssh] host and user: conch reaches every sandbox over ssh, so a provider with no ssh cannot be used") } - if f.SSH.Port < 0 || f.SSH.Port > 65535 { - add("[ssh] port %d", f.SSH.Port) + if f.SSH.Port.N < 0 || f.SSH.Port.N > 65535 { + add("[ssh] port %d", f.SSH.Port.N) + } + if f.SSH.Port.Reply != "" && f.SSH.Access.Call == "" { + add("[ssh] port %s: there is no [ssh.access] reply to read it from", f.SSH.Port) } switch f.SSH.Key { case "account", "none": @@ -308,6 +364,86 @@ func (f *ProviderFile) needs2() string { return "" } +// needs3 names the first thing in the file that format 2 did not have. +func (f *ProviderFile) needs3() string { + for _, c := range f.headerSets() { + if len(c.headers) > 0 { + return c.what + } + } + if f.SSH.Port.Reply != "" { + return "[ssh] port from the reply" + } + return "" +} + +type headerSet struct { + what string + headers map[string]string +} + +// headerSets is every place a file may give headers, named as a problem +// with them is said. +func (f *ProviderFile) headerSets() []headerSet { + return []headerSet{{"headers", f.Headers}, {"[create] headers", f.Create.Headers}, {"[get] headers", f.Get.Headers}, + {"[list] headers", f.List.Headers}, {"[start] headers", f.Start.Headers}, {"[stop] headers", f.Stop.Headers}, + {"[delete] headers", f.Delete.Headers}, {"[ssh.authorize] headers", f.SSH.Authorize.Headers}, + {"[ssh.access] headers", f.SSH.Access.Headers}} +} + +// placeholders a header value may use: {uuid} is new for every request. +// Never {key}: the key goes in auth_header alone, so a file cannot send it +// anywhere a person reading the file would not look for it. +var headerPlaceholders = map[string]bool{"{id}": true, "{name}": true, "{uuid}": true} + +var headerName = regexp.MustCompile("^[!#$%&'*+.^_`|~0-9A-Za-z-]+$") + +// conchHeaders are set by conch itself, or by Go for the request; a file +// that set them would break the request or move the key. +var conchHeaders = []string{"Host", "Content-Length", "Content-Type", "Accept", "Transfer-Encoding", "Connection"} + +func (f *ProviderFile) checkHeaders(h map[string]string) []string { + var p []string + auth := http.CanonicalHeaderKey(firstSet(f.AuthHeader, "Authorization")) + for _, k := range sortedKeys(h) { + val := h[k] + ck := http.CanonicalHeaderKey(k) + switch { + case !headerName.MatchString(k): + p = append(p, fmt.Sprintf("%q is not a header name", k)) + continue + case ck == auth: + p = append(p, fmt.Sprintf("%s carries the key: set it with auth_header and auth_value", k)) + continue + case slicesContains(conchHeaders, ck): + p = append(p, fmt.Sprintf("%s is conch's to set", k)) + continue + case strings.ContainsAny(val, "\r\n\x00"): + p = append(p, fmt.Sprintf("%s's value has a line break", k)) + continue + } + var bad []string + for _, ph := range placeholder.FindAllString(val, -1) { + if !headerPlaceholders[ph] { + bad = append(bad, ph) + } + } + if len(bad) > 0 { + p = append(p, fmt.Sprintf("%s: unknown placeholder %s (a header may use {id}, {name}, {uuid})", k, strings.Join(bad, ", "))) + } + } + return p +} + +func sortedKeys(m map[string]string) []string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + sort.Strings(keys) + return keys +} + func slicesContains(list []string, s string) bool { for _, x := range list { if x == s { @@ -394,11 +530,16 @@ type Mapping struct { // it reads back: the review a person does before saving the file. func (f *ProviderFile) Describe() []Mapping { body := func(c Call) string { + s := c.Call + h := f.headersOf(c) + for _, k := range sortedKeys(h) { + s += fmt.Sprintf(" [%s: %s]", k, h[k]) + } if len(c.Body) == 0 { - return c.Call + return s } b, _ := json.Marshal(c.Body) - return c.Call + " " + string(b) + return s + " " + string(b) } m := []Mapping{ {"New sandbox", body(f.Create.Call), f.Create.ID}, @@ -415,8 +556,8 @@ func (f *ProviderFile) Describe() []Mapping { sort.Strings(states) m = append(m, Mapping{"States", strings.Join(states, " "), ""}) way := fmt.Sprintf("ssh %s@%s", f.SSH.User, f.SSH.Host) - if f.SSH.Port != 0 && f.SSH.Port != 22 { - way += fmt.Sprintf(" -p %d", f.SSH.Port) + if f.SSH.Port.Reply != "" || f.SSH.Port.N != 0 && f.SSH.Port.N != 22 { + way += " -p " + f.SSH.Port.String() } switch f.SSH.Key { case "account": @@ -586,10 +727,58 @@ func (c *Custom) do(ctx context.Context, call Call, v vars) (any, error) { b, _ := v.fill(call.Body) body = b } - return c.request(ctx, method, path, body) + return c.request(ctx, method, path, body, c.headers(call, v)) +} + +// headersOf is the file's headers with the call's own laid over them, by +// name as HTTP compares names (Idempotency-Key and idempotency-key are one). +func (f *ProviderFile) headersOf(call Call) map[string]string { + if len(f.Headers) == 0 && len(call.Headers) == 0 { + return nil + } + out := map[string]string{} + for _, set := range []map[string]string{f.Headers, call.Headers} { + for k, val := range set { + out[http.CanonicalHeaderKey(k)] = val + } + } + return out } -func (c *Custom) request(ctx context.Context, method, path string, body any) (any, error) { +// headers fills a call's headers for one request. A header whose value is +// a placeholder with nothing to put there is left out, as a body's is; +// {uuid} is new for each request, so no two are taken for a retry. +func (c *Custom) headers(call Call, v vars) http.Header { + h := c.f.headersOf(call) + if len(h) == 0 { + return nil + } + hv := vars{"{uuid}": newUUID()} + for k, x := range v { + hv[k] = x + } + out := http.Header{} + for k, val := range h { + x, ok := hv.fill(val) + // Inside a longer value an unfilled placeholder would go out as its + // own letters; the header is left out instead. + if s := fmt.Sprint(x); ok && s != "" && !placeholder.MatchString(s) { + out.Set(k, s) + } + } + return out +} + +// newUUID is a random (version 4) UUID. +func newUUID() string { + b := make([]byte, 16) + _, _ = rand.Read(b) + b[6] = b[6]&0x0f | 0x40 + b[8] = b[8]&0x3f | 0x80 + return fmt.Sprintf("%x-%x-%x-%x-%x", b[0:4], b[4:6], b[6:8], b[8:10], b[10:]) +} + +func (c *Custom) request(ctx context.Context, method, path string, body any, headers http.Header) (any, error) { var rd io.Reader if body != nil { enc, err := json.Marshal(body) @@ -602,6 +791,11 @@ func (c *Custom) request(ctx context.Context, method, path string, body any) (an if err != nil { return nil, err } + // The file's headers first, so none can take the key's or conch's place + // (Problems refuses those names too). + for k, vals := range headers { + req.Header[k] = vals + } req.Header.Set(firstSet(c.f.AuthHeader, "Authorization"), strings.ReplaceAll(firstSet(c.f.AuthValue, "Bearer {key}"), "{key}", c.key)) req.Header.Set("Accept", "application/json") if body != nil { @@ -948,14 +1142,24 @@ func (c *Custom) SSHAccess(ctx context.Context, id string) (Access, error) { } fill := func(s string) string { s = v.str(s) - return regexp.MustCompile(`\{reply\.([^{}]+)\}`).ReplaceAllStringFunc(s, func(m string) string { + return replyRef.ReplaceAllStringFunc(s, func(m string) string { return pickStr(reply, strings.TrimSuffix(strings.TrimPrefix(m, "{reply."), "}")) }) } - a := Access{User: fill(c.f.SSH.User), Host: fill(c.f.SSH.Host), Port: c.f.SSH.Port, PlainUser: c.f.SSH.Key != "none"} + a := Access{User: fill(c.f.SSH.User), Host: fill(c.f.SSH.Host), Port: c.f.SSH.Port.N, PlainUser: c.f.SSH.Key != "none"} if a.User == "" || a.Host == "" { return Access{}, fmt.Errorf("%s gave no ssh user or host for %s", c.label(), id) } + if path := c.f.SSH.Port.Reply; path != "" { + // Said rather than left to ssh's default: the default is what a + // gateway that names its port does not listen on. + raw := pickStr(reply, path) + n, err := strconv.Atoi(raw) + if err != nil || n < 1 || n > 65535 { + return Access{}, fmt.Errorf("%s gave no ssh port for %s at %q (got %q)", c.label(), id, path, raw) + } + a.Port = n + } return a, nil } diff --git a/internal/sandbox/format2_test.go b/internal/sandbox/format2_test.go index cda803b3..ae038b20 100644 --- a/internal/sandbox/format2_test.go +++ b/internal/sandbox/format2_test.go @@ -235,8 +235,8 @@ func TestFormatTwoProblems(t *testing.T) { for _, c := range []struct{ from, to, want string }{ {"format = 2\n", "", "[get] item needs format = 2"}, {"format = 2\n", "format = 1\n", "[get] item needs format = 2"}, - {"format = 2\n", "format = 3\n", "reads up to format 2"}, - {"format = 2\n", "format = -1\n", "reads up to format 2"}, + {"format = 2\n", fmt.Sprintf("format = %d\n", ProviderFormat+1), fmt.Sprintf("reads up to format %d", ProviderFormat)}, + {"format = 2\n", "format = -1\n", fmt.Sprintf("reads up to format %d", ProviderFormat)}, } { path := customHome(t, "cloud", strings.Replace(good, c.from, c.to, 1)) _, problems, err := ReadProviderFile(path) diff --git a/internal/sandbox/format3_test.go b/internal/sandbox/format3_test.go new file mode 100644 index 00000000..1c695eaf --- /dev/null +++ b/internal/sandbox/format3_test.go @@ -0,0 +1,329 @@ +package sandbox + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + "time" + + "github.com/Amitgb14/conch/internal/config" +) + +// A provider shaped as instabox is: every change needs an Idempotency-Key +// (a key seen before on another request is refused), a delete must name +// the machine in X-Confirm-Delete, and a create answers with an operation. +type fakeInstabox struct { + mu sync.Mutex + next int + machines map[string]map[string]any + keys map[string]string // Idempotency-Key → the request it came with + port any // the gateway's port, as the token's reply says it + seen []http.Header +} + +func newFakeInstabox(t *testing.T) (*fakeInstabox, string) { + t.Helper() + f := &fakeInstabox{machines: map[string]map[string]any{}, keys: map[string]string{}, port: 2222} + srv := httptest.NewServer(http.HandlerFunc(f.serve)) + t.Cleanup(srv.Close) + return f, srv.URL +} + +func (f *fakeInstabox) serve(w http.ResponseWriter, r *http.Request) { + f.mu.Lock() + defer f.mu.Unlock() + f.seen = append(f.seen, r.Header.Clone()) + reply := func(code int, v any) { + w.WriteHeader(code) + _ = json.NewEncoder(w).Encode(v) + } + fail := func(code int, c string) { reply(code, map[string]any{"error": map[string]any{"code": c}}) } + if r.Header.Get("X-API-Key") != "ibk_test" { + fail(401, "invalid_key") + return + } + if r.Method != "GET" { + k := r.Header.Get("Idempotency-Key") + if k == "" { + fail(400, "bad_request") + return + } + if prev, ok := f.keys[k]; ok && prev != r.Method+r.URL.Path { + fail(409, "idempotency_conflict") + return + } + f.keys[k] = r.Method + r.URL.Path + } + parts := strings.Split(strings.Trim(r.URL.Path, "/"), "/") // v1 machines [id [verb]] + if len(parts) == 2 { + switch r.Method { + case "POST": + var body map[string]any + _ = json.NewDecoder(r.Body).Decode(&body) + f.next++ + id := fmt.Sprintf("m_%026d", f.next) + f.machines[id] = map[string]any{"id": id, "name": body["name"], "state": "creating"} + reply(202, map[string]any{"id": "op_1", "machineId": id, "state": "queued"}) + case "GET": + var all []any + for _, m := range f.machines { + all = append(all, m) + } + reply(200, map[string]any{"machines": all, "nextCursor": nil}) + } + return + } + m, ok := f.machines[parts[2]] + if !ok { + fail(404, "not_found") + return + } + switch { + case r.Method == "GET" && len(parts) == 3: + if m["state"] == "creating" { + defer func() { m["state"] = "running" }() + } + reply(200, m) + case r.Method == "DELETE": + if r.Header.Get("X-Confirm-Delete") != parts[2] { + fail(409, "delete_confirmation_required") + return + } + delete(f.machines, parts[2]) + reply(202, map[string]any{"id": "op_2", "state": "queued"}) + case r.Method == "POST" && parts[3] == "stop": + m["state"] = "stopped" + reply(202, map[string]any{"id": "op_3"}) + case r.Method == "POST" && parts[3] == "start": + m["state"] = "running" + reply(202, map[string]any{"id": "op_4"}) + case r.Method == "POST" && parts[3] == "ssh-tokens": + reply(201, map[string]any{"token": "ibssh_abc", "host": "ssh.box.test", "port": f.port}) + default: + fail(404, "not_found") + } +} + +func boxFile(api string) string { + return `format = 3 +name = "box" +api = "` + api + `" +key_env = "BOX_KEY" +auth_header = "X-API-Key" +auth_value = "{key}" +headers = { Idempotency-Key = "{uuid}", X-Client = "conch" } + +[create] +call = "POST /v1/machines" +body = { name = "{name}" } +id = "machineId" + +[get] +call = "GET /v1/machines/{id}" +state = "state" +name = "name" + +[list] +call = "GET /v1/machines" +items = "machines" +id = "id" +mine = "conch-" + +[start] +call = "POST /v1/machines/{id}/start" +[stop] +call = "POST /v1/machines/{id}/stop" +[delete] +call = "DELETE /v1/machines/{id}" +headers = { X-Confirm-Delete = "{id}", X-Note = "for {name}" } + +[states] +creating = "starting" +running = "started" +stopped = "stopped" +deleted = "gone" + +[ssh] +host = "{reply.host}" +user = "{reply.token}" +port = "{reply.port}" +key = "none" +[ssh.access] +call = "POST /v1/machines/{id}/ssh-tokens" +headers = { idempotency-key = "access-{id}-{uuid}", X-Client = "" } +` +} + +func TestFormatThreeSendsHeaders(t *testing.T) { + old := pollEvery + pollEvery = time.Millisecond + t.Cleanup(func() { pollEvery = old }) + f, api := newFakeInstabox(t) + t.Setenv("BOX_KEY", "ibk_test") + path := customHome(t, "box", boxFile(api)) + if _, problems, err := ReadProviderFile(path); err != nil || len(problems) != 0 { + t.Fatalf("file: %v %v", problems, err) + } + p, err := Open("box", config.SandboxCfg{}) + if err != nil { + t.Fatal(err) + } + ctx := context.Background() + s, err := p.Create(ctx, Spec{Name: "hull"}) + if err != nil || s.State != StateStarted { + t.Fatalf("create: %+v %v", s, err) + } + if err := p.Stop(ctx, s.ID); err != nil { + t.Fatal(err) + } + if _, err := p.Start(ctx, s.ID); err != nil { + t.Fatal(err) + } + a, err := p.SSHAccess(ctx, s.ID) + if err != nil || a.User != "ibssh_abc" || a.Host != "ssh.box.test" || a.Port != 2222 { + t.Fatalf("ssh: %+v %v", a, err) + } + // A reply with no port, or one that is not a port, is said: ssh's + // default is the port such a gateway does not listen on. + for _, bad := range []any{nil, "", "ssh", 0, 70000, 22.5} { + f.mu.Lock() + f.port = bad + f.mu.Unlock() + if a, err := p.SSHAccess(ctx, s.ID); err == nil || !strings.Contains(err.Error(), "no ssh port") { + t.Errorf("port %v: %+v %v", bad, a, err) + } + } + f.mu.Lock() + f.port = "2200" // a port written as a string reads too + f.mu.Unlock() + if a, err := p.SSHAccess(ctx, s.ID); err != nil || a.Port != 2200 { + t.Fatalf("string port: %+v %v", a, err) + } + if err := p.Delete(ctx, s.ID); err != nil { + t.Fatal(err) + } + if _, err := p.Get(ctx, s.ID); !errors.Is(err, ErrNotFound) { + t.Fatalf("gone: %v", err) + } + + f.mu.Lock() + defer f.mu.Unlock() + keys := map[string]bool{} + for _, h := range f.seen { + k := h.Get("Idempotency-Key") + if k == "" { + t.Fatalf("a request without its key: %v", h) + } + // A new key for every request: the provider would answer a second + // use with the first one's reply. + if keys[k] { + t.Fatalf("Idempotency-Key %q sent twice", k) + } + keys[k] = true + switch { + case strings.HasPrefix(k, "access-"): + // The call's own header, named in other letters, replaces the + // file's; an empty one leaves the file's out. + if !strings.HasPrefix(k, "access-"+s.ID+"-") || len(h.Values("Idempotency-Key")) != 1 || h.Get("X-Client") != "" { + t.Fatalf("ssh access headers: %v", h) + } + case len(k) != 36 || k[14] != '4': + t.Fatalf("not a v4 uuid: %q", k) + default: + if h.Get("X-Client") != "conch" { + t.Fatalf("the file's header missing: %v", h) + } + } + if h.Get("X-API-Key") != "ibk_test" { + t.Fatalf("the key: %v", h) + } + // {name} is not known to a delete, so that header is left out + // rather than sent as "for {name}". + if h.Get("X-Note") != "" { + t.Fatalf("an unfilled header was sent: %q", h.Get("X-Note")) + } + } +} + +func TestFormatThreeProblems(t *testing.T) { + good := boxFile("https://api.example.test") + for _, c := range []struct{ from, to, want string }{ + {"format = 3\n", "", "headers needs format = 3"}, + {"format = 3\n", "format = 2\n", "headers needs format = 3"}, + {`X-Client = "conch"`, `"X Client" = "conch"`, `"X Client" is not a header name`}, + {`X-Client = "conch"`, `x-api-key = "{uuid}"`, "x-api-key carries the key"}, + {`X-Client = "conch"`, `Content-Type = "text/plain"`, "Content-Type is conch's to set"}, + {`X-Client = "conch"`, `host = "evil.test"`, "host is conch's to set"}, + {`X-Client = "conch"`, `X-Secret = "{key}"`, "unknown placeholder {key}"}, + {`X-Client = "conch"`, `X-Client = "a\r\nX-Evil: 1"`, "line break"}, + {`X-Confirm-Delete = "{id}"`, `X-Confirm-Delete = "{snapshot}"`, "[delete] headers: X-Confirm-Delete: unknown placeholder {snapshot}"}, + {"[ssh.access]\ncall = \"POST /v1/machines/{id}/ssh-tokens\"\n", "", "no [ssh.access] reply to read it from"}, + {`port = "{reply.port}"`, `port = 70000`, "[ssh] port 70000"}, + } { + changed := strings.Replace(good, c.from, c.to, 1) + if changed == good { + t.Fatalf("%q is not in the file", c.from) + } + _, problems, err := ReadProviderFile(customHome(t, "box", changed)) + if err != nil || !strings.Contains(strings.Join(problems, "\n"), c.want) { + t.Errorf("%q → %q: %v %v", c.from, c.to, problems, err) + } + } + // Each place a header may go asks for format 3 alone. + tl := tensorlakeFile("https://api.example.test") + for _, where := range []string{"create", "get", "list", "delete"} { + changed := strings.Replace(tl, "["+where+"]\n", "["+where+"]\nheaders = { X-A = \"b\" }\n", 1) + if changed == tl { + t.Fatalf("[%s] is not in the file", where) + } + _, problems, _ := ReadProviderFile(customHome(t, "tlake", changed)) + if got := strings.Join(problems, "\n"); !strings.Contains(got, "["+where+"] headers needs format = 3") { + t.Errorf("[%s]: %v", where, problems) + } + } + // A port from the reply alone asks for format 3. + withPort := strings.Replace(strings.Replace(good, "format = 3\n", "format = 2\n", 1), "headers = { Idempotency-Key = \"{uuid}\", X-Client = \"conch\" }\n", "", 1) + withPort = strings.Replace(withPort, "headers = { X-Confirm-Delete = \"{id}\", X-Note = \"for {name}\" }\n", "", 1) + withPort = strings.Replace(withPort, "headers = { idempotency-key = \"access-{id}-{uuid}\", X-Client = \"\" }\n", "", 1) + if _, problems, _ := ReadProviderFile(customHome(t, "box", withPort)); strings.Join(problems, "\n") != "[ssh] port from the reply needs format = 3" { + t.Errorf("port alone: %v", problems) + } + // A port that is neither a number nor a reply path is the file's error. + for _, bad := range []string{`"22"`, `"{reply.port}x"`, `"{id}"`, `true`} { + if _, _, err := ReadProviderFile(customHome(t, "box", strings.Replace(good, `port = "{reply.port}"`, "port = "+bad, 1))); err == nil || !strings.Contains(err.Error(), "port") { + t.Errorf("port = %s: %v", bad, err) + } + } + // A header value that is not a string is the file's error, said. + if _, _, err := ReadProviderFile(customHome(t, "box", strings.Replace(good, `X-Client = "conch"`, `X-Client = 1`, 1))); err == nil { + t.Error("a number for a header value was taken") + } +} + +func TestFormatThreeDescribesHeaders(t *testing.T) { + f, problems, err := ReadProviderFile(customHome(t, "box", boxFile("https://api.example.test"))) + if err != nil || len(problems) != 0 { + t.Fatal(problems, err) + } + d := f.Describe() + if way := d[len(d)-1].Call; !strings.Contains(way, "@{reply.host} -p {reply.port}") { + t.Fatalf("way in: %q", way) + } + want := "DELETE /v1/machines/{id} [Idempotency-Key: {uuid}] [X-Client: conch] [X-Confirm-Delete: {id}] [X-Note: for {name}]" + if d[5].Does != "Delete" || d[5].Call != want { + t.Fatalf("%q", d[5].Call) + } + // A file without headers reads as it did. + tf, _, _ := ReadProviderFile(customHome(t, "tlake", tensorlakeFile("https://api.example.test"))) + for _, m := range tf.Describe()[:6] { + if strings.Contains(m.Call, " [") { + t.Fatalf("headers where there are none: %q", m.Call) + } + } +} diff --git a/internal/sandbox/providerfile.md b/internal/sandbox/providerfile.md index 8839dd75..5d096dd3 100644 --- a/internal/sandbox/providerfile.md +++ b/internal/sandbox/providerfile.md @@ -71,7 +71,7 @@ failed = "error" [ssh] host = "ssh.acme.dev" # may use {id} {name}, or {reply.PATH} after [ssh.access] user = "{id}" -port = 22 # 0 or 22: ssh's default +port = 22 # 0 or 22: ssh's default ("{reply.port}" in format 3) # Whose key opens it: # "account" keys the person registers with their account there (say how in note) # "authorize" conch hands its public key over with [ssh.authorize] @@ -107,5 +107,37 @@ them then refuses the file rather than misreading it. sandbox itself, `host = "{reply.droplet.networks.v4[type=public].ip_address}"` reaches a sandbox at its own address. +## Format 3 + +Two things for gateways and APIs that need more. A file that uses either +starts with `format = 3`. + +- **A port from the reply.** When `[ssh.access]`'s reply says which port + to connect on, `port = "{reply.port}"` (any path into that reply) reads + it there, as `host` and `user` do. A reply without a port is an error, + not port 22. + +**Headers**, for an API that needs more than the key on a request. `headers` at the top (before the +first `[table]`) go with every call; `headers` under a call go with that +call, laid over the top ones by name, and one set to `""` there leaves +the top one out of that call. + +```toml +format = 3 +headers = { Idempotency-Key = "{uuid}" } # a new value for every request + +[delete] +call = "DELETE /sandboxes/{id}" +headers = { X-Confirm-Delete = "{id}" } +``` + +A header value may use `{uuid}` (new for each request), `{id}` and +`{name}`. Conch knows `{name}` only on create and `[ssh.access]`. A +header whose placeholder has nothing to put there is left out. The key +goes only in `auth_header`, so a header may not carry `{key}` or reuse +that header's name. Conch sets `Host`, `Content-Type`, `Content-Length`, +`Accept`, `Transfer-Encoding` and `Connection` itself, so a file may not +name them. + Check it with `conch sandbox provider check -offline ./provider.toml`, and fix what it reports until it reports nothing.