diff --git a/README.md b/README.md index 54d8c25..6d972ba 100644 --- a/README.md +++ b/README.md @@ -250,7 +250,7 @@ a := api.New( ) ``` -Kinds without an override keep `api.StatusForKind`. Only statuses 400–599 are accepted: other calls are ignored and leave any earlier override for the kind in place, and a later valid call wins. Errors without a kind always answer 500. The title follows the decided status: the internal title for any 5xx, the unauthorized title for 401 and 403. The hook runs on the request goroutine with the original error and the status `WriteError` decided (a handler that already wrote a header, or a custom formatter or writer, can send another), before anything is written; it is shared by all requests the instance serves and may run concurrently, so it must be safe for concurrent use. A panic in it is not recovered: `api.ProcessRequest` answers 500 and logs it through the instance's `api.WithLogger` logger with the panic value (nothing is logged without a logger), and without `api.ProcessRequest` `net/http` aborts the connection. On the default instance the panic is logged the same way through `slog.Default`. Only `WriteError` calls the hook: not `Error400`/`Error500`/the other helpers, and not the library's own rejections (body-parse failures in `api.RequestBody`/`api.ProcessRequest`, the token middlewares' 401s, `api.ProcessRequest`'s panic 500). The default instance has no overrides and no hook. `ExampleWithKindStatus` and `ExampleWithErrorHook` are the compiled versions. +Kinds without an override keep `api.StatusForKind`. Only statuses 400–599 are accepted: other calls are ignored and leave any earlier override for the kind in place, and a later valid call wins. Errors without a kind always answer 500. The title follows the decided status: the internal title for any 5xx, the unauthorized title for 401 and 403. The hook runs on the request goroutine with the original error and the status `WriteError` decided (a handler that already wrote a header, or a custom formatter or writer, can send another), before anything is written; it is shared by all requests the instance serves and may run concurrently, so it must be safe for concurrent use. A panic in it is not recovered: `api.ProcessRequest` answers 500 and logs it through the instance's `api.WithLogger` logger with the panic type (nothing is logged without a logger), and without `api.ProcessRequest` `net/http` aborts the connection. On the default instance the panic is logged the same way through `slog.Default`. Only `WriteError` calls the hook: not `Error400`/`Error500`/the other helpers, and not the library's own rejections (body-parse failures in `api.RequestBody`/`api.ProcessRequest`, the token middlewares' 401s, `api.ProcessRequest`'s panic 500). The default instance has no overrides and no hook. `ExampleWithKindStatus` and `ExampleWithErrorHook` are the compiled versions. ## Logging @@ -264,7 +264,7 @@ a := api.New( // {"level":"INFO","msg":"request","event_id":"…","method":"GET","uri":"/login?access_token=[REDACTED]&page=2","headers":{"Pin":["[REDACTED]"]}} ``` -`api.ProcessRequest` writes a `request` record (`event_id`, `method`, `uri`, `headers`) and a `response` record (`event_id`, `method`, `uri`, `status`, `status_text`, `duration_ms`) at info level. A rejected body, a failed `Get*Value*`/`UnmarshalBody` helper, a recovered panic (`panic recovered`, with the panic value) and a response that cannot be encoded are error records with an `error` or `panic` attribute; a panic with `http.ErrAbortHandler` is a `handler aborted` debug record. A failed `Get*Value*` helper adds the name as `key`; for a sensitive name its `error` is only the kind (`invalid syntax`, `value out of range`, `invalid value`), never the value. A response that cannot be encoded (for example a `NaN` in the content) is answered with the `Error500()` body instead; the writer gets no request, so its records carry no request context or `event_id`. Bodies and the parsed form are never logged (query values appear only in `uri`). Header values, query values in `uri` and the user information of the `uri` are masked (see [Security notes](#security-notes)); secrets in path segments are not masked. The sensitive keys are the default list (`authorization`, `token`, `password`, `secret`, `credential`, `apikey`, `api-key`, `key`, `cookie`, `session`) plus the keys of `api.WithSensitiveLogKeys`, which can only add. `ExampleWithLogger` is the compiled version. +`api.ProcessRequest` writes a `request` record (`event_id`, `method`, `uri`, `headers`) and a `response` record (`event_id`, `method`, `uri`, `status`, `status_text`, `duration_ms`) at info level. A rejected body, a failed `Get*Value*`/`UnmarshalBody` helper, a recovered panic (`panic recovered`) and a response that cannot be encoded are error records with an `error` or `panic` attribute; a panic with `http.ErrAbortHandler` is a `handler aborted` debug record. A panic record always has `panic_type`, the Go type of the value (`string`, `*errors.errorString`, `runtime.boundsError`); its `panic` is the value only for a `runtime.Error` (nil map write, index out of range), which carries no request data, and `[REDACTED]` for any other value, which may. A failed `Get*Value*` helper adds the name as `key`; for a sensitive name its `error` is only the kind (`invalid syntax`, `value out of range`, `invalid value`), never the value. A response that cannot be encoded (for example a `NaN` in the content) is answered with the `Error500()` body instead; the writer gets no request, so its records carry no request context or `event_id`. Bodies and the parsed form are never logged (query values appear only in `uri`). Header values with a sensitive name, query values in `uri` and in other header values (after each `?`), and the user information of the `uri` and of URL-valued headers are masked (see [Security notes](#security-notes)); secrets in path segments are not masked. The sensitive keys are the default list (`authorization`, `token`, `password`, `secret`, `credential`, `apikey`, `api-key`, `key`, `cookie`, `session`) plus the keys of `api.WithSensitiveLogKeys`, which can only add. `ExampleWithLogger` is the compiled version. Since v1.0.0 the default instance writes the same records through `slog.Default()`, read at call time (route them with `slog.SetDefault`). Its sensitive keys are the default list plus `api.SensitiveLogKeys`, which can add keys but no longer remove a default one. Neither calls `api.Fatal`: a response that cannot be encoded is answered with 500. The package no longer calls `api.Print` or `api.PrintError` (only the deprecated `api.LogRequest`, `api.LogResponse` and `api.LogResponseWithDuration` call `api.Print`). `api.Print`, `api.PrintError`, `api.Fatal`, `api.SensitiveLogKeys` and the `api.Log*` functions are deprecated in favor of the options; `api.LogRequestBody`, `api.LogResponseBody`, `api.PrintFullEvent` and `api.MaxLoggedBodyBytes` are deprecated too: they affect only the deprecated `api.Log*` functions, since bodies are never logged. @@ -373,7 +373,7 @@ See [ADR 0002](docs/adr/0002-stdadapter-servemux.md). - `doc.Security` and `docs.SetDefaultSecurity` only document authentication; they do not enforce it. Enforce it with a middleware: `api.CustomToken` (validated by `api.CustomTokenValidatorFunc`), `api.BasicToken` (validated by `api.ValidateBasicAuthCredentialsFunc`), or your own. - `api.BasicToken` with the default validator answers 401 to every request while `api.Username` or `api.Password` is empty or `"default"` (since v0.11.0; before, it accepted `default:default`). Set both, or replace `api.ValidateBasicAuthCredentialsFunc`, before exposing a route. The default validator compares credentials in constant time; a custom one should too (`crypto/subtle`). After a successful check, `BasicToken` copies the user name into the `Basic-Client` request header. Since v1.0.0 it no longer copies the plain password into a `Basic-Secret` header, and it removes a `Basic-Secret` header the client sent; a handler that needs the password must take it from its own credential source. - The request token is never echoed (since v1.0.0, on the default instance too): a response body carries a `token` only when the handler sets `SecurityToken` on the response, and `api.RequestHeaderSession` does not copy the `SecurityToken` request header into the response header; a handler that wants to return a token sets it itself. -- `api.ProcessRequest` never logs bodies or the parsed form. It masks, by its sensitive keys (a name matches when it contains a key, ignoring case; the defaults include `authorization`, `token`, `password`, `key`, `cookie` and `session`, plus `api.WithSensitiveLogKeys` on an instance or `api.SensitiveLogKeys` on the default instance): header values and query parameter values in `uri` (names are found by splitting on `&` and `;`, and a masked value runs to the next `&`; a name that cannot be unescaped is masked too). It also masks the user information of an absolute-form, `//host` or CONNECT authority-form `uri` (`http://[REDACTED]@host/...`), up to the last `@`, so an `@` in the path or query hides the host and path too. Path segments are not masked. Masking is by name: a secret under a non-sensitive name (such as a URL with credentials in `?next=`) is logged as sent. See [Logging](#logging). +- `api.ProcessRequest` never logs bodies or the parsed form. It masks, by its sensitive keys (a name matches when it contains a key, ignoring case; the defaults include `authorization`, `token`, `password`, `key`, `cookie` and `session`, plus `api.WithSensitiveLogKeys` on an instance or `api.SensitiveLogKeys` on the default instance): header values and query parameter values in `uri` (names are found by splitting on `&` and `;`, and a masked value runs to the next `&`; a name that cannot be unescaped is masked too). It also masks the user information of an absolute-form, `//host` or CONNECT authority-form `uri` (`http://[REDACTED]@host/...`), up to the last `@`, so an `@` in the path or query hides the host and path too. Path segments are not masked. In a header value not masked by its name, every `?` starts a query that runs to the next `?`, and its values are masked by key as in `uri` (`Referer: https://app.example/cb?token=[REDACTED]`; each URL of a `Link` list); a value that starts with `/` or contains `://` also has its user information masked (`Origin: https://[REDACTED]@app.example`). Other values (`User-Agent`, `Accept`, `From: ops@example.com`) are kept. This goes by the value's shape, not a list of header names, so custom proxy headers are covered; a non-URL value with `://` or `?` may be over-masked. Otherwise masking is by name: a secret under a non-sensitive name (such as a URL with credentials in `?next=`) is logged as sent. See [Logging](#logging). - The deprecated `api.LogRequest`, `api.LogResponse` and `api.LogResponseWithDuration` log bodies only when `api.LogRequestBody` / `api.LogResponseBody` are true; they mask JSON body fields and `form` values by the keys in `api.SensitiveLogKeys` alone, cut bodies at `api.MaxLoggedBodyBytes`, and log a request body that is not JSON as `[non-JSON body, N bytes]` unless `api.PrintFullEvent` is true. - With `stdadapter`, a route variable can contain `/` and `..` (`/files/..%2F..%2Fetc` gives `name` = `../../etc`). Validate it before using it in a file path or another lookup that trusts it. - `api.ProcessRequest` stores the value of the `X-Forwarded-For` header up to its first `", "` as the `proxiedIPAddress` info value (and, when the request has no device UUID, as the event ID input). Only the first `X-Forwarded-For` header line is read. That value is set by the client: trust it only behind a proxy that overwrites the header, and do not use it on its own for rate limits, allow lists or audit. diff --git a/default_instance_test.go b/default_instance_test.go index 13607bd..f4f63fa 100644 --- a/default_instance_test.go +++ b/default_instance_test.go @@ -315,7 +315,7 @@ func TestDefaultInstanceLogsThroughSlogDefault(t *testing.T) { next: func(http.ResponseWriter, *http.Request) { panic("boom") }, want: []string{ `{"event_id":"","headers":{"Authorization":["[REDACTED]"],"Cookie":["[REDACTED]"]},"level":"INFO","method":"POST","msg":"request","uri":"/tasks?page=2&access_token=[REDACTED]"}`, - `{"event_id":"","level":"ERROR","msg":"panic recovered","panic":"boom"}`, + `{"event_id":"","level":"ERROR","msg":"panic recovered","panic":"[REDACTED]","panic_type":"string"}`, }, }, } diff --git a/docs/adr/0001-v1-instance-api.md b/docs/adr/0001-v1-instance-api.md index 8bd0842..a4dc2e9 100644 --- a/docs/adr/0001-v1-instance-api.md +++ b/docs/adr/0001-v1-instance-api.md @@ -114,14 +114,14 @@ sequenceDiagram - An instance without `WithLogger` writes no logs. - With the built-in writer, an instance never calls `Fatal` and never exits: a response that cannot be encoded is logged at error level through the instance logger. Since v0.11.0 (#46) the default instance does not call `Fatal` either: it reports the failure through `PrintError` and answers 500. *(Superseded by Amendment 1: the default instance logs it through `slog.Default`, not `PrintError`.)* - `WithSensitiveLogKeys(keys...)` adds keys to the default list; it cannot remove a default key (security defaults are not weakened). -- As landed in 5.5: `ProcessRequest` writes two info records, `request` (`event_id`, `method`, `uri`, `headers`) and `response` (`event_id`, `method`, `uri`, `status`, `status_text`, `duration_ms`). Errors met while serving a request (a rejected body in `ProcessRequest`/`RequestBody`, a failed `Get*Value*`/`UnmarshalBody` helper) are error records with an `error` attribute; a `Get*Value*` record adds the name as `key` and, for a sensitive name, gives only the kind of error (`invalid syntax`, `value out of range`, `invalid value`), never the value. A recovered panic is a `panic recovered` error record with exactly the panic value and `event_id`, never a dump of the request; a panic with `http.ErrAbortHandler` is a `handler aborted` debug record. An instance's key list is the built-in default list (the shipped value of `SensitiveLogKeys`, which has `cookie` and `session` since v0.11.0) plus its added keys; changes to the `SensitiveLogKeys` variable never reach an instance; added keys are lowercased and blank ones ignored; matching stays v0's case-insensitive substring match. The `uri` is masked with the default instance's helpers from v0.11.0 (#60), so both agree: query values by key with the literal `[REDACTED]` (a masked value runs to the next `&`, a key that cannot be unescaped is masked) and the user information of an absolute-form, `//host` or CONNECT authority-form target up to the last `@`; path segments are not. Bodies and the parsed form are never logged (no option, per section 5; query values still appear in `uri`). The built-in writer of an instance encodes before writing: a body that cannot be encoded is logged as `response cannot be encoded` and answered with 500 and the instance formatter's `Error500()` body (bare 500, with the response headers, if the formatter returns nil or that fails too); `http.ErrBodyNotAllowed` is not logged; the `Content-Type` comes only from the formatted headers (`application/json` when they set none), so one a handler set on `w.Header()` never labels the body; the encode-failure 500 drops the failed response's body headers (`Content-Type`, `Content-Encoding`, `Content-Length`, `Content-Range`, `Content-Disposition`, `Content-Language`, `Content-Location`, `ETag`, `Last-Modified`) and uses the error body's `Content-Type`; since 5.6 the default instance's writer also takes a `Content-Type` set in `Headers`, else `application/json` (section 8). `core.APIResponseWriter.Write` receives no request, so the writer's `response cannot be encoded`/`response cannot be written` records carry no request context or `event_id`. A custom writer, including a type that embeds `ResponseWriter`, is used as given and keeps the default instance's behavior. An instance never reads `Print`, `PrintError`, `Fatal` or `SensitiveLogKeys`, now deprecated with `LogRequest`, `LogResponse` and `LogResponseWithDuration`. *(Superseded by Amendment 1: the default instance writes these same records through `slog.Default`, and a type that embeds `ResponseWriter` logs through `slog.Default` with the registered formatter.)* +- As landed in 5.5: `ProcessRequest` writes two info records, `request` (`event_id`, `method`, `uri`, `headers`) and `response` (`event_id`, `method`, `uri`, `status`, `status_text`, `duration_ms`). Errors met while serving a request (a rejected body in `ProcessRequest`/`RequestBody`, a failed `Get*Value*`/`UnmarshalBody` helper) are error records with an `error` attribute; a `Get*Value*` record adds the name as `key` and, for a sensitive name, gives only the kind of error (`invalid syntax`, `value out of range`, `invalid value`), never the value. A recovered panic is a `panic recovered` error record with exactly the panic value and `event_id`, never a dump of the request; a panic with `http.ErrAbortHandler` is a `handler aborted` debug record. *(Amended in v1.1.1: the record carries `panic_type`, and `panic` is the value only for a `runtime.Error`, `[REDACTED]` otherwise, since an opaque value cannot be masked by key; header values are masked after each `?` and, when they carry a URL, in their user information, as `uri` is.)* An instance's key list is the built-in default list (the shipped value of `SensitiveLogKeys`, which has `cookie` and `session` since v0.11.0) plus its added keys; changes to the `SensitiveLogKeys` variable never reach an instance; added keys are lowercased and blank ones ignored; matching stays v0's case-insensitive substring match. The `uri` is masked with the default instance's helpers from v0.11.0 (#60), so both agree: query values by key with the literal `[REDACTED]` (a masked value runs to the next `&`, a key that cannot be unescaped is masked) and the user information of an absolute-form, `//host` or CONNECT authority-form target up to the last `@`; path segments are not. Bodies and the parsed form are never logged (no option, per section 5; query values still appear in `uri`). The built-in writer of an instance encodes before writing: a body that cannot be encoded is logged as `response cannot be encoded` and answered with 500 and the instance formatter's `Error500()` body (bare 500, with the response headers, if the formatter returns nil or that fails too); `http.ErrBodyNotAllowed` is not logged; the `Content-Type` comes only from the formatted headers (`application/json` when they set none), so one a handler set on `w.Header()` never labels the body; the encode-failure 500 drops the failed response's body headers (`Content-Type`, `Content-Encoding`, `Content-Length`, `Content-Range`, `Content-Disposition`, `Content-Language`, `Content-Location`, `ETag`, `Last-Modified`) and uses the error body's `Content-Type`; since 5.6 the default instance's writer also takes a `Content-Type` set in `Headers`, else `application/json` (section 8). `core.APIResponseWriter.Write` receives no request, so the writer's `response cannot be encoded`/`response cannot be written` records carry no request context or `event_id`. A custom writer, including a type that embeds `ResponseWriter`, is used as given and keeps the default instance's behavior. An instance never reads `Print`, `PrintError`, `Fatal` or `SensitiveLogKeys`, now deprecated with `LogRequest`, `LogResponse` and `LogResponseWithDuration`. *(Superseded by Amendment 1: the default instance writes these same records through `slog.Default`, and a type that embeds `ResponseWriter` logs through `slog.Default` with the registered formatter.)* ### 7. Errors: kind-to-status override and error hook (API-01) - `WriteError(w, r, err)` keeps its signature. With an instance it takes the status from the instance's overrides, else `StatusForKind`. `StatusForKind` stays the documented default mapping and is not deprecated. - `WithKindStatus(kind, status)` overrides one kind; repeat it for more. A per-kind override instead of a whole mapping function, so a service cannot forget the default rows. - `WithErrorHook(fn)` is called by `WriteError` after the status is decided and before the response is written, with the original error, so a service can log or count it without wrapping every handler. Only `WriteError` calls it (the other error helpers receive no `error` to report); a panic in the hook is not recovered (it is the caller's code). -- As landed in 5.4: an override status outside 400–599 is ignored (an error never answers 2xx/3xx, and `New` returns no error), and the empty kind cannot be overridden, so errors without a kind and nil errors always answer 500. A later valid override for the same kind wins; an ignored call leaves any earlier override in place. A later `WithErrorHook` replaces an earlier one, and nil means no hook. The response title follows the decided status: the internal title for any 5xx (like `Error501`), the unauthorized title for 401/403, the default title otherwise. The hook receives the status `WriteError` decided, which may differ from the one sent if the handler already wrote a header or a custom formatter or writer changes it. Only `WriteError` calls the hook: the library's own rejections (body-parse failures in `RequestBody`/`ProcessRequest`, the token middlewares' 401s, `ProcessRequest`'s panic 500) do not. The hook is shared by all requests and must be safe for concurrent use. A hook panic propagates out of `WriteError` with nothing written, reaching `ProcessRequest`'s recovery (a 500, logged with the panic value through the instance's `WithLogger` logger, and not logged without one) or `net/http`'s (the connection is aborted). +- As landed in 5.4: an override status outside 400–599 is ignored (an error never answers 2xx/3xx, and `New` returns no error), and the empty kind cannot be overridden, so errors without a kind and nil errors always answer 500. A later valid override for the same kind wins; an ignored call leaves any earlier override in place. A later `WithErrorHook` replaces an earlier one, and nil means no hook. The response title follows the decided status: the internal title for any 5xx (like `Error501`), the unauthorized title for 401/403, the default title otherwise. The hook receives the status `WriteError` decided, which may differ from the one sent if the handler already wrote a header or a custom formatter or writer changes it. Only `WriteError` calls the hook: the library's own rejections (body-parse failures in `RequestBody`/`ProcessRequest`, the token middlewares' 401s, `ProcessRequest`'s panic 500) do not. The hook is shared by all requests and must be safe for concurrent use. A hook panic propagates out of `WriteError` with nothing written, reaching `ProcessRequest`'s recovery (a 500, logged with the panic type, and the value only for a `runtime.Error`, through the instance's `WithLogger` logger, and not logged without one) or `net/http`'s (the connection is aborted). ### 8. RFC 9457 problem details (API-12) diff --git a/docs/migration-v1.md b/docs/migration-v1.md index 0aaead5..d3f0fc9 100644 --- a/docs/migration-v1.md +++ b/docs/migration-v1.md @@ -33,10 +33,10 @@ Requests that no `a.Handler` wraps use the default instance. Since v1.0.0 it ans **Logging changes:** -- `ProcessRequest` writes `slog` records through `slog.Default()`, read at call time: `request` and `response` info records, `request body rejected`, `panic recovered` and the `Get*Value*` helper failures as error records, the same records an instance writes with `WithLogger` (see the README "Logging" section). `slog.Default()` writes through the standard `log` package unless you call `slog.SetDefault`. +- `ProcessRequest` writes `slog` records through `slog.Default()`, read at call time: `request` and `response` info records, `request body rejected`, `panic recovered` (with `panic_type`; the value itself only for a `runtime.Error`, `[REDACTED]` otherwise) and the `Get*Value*` helper failures as error records, the same records an instance writes with `WithLogger` (see the README "Logging" section). `slog.Default()` writes through the standard `log` package unless you call `slog.SetDefault`. - `Print` and `PrintError` are never called by the package any more; only the deprecated `LogRequest`, `LogResponse` and `LogResponseWithDuration` call `Print`. A service that replaced them to route logs must call `slog.SetDefault` with its handler, or move to `api.WithLogger` ([step 3](#step-3-move-to-apinew)). `Fatal` is never called, as since v0.11.0. - Bodies and the parsed form are never logged. `LogRequestBody`, `LogResponseBody`, `PrintFullEvent` and `MaxLoggedBodyBytes` now affect only the deprecated `Log*` functions. -- Header values and the query values in `uri` whose key is sensitive are masked, as is the `uri`'s user information. The keys are the built-in list plus `SensitiveLogKeys`: the variable can add keys but no longer remove a built-in one. +- Header values and the query values in `uri` whose key is sensitive are masked, as is the `uri`'s user information. Other header values have the query values after each `?` masked the same way, and the user information too when they carry a URL (`Referer`, `Origin`, `X-Original-URL`, …). The keys are the built-in list plus `SensitiveLogKeys`: the variable can add keys but no longer remove a built-in one. - A response that cannot be encoded is answered with 500 and the registered formatter's `Error500()` body (bare 500 if it returns nil), and logged as `response cannot be encoded`. **What still configures the default instance:** `SetRequestMode`, `SetResponseMode` (`CurrentRequestMode`, `CurrentResponseMode`), `TraceVisibility`, `TraceIDLength`, `BlankSuccess`, `SensitiveLogKeys` (adds only) and the components set with `RegisterNewAPIRequestReceiver`, `RegisterNewAPIResponseFormatter` and `RegisterNewAPIResponseWriter`. @@ -206,7 +206,8 @@ a := api.New( **Logging.** An instance logs only through its `WithLogger` logger, with the same `slog` records as the default instance (`request`, `response`, and error records). Without a logger it logs nothing; the default instance logs through `slog.Default` instead. Both: - mask the built-in sensitive keys plus the added ones: `WithSensitiveLogKeys` on an instance, `SensitiveLogKeys` on the default instance; changing `SensitiveLogKeys` does not reach an instance; -- mask header values, query values in `uri` (`/login?access_token=[REDACTED]&page=2`) and the user information of the `uri` (`http://[REDACTED]@host/...`); path segments are not; +- mask header values, query values in `uri` (`/login?access_token=[REDACTED]&page=2`) and after each `?` in other header values (`Referer`, `Origin`, `Link`, …), and the user information of the `uri` (`http://[REDACTED]@host/...`); path segments are not; +- log a recovered panic's `panic_type` and, as `panic`, the value only for a `runtime.Error` (`[REDACTED]` otherwise); - never log bodies or the parsed form; - log a failed `Get*Value*` helper with the name as `key`; for a sensitive name the `error` gives only the kind of failure, never the value; - never call `Fatal` or exit from the built-in writer: a body that cannot be encoded is logged and answered with 500 and the `Error500()` body, with the formatter's `Content-Type`. The failed response's headers that describe its body (`Content-Type`, `Content-Encoding`, `Content-Length`, `ETag`, …) are dropped from that 500; the others are kept. diff --git a/functions.go b/functions.go index a272184..f9bb8c7 100644 --- a/functions.go +++ b/functions.go @@ -431,9 +431,11 @@ func defaultSensitiveLogKeys() []string { // LogRequest prints API request in log. The values of sensitive headers, // JSON body fields, query parameters in uri and form fields are masked (see -// SensitiveLogKeys), as is the user information of an absolute-form uri. A -// request body that is not JSON is logged as its size unless PrintFullEvent -// is true. The package no longer calls it: ProcessRequest logs through +// SensitiveLogKeys), as is the user information of an absolute-form uri. +// Other header values have the query values after each "?" masked the same +// way, and the user information too when they carry a URL (Referer, Origin, +// X-Original-URL, ...). A request body that is not JSON is logged as its +// size unless PrintFullEvent is true. The package no longer calls it: ProcessRequest logs through // slog.Default, or through the instance's WithLogger logger. // // Deprecated: use api.New(api.WithLogger(...)) instead. @@ -518,7 +520,8 @@ func sanitizeHeadersForLog(headers http.Header) http.Header { } // redactHeaders returns a copy of headers with the values of the headers -// whose name matches sensitiveKeys replaced by redactedLogValue. +// whose name matches sensitiveKeys replaced by redactedLogValue, and the other +// values masked by redactHeaderValue. func redactHeaders(headers http.Header, sensitiveKeys []string) http.Header { sanitized := make(http.Header, len(headers)) for key, values := range headers { @@ -526,7 +529,11 @@ func redactHeaders(headers http.Header, sensitiveKeys []string) http.Header { sanitized[key] = []string{redactedLogValue} continue } - sanitized[key] = append([]string(nil), values...) + redacted := make([]string, len(values)) + for i, value := range values { + redacted[i] = redactHeaderValue(value, sensitiveKeys) + } + sanitized[key] = redacted } return sanitized } @@ -585,6 +592,36 @@ func redactURI(uri string, sensitiveKeys []string) string { return redactUserInfo(redacted, uri) } +// redactHeaderValue masks a header value that carries a URL as uri is +// masked. The shape decides, not the header name, so custom proxy headers +// (X-Original-URI, X-Envoy-Original-Path, ...) are covered too. Every "?" +// starts a query that runs to the next "?", so each URL of a value that holds +// several (a Link list) has its sensitive query values masked; a value that +// starts with "/" or contains "://" also has its user information masked, as +// in uri. A value with neither "?" nor "@" is kept as is, and so is an "@" +// without URL shape (From: ops@example.com). A non-URL value with "?" or +// "://" may be over-masked, the safe direction. +func redactHeaderValue(value string, sensitiveKeys []string) string { + if !strings.ContainsAny(value, "?@") { + return value + } + redacted := redactEveryQuery(value, sensitiveKeys) + if strings.HasPrefix(value, "/") || strings.Contains(value, "://") { + return redactUserInfo(redacted, value) + } + return redacted +} + +// redactEveryQuery masks, with redactQuery, the text after each "?" of value +// up to the next "?". +func redactEveryQuery(value string, sensitiveKeys []string) string { + parts := strings.Split(value, "?") + for i := 1; i < len(parts); i++ { + parts[i] = redactQuery(parts[i], sensitiveKeys) + } + return strings.Join(parts, "?") +} + // redactUserInfo replaces the user information of target with // redactedLogValue: everything from the start of the authority to the last // "@", so a raw "/", "?" or "#" in a password is masked too (an "@" in the diff --git a/functions_logging_test.go b/functions_logging_test.go index 830d40e..b965948 100644 --- a/functions_logging_test.go +++ b/functions_logging_test.go @@ -53,6 +53,21 @@ func TestLogRequestRedactsSensitiveHeadersAndJSONBody(t *testing.T) { } } +func TestLogRequestRedactsURLValuedHeaders(t *testing.T) { + headers := http.Header{"Referer": []string{"https://app.example/cb?token=referer-token&page=2"}} + + output := captureLogOutput(t, func() { + LogRequest(http.MethodGet, "/tasks", "event-id", "", headers, nil) + }) + + if strings.Contains(output, "referer-token") { + t.Fatalf("expected the referer token to be redacted, got %q", output) + } + if !strings.Contains(output, `cb?token=[REDACTED]\u0026page=2`) { + t.Fatalf("expected the referer query masked as in uri, got %q", output) + } +} + func TestFormatBodyForLogTruncatesSanitizedBody(t *testing.T) { previousMaxLoggedBodyBytes := MaxLoggedBodyBytes previousPrintFullEvent := PrintFullEvent diff --git a/instance.go b/instance.go index db32477..7d6eaa5 100644 --- a/instance.go +++ b/instance.go @@ -243,7 +243,7 @@ const maxErrorStatus = 599 // rejections (body-parse failures in RequestBody and ProcessRequest, the // token middlewares' 401s, ProcessRequest's panic 500) do not. A panic in fn // is not recovered: it propagates out of WriteError with nothing written. -// Under ProcessRequest it answers 500 and is logged, with the panic value, +// Under ProcessRequest it answers 500 and is logged, with the panic type, // through the instance's WithLogger logger; nothing is logged without one. // fn is shared by all // requests the instance serves and may run concurrently, so it must be safe @@ -261,10 +261,13 @@ func WithErrorHook(fn func(r *http.Request, err error, status int)) Option { // panic, a response that cannot be encoded) are written as error records. // Request and response bodies and the parsed form are never logged. Header // and query values whose key is sensitive (see WithSensitiveLogKeys) are -// masked, as is the user information of an absolute-form URI; a failed -// Get*Value* helper logs the name as "key" and, for a sensitive name, only -// the kind of error, never the value. A panic with http.ErrAbortHandler is a -// "handler aborted" debug record instead of an error record. +// masked, as is the user information of an absolute-form URI; other header +// values have the query values after each "?" masked the same way, and the +// user information too when they carry a URL; a failed Get*Value* helper logs the name as "key" and, for a sensitive name, only +// the kind of error, never the value. A recovered panic logs its Go type as +// "panic_type" and its value only for a runtime.Error. A panic with +// http.ErrAbortHandler is a "handler aborted" debug record instead of an +// error record. // The built-in writer's records carry no request context or event_id, since // core.APIResponseWriter.Write receives no request. // Without it, or with nil, the instance writes no logs. A later WithLogger diff --git a/instance_bench_test.go b/instance_bench_test.go index 4b47a05..6c8f596 100644 --- a/instance_bench_test.go +++ b/instance_bench_test.go @@ -102,3 +102,20 @@ func BenchmarkProcessRequest(b *testing.B) { }) } } + +// BenchmarkRedactHeaders measures masking a typical request header set: a +// sensitive name, two values without URL shape and a Referer with a query. +func BenchmarkRedactHeaders(b *testing.B) { + headers := http.Header{ + "Authorization": []string{"Bearer placeholder"}, + "Accept": []string{"application/json"}, + "User-Agent": []string{"Mozilla/5.0 (X11; Linux) Firefox/120.0"}, + "Referer": []string{"https://app.example/cb?token=placeholder&page=2"}, + } + sensitiveKeys := defaultSensitiveLogKeys() + + b.ReportAllocs() + for b.Loop() { + redactHeaders(headers, sensitiveKeys) + } +} diff --git a/instance_log.go b/instance_log.go index 363ce5a..ac4b465 100644 --- a/instance_log.go +++ b/instance_log.go @@ -5,9 +5,11 @@ import ( "context" "encoding/json" "errors" + "fmt" "log/slog" "net/http" "net/http/httptest" + "runtime" "strconv" "strings" "time" @@ -82,7 +84,8 @@ func sensitiveValueError(err error) string { // logRequest logs the request ProcessRequest received. It never logs the // body or the parsed form (it may come from the body) and masks the query -// values whose key is sensitive. +// values whose key is sensitive, in uri and in header values that carry a URL +// (see redactHeaders). func (a *API) logRequest(r *http.Request, eventID string) { if !a.logs(r.Context(), slog.LevelInfo) { return @@ -114,9 +117,13 @@ func (a *API) logResponse(r *http.Request, eventID string, rec *httptest.Respons ) } -// logPanic logs a panic ProcessRequest recovered, with the panic value and -// the event ID. A panic with http.ErrAbortHandler is a deliberate abort, so -// it is logged at debug level as "handler aborted". +// logPanic logs a panic ProcessRequest recovered, with the event ID, the +// panic value's Go type as "panic_type" and, as "panic", the value itself +// only for a runtime.Error. Any other value (panic(err) of a failed parse, +// a formatted string) may carry request data, and masking by key cannot +// apply to an opaque value, so it is logged as redactedLogValue. A panic +// with http.ErrAbortHandler is a deliberate abort, so it is logged at debug +// level as "handler aborted", under the same rule. func (a *API) logPanic(r *http.Request, eventID string, recovered any) { level, msg := slog.LevelError, "panic recovered" if err, ok := recovered.(error); ok && errors.Is(err, http.ErrAbortHandler) { @@ -125,9 +132,14 @@ func (a *API) logPanic(r *http.Request, eventID string, recovered any) { if !a.logs(r.Context(), level) { return } + panicAttr := slog.String("panic", redactedLogValue) + if runtimeErr, ok := recovered.(runtime.Error); ok { + panicAttr = slog.Any("panic", runtimeErr) + } a.log().LogAttrs(r.Context(), level, msg, slog.String("event_id", eventID), - slog.Any("panic", recovered), + panicAttr, + slog.String("panic_type", fmt.Sprintf("%T", recovered)), ) } diff --git a/instance_log_test.go b/instance_log_test.go index 57a3616..2f20fa1 100644 --- a/instance_log_test.go +++ b/instance_log_test.go @@ -203,6 +203,63 @@ func TestInstanceURIMaskingAgreesWithTheDefaultInstance(t *testing.T) { } } +// TestHeaderMaskingAgreesWithTheDefaultInstance checks that the default +// instance and an instance mask URL-valued header values as they mask uri, +// and keep the other values. +func TestHeaderMaskingAgreesWithTheDefaultInstance(t *testing.T) { + tests := []struct { + name string + header string + values []string + want []string + secret string + }{ + {name: "referer with token", header: "Referer", values: []string{"https://app.example/cb?token=referer-secret&page=2"}, want: []string{"https://app.example/cb?token=[REDACTED]&page=2"}, secret: "referer-secret"}, + {name: "origin with user info", header: "Origin", values: []string{"https://u:origin-secret@app.example"}, want: []string{"https://[REDACTED]@app.example"}, secret: "origin-secret"}, + {name: "origin-form", header: "X-Original-Url", values: []string{"/login?password=original-secret"}, want: []string{"/login?password=[REDACTED]"}, secret: "original-secret"}, + {name: "network-path", header: "X-Forwarded-Uri", values: []string{"//u:forwarded-secret@host/p?code=1"}, want: []string{"//[REDACTED]@host/p?code=1"}, secret: "forwarded-secret"}, + {name: "link", header: "Link", values: []string{`; rel="next"`}, want: []string{"; rel="next", ; rel="last"`}, want: []string{`