Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 122 additions & 1 deletion crates/data/src/diagnostics.rs
Original file line number Diff line number Diff line change
Expand Up @@ -604,11 +604,55 @@ fn ambiguous_userinfo(address: &str) -> bool {
!authority.contains('@') && !authority.starts_with('[') && (authority.contains(':') || before_at.contains(':'))
}

/// How many words after a bare `scheme://user:pass` token may still be its password's.
const PASSWORD_WORDS: usize = 4;

/// Whether an address token that ends at whitespace may be the front of `scheme://user:pass word@host`:
/// nothing but `x:y` after the scheme (no path, query or `@` yet), and a word with an `@` before a
/// slash on the same line. `tail` is the text after the token. When `y` can't be a port (`pass`),
/// the word may be any of the next [`PASSWORD_WORDS`]. When it can (`12`, `8080`), `x:y` may just
/// as well be a host and port, so only the very next word counts: `https://host:8080 failed,
/// contact admin@example.org` keeps its reason. The caller omits the rest. `https://host:8080/x
/// user@example.org` has a path and keeps its words. An empty `y` (`https://example.org:`) is no
/// password, and a host in brackets (IPv6) is no `x:y`, so neither is cut; an empty `x`
/// (`redis://:pa ss@host`) is. A token that ended at a quote or bracket, not whitespace, is part of
/// structured text and is left to the other rules.
fn password_runs_on(address: &str, tail: &str) -> bool {
let Some((_, rest)) = address.split_once("://") else { return false };
let Some((_, pass)) = rest.split_once(':') else { return false };
if !tail.starts_with(char::is_whitespace) || pass.is_empty() || rest.starts_with('[') || rest.contains(['/', '\\', '?', '#', '@']) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Residual credential leak: an @ inside the password defeats the new cut. (Confidence: medium, traced by reading, not run. Pre-existing in its effect, but in the class this PR claims to close.)

password_runs_on returns false when the address contains @. For https://alice:pa@ss word@host/x failed, the token is https://alice:pa@ss. ambiguous_userinfo is false (no / before the @), so the cut is skipped. The token then parses as user alice, host ss, and is emitted as https://ss/. The scan continues with word@host/x failed. The output keeps word (part of the password) and ss, plus word@host/x.

Suggestion: split the rest on the last @ before the next whitespace (or allow @ in pass and check the word after the final @), and add a regression test with https://alice:pa@ss word@host/x in both redact and redact_parameter. Or state the limit in docs/privacy.md next to the / case.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of scope for this PR. https://alice:pa@ss word@host can't be told apart from a credentialed address followed by prose with an email, and main behaves the same. The limit is now stated in docs/privacy.md in 24c42d8. (reviewed at 24c42d8)

return false;
}
let digits = pass.trim_end_matches([',', ';', '.', ')', ':']);
if digits.is_empty() {
return false;
}
let port = digits.len() <= 5 && digits.bytes().all(|b| b.is_ascii_digit());
// Words of the token's own line, a few of them, found without scanning the rest of the text.
let mut rest = tail;
for _ in 0..if port { 1 } else { PASSWORD_WORDS } {
rest = rest.trim_start_matches(|c: char| c.is_whitespace() && c != '\n');
if rest.is_empty() || rest.starts_with('\n') {
return false;
}
let end = rest.find(char::is_whitespace).unwrap_or(rest.len());
let word = rest.get(..end).unwrap_or("");
if word.split_once('@').is_some_and(|(before, _)| !before.is_empty() && !before.contains(['/', '\\']) && !word.contains("://")) {
return true;
}
rest = rest.get(end..).unwrap_or("");
}
false
}

/// Whether a word of text too long to scan names an address whose query or userinfo may hold
/// a credential: a linear look at each `://` word, without the full scanner.
fn may_hold_secret(text: &str) -> bool {
text.split_whitespace().filter(|word| word.contains("://")).any(|word| {
// Where the word ends in the text, so the words after it can be looked at.
let after = (word.as_ptr() as usize).checked_sub(text.as_ptr() as usize).and_then(|at| text.get(at + word.len()..)).unwrap_or("");
ambiguous_userinfo(word)
|| password_runs_on(word, after)
|| word
.split_once(['?', '#'])
.is_some_and(|(_, query)| query.split(['&', ';', '#']).any(|pair| secret_name(pair.split('=').next().unwrap_or(""))))
Expand Down Expand Up @@ -881,7 +925,7 @@ fn redact_scan(text: &str, diagnostic: bool) -> String {
field_before = false;
continue;
}
if ambiguous_userinfo(address) {
if ambiguous_userinfo(address) || password_runs_on(address, rest.get(url_end..).unwrap_or("")) {
// Its password may run on past whitespace, as a secret query's may: the rest
// goes too, and the marker says so.
out.push_str("[address and the rest omitted]");
Expand Down Expand Up @@ -1213,6 +1257,11 @@ mod tests {
let ambiguous = format!("{} https://u:123?w@host/x tail", "word ".repeat(4000));
assert!(secret_continuation_cut(&ambiguous).is_some());
assert_eq!(secret_continuation_cut(&format!("{} see https://example.org/a?page=2 then", "word ".repeat(4000))), None);
// A password with a space goes whole in long text too, as in short; a port and prose keep.
let open = format!("{} see https://alice:pass word@host/x failed", "word ".repeat(4000));
assert!(secret_continuation_cut(&open).is_some_and(|cut| !cut.contains("word@")));
let port = format!("{} see https://host:8080 failed, contact admin@example.org", "word ".repeat(4000));
assert_eq!(secret_continuation_cut(&port), None);
assert_eq!(secret_continuation_cut(&"word ".repeat(4000)), None);
assert_eq!(secret_continuation_cut("failed at https://example.org/a: HTTP 500"), None);
}
Expand Down Expand Up @@ -1342,6 +1391,78 @@ mod tests {
}
}

#[test]
fn unknown_address_passwords_with_spaces_are_cut_through_their_at_sign() {
for text in [
"see https://alice:pass word@host/x failed",
"see https://alice:12 word@host/x failed",
"https://alice:pass wo rd@host/x failed",
"https://alice:pa ss word@host:80/x failed",
"https://alice:12 word@example.org",
// An empty name, a `]` in the text, the fourth word, and a number too long for a port.
"redis://:pa ss@host/x failed",
"https://alice:pa]ss word@host failed",
"https://alice:pa[ss word@host failed",
"https://alice:[pa ss]@host/x failed",
"https://alice:pass a b c d@host failed",
"https://alice:123456 a b c@host failed",
// `host:8080` reads as `alice:12`: an email right after it is cut, and the marker says so.
"see https://host:8080 user@example.org",
"see https://alice:pa\tss@host/x failed",
] {
for clean in [redact(text), redact_parameter(text)] {
assert!(clean.ends_with("[address and the rest omitted]"), "{text} -> {clean}");
for private in ["alice", "pass", "pa ", "12", "word", "wo ", "rd@", "ss@", "@", "failed"] {
assert!(!clean.contains(private), "{private} in {text} -> {clean}");
}
assert_eq!(redact(&clean), clean);
}
// A saved copy is cut the same way, and says so.
let saved = secret_continuation_cut(text).unwrap_or_default();
assert!(saved.ends_with("[address and the rest omitted]") && !saved.contains('@'), "{text} -> {saved}");
}
// The cut is on one line: the next one is another message.
assert_eq!(redact("https://alice:pass word\nthen user@example.org"), "[address omitted] word\nthen user@example.org");
}

#[test]
fn addresses_without_an_open_password_keep_the_words_after_them() {
for text in [
// A path ends the authority: the colon is a port's, and the email is prose.
"see https://host:8080/x user@example.org failed",
"see https://host:8080/ user@example.org failed",
// No colon in the authority, so no password to run on.
"see https://host user@example.org failed",
"https://host/x and mail user@example.org",
// No `@` within reach of the address.
"see https://host:8080 is down, ask your admin to restart it at ops@example.org",
"see https://host:8080 failed",
// A host and port is ambiguous with a short password: only the next word counts.
"see https://host:8080 failed, contact admin@example.org",
"see https://[::1]:8080 user@example.org failed",
"see https://[::1]:8080 failed, contact admin@example.org",
"https://x:12345 a b@example.org failed",
"see https://host:8080\nuser@example.org",
"https://example.org: timeout, mail ops@example.org",
// The local part of a word before `@` is a path, not a password's tail.
"see https://host:8080 /tmp/a@2x.png failed",
] {
assert_eq!(redact(text), text, "{text}");
assert_eq!(redact_parameter(text), text, "{text}");
assert_eq!(secret_continuation_cut(text), None, "{text}");
}
// A later address is its own: only its user information goes, not the words around it.
assert_eq!(redact("see https://host:8080 https://u@example.org/x failed"), "see https://host:8080 https://example.org/x failed");
// Nothing left of the text once its punctuation is trimmed is no password to run on.
assert!(redact("https://example.org:, mail ops@example.org").ends_with("mail ops@example.org"));
// A port followed by `:` is no valid address (omitted whole), but its reason stays.
let clean = redact("https://host:8080: HTTP 503 (ask ops@example.org)");
assert_eq!(clean, "[address omitted] HTTP 503 (ask ops@example.org)");
// A token that ends at a quote is part of structured text, not a front of userinfo.
let json = r#"{"url":"https://h:8080","b":"a@b"}"#;
assert_eq!(redact_parameter(json), json);
}

#[test]
fn unambiguous_ports_and_path_at_signs_keep_address_context() {
for address in
Expand Down
154 changes: 144 additions & 10 deletions crates/engine/src/cmd/history.rs
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ pub(crate) fn redacted_parameter(t: &str) -> String {
} else if structured || t.contains("://") {
rustgis_data::diagnostics::redact_parameter(t)
} else {
t.to_string()
schemeless_query_cut(t)
};
query_pairs_emptied(&private_keys_removed(&text))
}
Expand All @@ -182,12 +182,43 @@ pub(crate) fn redacted_text(t: &str) -> String {
} else if t.contains("://") {
rustgis_data::diagnostics::redact(t)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: a scheme-less secret query is not cut when the same text also holds a :// address. (Confidence: medium, traced by reading, not run.)

The t.contains("://") branch sends the whole text to diagnostics::redact, and schemeless_query_cut is only reached in the final else. So see https://example.org/ok and data.tif?token=abc def goes to redact, which knows nothing of data.tif?token=… as a scheme-less word. query_pairs_emptied then empties abc, and def stays in copied history errors and the project log. The same gap is in redacted_parameter (line 167: structured || t.contains("://")). That contradicts docs/privacy.md, which says a secret query in scheme-less text "goes with the rest of the text".

Suggested fix for this branch: cut the scheme-less word first, then redact. URLs before the cut are still redacted by redact, and the cut marker is kept.

Suggested change
rustgis_data::diagnostics::redact(t)
rustgis_data::diagnostics::redact(&schemeless_query_cut(t))

The same change in redacted_parameter needs the structured case kept on the redact_parameter(t) path, so it is not a one-line edit there. Please add a test for "see https://example.org/ok and data.tif?token=abc def" in both redacted_text and redacted_parameter, asserting that def is gone and the marker is present, and that clean mixed text ("see https://example.org/ok and data.tif?v=2 def") is unchanged.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not reproduced: redact's field rule handles it. In a text that also holds a :// address, data.tif?token=abc def becomes …data.tif?[REDACTED] and def is dropped; the same for ?pwd=, a key-shaped ?sig=, ;token= and #token=, and data.tif?v=2 def is unchanged. The engine functions only empty values after redact, so nothing comes back. Pinned in 9333137 (a_schemeless_secret_beside_an_address_is_cut_by_the_field_rule). No code change. (reviewed at 8abef92)

} else {
t.to_string()
schemeless_query_cut(t)
};
// After `redact` too: a prefix it knows may come in a form it doesn't (GDAL's `MYSQL:`).
query_pairs_emptied(&private_keys_removed(&r))
}

/// Whether `name=value` is a secret pair. A bare `key`, `sig` or `sas` is as likely a field in an
/// expression (`key=5`): a secret only when its value reads as one.
fn secret_pair(name: &str, value: &str) -> bool {
let k = name.trim();
!k.is_empty()
&& rustgis_data::source::credentials::secret_key(k)
&& (!["key", "sig", "sas"].iter().any(|p| k.eq_ignore_ascii_case(p))
|| crate::sources::looks_like_a_key(value.trim_matches(['\'', '"', '{', '}'])))
}

/// Text with no scheme that has a secret pair in the query or fragment of one of its words
/// (`data.tif?token=abc def`, `a.tif?x=1&token=abc def`, `a.tif?x=1;token=abc def`): a URL query
/// holds no raw space, so the words after it may be the value's. The word's query and the rest of
/// the text go, said, as for an address with a scheme. Only the names `secret_key` knows count,
/// as everywhere in this file. A libpq-style `password=abc` has no `?` or `#`: its value ends at
/// the space.
fn schemeless_query_cut(t: &str) -> String {
let mut start = 0;
for word in t.split_inclusive(char::is_whitespace) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low: a secret-query word cuts every later line, not just its own. (Confidence: medium, traced by reading, not run.)

split_inclusive(char::is_whitespace) treats \n as a word break, so data.tif?token=abc followed by a newline makes the cut run through all later lines of the copied error, the saved project-log error, or the replay parameter. The password rule in password_runs_on deliberately stops at the newline ("The cut is on one line"), so the two rules disagree.

The argument for cutting past a space (the value may continue) doesn't carry over to a newline, which a URL query can't hold and which this file already treats as a message boundary. If you agree, stop the scan at \n and keep the lines after it, for example by cutting only the rest of the line and keeping &t[line_end..]. Add a test that "a.tif?token=abc\nnext line" keeps next line. If the whole-text cut is intended, a sentence in docs/privacy.md saying so would settle it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept as is. A secret query already cuts the rest of the text for scheme:// URLs, newlines included (checked: https://x/a?token=abc\nnext line becomes https://x/a[query and the rest omitted]), and a value can run on past what we can see. The scheme-less cut now matches that rule, and over-cutting is marked and fail-safe, so changing it is left for the release quality pass. (reviewed at 48dae37)

let w = word.trim_end_matches(char::is_whitespace);
if let Some(q) = w.find(['?', '#']) {
let query = w.get(q + 1..).unwrap_or("");
if query.split(['&', ';', '#', '?']).filter_map(|pair| pair.split_once('=')).any(|(k, v)| secret_pair(k, v)) {
return format!("{}[query and the rest omitted]", t.get(..start + q).unwrap_or(""));
}
}
start += word.len();
}
t.to_string()
}

/// `t` with its secret pairs emptied (`secret_pairs_emptied`), parted first at each `&` that
/// starts another `name=` pair, as a query string's do (`a=1&password=…`, with no `://` for
/// `redact` to read it by). An `&` inside a value (`password=ab&cd`), or in a value quoted or
Expand Down Expand Up @@ -291,16 +322,14 @@ fn secret_pairs_emptied(t: &str) -> String {
let next = word.get(..word.find(parts).unwrap_or(word.len())).unwrap_or("");
if word.len() < value.len() && !next.is_empty() && !next.contains('=') { (k, at + value.len() - word.len()) } else { (k, at) }
});
// A bare `key`, `sig` or `sas` is as likely a field in an expression (`key=5`): a secret
// only when its value reads as one.
let secret = |&(k, at): &(&str, usize)| {
let k = k.trim();
let value = rest.get(at..).unwrap_or("");
let v = value.get(..value.find(parts).unwrap_or(value.len())).unwrap_or("");
!k.is_empty()
&& rustgis_data::source::credentials::secret_key(k)
&& (!["key", "sig", "sas"].iter().any(|p| k.eq_ignore_ascii_case(p))
|| crate::sources::looks_like_a_key(v.trim_matches(['\'', '"', '{', '}'])))
// In a word with a `?`, the name after the last `?`, `&` or `#`: its path before is no part
// of the name. Without one, the whole field name (`password#1`) or the part after its last
// `&` or `#` (`data.tif#pwd`) makes it a secret.
let tail = k.rsplit(['?', '&', '#']).next().unwrap_or(k);
let value = value.get(..value.find(parts).unwrap_or(value.len())).unwrap_or("");
if k.contains('?') { secret_pair(tail, value) } else { secret_pair(k, value) || secret_pair(tail, value) }
};
let Some((_, at)) = pair.filter(secret) else {
// Not a secret: the item and the separator after it, as they are.
Expand Down Expand Up @@ -445,3 +474,108 @@ fn shell(t: &str) -> String {
fn line(t: &str) -> String {
t.chars().map(|c| if c.is_control() || c == '\u{2028}' || c == '\u{2029}' { ' ' } else { c }).take(300).collect()
}

#[cfg(test)]
mod tests {
use super::*;

/// A URL query holds no raw space: a secret in a scheme-less address's query may run on past
/// one, so the rest is cut, said. Spaces in other text and libpq pairs stay as they were.
#[test]
fn a_secret_query_in_a_schemeless_address_cuts_the_rest() {
for text in ["data.tif?token=abc def", "data.tif?token=abc def ghi", "see data.tif?access_token=abc def"] {
for clean in [redacted_parameter(text), redacted_text(text)] {
assert!(clean.ends_with("[query and the rest omitted]"), "{text} -> {clean}");
for private in ["abc", "def", "ghi", "token"] {
assert!(!clean.contains(private), "{private} in {text} -> {clean}");
}
}
}
assert_eq!(redacted_parameter("see data.tif?token=abc def"), "see data.tif[query and the rest omitted]");
// A secret anywhere in the word's query, and the exact names a path word would hide.
for text in [
"a.tif?x=1&token=abc def",
"a.tif?token=abc&x=1 def",
"data.tif?pwd=hunter2 def",
"data.tif?sig=Ab3dE9fG7hJ2kL5mN8pQ4rS6tU1vW0xY def",
"a.tif?x=1;token=abc def",
"a.tif?x=1#token=abc def",
"a.tif?x=1?token=abc def",
"data.tif#token=abc def",
] {
let clean = redacted_parameter(text);
assert!(
clean.ends_with("[query and the rest omitted]") && !clean.contains("def") && !clean.contains("hunter2"),
"{text} -> {clean}"
);
assert!(!clean.contains("x=1"), "{text} -> {clean}");
}
// A credential word in the path is no secret query: the words after it stay.
for text in ["/data/tokens/a.tif?v=2 failed", r"C:\Users\me\secrets\a.tif?x=1 done"] {
let clean = redacted_parameter(text);
assert!(!clean.contains("omitted]") && (clean.ends_with("failed") || clean.ends_with("done")), "{text} -> {clean}");
}
for text in [
"/data/a b.tif?x=1",
"/data/a.tif?sig=abc def&x=1 y",
"a.tif?key=abc def",
"data.tif?v=2 and more words",
"a.tif?x=1;y=2 def",
"host=h dbname=g user=u",
] {
assert_eq!(redacted_parameter(text), text, "{text}");
}
// A libpq pair has no `?`: its value ends at the space, the next word is a pair of its own.
assert_eq!(redacted_parameter("host=h password=abc dbname=g"), "host=h password= dbname=g");
assert_eq!(redacted_parameter("password=a b"), "password= b");
}

/// A secret in the query of one word is judged by its own name, not by the whole word: an exact
/// name (`pwd`, a key-shaped `sig`) is no less a secret behind a path, and a credential word in
/// a folder name is no secret query.
#[test]
fn a_secret_in_the_query_of_one_word_is_emptied_by_its_own_name() {
for (text, kept) in [
("data.tif?pwd=hunter2", "data.tif?pwd="),
("data.tif?sig=Ab3dE9fG7hJ2kL5mN8pQ4rS6tU1vW0xY", "data.tif?sig="),
("data.tif?token=abc", "data.tif?token="),
("a.tif?x=1&pwd=hunter2", "a.tif?x=1&pwd="),
("data.tif?v=2", "data.tif?v=2"),
("/data/tokens/a.tif?v=2", "/data/tokens/a.tif?v=2"),
("data.tif?sig=abc", "data.tif?sig=abc"),
// A field name that holds a `#` or an `&` but no `?` is judged whole.
("password#1=hunter2", "password#1="),
("token#x=abc", "token#x="),
("password#1=hunter2 failed", "password#1= failed"),
// A word with no `?` is secret by its whole name or by the part after its last `&` or `#`.
("data.tif#pwd=hunter2", "data.tif#pwd="),
("data.tif#v=2", "data.tif#v=2"),
] {
assert_eq!(redacted_parameter(text), kept, "{text}");
assert_eq!(redacted_text(text), kept, "{text}");
}
}

/// The project log passes every string of a parameter through the same cut. Copy as code does
/// not: it keeps opaque text and says so (see `durable::output`).
#[test]
fn text_parameters_of_the_project_log_get_the_schemeless_cut() {
let params = serde_json::json!({"text": "data.tif?token=abc def", "n": 1});
assert_eq!(redacted(&params), serde_json::json!({"text": "data.tif[query and the rest omitted]", "n": 1}));
let clean = serde_json::json!({"text": "data.tif?v=2 def", "n": 1});
assert_eq!(redacted(&clean), clean);
}

/// A scheme-less secret query in text that also holds a `://` address goes through `redact`'s
/// field rule: the secret and the words after it are gone.
#[test]
fn a_schemeless_secret_beside_an_address_is_cut_by_the_field_rule() {
let secret = "see https://example.org/ok and data.tif?token=abc def";
let clean = "see https://example.org/ok and data.tif?v=2 def";
for cut in [redacted_text(secret), redacted_parameter(secret)] {
assert_eq!(cut, "see https://example.org/ok and data.tif?[REDACTED]");
}
assert_eq!(redacted_text(clean), clean);
assert_eq!(redacted_parameter(clean), clean);
}
}
4 changes: 2 additions & 2 deletions crates/engine/src/cmd/layer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3661,7 +3661,7 @@ fn parts_left_out(url: &str, bytes: &[u8], p: &Value) -> Option<String> {
if more > 0 {
said.push(format!("and {more} more Shapefiles without some of theirs"));
}
(!said.is_empty()).then(|| format!("{}: {}", crate::sources::redact(url), said.join("; ")))
(!said.is_empty()).then(|| format!("{}: {}", crate::sources::address_in_message(url), said.join("; ")))
}

/// A Shapefile by its `.shp` address, waited for (#1238): the `.shp`, then its `.dbf`, `.shx`,
Expand Down Expand Up @@ -4632,7 +4632,7 @@ fn add_download(s: &mut Session, p: &Value) -> CmdResult {
}
// No window landed for it: a whole file isn't added in a window's place.
if p.get("window").is_some_and(Value::is_object) {
let shown = crate::sources::redact(&url);
let shown = crate::sources::address_in_message(&url);
return Err(format!("{shown}: \"window\" is for a window the app read off the frame: use layer.addUrl with \"bbox\""));
}
// A download read already (an ArcGIS layer's protocol buffer pages, joined): added as read,
Expand Down
Loading
Loading