feat(go): test several members of one array element of a JsonSearch field

languages/golang/CHANGELOG.md

@@ -12,6 +12,29 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
 
 ### Added
 
+- **Test several members of one array element of a `JsonSearch` field
+  (#1442).** A field whose `ordered=` paths name members beneath an array's
+  wildcard (`$.contacts[*].relationship;$.contacts[*].calls`) gets
+  `Fields.<Field>.Elements.<Array>.Match`, one accessor per array, named as
+  `At` names a path (`Elements.Contacts`). `Match` takes
+  `encrypt.ElementPredicates` made with `encrypt.MemberEq`, `MemberLt`,
+  `MemberLte`, `MemberGt`, `MemberGte` and `MemberBetween`, and returns an
+  `eql.JSONSearchElementsQuery` for
+  `eql_v4.json_tree_element_match(col, $1::eql_v4.query_json_elements)`,
+  which finds a row where one element passes every predicate. An `EqAt` and
+  an ordering at the array's paths can each be answered by a different
+  element. Each comparison is of the member's ordering term, so `MemberEq`
+  compares text case- and accent-folded and integers past 2^53 can tie;
+  `EqAt` stays exact. A member the field does not order is refused before
+  the engine is asked, with `encrypt.ErrTerm`, naming the member and the
+  array; no predicates, a zero `ElementPredicate` and a member with a
+  wildcard are refused with `encrypt.ErrEncoding`. A row whose array is
+  longer than `JSONReadOptions.WildcardBound` (64 by default) fails the
+  query with SQLSTATE 22023. An `ElementPredicate` prints and logs its
+  member and comparison, never its values. The embedded EQL guest now names
+  the `eql_v4.query_json_elements` domain, so `Checker.Operators` lists
+  `element_match` (input `path_predicates`) on a field that orders a path.
+
 - **A two-letter word is searchable in `TextMatch`, `TextSearch` and
   `TextSearchOre` (#1309).** The EQL v4 match types index each two-letter
   word whole, beside the 3-grams of the text, so a two-letter surname such

languages/golang/cmd/stashgen/README.md

@@ -178,6 +178,7 @@ The field's `Fields` entry has a method per operation:
 | `Selector(ctx, c, path)` | `encrypt.JSONSelection`: `Selector()` and `Open` | `col -> $1::text` |
 | `Subtree(ctx, c, path, opts)` | `encrypt.JSONRead`: `Tokens()`, `Open` and `OpenOne` | `eql_v4.json_tree_subtree(col, $1::text[], 64)` |
 | `Children(ctx, c, path)`, `ChildCount(ctx, c, path)` | `encrypt.JSONToken` | `eql_v4.json_tree_children(col, $1::text)`, `eql_v4.json_tree_child_count(col, $1::text)` |
+| `Elements.Visits.Match(ctx, c, predicates, opts)` | `eql.JSONSearchElementsQuery` | `eql_v4.json_tree_element_match(col, $1::eql_v4.query_json_elements)` |
 
 `At` has one accessor per ordered path, named for its keys (`$.visits[*].at` is `At.VisitsAt`, `$.a.1` is `At.A1`); two paths with one name, or one path given twice, stop `stashgen`.
 An ordering compares the value `->` extracts at its path, and EQL's `<` compares ordering terms alone, so take the selector from the same accessor (`At.Age.Selector`) rather than spelling the path again.
@@ -193,6 +194,32 @@ SELECT id FROM customers ORDER BY (
 ```
 
 with `$1` from `At.VisitsAt.Selector(ctx, c)` and `$2` from `At.VisitsAt.Gt(ctx, c, v)`; for the latest visit, order by `ORDER BY 1 DESC NULLS LAST` inside and `DESC` outside.
+
+Two queries at a wildcard path, an `EqAt` and an ordering say, can each be answered by a different element.
+To test several members of one element together, use `Match` under `Elements`.
+`Elements` has one accessor per array whose elements have an ordered member, named for the array's keys as `At` names a path: with `ordered=$.visits[*].at;$.visits[*].place`, `$.visits[*]` is `Elements.Visits`, with the members `at` and `place`.
+A root array (`ordered=$[*].name`) is `Elements.Root`.
+An ordered path that ends at its wildcard (`$.tags[*]`), or has none, gives no accessor, and a field without one has no `Elements`.
+Two arrays whose keys give one name (`$.a_b[*]` and `$.aB[*]` are both `AB`), and an array whose keys give no name, get no accessor: `stashgen` still writes the field and its `At`, and prints a notice naming the ordered paths, which the generated file repeats in a comment.
+
+```go
+// rows with one visit that is at home and after 15
+q, err := customers.Fields.Profile.Elements.Visits.Match(ctx, cipher, encrypt.ElementPredicates{
+	encrypt.MemberEq("place", "home"),
+	encrypt.MemberGt("at", 15),
+}, encrypt.JSONReadOptions{})
+```
+
+```sql
+SELECT id FROM customers WHERE eql_v4.json_tree_element_match(profile, $1::eql_v4.query_json_elements)
+```
+
+The predicates are `encrypt.MemberEq`, `MemberLt`, `MemberLte`, `MemberGt`, `MemberGte` and `MemberBetween` (inclusive), each with a member written as it is below the array in the tag (`place`, or `address.city`) and Go values, as the orderings take.
+A member the field does not order is refused before the engine is asked, with `encrypt.ErrTerm`, naming the member, the array and the members it orders; no predicates, and a member with a wildcard, are refused with `encrypt.ErrEncoding`.
+Each comparison is of the member's ordering term, so `MemberEq` is an equality of folded values: text compares case- and accent-folded (`"home"` equals `"Home"`), and integers past 2^53 can tie.
+`EqAt` at the member's full path is exact, of any element.
+`Match` reads each array to `opts.WildcardBound` elements, 64 when it is 0, and a row whose array is longer fails the query with SQLSTATE 22023 rather than answer over part of it: set the bound past the longest array.
+The function runs on each row a scan gives it, so narrow the rows first with an indexed predicate, such as `@>` on the same values.
 `OpenOne` reads one value and refuses a read whose path has a wildcard before anything is opened; open a wildcard read with `Open`.
 A needle and a value are Go values: a `json.RawMessage` of JSON text crosses as it is, and anything else is marshalled with `encoding/json`.
 `EqAt` and the orderings take a scalar: a string, a number, a bool, or `nil` for JSON `null`.

languages/golang/cmd/stashgen/main_test.go

@@ -442,6 +442,90 @@ func TestRunGeneratesJSONSearchFields(t *testing.T) {
 	if !strings.Contains(written, "func (p ProfileFieldPath) Selector(ctx context.Context, c *encrypt.Cipher) (encrypt.JSONSelection[eql.JSONEntry], error)") {
 		t.Error("an ordered path has no Selector of its own")
 	}
+	// The array with an ordered member gets an accessor under Elements,
+	// named as At names its path, whose Match derives the element match.
+	for _, want := range []string{
+		"Elements ProfileFieldElements",
+		"Visits ProfileFieldArray // $.visits[*]: at",
+		`Visits: ProfileFieldArray{field: f, path: "$.visits[*]"},`,
+		"func (a ProfileFieldArray) Match(ctx context.Context, c *encrypt.Cipher, predicates encrypt.ElementPredicates, opts encrypt.JSONReadOptions) (eql.JSONSearchElementsQuery, error)",
+		"gensupport.JSONElementMatch[eql.JSONSearchElementsQuery](ctx, a.field, c, a.path, predicates, opts)",
+	} {
+		if !strings.Contains(written, want) {
+			t.Errorf("the generated file lacks %q", want)
+		}
+	}
+	if strings.Contains(written, "PrefsFieldElements") {
+		t.Error("Prefs orders no member, and got Elements")
+	}
+	// The members of one array, named as At names their paths; an ordered
+	// path that ends at its wildcard, or has none, is no array's member.
+	written, stderr, code = generate("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.extra.emergency_contacts[*].relationship;$.extra.emergency_contacts[*].previous_call_count;$.tags[*];$.age)\"`")
+	if code != 0 {
+		t.Fatalf("exit %d\n%s", code, stderr)
+	}
+	// gofmt aligns the struct fields: read each line's words.
+	words := strings.Join(strings.Fields(written), " ")
+	for _, want := range []string{
+		"ExtraEmergencyContactsRelationship ProfileFieldPath // $.extra.emergency_contacts[*].relationship",
+		"ExtraEmergencyContacts ProfileFieldArray // $.extra.emergency_contacts[*]: relationship, previous_call_count",
+	} {
+		if !strings.Contains(words, want) {
+			t.Errorf("the generated file lacks %q", want)
+		}
+	}
+	if strings.Contains(written, "Tags ProfileFieldArray") {
+		t.Error("$.tags[*] orders the elements themselves, no member, and got an accessor under Elements")
+	}
+	// A field that orders no member beneath an array has no Elements.
+	written, stderr, code = generate("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.age;$.tags[*])\"`")
+	if code != 0 {
+		t.Fatalf("exit %d\n%s", code, stderr)
+	}
+	if strings.Contains(written, "Elements") || strings.Contains(written, "JSONElementMatch") {
+		t.Error("a field with no ordered member beneath an array got Elements")
+	}
+	// A root array's accessor is Root, beside its paths' At accessors.
+	written, stderr, code = generate("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$[*].name;$[*].age)\"`")
+	if code != 0 {
+		t.Fatalf("a root array: exit %d\n%s", code, stderr)
+	}
+	words = strings.Join(strings.Fields(written), " ")
+	for _, want := range []string{
+		"Name ProfileFieldPath // $[*].name",
+		"Root ProfileFieldArray // $[*]: name, age",
+		`Root: ProfileFieldArray{field: f, path: "$[*]"},`,
+	} {
+		if !strings.Contains(words, want) {
+			t.Errorf("a root array: the generated file lacks %q", want)
+		}
+	}
+	// Arrays whose keys give one accessor name, or none, get no accessor:
+	// the tag still generates, At is as it is without them, and a notice
+	// in the file and on stderr names the paths as the tag writes them.
+	for _, c := range []struct{ name, ordered, at, notice string }{
+		{"two arrays, one name", "$.a_b[*].x;$.aB[*].y;$.c[*].z", "ABX ProfileFieldPath // $.a_b[*].x", "no Elements accessor for the elements of $.a_b[*] (ordered $.a_b[*].x): another array's keys give the same accessor name, Elements.AB"},
+		{"an array with no name", "$.__[*].x;$.c[*].z", "X ProfileFieldPath // $.__[*].x", "no Elements accessor for the elements of $.__[*] (ordered $.__[*].x): its keys give no Go accessor name"},
+	} {
+		written, stderr, code := generate("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=" + c.ordered + ")\"`")
+		if code != 0 {
+			t.Fatalf("%s: exit %d\n%s", c.name, code, stderr)
+		}
+		// A wrapped comment reads as one line.
+		words := strings.Join(strings.Fields(strings.ReplaceAll(written, "\n// ", " ")), " ")
+		for _, want := range []string{c.at, "C ProfileFieldArray // $.c[*]: z", "Profile: " + c.notice} {
+			if !strings.Contains(words, want) {
+				t.Errorf("%s: the generated file lacks %q", c.name, want)
+			}
+		}
+		if !strings.Contains(stderr, "stashgen: Customer.Profile: "+c.notice) {
+			t.Errorf("%s: stderr = %q", c.name, stderr)
+		}
+	}
+	written, _, code = generate("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.a_b[*].x;$.aB[*].y)\"`")
+	if code != 0 || strings.Contains(written, "ProfileFieldElements") || !strings.Contains(written, "ABY ProfileFieldPath // $.aB[*].y") {
+		t.Errorf("two colliding arrays and no other: exit %d, Elements written or At changed", code)
+	}
 	// A json.RawMessage sealed any other way is bytes, as it was before
 	// documents; an object key of digits names its accessor.
 	written, stderr, code = generate("\tData json.RawMessage `stash:\"data,encrypt,index=equality\"`\n\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.a.1)\"`")
@@ -473,6 +557,19 @@ func TestRunGeneratesJSONSearchFields(t *testing.T) {
 	if _, err := os.Stat(filepath.Join(dir, "customer_stash.go")); !os.IsNotExist(err) {
 		t.Fatal("a file was written after the refusal")
 	}
+	// The same for an array whose elements have an ordered member, and an
+	// engine that answers no element_match.
+	dir = writeModule(t, map[string]string{"model.go": source("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.visits[*].at)\"`")})
+	stderrBuf.Reset()
+	if code := run([]string{"-type", "Customer"}, dir, &bytes.Buffer{}, &stderrBuf, withoutOperator("element_match")); code != 1 {
+		t.Fatalf("exit %d, want 1\n%s", code, stderrBuf.String())
+	}
+	if got := stderrBuf.String(); !strings.Contains(got, "Customer.Profile") || !strings.Contains(got, "the engine answers no element_match") {
+		t.Fatalf("stderr = %q", got)
+	}
+	if _, err := os.Stat(filepath.Join(dir, "customer_stash.go")); !os.IsNotExist(err) {
+		t.Fatal("a file was written after the refusal")
+	}
 	for _, c := range []struct{ name, fields, want string }{
 		{"a sealed null", "\tPrefs *json.RawMessage `stash:\"prefs,encrypt_into=JsonSearch,null=sealed\"`", "Customer.Prefs: a nil has an equality term and no other, so another index's column would show which values are absent; write null=sql"},
 		{"a nil document that does not say null=sql", "\tPrefs *json.RawMessage `stash:\"prefs,encrypt_into=JsonSearch\"`", "null=sql"},

languages/golang/encrypt/README.md

@@ -520,7 +520,7 @@ The EQL types the engine produces, and the Go field types each takes:
 | `Numeric` | `encrypt.Decimal` | none |
 | `NumericEq` | `encrypt.Decimal` | `=`, `<>` (`1.5 = 1.50`) |
 | `NumericOrd`, `NumericOrdOpe`, `NumericOrdOre` | `encrypt.Decimal` | `<`, `<=`, `>`, `>=`, `ORDER BY` |
-| `JsonSearch` (`eql.JSONSearch`) | struct, map, slice, or `json.RawMessage` | containment, equality at a path, ordering at declared paths, selection, subtree reads; see [Encrypted JSON documents](#encrypted-json-documents) |
+| `JsonSearch` (`eql.JSONSearch`) | struct, map, slice, or `json.RawMessage` | containment, equality at a path, ordering at declared paths, several members of one array element, selection, subtree reads; see [Encrypted JSON documents](#encrypted-json-documents) |
 
 Any built-in Go integer field takes an integer EQL type (ADR-0002): the field
 is sealed at the type's width, and opens back into the field's type. A type
@@ -658,6 +658,26 @@ element of each row, so an ordering over every element is written over
 `eql_v4.json_tree_path_query(col, $sel)`, its matches (the stashgen README
 shows the SQL).
 
+Two such queries can each be answered by a different element. To test
+several members of one array element together, use `Match` on the array's
+accessor under `Elements`. A field has one for each array whose elements
+have an ordered member (`ordered=$.visits[*].at;$.visits[*].place`):
+
+```go
+q, err := users.Fields.Profile.Elements.Visits.Match(ctx, cipher, encrypt.ElementPredicates{
+	encrypt.MemberEq("place", "home"),
+	encrypt.MemberGt("at", 15),
+}, encrypt.JSONReadOptions{})
+rows, err := db.QueryContext(ctx, `SELECT id FROM users WHERE eql_v4.json_tree_element_match(profile, $1::eql_v4.query_json_elements)`, q)
+```
+
+Each comparison is of the member's ordering term, so `MemberEq` compares
+text case- and accent-folded, and integers past 2^53 can tie; `EqAt` is
+exact. A member the field does not order is refused before the engine is
+asked, naming it. A row whose array is longer than
+`JSONReadOptions.WildcardBound` (64 by default) fails the query with
+SQLSTATE 22023.
+
 ## Building
 
 The package embeds `wasm/stack_encrypt_guest.wasm`, a build artefact of the

languages/golang/encrypt/checker.go

@@ -258,7 +258,7 @@ func parseOperators(value any) ([]record.Operator, error) {
 				op.Index, err = parseOperatorIndex(s)
 			case "input":
 				switch s {
-				case "value", "pair", "json", "path_json", "path_pair", "path", "path_bound":
+				case "value", "pair", "json", "path_json", "path_pair", "path", "path_bound", "path_predicates":
 					op.Input = s
 				default:
 					return nil, fmt.Errorf("operator %d: unknown input %q", i, s)
@@ -282,14 +282,14 @@ func parseOperators(value any) ([]record.Operator, error) {
 // of the names the engine's OpSpec::parse accepts, the scalar operators
 // (eq, lt, lte, gt, gte, between, matches) and a document's (contains,
 // contained_by, eq_at, the orderings at a path, between_at, selector,
-// subtree, children, child_count). The catalog (se_targets) and se_query_term are
+// subtree, children, child_count, element_match). The catalog (se_targets) and se_query_term are
 // read through this, parseFidelity and parseOperatorIndex, so the two
 // readers refuse the same values.
 func parseOperatorName(s string) (string, error) {
 	switch s {
 	case "eq", "lt", "lte", "gt", "gte", "between", "matches",
 		"contains", "contained_by", "eq_at", "lt_at", "lte_at", "gt_at", "gte_at", "between_at",
-		"selector", "subtree", "children", "child_count":
+		"selector", "subtree", "children", "child_count", "element_match":
 		return s, nil
 	}
 	return "", fmt.Errorf("unknown operator %q", s)

languages/golang/encrypt/doc.go

@@ -242,6 +242,22 @@
 // A selection or a subtree read opens values as JSON text ([JSONValue],
 // [JSONRead.Open]). [DecodeJSON] reads one into a Go type.
 //
+// Two queries at an array's paths can each be answered by a different
+// element. To test several members of one element together, use Match on
+// the array's accessor under Elements, which a field has for each array
+// whose elements have an ordered member:
+//
+//	q, err := customers.Fields.Profile.Elements.Visits.Match(ctx, cipher, encrypt.ElementPredicates{
+//		encrypt.MemberEq("place", "home"),
+//		encrypt.MemberGt("at", 15),
+//	}, encrypt.JSONReadOptions{})
+//	// WHERE eql_v4.json_tree_element_match(profile, $1::eql_v4.query_json_elements)
+//
+// Each comparison is of the member's ordering term, so [MemberEq] compares
+// text case- and accent-folded, and integers past 2^53 can tie. A row whose
+// array is longer than [JSONReadOptions.WildcardBound] (64 by default)
+// fails the query with SQLSTATE 22023.
+//
 // # Cancellation
 //
 // Every call takes a [context.Context], and a call whose context is

languages/golang/encrypt/example/json/README.md

@@ -1,8 +1,9 @@
 # Typed encrypted JSON document example
 
 This program stores a Go `Profile` struct as one encrypted EQL v4 JSON column.
-It finds a row by containment, compares an ordered path, decrypts a whole
-profile, and opens only the visit times from another query.
+It finds a row by containment, compares an ordered path, finds a row where one
+visit passes two tests, decrypts a whole profile, and opens only the visit
+times from another query.
 
 ## Run
 
@@ -35,8 +36,25 @@ its `stash` tags. After changing the struct or tags, run this from
 go generate ./encrypt/example/json
 ```
 
-`Profile` uses `encoding/json` for the document shape. The `ordered=$.age`
-tag adds an ordering term for that path. A containment needle made from a
+`Profile` uses `encoding/json` for the document shape. The
+`ordered=$.age;$.visits[*].at;$.visits[*].place` tag adds an ordering term
+for each of those paths. Because `at` and `place` are ordered members of
+each visit, the generated `Fields.Profile.Elements.Visits.Match` tests both
+on one visit:
+
+```go
+q, err := Fields.Profile.Elements.Visits.Match(ctx, cipher, encrypt.ElementPredicates{
+	encrypt.MemberEq("place", "home"),
+	encrypt.MemberGt("at", 15),
+}, encrypt.JSONReadOptions{})
+// SELECT id FROM customers WHERE eql_v4.json_tree_element_match(profile, $1::eql_v4.query_json_elements)
+```
+
+Ada has a visit at home and a visit after 15, but no one visit that is both,
+so only Bea's row matches. Each comparison is of the ordering term, so text
+compares case- and accent-folded. A row with more visits than
+`JSONReadOptions.WildcardBound` (64 by default) fails the query with SQLSTATE
+22023. A containment needle made from a
 struct includes every field without `omitempty`, including zero values;
 `main.go` uses a map to name only the member being queried. The prepared
 subtree read returns JSON text, which `encrypt.DecodeJSON[int64]` decodes.