diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index e382bac..1ae2dac 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -16,9 +16,9 @@ In `packages/core/src`, a module uses another only through its `index.ts`, never - `parser/`: `check()`, `roles()`, and the Expected set at a position, which completion uses. - `highlight/`: `highlight()`. - `format/`: `format()`. -- `complete/`: `complete()`, `describe()`, `engineArgument()` and `functionArgument()`. +- `complete/`: `complete()`, `describe()`, `engineArgument()`, `functionArgument()`, `parameterHints()` and `keywordDocumentation()`. - `options/`: `checkOptions()`, and the settings, codecs and other options that completion offers. -- `names/`: `definition()`, `references()`, `prepareRename()` and `rename()`. +- `names/`: `definition()`, `references()`, `prepareRename()` and `rename()`, and `objects()`, `objectReferences()` and `objectDefinitions()` across files. - `structure/`: `outline()`, `foldingRanges()` and `selectionRanges()`. - `generated/`: lists from ClickHouse: keywords, data types, privileges, and the functions, settings, table engines, formats, table functions, combinators and time zones that completion offers. @@ -77,21 +77,43 @@ A layer apart from `check()`, which it never changes: it looks only at the state `backup`, ...); names analysis skips these marks. `options/catalog.ts` gives the settings each place takes, and ClickHouse's message for a name it does not take. 2. Each `name = value` is checked by name, and its literal by the conversions of `src/Core/SettingsFields.cpp` - (`options/values.ts`): numbers in range, strings that parse, booleans, enums and their lists, one character, maps - and time zones. DEFAULT, query parameters, `param_` and `custom_` names are left alone. + (`options/values.ts`): numbers in range, strings that parse, booleans, enums and their lists, one character, maps, + time zones, URIs and the version of `compatibility`; a map given to another type gets ClickHouse's message for it. + DEFAULT, query parameters, `param_` and `custom_` names are left alone. 3. The roles find `CODEC(...)`, `INDEX ... TYPE`, `STATISTICS`, `FORMAT` and `COMPRESSION`. A parameter of a codec or index type is checked against the values that ClickHouse was asked about: a number between two probes with the same - verdict gets that verdict, any other is left alone. -4. An error is what ClickHouse refuses before the statement runs, whatever its server. What a setting of the server or + verdict gets that verdict, any other is left alone. The `name = value` arguments of the `text` index are checked by + name. A codec is checked against the types that the column's type holds, through Nullable, LowCardinality, Array, + Map and Tuple, when each is a type that ClickHouse was asked about with that codec; a column without a type written + is left alone. The `LEVEL` of `INTO OUTFILE` is checked against the range of its compression method. +4. After `SET default_table_engine = 'None'`, a `CREATE TABLE` without `ENGINE` that does not take one from `AS` + another table is an error, as ClickHouse refuses it with ENGINE_REQUIRED. +5. An error is what ClickHouse refuses before the statement runs, whatever its server. What a setting of the server or session can allow (`allow_suspicious_codecs`, the settings of experimental codecs) and what fails only as the statement runs (a compression method) is a warning. `ATTACH` is not checked, as ClickHouse does not check it again. +## Quick fixes + +A diagnostic that knows the answer carries `fixes`, each a title and the edits that apply it (`options/fixes.ts`): + +- The names within a few typos of an unknown setting, codec, index or statistics type, format, compression method, + enum value or test hint error name, found as ClickHouse's `NamePrompter` finds them, closest first. A codec or index + type must take the parameters written, and a setting the value written; for settings only the closest are kept. +- The keyword of check()'s "Maybe you meant", in the case of the word, kept only when the statement then passes check(). +- A number in quotes for a setting of one character, and `ENGINE = MergeTree`, with `ORDER BY tuple()` when the + table has no ORDER BY or PRIMARY KEY, for ENGINE_REQUIRED. +- `messageFixes()` reads the names of "Maybe you meant" in an error of the server for the extension's server check. + +The fix that is closer than all the others is the preferred one. `npm run oracle:fixes` applies every fix to +generated statements and to the sample of the oracle and runs the results in `clickhouse local`. + The catalog comes from ClickHouse: `scripts/generate-catalog.ts` asks `system.settings` and the enum values of each setting (from ClickHouse's "Must be one of" for a value no enum has), and `scripts/generate-options.ts` reads `system.merge_tree_settings`, `system.codecs`, `system.data_skipping_index_types`, `system.database_engines`, the -lists of settings that the sources declare for the other engines, BACKUP and RESTORE, and the docs of EXPLAIN. It -asks a binary, in `clickhouse local`, which names each engine takes, which values each enum, codec parameter and index -parameter takes, and the messages for the others, so a new release updates all of it. +lists of settings that the sources declare for the other engines, BACKUP and RESTORE, the argument names of the text +index, and the docs of EXPLAIN. It asks a binary, in `clickhouse local`, which names each engine takes, which values +each enum, codec parameter and index parameter takes, which data types each codec refuses, which levels each +compression method takes, and the messages for the others, so a new release updates all of it. ## roles() and highlight() @@ -162,6 +184,18 @@ In the brackets of a function, the suggestions for the argument at the cursor co the statement makes and of the tables it reads in the schema follow, those whose type fits the documented types of the argument first. +`parameterHints()` names the argument of each call with two arguments or more, in the statements that parse and where +the parser reads the name as a function or an engine, by the same reading as `functionArgument()` and +`engineArgument()` at the start of the argument. An argument that is a name equal to the parameter, one that the +syntax leaves open and the arguments of a repeated parameter get no name. + +`keywordDocumentation()` gives the section of ClickHouse's documentation of the SYSTEM command at an offset, by the +longest command that the words after SYSTEM start with, and in `CREATE` and `ALTER` of a user, role, quota, row policy +or settings profile, of the clause whose keywords are there, or the statement's syntax for another keyword. +`scripts/generate-statements.ts` reads them from `docs/reference/statements`: the commands from the headings of +`system.mdx` and the forms its syntax adds, the clauses from the sections named after them and the paragraphs that +start with them. + In a list of settings, `options/complete.ts` reads from the words before it which place it is (a query, `SET`, the engine of `ENGINE = Name ... SETTINGS`, a database, a column, `MODIFY SETTING`, `EXPLAIN kind`, `BACKUP`, ...) and offers that place's settings, its own before those of queries, and after `name =` the values of an enum or a boolean. @@ -229,6 +263,8 @@ In `packages/vscode/src`: - `selection.ts`: Expand and Shrink Selection from `selectionRanges()`, with the `ParseCache` of the document. - `format.ts`: Format Document and Format Selection through `format()`, with the editor's indentation. Format Selection matches the units of `split()` in the text and in its formatted copy, and replaces only the ones the selection touches. +- `inlay.ts`: inlay hints, off by default, from `parameterHints()` for the statements on the screen, and the type of + each `param_name` on it from the `{name:Type}` of the document. - `complete.ts`, `hover.ts`, `signature.ts`: completion, hover and signature help through `complete()`, `describe()` and, in the brackets of an engine or a function, `engineArgument()` or `functionArgument()`; hover takes the role of the word from the parser where completion does not tell what it is. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9fb3ac6..9379c71 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -36,6 +36,7 @@ npm run oracle:format -- --clickhouse --corpus --corpus /tests/queries/0_stateless npm run oracle:arguments -- --clickhouse --corpus /tests/queries/0_stateless npm run oracle:options -- --clickhouse --corpus /tests/queries/0_stateless +npm run oracle:fixes -- --clickhouse ``` The second command labels every statement of the tests with that binary and compares `check()` with it. The third @@ -44,7 +45,9 @@ whole, as LF and as CRLF text. The fourth renames the names of each SELECT that and checks that the renamed query gives the same result. The fifth checks what completion offers in the arguments of functions against the arguments that the tests write, and runs each suggestion in a call. The sixth runs each statement of the tests that `checkOptions()` flags in `clickhouse local`, compares its checks of settings, codecs and index types with ClickHouse on generated values, and -runs every option that completion offers. `npm test` fails when the sample's rates fall below +runs every option that completion offers. The seventh applies every quick fix to statements with typos of the +names of the catalogs, wrong values, CREATE TABLE without ENGINE, test hints with unknown error names and the +invalid statements of the sample, and counts the fixed statements that ClickHouse takes. `npm test` fails when the sample's rates fall below `oracle/thresholds.json`; raise them when `check()` gets better. [oracle/README.md](oracle/README.md) describes the sample and how to rebuild it. @@ -56,6 +59,8 @@ The lists that come from ClickHouse are generated; do not edit them by hand. - `npx tsx scripts/generate-catalog.ts --clickhouse `: the functions, settings, table engines, formats, table functions, combinators and time zones that completion offers. Signatures missing from the server metadata are maintained in `scripts/catalog-signatures.ts`. +- `npx tsx scripts/generate-statements.ts --docs /docs/reference`: the documentation of SYSTEM commands + and of the clauses of the access statements that hover shows, in `packages/core/src/generated/statements.ts`. - `npx tsx scripts/generate-errors.ts --clickhouse ` in `packages/vscode`: the error codes of test hints in `packages/vscode/src/generated/errors.ts`. - `npx tsx scripts/generate-readme.ts` in `packages/vscode`: the settings and commands of its README, from its diff --git a/README.md b/README.md index c80c218..3270f1f 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,7 @@ A VS Code extension and a TypeScript library that read [ClickHouse](https://gith - **Highlighting**: keywords, functions, types and names get their colours from how the parser reads them. - **Error checking**: the syntax errors that ClickHouse would report, shown as you type. - **Options**: settings, engine and database settings, codecs, index and statistics types, formats and the settings of EXPLAIN and BACKUP completed and documented, and their names and values checked as ClickHouse checks them. +- **Quick fixes**: the name or keyword that a typo is closest to, the server's "Maybe you meant" hints, and a missing `ENGINE` where ClickHouse requires one. - **Formatting**: queries laid out as `clickhouse format` does. - **Autocomplete**: keywords, functions, types, settings, engines and formats that fit where you type. - **Engine arguments**: in `ENGINE = Name(...)`, what the argument at the cursor takes, and the engine's parameters as you type. @@ -12,7 +13,10 @@ A VS Code extension and a TypeScript library that read [ClickHouse](https://gith - **Outline**: each statement with its CTEs and columns, in the Outline view and breadcrumbs. - **Folding**: statements, subqueries and comments of several lines, and `-- #region` blocks. - **Expand Selection**: grows from a word to its expression, call, clause, subquery and statement. +- **Hover**: ClickHouse's documentation of functions, settings, engines, formats, SYSTEM commands and the clauses of the statements that make and change users, roles, quotas, row policies and settings profiles. +- **Inlay hints** (optional): the names of the parameters before the arguments of calls, such as `dateDiff(unit: 'day', ...)`. - **Go to definition and rename**: aliases, CTEs, table aliases and lambda and query parameters, scoped as ClickHouse scopes them. +- **Workspace symbols**: the tables, views, dictionaries, functions and databases that the CREATE statements of the folder make, and go to definition and references of a table from the queries of any file. - **Server check** (optional): your own ClickHouse server checks the names, functions and types in your queries. ## Use it in VS Code @@ -29,7 +33,7 @@ To build it yourself instead (needs Node.js 22 or newer): git clone https://github.com/valerypetrov/clickhouse-sql-syntax.git cd clickhouse-sql-syntax npm install && npm run build && npm run package:vscode -code --install-extension packages/vscode/clickhouse-sql-0.4.0.vsix +code --install-extension packages/vscode/clickhouse-sql-0.5.0.vsix ``` ## Use the library diff --git a/oracle/package.json b/oracle/package.json index e7211e2..2ae7be0 100644 --- a/oracle/package.json +++ b/oracle/package.json @@ -12,10 +12,11 @@ "oracle:rename": "tsx src/rename-oracle.ts", "oracle:arguments": "tsx src/arguments-oracle.ts", "oracle:options": "tsx src/options-oracle.ts", + "oracle:fixes": "tsx src/fixes-oracle.ts", "typecheck": "tsc -p tsconfig.json", "test": "npm run typecheck && tsx --test \"test/**/*.test.ts\"" }, "dependencies": { - "clickhouse-sql-syntax": "0.4.0" + "clickhouse-sql-syntax": "0.5.0" } } diff --git a/oracle/src/fixes-oracle.ts b/oracle/src/fixes-oracle.ts new file mode 100644 index 0000000..20914a7 --- /dev/null +++ b/oracle/src/fixes-oracle.ts @@ -0,0 +1,241 @@ +// Applies the quick fixes of check(), checkOptions(), test hints and ClickHouse's own hints to erroring statements and +// runs the results in clickhouse local, as `npm run oracle:fixes -- --help` describes. +import { spawnSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { parseArgs } from "node:util"; + +import { check, checkOptions, type Diagnostic, type Fix, messageFixes, nameHints, replaceFixes } from "clickhouse-sql-syntax"; + +import { CODECS, INDEX_TYPES, STATISTICS_TYPES } from "../../packages/core/src/generated/options.js"; +import { FORMATS } from "../../packages/core/src/generated/formats.js"; +import { FUNCTIONS } from "../../packages/core/src/generated/functions.js"; +import { MERGE_TREE_SETTINGS } from "../../packages/core/src/generated/merge-tree-settings.js"; +import { SETTINGS } from "../../packages/core/src/generated/settings.js"; +import { type Outcome, probe } from "../../scripts/probe.js"; +import { readCorpus } from "./corpus.js"; + +const USAGE = `Usage: npm run oracle:fixes -w oracle -- --clickhouse [--report ] + +Makes erroring statements (typos of the names in the catalogs, wrong values, CREATE TABLE without ENGINE, test hints +with unknown error names, and the invalid statements of fixtures/corpus.jsonl.gz), applies every quick fix offered for +them, and runs each result in clickhouse local. It prints how many fixed statements ClickHouse accepts, by kind of fix.`; + +const { values: args } = parseArgs({ + options: { + clickhouse: { type: "string" }, + report: { type: "string", default: new URL("../out/fixes-report.json", import.meta.url).pathname }, + help: { type: "boolean", default: false }, + }, +}); +if (args.help || !args.clickhouse) { + console.log(USAGE); + process.exit(args.help ? 0 : 2); +} +const clickhouse = args.clickhouse; + +function apply(sql: string, fix: Fix): string { + let text = sql; + for (const edit of [...fix.edits].sort((a, b) => b.start - a.start)) text = text.slice(0, edit.start) + edit.newText + text.slice(edit.end); + return text; +} + +/** A typo of a name: a letter dropped from its middle, or two swapped in a short one; deterministic. */ +function typo(name: string, i: number): string { + if (name.length > 5) { + const at = 1 + (i % (name.length - 2)); + return name.slice(0, at) + name.slice(at + 1); + } + return name.length > 2 ? name[0] + name[2] + name[1] + name.slice(3) : `${name}q`; +} + +/** One erroring statement with its fixes, and what ClickHouse said about each fixed text. */ +interface Case { + kind: string; + sql: string; + setup?: string[]; + /** For the statements of the sample, the word that the fixes replace. */ + word?: string; + fixes: { text: string; preferred: boolean; outcome?: Outcome }[]; +} + +const cases: Case[] = []; +const id = () => Math.floor(Math.random() * 1e12); +const quote = (s: string) => `'${s.replace(/\\/g, "\\\\").replace(/'/g, "\\'")}'`; + +/** The cases of the statements that ClickHouse runs before a typo, from the diagnostics of the typo that have fixes. */ +function fromTypos(kind: string, pairs: { valid: string; broken: string }[], setup?: string[], diagnose = (sql: string) => checkOptions(sql), chunk = 2000): void { + const controls = probe(clickhouse, pairs.map((p) => p.valid), { setup, chunk }); + let skipped = 0; + pairs.forEach((pair, i) => { + if (!controls[i]!.ok) return void skipped++; + const fixes = diagnose(pair.broken).flatMap((d: Diagnostic) => d.fixes ?? []); + cases.push({ kind, sql: pair.broken, setup, fixes: fixes.map((f) => ({ text: apply(pair.broken, f), preferred: !!f.preferred })) }); + }); + console.log(`${kind}: ${pairs.length - skipped} of ${pairs.length} statements run before the typo`); +} + +const literal = (value: string | undefined) => quote(value ?? ""); + +// Settings of queries and of MergeTree, by their default values. +const settings = SETTINGS.filter((s) => !s.obsolete && !s.aliasTo); +fromTypos("query setting", settings.map((s, i) => ({ valid: `SELECT 1 SETTINGS ${s.name} = ${literal(s.default)} FORMAT Null`, broken: `SELECT 1 SETTINGS ${typo(s.name, i)} = ${literal(s.default)} FORMAT Null` }))); +fromTypos("MergeTree setting", MERGE_TREE_SETTINGS.filter((s) => !s.obsolete).map((s, i) => { + const make = (name: string) => `CREATE TABLE t_${id()} (x UInt64) ENGINE = MergeTree ORDER BY x SETTINGS ${name} = ${literal(s.default)}`; + return { valid: make(s.name), broken: make(typo(s.name, i)) }; +})); + +// Codecs, index types and statistics. +const codecs = CODECS.filter((c) => !c.refused && !c.encryption && !c.setting); +fromTypos("codec", codecs.flatMap((c, i) => ["UInt64", "Float64", "String", "DateTime"].flatMap((type) => { + const make = (name: string) => `CREATE TABLE t_${id()} (x ${type} CODEC(${name}${c.rule?.max ? "(1)" : ""})) ENGINE = MergeTree ORDER BY tuple()`; + return [0, 1].map((n) => ({ valid: make(c.name), broken: make(typo(c.name, i + n)) })); +})), ["SET allow_suspicious_codecs = 1"]); +const indexes = ["minmax", "set(100)", "bloom_filter(0.01)", "ngrambf_v1(3, 256, 2, 0)", "tokenbf_v1(256, 2, 0)", "bloom_filter", "set(0)"]; +fromTypos("index type", indexes.flatMap((call, i) => { + const name = /^\w+/.exec(call)![0]; + const make = (type: string) => `CREATE TABLE t_${id()} (s String, INDEX i s TYPE ${type}${call.slice(name.length)}) ENGINE = MergeTree ORDER BY tuple()`; + return [0, 1, 2].map((n) => ({ valid: make(name), broken: make(typo(name, i + n)) })); +})); +fromTypos("statistics type", STATISTICS_TYPES.flatMap((t, i) => { + const make = (name: string) => `CREATE TABLE t_${id()} (x UInt64 STATISTICS(${name})) ENGINE = MergeTree ORDER BY tuple()`; + return [0, 1].map((n) => ({ valid: make(t.name), broken: make(typo(t.name, i + n)) })); +}), ["SET allow_experimental_statistics = 1"]); + +// Output formats and compression methods. +const outputs = FORMATS.filter((f) => f.output); +fromTypos("format", outputs.map((f, i) => ({ valid: `SELECT 1 AS x FORMAT ${f.name}`, broken: `SELECT 1 AS x FORMAT ${typo(f.name, i)}` })), undefined, undefined, 1); +const out = mkdtempSync(join(tmpdir(), "fixes-oracle-")); +fromTypos("compression method", ["gzip", "deflate", "brotli", "zstd", "lz4", "bz2", "snappy", "lzma", "auto", "none"].map((m, i) => { + const file = join(out, `f_${i}`); + return { valid: `SELECT 1 INTO OUTFILE '${file}' TRUNCATE COMPRESSION '${m}'`, broken: `SELECT 1 INTO OUTFILE '${file}' TRUNCATE COMPRESSION '${typo(m, i)}'` }; +})); + +// Values: typos of enum values, and numbers for settings that take strings. +const enums = settings.filter((s) => s.values?.length); +fromTypos("enum value", enums.flatMap((s, i) => s.values!.filter((v) => /^\w{3,}$/.test(v)).slice(0, 3).map((v, n) => ({ valid: `SET ${s.name} = '${v}'`, broken: `SET ${s.name} = '${typo(v, i + n)}'` })))); +fromTypos("quoted value", settings.flatMap((s) => ["1", "0", "-1", "1.5"].map((v) => ({ valid: `SELECT 1`, broken: `SELECT 1 SETTINGS ${s.name} = ${v} FORMAT Null` })))); + +// CREATE TABLE without ENGINE after SET default_table_engine = 'None'. +const noEngine = ["SET default_table_engine = 'None'"]; +fromTypos("missing ENGINE", [ + "CREATE TABLE t_{} (x UInt64)", "CREATE TABLE t_{} (x UInt64) ORDER BY x", "CREATE TABLE t_{} (x UInt64) PRIMARY KEY x", "CREATE TABLE IF NOT EXISTS t_{} (x UInt64) PARTITION BY x", + "CREATE OR REPLACE TABLE t_{} (x UInt64) COMMENT 'c'", "CREATE TABLE t_{} (x UInt64) SETTINGS index_granularity = 128", "CREATE TABLE t_{} AS SELECT 1 AS x", + "CREATE TABLE t_{} ORDER BY x AS SELECT 1 AS x", "CREATE TABLE t_{} EMPTY AS SELECT 1 AS x ORDER BY x", "CREATE TABLE t_{} (x UInt64, INDEX i x TYPE minmax) TTL toDate(x) + 1", + "CREATE TABLE t_{} (x UInt64) ORDER BY x SAMPLE BY x", "CREATE TABLE t_{} (x UInt64, y UInt64) PRIMARY KEY x ORDER BY (x, y)", +].map((sql) => ({ valid: "SELECT 1", broken: sql.replace("{}", String(id())) })), noEngine, (sql) => checkOptions(`${noEngine[0]};\n${sql}`).map((d) => ({ + ...d, fixes: d.fixes?.map((f) => ({ ...f, edits: f.edits.map((e) => ({ ...e, start: e.start - noEngine[0]!.length - 2, end: e.end - noEngine[0]!.length - 2 })) })), +}))); + +// Keywords: the invalid statements of the stored sample, whose fixes must parse. +const invalid = readCorpus(new URL("../fixtures/corpus.jsonl.gz", import.meta.url).pathname).entries.filter((e) => !e.valid); +let keywordCases = 0; +for (const entry of invalid) { + const fixes = check(entry.sql).flatMap((d) => d.fixes ?? []); + if (!fixes.length) continue; + keywordCases++; + cases.push({ kind: "keyword", sql: entry.sql, fixes: fixes.map((f) => ({ text: apply(entry.sql, f), preferred: !!f.preferred })) }); +} +console.log(`keyword: ${keywordCases} of ${invalid.length} invalid statements of the sample have a fix`); + +// The statements of the sample that parse and that checkOptions() fixes; they run without their tables, so a fix is +// taken when ClickHouse's error no longer names the word it replaced. +for (const e of readCorpus(new URL("../fixtures/corpus.jsonl.gz", import.meta.url).pathname).entries.filter((e) => e.valid)) { + const fixes = checkOptions(e.sql).flatMap((d) => d.fixes ?? []); + if (!fixes.length) continue; + const edit = fixes[0]!.edits[0]!; + cases.push({ kind: "sample option", sql: e.sql, word: e.sql.slice(edit.start, edit.end), fixes: fixes.map((f) => ({ text: apply(e.sql, f), preferred: !!f.preferred })) }); +} + +// ---------------------------------------------------------------- Running the fixed statements + +const parsing = cases.filter((c) => c.kind === "keyword"); +const running = cases.filter((c) => c.kind !== "keyword"); +let fresh = 0; +const runAll = (list: Case[], wrap: (text: string) => string, chunk = 500) => { + const bySetup = new Map(); + for (const c of list) bySetup.set(JSON.stringify(c.setup ?? []), [...(bySetup.get(JSON.stringify(c.setup ?? [])) ?? []), c]); + for (const [setup, group] of bySetup) { + const fixes = group.flatMap((c) => c.fixes); + // Each fix of a CREATE makes a table of its own. + const outcomes = probe(clickhouse, fixes.map((f) => wrap(f.text.replace(/\bt_\d+/, () => `f_${fresh++}`))), { setup: JSON.parse(setup) as string[], chunk }); + fixes.forEach((f, i) => (f.outcome = outcomes[i])); + } +}; +// The parser of ClickHouse, without running anything. +runAll(parsing, (text) => `SELECT throwIf(formatQueryOrNull(${quote(text)}) IS NULL, 'does not parse') FORMAT Null`); +runAll(running.filter((c) => c.kind !== "format"), (text) => text); +// Binary formats can break the markers between the outputs, so each runs alone. +runAll(running.filter((c) => c.kind === "format"), (text) => text, 1); + +// Test hints with a typo of an error name, fixed as the extension's hints.ts does, each file run as clickhouse-client reads its hints. +const errorNames = [...readFileSync(new URL("../../packages/vscode/src/generated/errors.ts", import.meta.url), "utf8").matchAll(/^ {2}(\w+): \d+,$/gm)].map((m) => m[1]!); +const names = errorNames.filter((_, i) => i % 10 === 0); +const dir = mkdtempSync(join(tmpdir(), "fixes-hints-")); +for (const [i, name] of names.entries()) { + const wrong = typo(name, i); + const text = `SELECT throwIf(1); -- { serverError ${wrong} }\nSELECT 'done';\n`; + const at = text.indexOf(wrong); + const fixes = errorNames.includes(wrong) ? [] : replaceFixes(at, at + wrong.length, nameHints(wrong, errorNames)); + const c: Case = { kind: "test hint", sql: text, fixes: fixes.map((f) => ({ text: apply(text, f), preferred: !!f.preferred })) }; + for (const f of c.fixes) { + writeFileSync(join(dir, "q.sql"), f.text); + const run = spawnSync(clickhouse, ["local", "--queries-file", join(dir, "q.sql")], { encoding: "utf8", timeout: 60_000, killSignal: "SIGKILL" }); + const output = `${run.stdout}${run.stderr}`; + f.outcome = /No error code with name/.test(output) || run.signal ? { ok: false, message: output.trim().slice(0, 300) } : { ok: true }; + } + cases.push(c); +} +rmSync(dir, { recursive: true, force: true }); + +// ClickHouse's own hints in its errors: misspelled functions, types, table functions, databases and tables. +const functions = FUNCTIONS.filter((f) => !f.aggregate && /^[a-z]\w{4,}$/i.test(f.name)).filter((_, i) => i % 15 === 0); +const named: { kind: string; sql: string; word: string }[] = [ + ...functions.map((f, i) => ({ kind: "server hint: function", word: typo(f.name, i), sql: `SELECT ${typo(f.name, i)}(1)` })), + ...["String", "UInt64", "DateTime", "Decimal", "Float32", "LowCardinality"].map((t, i) => ({ kind: "server hint: data type", word: typo(t, i), sql: `SELECT CAST('1' AS ${typo(t, i)})` })), + ...["numbers", "zeros", "generateRandom", "values"].map((t, i) => ({ kind: "server hint: table function", word: typo(t, i), sql: `SELECT * FROM ${typo(t, i)}(1) LIMIT 1` })), + { kind: "server hint: database", word: "sytem", sql: "SELECT * FROM sytem.one" }, + { kind: "server hint: table", word: "system.numbrs", sql: "SELECT * FROM system.numbrs LIMIT 1" }, + { kind: "server hint: table", word: "system.setings", sql: "SELECT * FROM system.setings LIMIT 1" }, +]; +const first = probe(clickhouse, named.map((n) => n.sql)); +const hinted: Case[] = []; +named.forEach((n, i) => { + const at = n.sql.indexOf(n.word); + const fixes = messageFixes(n.sql, at, at + n.word.length, first[i]!.message ?? ""); + if (fixes.length) hinted.push({ kind: n.kind, sql: n.sql, fixes: fixes.map((f) => ({ text: apply(n.sql, f), preferred: !!f.preferred })) }); +}); +console.log(`server hints: ${hinted.length} of ${named.length} statements have hints in ClickHouse's error`); +runAll(hinted, (text) => text); +cases.push(...hinted); + +// ---------------------------------------------------------------- The numbers + +/** Whether ClickHouse took the fix: it ran, or for ClickHouse's hints, its error is no longer about an unknown name. */ +const UNKNOWN_NAME = new Set([46, 50, 60, 81]); +const accepted = (c: Case, outcome: Outcome) => + outcome.ok || (c.kind.startsWith("server hint") && !UNKNOWN_NAME.has(outcome.code ?? 0)) || (c.word !== undefined && !outcome.message?.includes(c.word)); +const kinds = [...new Set(cases.map((c) => c.kind))]; +const summary = kinds.map((kind) => { + const of = cases.filter((c) => c.kind === kind); + const fixes = of.flatMap((c) => c.fixes.map((f) => ({ c, f }))); + const preferred = fixes.filter(({ f }) => f.preferred); + return { + kind, + statements: of.length, + withFixes: of.filter((c) => c.fixes.length).length, + fixes: fixes.length, + accepted: fixes.filter(({ c, f }) => accepted(c, f.outcome!)).length, + preferred: preferred.length, + preferredAccepted: preferred.filter(({ c, f }) => accepted(c, f.outcome!)).length, + refused: fixes.filter(({ c, f }) => !accepted(c, f.outcome!)).slice(0, 20).map(({ f }) => `${f.text.slice(0, 200)} -- ${f.outcome!.message?.slice(0, 200)}`), + }; +}); +for (const s of summary) { + console.log(`${s.kind}: ${s.accepted} of ${s.fixes} fixes accepted (${s.preferredAccepted} of ${s.preferred} preferred), ${s.withFixes} of ${s.statements} statements with fixes`); + for (const r of s.refused.slice(0, 8)) console.log(` refused: ${r}`); +} +rmSync(out, { recursive: true, force: true }); +mkdirSync(dirname(args.report!), { recursive: true }); +writeFileSync(args.report!, JSON.stringify({ summary, cases }, null, 1)); diff --git a/oracle/src/options-oracle.ts b/oracle/src/options-oracle.ts index 7eeb8e2..05b3b18 100644 --- a/oracle/src/options-oracle.ts +++ b/oracle/src/options-oracle.ts @@ -1,6 +1,7 @@ // Compares checkOptions() and the option suggestions of complete() with a ClickHouse binary, // as `npm run oracle:options -- --help` describes. -import { mkdirSync, readdirSync, readFileSync, writeFileSync } from "node:fs"; +import { mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import { parseArgs } from "node:util"; @@ -63,8 +64,9 @@ function corpus(): void { if (!own.length) continue; const lineEnd = source.indexOf("\n", unit.statement!.end); const after = source.slice(unit.statement!.end, lineEnd < 0 ? undefined : lineEnd); - const expected = /\b(serverError|clientError)\b/.test(after); const sql = source.slice(unit.statement!.start, unit.statement!.end).replace(/;\s*$/, ""); + // A hint inside the statement counts too, as clickhouse-client reads it there. + const expected = /\b(serverError|clientError)\b/.test(after) || /--\s*\{\s*(serverError|clientError)\b/.test(sql); flagged.push({ file, sql, diagnostics: own.map((d) => `${d.severity}: ${d.message}`), severity: own.some(isError) ? "error" : "warning", expected }); } } @@ -164,17 +166,31 @@ function values(): void { function codecs(): void { const codecNames = complete("CREATE TABLE t (x UInt64 CODEC(", 31).filter((c) => c.kind === "codec").map((c) => c.label); const params = ["", "(0)", "(1)", "(2)", "(3)", "(4)", "(5)", "(6)", "(8)", "(9)", "(12)", "(13)", "(15)", "(19)", "(22)", "(23)", "(28)", "(29)", "(-1)", "(1.5)", "('byte')", "('bit')", "('zz')", "(AUTO)", "(zz)", "(1, 1)", "(3, 10)", "(3, 9)", "(3, 31)", "(3, 32)", "(12, 4)", "(12, 5)", "(1, 2, 3)"]; - const types = ["UInt64", "Float64"]; + const types = ["UInt64", "Float64", "String", "LowCardinality(String)", "Nullable(Float32)", "Array(UInt128)", "Map(String, UInt64)", "Tuple(a Int32, b DateTime)"]; const codec = codecNames.flatMap((name) => types.flatMap((type) => params.map((p) => `CREATE TABLE p_${id()} (x ${type} CODEC(${name}${p}, LZ4)) ENGINE = Memory`))); const setup = ["SET allow_suspicious_codecs = 1", ...["enable_pco_codec", "enable_alp_codec", "enable_sz3_codec", "enable_zxc_codec", "enable_gdeflate_codec", "enable_quantized_codec"].map((s) => `SET ${s} = 1`)]; - const codecVerdict = (o: Outcome) => (o.ok ? false : /codec|Codec|Bad get|parameter/.test(o.message ?? "") && !/not applicable|is not supported for|cannot be applied/.test(o.message ?? "") ? true : undefined); + const codecVerdict = (o: Outcome) => (o.ok ? false : /codec|Codec|Bad get|parameter/.test(o.message ?? "") ? true : undefined); summarize("codecs", compare([...codec, `CREATE TABLE p_${id()} (x UInt64 CODEC(Zzz)) ENGINE = Memory`], codecVerdict, setup)); const indexes = complete("CREATE TABLE t (x String, INDEX i x TYPE ", 41).filter((c) => c.kind === "index type").map((c) => c.label); const indexParams = ["", "(0)", "(1)", "(0.5)", "(2)", "(100)", "(-1)", "(3, 256, 2, 0)", "(3, 256, 2)", "(256, 2, 0)", "(0, 2, 0)", "(3, 100, 256, 2, 0)", "('a')"]; - const index = [...indexes, "zz_type"].flatMap((name) => indexParams.map((p) => `CREATE TABLE p_${id()} (x String, INDEX i x TYPE ${name}${p} GRANULARITY 1) ENGINE = MergeTree ORDER BY tuple()`)); + const textParams = ["(tokenizer = splitByNonAlpha)", "(tokenizer = zz)", "(tokenizer = ngrams(3))", "(tokenizer = 'array', posting_list_block_size = 0)", "(tokenizer = array, zz = 1)", + "(tokenizer = array, tokenizer = array)", "(tokenizer = array, dictionary_block_size = 'a')", "(tokenizer = array, support_phrase_search = 2)", "(preprocessor = lower(x))"]; + const vectorParams = ["('hnsw', 'L2Distance', 3)", "('hnsw', 'L2Distance', 0)", "('hnsw', 'cosineDistance', 3, 'bf16', 16, 100)", "('hnsw', 'L2Distance', 3, 'bf16', 1, 100)", + "('hnsw', 'L2Distance', 3, 'bf16')", "('zz', 'L2Distance', 3)", "('hnsw', 'zz', 3)", "('hnsw', 'L2Distance', 3, 'zz', 16, 100)", "('hnsw', 'L2Distance', -1)"]; + const index = [ + ...[...indexes, "zz_type"].flatMap((name) => indexParams.map((p) => `CREATE TABLE p_${id()} (x String, INDEX i x TYPE ${name}${p} GRANULARITY 1) ENGINE = MergeTree ORDER BY tuple()`)), + ...textParams.map((p) => `CREATE TABLE p_${id()} (x String, INDEX i x TYPE text${p} GRANULARITY 1) ENGINE = MergeTree ORDER BY tuple()`), + ...[...indexParams, ...vectorParams].map((p) => `CREATE TABLE p_${id()} (x Array(Float32), INDEX i x TYPE vector_similarity${p} GRANULARITY 1) ENGINE = MergeTree ORDER BY tuple()`), + ]; summarize("index types", compare(index, (o) => (o.ok ? false : /index|Index|argument|parameter/.test(o.message ?? "") && !/does not support|cannot be used|data type/.test(o.message ?? "") ? true : undefined))); const statistics = [...complete("CREATE TABLE t (x UInt64 STATISTICS(", 36).filter((c) => c.kind === "statistics type").map((c) => c.label), "zz", "TDigest"]; summarize("statistics types", compare(statistics.map((s) => `CREATE TABLE p_${id()} (x UInt64 STATISTICS(${s})) ENGINE = MergeTree ORDER BY x`), (o) => (o.ok ? false : /Unknown statistics/.test(o.message ?? "") ? true : undefined))); + // The levels of INTO OUTFILE, into a file of a folder of its own. + const folder = mkdtempSync(join(tmpdir(), "options-oracle-")); + const levels = ["none", "gzip", "zstd", "lz4", "br", "xz", "bz2", "snappy", "deflate", "gz"].flatMap((m) => ["0", "1", "9", "10", "12", "13", "22", "23", "-1", "1.5", "0x05"].map((l) => + `SELECT 1 INTO OUTFILE '${join(folder, "f")}' TRUNCATE COMPRESSION '${m}' LEVEL ${l}`)); + summarize("compression levels", compare(levels, (o) => (o.ok ? false : /compression level/.test(o.message ?? "") ? true : undefined))); + rmSync(folder, { recursive: true, force: true }); const formats = ["JSON", "json", "TSV", "Zzz", "Null", "CSVWithNames", "pretty"]; summarize("formats", compare(formats.map((f) => `SELECT 1 FORMAT ${f}`), (o) => (o.ok ? false : /Unknown format/.test(o.message ?? "") ? true : undefined))); } diff --git a/package-lock.json b/package-lock.json index 1e62d02..e2547c8 100644 --- a/package-lock.json +++ b/package-lock.json @@ -3513,12 +3513,12 @@ "version": "0.0.0", "license": "SEE LICENSE IN LICENSE", "dependencies": { - "clickhouse-sql-syntax": "0.4.0" + "clickhouse-sql-syntax": "0.5.0" } }, "packages/core": { "name": "clickhouse-sql-syntax", - "version": "0.4.0", + "version": "0.5.0", "license": "SEE LICENSE IN LICENSE", "engines": { "node": ">=22" @@ -3526,14 +3526,14 @@ }, "packages/vscode": { "name": "clickhouse-sql", - "version": "0.4.0", + "version": "0.5.0", "license": "SEE LICENSE IN LICENSE", "devDependencies": { "@types/mocha": "^10.0.10", "@types/vscode": "~1.90.0", "@vscode/test-electron": "^3.1.0", "@vscode/vsce": "^4.1.0", - "clickhouse-sql-syntax": "0.4.0", + "clickhouse-sql-syntax": "0.5.0", "esbuild": "^0.28.2", "mocha": "^12.0.3", "vscode-oniguruma": "^2.0.1", diff --git a/package.json b/package.json index 5d1d08f..7bc7afe 100644 --- a/package.json +++ b/package.json @@ -24,6 +24,7 @@ "oracle:rename": "npm run oracle:rename -w oracle --", "oracle:arguments": "npm run oracle:arguments -w oracle --", "oracle:options": "npm run oracle:options -w oracle --", + "oracle:fixes": "npm run oracle:fixes -w oracle --", "generate": "tsx scripts/generate-names.ts", "package:vscode": "npm run package -w packages/vscode", "test:vscode": "npm run test:e2e -w packages/vscode", diff --git a/packages/core/README.md b/packages/core/README.md index f75f9b7..31552bf 100644 --- a/packages/core/README.md +++ b/packages/core/README.md @@ -143,6 +143,22 @@ the database may have a column of that name, there are no results, and `rename() the new name where ClickHouse would, and refuses one that would change what the query means. With `{ cache }`, the statements come from the `ParseCache` of `checkWithRoles()`. +### objects(sql), objectReferences(sql, options?) and objectDefinitions(reference, file, files) + +For tables, views and dictionaries across files: what the CREATE statements of a file make, where statements name a +table or a database, and which objects a name may mean. + +```ts +objects("USE db; CREATE TABLE t (x UInt8) ENGINE = Memory"); // [{ kind: "table", name: "t", database: "db", temporary: false, ... }] +objectReferences("SELECT * FROM db.t"); // [{ kind: "database", name: "db", ... }, { kind: "table", name: "t", database: "db", qualified: true, ... }] +``` + +`objects()` reads only the tokens, so it is fast enough for every file of a folder. `objectReferences()` takes the +names from the parse, as in `FROM db.t`, `INSERT INTO t` or `ALTER TABLE t`, leaves out a name that a CTE of the +statement takes, and gives a name without a database the one that an earlier `USE` sets. `objectDefinitions()` runs +each file as a session of its own: a temporary table first, then in the same file the last CREATE before the name, +and where the database is not known, every object of that name. + ### highlight(sql) and tokenize(sql) ```ts diff --git a/packages/core/package.json b/packages/core/package.json index d786a75..de17579 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -1,6 +1,6 @@ { "name": "clickhouse-sql-syntax", - "version": "0.4.0", + "version": "0.5.0", "description": "Tokenizer, syntax highlighter and syntax checker for ClickHouse SQL", "keywords": [ "clickhouse", diff --git a/packages/core/src/complete/hints.ts b/packages/core/src/complete/hints.ts new file mode 100644 index 0000000..4e24f5b --- /dev/null +++ b/packages/core/src/complete/hints.ts @@ -0,0 +1,79 @@ +/** + * parameterHints(): the documented parameter that each argument of a call is, for inlay hints. + */ +import { split } from "../format/index.js"; +import { checkWithRoles, type ParseCache } from "../parser/index.js"; +import { isSignificant, type ParameterHint, type Token, TokenKind as T } from "../types.js"; +import { engineCall } from "./engines.js"; +import { functionCall } from "./functions.js"; +import { nameOf } from "./schema.js"; + +const OPENING = new Set([T.OpeningRoundBracket, T.OpeningSquareBracket, T.OpeningCurlyBrace]); +const CLOSING = new Set([T.ClosingRoundBracket, T.ClosingSquareBracket, T.ClosingCurlyBrace]); + +/** The tokens of each argument at the top level of the brackets opened at `open`, as [start, end) indices. */ +function argumentSpans(tokens: Token[], open: number): [number, number][] { + const spans: [number, number][] = []; + let start = open + 1; + for (let i = open + 1, depth = 0; i < tokens.length; i++) { + const kind = tokens[i]!.kind; + if (OPENING.has(kind)) depth++; + else if (CLOSING.has(kind) && depth-- === 0) return [...spans, [start, i]]; + else if (kind === T.Comma && depth === 0) { + spans.push([start, i]); + start = i + 1; + } + } + return [...spans, [start, tokens.length]]; +} + +/** The index of the bracket that opens the one that closes at `close`. */ +function openingOf(tokens: Token[], close: number): number { + for (let i = close, depth = 0; i >= 0; i--) { + if (CLOSING.has(tokens[i]!.kind)) depth++; + else if (OPENING.has(tokens[i]!.kind) && --depth === 0) return i; + } + return -1; +} + +/** Whether the argument is already a name equal to the parameter, as `unit` or `t.unit` for unit. */ +function saysIt(sql: string, tokens: Token[], [start, end]: [number, number], name: string): boolean { + for (let i = start; i < end; i++) if ((end - 1 - i) % 2 === 1 ? tokens[i]!.kind !== T.Dot : nameOf(sql, tokens[i]) === undefined) return false; + return nameOf(sql, tokens[end - 1])?.toLowerCase() === name.toLowerCase(); +} + +/** + * The parameter of each argument of the function, table function and engine calls with two arguments or more, in the + * statements that parse, where the parser reads the name as a function or an engine. + */ +export function parameterHints(sql: string, cache?: ParseCache): ParameterHint[] { + const calls = new Set(checkWithRoles(sql, cache).spans.filter((s) => s.role === "function" || s.role === "engine").map((s) => s.start)); + const hints: ParameterHint[] = []; + for (const unit of split(sql)) { + if (!unit.statement) continue; + const statement = unit.tokens.filter((t) => isSignificant(t.kind) && t.kind !== T.Semicolon && t.kind !== T.EndOfStream); + for (let open = 1; open < statement.length; open++) { + if (statement[open]!.kind !== T.OpeningRoundBracket) continue; + // The second brackets of name(parameters)(arguments) follow the first ones. + const name = statement[open - 1]!.kind === T.ClosingRoundBracket ? statement[openingOf(statement, open - 1) - 1] : statement[open - 1]; + if (!name || !calls.has(name.start)) continue; + const spans = argumentSpans(statement, open); + if (spans.length < 2) continue; + const found: ParameterHint[] = []; + for (const span of spans) { + const first = statement[span[0]]; + if (span[0] === span[1] || !first) continue; + const before = statement.slice(0, span[0]); + const engine = engineCall(sql, statement, before); + // An engine argument written `name = value` already says which it is. + const names = engine + ? statement[span[0] + 1]?.kind === T.Equals ? [] : engine.parameters.map((p) => p.name) + : (functionCall(sql, statement, before)?.arguments.map((a) => a.arg.name) ?? []); + if (names.length === 1 && /^\w+$/.test(names[0]!) && !saysIt(sql, statement, span, names[0]!)) found.push({ offset: first.start, name: names[0]! }); + } + // A parameter that repeats, as the values of tuple(), names no argument in particular. + hints.push(...found.filter((h) => found.filter((o) => o.name === h.name).length === 1)); + } + } + return hints; +} diff --git a/packages/core/src/complete/index.ts b/packages/core/src/complete/index.ts index bdf867d..ec41485 100644 --- a/packages/core/src/complete/index.ts +++ b/packages/core/src/complete/index.ts @@ -1,2 +1,4 @@ export { describe } from "./catalog.js"; +export { parameterHints } from "./hints.js"; export { complete, engineArgument, functionArgument } from "./complete.js"; +export { keywordDocumentation } from "./statements.js"; diff --git a/packages/core/src/complete/statements.ts b/packages/core/src/complete/statements.ts new file mode 100644 index 0000000..af906d2 --- /dev/null +++ b/packages/core/src/complete/statements.ts @@ -0,0 +1,62 @@ +/** + * keywordDocumentation(): the documentation of the SYSTEM command, or of the clause of a statement that makes or + * changes a user, role, quota, row policy or settings profile, whose keyword is at an offset. + */ +import { split } from "../format/index.js"; +import { ACCESS_DOCS, type AccessStatementDoc, type KeywordDoc, SYSTEM_DOCS } from "../generated/statements.js"; +import { checkWithRoles, type ParseCache } from "../parser/index.js"; +import { isSignificant, type KeywordDocumentation, type Token, TokenKind as T } from "../types.js"; + +/** The words after CREATE or ALTER that name each kind of access entity, as ClickHouse takes them. */ +const ENTITIES: Record = { USER: "USER", ROLE: "ROLE", QUOTA: "QUOTA", "ROW POLICY": "ROW POLICY", POLICY: "ROW POLICY", "SETTINGS PROFILE": "SETTINGS PROFILE", PROFILE: "SETTINGS PROFILE" }; + +const upper = (sql: string, token: Token | undefined) => (token?.kind === T.BareWord ? sql.slice(token.start, token.end).toUpperCase() : undefined); + +/** The doc whose keywords the words at `from` start with, the longest one, and how many words it has. */ +function longest(sql: string, tokens: Token[], from: number, docs: readonly KeywordDoc[]): { doc: KeywordDoc; length: number } | undefined { + let found: { doc: KeywordDoc; length: number } | undefined; + for (const doc of docs) { + for (const keyword of doc.keywords) { + const words = keyword.split(" "); + if (words.length > (found?.length ?? 0) && words.every((w, i) => upper(sql, tokens[from + i]) === w)) found = { doc, length: words.length }; + } + } + return found; +} + +/** The access statement that the tokens make or change, by its first words. */ +function accessStatement(sql: string, tokens: Token[]): AccessStatementDoc | undefined { + const verb = upper(sql, tokens[0]); + if (verb !== "CREATE" && verb !== "ALTER") return undefined; + const entity = ENTITIES[`${upper(sql, tokens[1])} ${upper(sql, tokens[2])}`] ?? ENTITIES[upper(sql, tokens[1]) ?? ""]; + return entity ? ACCESS_DOCS.find((d) => d.statement === `${verb} ${entity}`) : undefined; +} + +/** The documentation of the SYSTEM command or access statement clause whose keyword is at `offset`. */ +export function keywordDocumentation(sql: string, offset: number, cache?: ParseCache): KeywordDocumentation | undefined { + const unit = split(sql).find((u) => u.statement && u.statement.start <= offset && offset <= u.statement.end); + if (!unit) return undefined; + const tokens = unit.tokens.filter((t) => isSignificant(t.kind) && t.kind !== T.Semicolon && t.kind !== T.EndOfStream); + const at = tokens.findIndex((t) => t.start <= offset && offset <= t.end && t.kind === T.BareWord); + if (at < 0) return undefined; + const span = (from: number, length: number) => ({ start: tokens[from]!.start, end: tokens[from + length - 1]!.end }); + + if (upper(sql, tokens[0]) === "SYSTEM") { + const found = longest(sql, tokens, 1, SYSTEM_DOCS); + return found && at <= found.length ? { ...span(0, found.length + 1), documentation: found.doc.markdown } : undefined; + } + + const statement = accessStatement(sql, tokens); + if (!statement) return undefined; + // Only a word that the parser reads as a keyword, so not the name of a user called host. + const start = unit.statement!.start; + const role = checkWithRoles(sql.slice(start, unit.statement!.end), cache).spans.find((s) => s.start + start === tokens[at]!.start)?.role; + if (role !== "keyword") return undefined; + let best: { from: number; doc: KeywordDoc; length: number } | undefined; + for (let from = Math.max(0, at - 3); from <= at; from++) { + const found = longest(sql, tokens, from, statement.clauses); + if (found && from + found.length > at && found.length > (best?.length ?? 0)) best = { from, ...found }; + } + if (best) return { ...span(best.from, best.length), documentation: best.doc.markdown }; + return { ...span(at, 1), documentation: statement.markdown }; +} diff --git a/packages/core/src/generated/options.ts b/packages/core/src/generated/options.ts index 5c07684..1ae7976 100644 --- a/packages/core/src/generated/options.ts +++ b/packages/core/src/generated/options.ts @@ -20,6 +20,8 @@ export interface CallRule { /** ClickHouse's message for too many parameters, with {} for their count. */ tooMany?: string; tooFew?: string; + /** Counts between min and max that ClickHouse refuses, with its message. */ + gaps?: readonly (readonly [number, string])[]; parameters: readonly ParameterRule[]; } @@ -40,6 +42,30 @@ export interface CodecInfo { /** ClickHouse's message when it cannot be written in CODEC at all. */ refused?: string; rule?: CallRule; + /** The column types it refuses, by ClickHouse's message with {} for the type, written without parameters and with them. */ + types?: readonly TypeRefusal[]; + typesWithParameters?: readonly TypeRefusal[]; +} + +export interface TypeRefusal { + message: string; + types: readonly string[]; + /** The number of parameters it was written with; none when any parameters give the same refusals. */ + parameters?: number; +} + +/** The arguments of an index type written as name = value, as those of text. */ +export interface NamedArguments { + required: readonly string[]; + /** ClickHouse's messages, with {} for the name or names. */ + missing?: string; + positional?: string; + unknown?: string; + duplicate?: string; + /** The names it takes, with what each value may be; null when its values are not known. */ + arguments: Readonly>; + tokenizers: readonly string[]; + unknownTokenizer?: string; } export interface IndexTypeInfo { @@ -47,6 +73,7 @@ export interface IndexTypeInfo { syntax?: string; description?: string; rule?: CallRule; + named?: NamedArguments; /** ClickHouse's message for a type that no table can use any more. */ refused?: string; } @@ -74,25 +101,28 @@ export const STATISTICS_MESSAGE = "Unknown statistics type: {}. Supported statis /** The settings that a column can have in SETTINGS (...), and ClickHouse's message for others, with {} for the name. */ export const COLUMN_SETTINGS: readonly string[] = ["max_compress_block_size","min_compress_block_size"]; export const COLUMN_SETTING_MESSAGE = "Setting {} is unknown or not supported at column level, supported settings: max_compress_block_size, min_compress_block_size."; +/** The levels that INTO OUTFILE takes after COMPRESSION, by method, and ClickHouse's message for others. */ +export const COMPRESSION_LEVELS: Readonly> = {"none":[1,9],"gzip":[1,12],"deflate":[1,12],"br":[1,9],"xz":[1,9],"zstd":[1,22],"lz4":[1,12],"bz2":[1,9],"snappy":[1,9]}; +export const COMPRESSION_LEVEL_MESSAGE = "Invalid compression level, must be positive integer in range {min}-{max}."; export const CODECS: readonly CodecInfo[] = [ {"name":"AES_128_GCM_SIV","description":"Encrypts and decrypts blocks with AES-128 in GCM-SIV mode (RFC-8452).","encryption":true,"type":"UInt64","rule":{"min":0,"max":0,"tooMany":"Codec AES_128_GCM_SIV must not have parameters, given {}.","parameters":[]}}, {"name":"AES_256_GCM_SIV","description":"Encrypts and decrypts blocks with AES-256 in GCM-SIV mode (RFC-8452).","encryption":true,"type":"UInt64","rule":{"min":0,"max":0,"tooMany":"Codec AES_256_GCM_SIV must not have parameters, given {}.","parameters":[]}}, - {"name":"ALP","description":"Adaptive Lossless floating-Point; suitable for time series data.","compression":true,"timeSeries":true,"tier":"Beta","setting":"enable_alp_codec","type":"Float64","rule":{"min":0,"max":1,"tooMany":"ALP codec must have zero or one parameter, given {}.","parameters":[{"ints":[[-1,false],[0,false],[1,false],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"ALP codec variant must be an identifier: AUTO, STD or RD.","int":"ALP codec variant must be an identifier: AUTO, STD or RD.","float":"ALP codec variant must be an identifier: AUTO, STD or RD.","string":"ALP codec variant must be an identifier: AUTO, STD or RD.","word":"ALP codec variant must be AUTO, STD or RD, given {}."},"stringsRefused":true,"words":["auto","std","rd"],"wordsRefused":true}]}}, - {"name":"Delta","description":"Preprocessor (should be followed by some compression codec). Stores difference between neighboring values; good for monotonically increasing or decreasing data.","type":"UInt64","rule":{"min":0,"max":1,"tooMany":"Delta codec must have 1 parameter, given {}.","parameters":[{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,false],[4,true],[5,false],[7,false],[8,true],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Delta codec argument must be unsigned integer.","int":"Delta value for delta codec can be 1, 2, 4 or 8, given {}.","float":"Delta codec argument must be unsigned integer.","string":"Delta codec argument must be unsigned integer.","word":"Delta codec argument must be unsigned integer."},"stringsRefused":true,"wordsRefused":true}]}}, - {"name":"DoubleDelta","description":"Stores difference between neighboring delta values; suitable for time series data.","compression":true,"type":"UInt64","rule":{"min":0,"max":1,"tooMany":"DoubleDelta codec must have 1 parameter, given {}.","parameters":[{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,false],[4,true],[5,false],[7,false],[8,true],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"DoubleDelta codec argument must be unsigned integer.","int":"Argument value for DoubleDelta codec can be 1, 2, 4 or 8, given {}.","float":"DoubleDelta codec argument must be unsigned integer.","string":"DoubleDelta codec argument must be unsigned integer.","word":"DoubleDelta codec argument must be unsigned integer."},"stringsRefused":true,"wordsRefused":true}]}}, - {"name":"FPC","description":"High Throughput Compression of Double-Precision Floating-Point Data.","compression":true,"timeSeries":true,"type":"Float64","rule":{"min":0,"max":2,"tooMany":"Codec 'FPC' must have from 0 to 2 parameters, given {}.","parameters":[{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,true],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"1st argument of codec 'FPC' must be unsigned integer.","int":"Codec 'FPC' level must be between 1 and 28.","float":"1st argument of codec 'FPC' must be unsigned integer.","string":"1st argument of codec 'FPC' must be unsigned integer.","word":"1st argument of codec 'FPC' must be unsigned integer."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,false],[1,false],[2,false],[3,false],[4,true],[5,false],[7,false],[8,true],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Float size for codec 'FPC' can be 4 or 8, given 18446744073709551615.","int":"Float size for codec 'FPC' can be 4 or 8, given {}.","float":"2nd argument of codec 'FPC' must be unsigned integer.","string":"2nd argument of codec 'FPC' must be unsigned integer.","word":"2nd argument of codec 'FPC' must be unsigned integer."},"stringsRefused":true,"wordsRefused":true}]}}, - {"name":"GCD","description":"Preprocessor. Greatest common divisor compression; divides values by a common divisor; effective for divisible integer sequences.","type":"UInt64","rule":{"min":0,"max":0,"tooMany":"GCD codec must have 0 parameters, given {}.","parameters":[]}}, + {"name":"ALP","description":"Adaptive Lossless floating-Point; suitable for time series data.","compression":true,"timeSeries":true,"tier":"Beta","setting":"enable_alp_codec","type":"Float64","rule":{"min":0,"max":1,"tooMany":"ALP codec must have zero or one parameter, given {}.","parameters":[{"ints":[[-1,false],[0,false],[1,false],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"ALP codec variant must be an identifier: AUTO, STD or RD.","int":"ALP codec variant must be an identifier: AUTO, STD or RD.","float":"ALP codec variant must be an identifier: AUTO, STD or RD.","string":"ALP codec variant must be an identifier: AUTO, STD or RD.","word":"ALP codec variant must be AUTO, STD or RD, given {}."},"words":["auto","std","rd"],"stringsRefused":true,"wordsRefused":true}]},"types":[{"message":"Codec ALP is not applicable for {} because the data type is not Float*.","types":["BFloat16","Bool","Date","Date32","DateTime","IPv4","IPv6","Int128","Int16","Int256","Int32","Int64","Int8","IntervalDay","IntervalHour","IntervalMicrosecond","IntervalMillisecond","IntervalMinute","IntervalMonth","IntervalNanosecond","IntervalQuarter","IntervalSecond","IntervalWeek","IntervalYear","String","Time","UInt128","UInt16","UInt256","UInt32","UInt64","UInt8","UUID"]},{"message":"Codec ALP is not applicable for {}(3) because the data type is not Float*.","types":["DateTime64","Time64"]},{"message":"Codec ALP is not applicable for {}(10, 0) because the data type is not Float*.","types":["Decimal"]},{"message":"Codec ALP is not applicable for {}() because the data type is not Float*.","types":["Tuple"]}],"typesWithParameters":[{"message":"Codec ALP is not applicable for {} because the data type is not Float*.","types":["BFloat16","Bool","Date","Date32","DateTime","IPv4","IPv6","Int128","Int16","Int256","Int32","Int64","Int8","IntervalDay","IntervalHour","IntervalMicrosecond","IntervalMillisecond","IntervalMinute","IntervalMonth","IntervalNanosecond","IntervalQuarter","IntervalSecond","IntervalWeek","IntervalYear","String","Time","UInt128","UInt16","UInt256","UInt32","UInt64","UInt8","UUID"]},{"message":"Codec ALP is not applicable for {}(3) because the data type is not Float*.","types":["DateTime64","Time64"]},{"message":"Codec ALP is not applicable for {}(10, 0) because the data type is not Float*.","types":["Decimal"]},{"message":"Codec ALP is not applicable for {}() because the data type is not Float*.","types":["Tuple"]}]}, + {"name":"Delta","description":"Preprocessor (should be followed by some compression codec). Stores difference between neighboring values; good for monotonically increasing or decreasing data.","type":"UInt64","rule":{"min":0,"max":1,"tooMany":"Delta codec must have 1 parameter, given {}.","parameters":[{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,false],[4,true],[5,false],[7,false],[8,true],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Delta codec argument must be unsigned integer.","int":"Delta value for delta codec can be 1, 2, 4 or 8, given {}.","float":"Delta codec argument must be unsigned integer.","string":"Delta codec argument must be unsigned integer.","word":"Delta codec argument must be unsigned integer."},"stringsRefused":true,"wordsRefused":true}]},"types":[{"message":"Codec Delta is only applicable for data types of size 1, 2, 4, 8 bytes. Given type {}.","types":["IPv6","Int128","Int256","UInt128","UInt256","UUID"]},{"message":"Codec Delta is not applicable for {} because the data type is not of fixed size.","types":["String"]},{"message":"Codec Delta is not applicable for {}() because the data type is not of fixed size.","types":["Tuple"]}]}, + {"name":"DoubleDelta","description":"Stores difference between neighboring delta values; suitable for time series data.","compression":true,"type":"UInt64","rule":{"min":0,"max":1,"tooMany":"DoubleDelta codec must have 1 parameter, given {}.","parameters":[{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,false],[4,true],[5,false],[7,false],[8,true],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"DoubleDelta codec argument must be unsigned integer.","int":"Argument value for DoubleDelta codec can be 1, 2, 4 or 8, given {}.","float":"DoubleDelta codec argument must be unsigned integer.","string":"DoubleDelta codec argument must be unsigned integer.","word":"DoubleDelta codec argument must be unsigned integer."},"stringsRefused":true,"wordsRefused":true}]},"types":[{"message":"Codec DoubleDelta is not applicable for {} because the data type is not numeric.","types":["IPv6","String","UUID"]},{"message":"Codec DoubleDelta is only applicable for data types of size 1, 2, 4, 8 bytes. Given type {}.","types":["Int128","Int256","UInt128","UInt256"]},{"message":"Codec DoubleDelta is not applicable for {}() because the data type is not numeric.","types":["Tuple"]}],"typesWithParameters":[{"message":"Codec DoubleDelta is not applicable for {} because the data type is not numeric.","types":["IPv6","String","UUID"]},{"message":"Codec DoubleDelta is only applicable for data types of size 1, 2, 4, 8 bytes. Given type {}.","types":["Int128","Int256","UInt128","UInt256"]},{"message":"Codec DoubleDelta is not applicable for {}() because the data type is not numeric.","types":["Tuple"]}]}, + {"name":"FPC","description":"High Throughput Compression of Double-Precision Floating-Point Data.","compression":true,"timeSeries":true,"type":"Float64","rule":{"min":0,"max":2,"tooMany":"Codec 'FPC' must have from 0 to 2 parameters, given {}.","parameters":[{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,true],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"1st argument of codec 'FPC' must be unsigned integer.","int":"Codec 'FPC' level must be between 1 and 28.","float":"1st argument of codec 'FPC' must be unsigned integer.","string":"1st argument of codec 'FPC' must be unsigned integer.","word":"1st argument of codec 'FPC' must be unsigned integer."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,false],[1,false],[2,false],[3,false],[4,true],[5,false],[7,false],[8,true],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Float size for codec 'FPC' can be 4 or 8, given 18446744073709551615.","int":"Float size for codec 'FPC' can be 4 or 8, given {}.","float":"2nd argument of codec 'FPC' must be unsigned integer.","string":"2nd argument of codec 'FPC' must be unsigned integer.","word":"2nd argument of codec 'FPC' must be unsigned integer."},"stringsRefused":true,"wordsRefused":true}]},"types":[{"message":"Codec 'FPC' is not applicable for {} because the data type is not Float*.","types":["BFloat16","Bool","Date","Date32","DateTime","IPv4","IPv6","Int128","Int16","Int256","Int32","Int64","Int8","IntervalDay","IntervalHour","IntervalMicrosecond","IntervalMillisecond","IntervalMinute","IntervalMonth","IntervalNanosecond","IntervalQuarter","IntervalSecond","IntervalWeek","IntervalYear","String","Time","UInt128","UInt16","UInt256","UInt32","UInt64","UInt8","UUID"]},{"message":"Codec 'FPC' is not applicable for {}(3) because the data type is not Float*.","types":["DateTime64","Time64"]},{"message":"Codec 'FPC' is not applicable for {}(10, 0) because the data type is not Float*.","types":["Decimal"]},{"message":"Codec 'FPC' is not applicable for {}() because the data type is not Float*.","types":["Tuple"]}],"typesWithParameters":[{"message":"Codec 'FPC' is not applicable for {} because the data type is not Float*.","types":["BFloat16","Bool","Date","Date32","DateTime","IPv4","IPv6","Int128","Int16","Int256","Int32","Int64","Int8","IntervalDay","IntervalHour","IntervalMicrosecond","IntervalMillisecond","IntervalMinute","IntervalMonth","IntervalNanosecond","IntervalQuarter","IntervalSecond","IntervalWeek","IntervalYear","String","Time","UInt128","UInt16","UInt256","UInt32","UInt64","UInt8","UUID"]},{"message":"Codec 'FPC' is not applicable for {}(3) because the data type is not Float*.","types":["DateTime64","Time64"]},{"message":"Codec 'FPC' is not applicable for {}(10, 0) because the data type is not Float*.","types":["Decimal"]},{"message":"Codec 'FPC' is not applicable for {}() because the data type is not Float*.","types":["Tuple"]}]}, + {"name":"GCD","description":"Preprocessor. Greatest common divisor compression; divides values by a common divisor; effective for divisible integer sequences.","type":"UInt64","rule":{"min":0,"max":0,"tooMany":"GCD codec must have 0 parameters, given {}.","parameters":[]},"types":[{"message":"Codec GCD cannot be applied to column {} because it can only be used with Int*, UInt*, Decimal*, Date* or DateTime* types.","types":["BFloat16","Float32","Float64","IPv4","IPv6","IntervalDay","IntervalHour","IntervalMicrosecond","IntervalMillisecond","IntervalMinute","IntervalMonth","IntervalNanosecond","IntervalQuarter","IntervalSecond","IntervalWeek","IntervalYear","String","Time","UUID"]},{"message":"Codec GCD cannot be applied to column {}(3) because it can only be used with Int*, UInt*, Decimal*, Date* or DateTime* types.","types":["Time64"]},{"message":"Codec GCD cannot be applied to column {}() because it can only be used with Int*, UInt*, Decimal*, Date* or DateTime* types.","types":["Tuple"]}]}, {"name":"GDeflate","description":"GDeflate, a Deflate that a GPU can expand in parallel, in chunks of 64 KiB. Levels 1..12.","compression":true,"generic":true,"tier":"Experimental","setting":"enable_gdeflate_codec","type":"UInt64","rule":{"min":0,"max":1,"tooMany":"GDeflate codec must have 1 parameter, given {}.","parameters":[{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"GDeflate codec argument must be a positive integer.","int":"GDeflate codec level must be between 1 and 12, given {}.","float":"GDeflate codec argument must be a positive integer.","string":"GDeflate codec argument must be a positive integer.","word":"GDeflate codec argument must be a positive integer."},"stringsRefused":true,"wordsRefused":true}]}}, - {"name":"Gorilla","description":"Calculates XOR between current and previous value; suitable for slowly changing numbers.","compression":true,"timeSeries":true,"type":"UInt64","rule":{"min":0,"max":1,"tooMany":"Gorilla codec must have 1 parameter, given {}.","parameters":[{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,false],[4,true],[5,false],[7,false],[8,true],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Gorilla codec argument must be unsigned integer.","int":"Argument value for Gorilla codec can be 1, 2, 4 or 8, given {}.","float":"Gorilla codec argument must be unsigned integer.","string":"Gorilla codec argument must be unsigned integer.","word":"Gorilla codec argument must be unsigned integer."},"stringsRefused":true,"wordsRefused":true}]}}, + {"name":"Gorilla","description":"Calculates XOR between current and previous value; suitable for slowly changing numbers.","compression":true,"timeSeries":true,"type":"UInt64","rule":{"min":0,"max":1,"tooMany":"Gorilla codec must have 1 parameter, given {}.","parameters":[{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,false],[4,true],[5,false],[7,false],[8,true],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Gorilla codec argument must be unsigned integer.","int":"Argument value for Gorilla codec can be 1, 2, 4 or 8, given {}.","float":"Gorilla codec argument must be unsigned integer.","string":"Gorilla codec argument must be unsigned integer.","word":"Gorilla codec argument must be unsigned integer."},"stringsRefused":true,"wordsRefused":true}]},"types":[{"message":"Codec Gorilla is only applicable for data types of size 1, 2, 4, 8 bytes. Given type {}.","types":["IPv6","Int128","Int256","UInt128","UInt256","UUID"]},{"message":"Codec Gorilla is not applicable for {} because the data type is not of fixed size.","types":["String"]},{"message":"Codec Gorilla is not applicable for {}() because the data type is not of fixed size.","types":["Tuple"]}]}, {"name":"LZ4","description":"Extremely fast; good compression; balanced speed and efficiency.","compression":true,"generic":true,"type":"UInt64","rule":{"min":0,"max":0,"tooMany":"Compression codec LZ4 cannot have arguments.","parameters":[]}}, {"name":"LZ4HC","description":"LZ4 High Compression algorithm with configurable level; slower but better compression than LZ4, but decompression is still fast.","compression":true,"generic":true,"type":"UInt64","rule":{"min":0,"max":1,"tooMany":"LZ4HC codec must have 1 parameter, given {}.","parameters":[{"ints":[[-1,true],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"float":"Bad get: has Float64, requested UInt64.","string":"Bad get: has String, requested UInt64.","word":"LZ4HC codec argument must be integer."},"stringsRefused":true,"wordsRefused":true}]}}, {"name":"Multiple","description":"Apply multiple codecs consecutively defined by user.","refused":"Codec Multiple cannot be specified directly."}, {"name":"NONE","description":"No compression. Can be used on columns that can not be compressed anyway.","type":"UInt64","rule":{"min":0,"max":0,"tooMany":"Compression codec NONE cannot have arguments.","parameters":[]}}, - {"name":"PCO","description":"Lossless compression of numeric sequences (pcodec); excellent on numeric columns.","compression":true,"tier":"Beta","setting":"enable_pco_codec","type":"UInt64","rule":{"min":0,"max":1,"tooMany":"Codec 'PCO' must have 0 or 1 parameters, given {}.","parameters":[{"ints":[[-1,false],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Codec 'PCO' parameter (compression level) must be an unsigned integer.","int":"Codec 'PCO' compression level must be between 0 and 12, given {}.","float":"Codec 'PCO' parameter (compression level) must be an unsigned integer.","string":"Codec 'PCO' parameter (compression level) must be an unsigned integer.","word":"Codec 'PCO' parameter (compression level) must be an unsigned integer."},"stringsRefused":true,"wordsRefused":true}]}}, - {"name":"Quantized","description":"Stores a compact quantized companion stream of a dense vector column for fast nearest-neighbour search; the full-precision data is stored uncompressed.","tier":"Experimental","setting":"enable_quantized_codec"}, - {"name":"SZ3","description":"SZ3 is a lossy compressor for floating-point data with error bounds.","compression":true,"timeSeries":true,"tier":"Experimental","setting":"enable_sz3_codec","type":"Float64","rule":{"min":0,"max":99,"parameters":[]}}, - {"name":"T64","description":"Preprocessor. Crops unused high bits; puts them into a 64x64 bit matrix; optimized for 64-bit data types.","compression":true,"type":"UInt64","rule":{"min":0,"max":1,"tooMany":"T64 support zero or one parameter, given {}.","parameters":[{"ints":[[-1,false],[0,false],[1,false],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Bad get: has Int64, requested String.","int":"Bad get: has UInt64, requested String.","float":"Bad get: has Float64, requested String.","string":"Wrong modification for T64: {}.","word":"Wrong modification for T64. Expected: 'bit', 'byte')."},"strings":["byte","bit"],"stringsRefused":true,"wordsRefused":true}]}}, + {"name":"PCO","description":"Lossless compression of numeric sequences (pcodec); excellent on numeric columns.","compression":true,"tier":"Beta","setting":"enable_pco_codec","type":"UInt64","rule":{"min":0,"max":1,"tooMany":"Codec 'PCO' must have 0 or 1 parameters, given {}.","parameters":[{"ints":[[-1,false],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Codec 'PCO' parameter (compression level) must be an unsigned integer.","int":"Codec 'PCO' compression level must be between 0 and 12, given {}.","float":"Codec 'PCO' parameter (compression level) must be an unsigned integer.","string":"Codec 'PCO' parameter (compression level) must be an unsigned integer.","word":"Codec 'PCO' parameter (compression level) must be an unsigned integer."},"stringsRefused":true,"wordsRefused":true}]},"types":[{"message":"Codec 'PCO' is not applicable for {} because it is not a number.","types":["IPv6","String","UUID"]},{"message":"Codec 'PCO' is only applicable for numeric types of size 1, 2, 4 or 8 bytes, but {} has size 16.","types":["Int128","UInt128"]},{"message":"Codec 'PCO' is only applicable for numeric types of size 1, 2, 4 or 8 bytes, but {} has size 32.","types":["Int256","UInt256"]},{"message":"Codec 'PCO' is not applicable for {}() because it is not a number.","types":["Tuple"]}],"typesWithParameters":[{"message":"Codec 'PCO' is not applicable for {} because it is not a number.","types":["IPv6","String","UUID"]},{"message":"Codec 'PCO' is only applicable for numeric types of size 1, 2, 4 or 8 bytes, but {} has size 16.","types":["Int128","UInt128"]},{"message":"Codec 'PCO' is only applicable for numeric types of size 1, 2, 4 or 8 bytes, but {} has size 32.","types":["Int256","UInt256"]},{"message":"Codec 'PCO' is not applicable for {}() because it is not a number.","types":["Tuple"]}]}, + {"name":"Quantized","description":"Stores a compact quantized companion stream of a dense vector column for fast nearest-neighbour search; the full-precision data is stored uncompressed.","tier":"Experimental","setting":"enable_quantized_codec","rule":{"min":2,"max":4,"tooMany":"Codec Quantized requires 2 to 4 parameters: Quantized(method, dimensions[, bits[, m]]) .","tooFew":"Codec Quantized requires 2 to 4 parameters: Quantized(method, dimensions[, bits[, m]]) .","parameters":[{"ints":[[-1,false],[0,false],[1,false],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"First argument of codec Quantized (method) must be a string literal.","int":"First argument of codec Quantized (method) must be a string literal.","float":"First argument of codec Quantized (method) must be a string literal.","string":"Unknown quantization method '{}'.","word":"First argument of codec Quantized (method) must be a string literal."},"strings":["int8"],"wordsRefused":true},{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Second argument of codec Quantized (dimensions) must be an unsigned integer.","int":"Codec Quantized: The number of dimensions must be greater than zero.","float":"Second argument of codec Quantized (dimensions) must be an unsigned integer.","string":"Second argument of codec Quantized (dimensions) must be an unsigned integer.","word":"Second argument of codec Quantized (dimensions) must be an unsigned integer."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Third argument of codec Quantized (bits) must be an unsigned integer.","float":"Third argument of codec Quantized (bits) must be an unsigned integer.","string":"Third argument of codec Quantized (bits) must be an unsigned integer.","word":"Third argument of codec Quantized (bits) must be an unsigned integer."},"stringsRefused":true,"wordsRefused":true}]}}, + {"name":"SZ3","description":"SZ3 is a lossy compressor for floating-point data with error bounds.","compression":true,"timeSeries":true,"tier":"Experimental","setting":"enable_sz3_codec","type":"Float64","rule":{"min":0,"max":3,"tooMany":"Codec SZ3 must have 0 or 3 arguments but 4 arguments are given.","gaps":[[1,"Codec SZ3 must have 0 or 3 arguments but 1 arguments are given."],[2,"Codec SZ3 must have 0 or 3 arguments but 2 arguments are given."]],"parameters":[{"ints":[[-1,false],[0,false],[1,false],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"1st argument of codec 'SZ3' must be a String.","int":"1st argument of codec 'SZ3' must be a String.","float":"1st argument of codec 'SZ3' must be a String.","string":"Unsupported algorithm '{}' for codec 'SZ3'. Supported algorithms are 'ALGO_LORENZO_REG', 'ALGO_INTERP_LORENZO' and 'ALGO_INTERP'.","word":"1st argument of codec 'SZ3' must be a String."},"strings":["algo_lorenzo_reg"],"wordsRefused":true},{"ints":[[-1,false],[0,false],[1,false],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"2nd argument of codec 'SZ3' be a String.","int":"2nd argument of codec 'SZ3' be a String.","float":"2nd argument of codec 'SZ3' be a String.","string":"Unsupported error bound mode '{}' for codec 'SZ3'. Supported modes are 'ABS', 'REL', 'PSNR' and 'ABS_AND_REL'.","word":"2nd argument of codec 'SZ3' be a String."},"strings":["rel"],"wordsRefused":true},{"ints":[[-1,false],[0,false],[1,false],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,true],[0.01,true],[0.025,true],[0.5,true],[0.999,true],[1,true],[1.5,true],[2,true],[4,true],[8,true]],"messages":{"negative":"3rd argument of codec 'SZ3' be a Float64.","int":"3rd argument of codec 'SZ3' be a Float64.","float":"The error bound (3rd argument) of codec 'SZ3' must be a finite positive number, got -0.5.","string":"3rd argument of codec 'SZ3' be a Float64.","word":"3rd argument of codec 'SZ3' be a Float64."},"stringsRefused":true,"wordsRefused":true}]},"types":[{"message":"Codec 'SZ3' is not applicable for {} because the data type is not Float*.","types":["BFloat16","Bool","Date","Date32","DateTime","IPv4","IPv6","Int128","Int16","Int256","Int32","Int64","Int8","IntervalDay","IntervalHour","IntervalMicrosecond","IntervalMillisecond","IntervalMinute","IntervalMonth","IntervalNanosecond","IntervalQuarter","IntervalSecond","IntervalWeek","IntervalYear","String","Time","UInt128","UInt16","UInt256","UInt32","UInt64","UInt8","UUID"]},{"message":"Codec 'SZ3' is not applicable for {}(3) because the data type is not Float*.","types":["DateTime64","Time64"]},{"message":"Codec 'SZ3' is not applicable for {}(10, 0) because the data type is not Float*.","types":["Decimal"]},{"message":"Codec 'SZ3' is not applicable for {}() because the data type is not Float*.","types":["Tuple"]}],"typesWithParameters":[{"message":"Codec 'SZ3' is not applicable for {} because the data type is not Float*.","types":["BFloat16","Bool","Date","Date32","DateTime","IPv4","IPv6","Int128","Int16","Int256","Int32","Int64","Int8","IntervalDay","IntervalHour","IntervalMicrosecond","IntervalMillisecond","IntervalMinute","IntervalMonth","IntervalNanosecond","IntervalQuarter","IntervalSecond","IntervalWeek","IntervalYear","String","Time","UInt128","UInt16","UInt256","UInt32","UInt64","UInt8","UUID"]},{"message":"Codec 'SZ3' is not applicable for {}(3) because the data type is not Float*.","types":["DateTime64","Time64"]},{"message":"Codec 'SZ3' is not applicable for {}(10, 0) because the data type is not Float*.","types":["Decimal"]},{"message":"Codec 'SZ3' is not applicable for {}() because the data type is not Float*.","types":["Tuple"]}]}, + {"name":"T64","description":"Preprocessor. Crops unused high bits; puts them into a 64x64 bit matrix; optimized for 64-bit data types.","compression":true,"type":"UInt64","rule":{"min":0,"max":1,"tooMany":"T64 support zero or one parameter, given {}.","parameters":[{"ints":[[-1,false],[0,false],[1,false],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Bad get: has Int64, requested String.","int":"Bad get: has UInt64, requested String.","float":"Bad get: has Float64, requested String.","string":"Wrong modification for T64: {}.","word":"Wrong modification for T64. Expected: 'bit', 'byte')."},"strings":["byte","bit"],"stringsRefused":true,"wordsRefused":true}]},"types":[{"message":"T64 codec is not supported for specified type {}.","types":["BFloat16","Float32","Float64","IPv6","Int128","Int256","IntervalDay","IntervalHour","IntervalMicrosecond","IntervalMillisecond","IntervalMinute","IntervalMonth","IntervalNanosecond","IntervalQuarter","IntervalSecond","IntervalWeek","IntervalYear","String","UInt128","UInt256","UUID"]},{"message":"T64 codec is not supported for specified type {}().","types":["Tuple"]}],"typesWithParameters":[{"message":"T64 codec is not supported for specified type {}.","types":["BFloat16","Float32","Float64","IPv6","Int128","Int256","IntervalDay","IntervalHour","IntervalMicrosecond","IntervalMillisecond","IntervalMinute","IntervalMonth","IntervalNanosecond","IntervalQuarter","IntervalSecond","IntervalWeek","IntervalYear","String","UInt128","UInt256","UUID"]},{"message":"T64 codec is not supported for specified type {}().","types":["Tuple"]}]}, {"name":"ZSTD","description":"Good compression; pretty fast; best for high compression needs. Don’t use levels higher than 6.","compression":true,"generic":true,"type":"UInt64","rule":{"min":0,"max":2,"tooMany":"ZSTD codec must have 1 or 2 parameters, given {}.","parameters":[{"ints":[[-1,true],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"int":"ZSTD codec can't have level more than 22, given {}.","float":"Bad get: has Float64, requested UInt64.","string":"Bad get: has String, requested UInt64.","word":"ZSTD codec argument must be integer."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,true],[1,false],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"ZSTD codec can't have window log more than 31 and lower than 10, given {}.","int":"ZSTD codec can't have window log more than 31 and lower than 10, given {}.","float":"Bad get: has Float64, requested UInt64.","string":"Bad get: has String, requested UInt64.","word":"ZSTD codec second argument must be integer."},"stringsRefused":true,"wordsRefused":true}]}}, {"name":"ZXC","description":"Asymmetric LZ codec (zxc): slow compression, very fast decompression, ratio between LZ4 and ZSTD. Levels 1..7.","compression":true,"generic":true,"tier":"Experimental","setting":"enable_zxc_codec","type":"UInt64","rule":{"min":0,"max":1,"tooMany":"ZXC codec must have 1 parameter, given {}.","parameters":[{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"ZXC codec level must be between 1 and 7, given {}.","int":"ZXC codec level must be between 1 and 7, given {}.","float":"Bad get: has Float64, requested UInt64.","string":"Bad get: has String, requested UInt64.","word":"ZXC codec argument must be integer."},"stringsRefused":true,"wordsRefused":true}]}}, ]; @@ -100,13 +130,13 @@ export const CODECS: readonly CodecInfo[] = [ export const INDEX_TYPES: readonly IndexTypeInfo[] = [ {"name":"bloom_filter","syntax":"INDEX name expr TYPE bloom_filter([false_positive_rate]) GRANULARITY g","description":"A Bloom filter over the values of the index expression, for speeding up equality and IN conditions on columns, including Array and Map elements.","rule":{"min":0,"max":1,"tooMany":"BloomFilter index cannot have more than one parameter.","parameters":[{"ints":[[-1,false],[0,false],[1,false],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,true],[0.001,true],[0.025,true],[0.5,true],[0.999,true],[1,true],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"The BloomFilter false positive must be a double number between 0 and 1.","int":"The BloomFilter false positive must be a double number between 0 and 1.","float":"The BloomFilter false positive must be a double number between 0 and 1.","string":"The BloomFilter false positive must be a double number between 0 and 1.","word":"The BloomFilter false positive must be a double number between 0 and 1."},"stringsRefused":true,"wordsRefused":true}]}}, {"name":"hypothesis","syntax":"INDEX name expr TYPE hypothesis GRANULARITY g","description":"Deprecated and no longer supported. It is retained only so that tables which still reference it can be attached.","refused":"Index of type 'hypothesis' is no longer supported. Please drop the index."}, - {"name":"minmax","syntax":"INDEX name expr TYPE minmax GRANULARITY n","description":"Stores the minimum and maximum values of the index expression for each granule, allowing granules to be skipped when a query's range condition cannot match.","rule":{"min":0,"max":6,"parameters":[{"ints":[[-1,true],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,true],[0,true],[0.001,true],[0.025,true],[0.5,true],[0.999,true],[1,true],[1.5,true],[2,true],[4,true],[8,true]],"messages":{},"strings":["byte","bit","hnsw"],"stringsRefused":false,"words":["auto","std","rd"],"wordsRefused":false},{"ints":[[-1,true],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,true],[0,true],[0.001,true],[0.025,true],[0.5,true],[0.999,true],[1,true],[1.5,true],[2,true],[4,true],[8,true]],"messages":{},"strings":["byte","bit","hnsw"],"stringsRefused":false,"words":["auto","std","rd"],"wordsRefused":false},{"ints":[[-1,true],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,true],[0,true],[0.001,true],[0.025,true],[0.5,true],[0.999,true],[1,true],[1.5,true],[2,true],[4,true],[8,true]],"messages":{},"strings":["byte","bit","hnsw"],"stringsRefused":false,"words":["auto","std","rd"],"wordsRefused":false},{"ints":[[-1,true],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,true],[0,true],[0.001,true],[0.025,true],[0.5,true],[0.999,true],[1,true],[1.5,true],[2,true],[4,true],[8,true]],"messages":{},"strings":["byte","bit","hnsw"],"stringsRefused":false,"words":["auto","std","rd"],"wordsRefused":false},{"ints":[[-1,true],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,true],[0,true],[0.001,true],[0.025,true],[0.5,true],[0.999,true],[1,true],[1.5,true],[2,true],[4,true],[8,true]],"messages":{},"strings":["byte","bit","hnsw"],"stringsRefused":false,"words":["auto","std","rd"],"wordsRefused":false},{"ints":[[-1,true],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,true],[0,true],[0.001,true],[0.025,true],[0.5,true],[0.999,true],[1,true],[1.5,true],[2,true],[4,true],[8,true]],"messages":{},"strings":["byte","bit","hnsw"],"stringsRefused":false,"words":["auto","std","rd"],"wordsRefused":false}]}}, + {"name":"minmax","syntax":"INDEX name expr TYPE minmax GRANULARITY n","description":"Stores the minimum and maximum values of the index expression for each granule, allowing granules to be skipped when a query's range condition cannot match.","rule":{"min":0,"max":7,"parameters":[{"ints":[[-1,true],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,true],[0,true],[0.001,true],[0.025,true],[0.5,true],[0.999,true],[1,true],[1.5,true],[2,true],[4,true],[8,true]],"messages":{},"strings":["byte","bit","hnsw"],"words":["auto","std","rd"]},{"ints":[[-1,true],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,true],[0,true],[0.001,true],[0.025,true],[0.5,true],[0.999,true],[1,true],[1.5,true],[2,true],[4,true],[8,true]],"messages":{},"strings":["byte","bit","hnsw"],"words":["auto","std","rd"]},{"ints":[[-1,true],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,true],[0,true],[0.001,true],[0.025,true],[0.5,true],[0.999,true],[1,true],[1.5,true],[2,true],[4,true],[8,true]],"messages":{},"strings":["byte","bit","hnsw"],"words":["auto","std","rd"]},{"ints":[[-1,true],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,true],[0,true],[0.001,true],[0.025,true],[0.5,true],[0.999,true],[1,true],[1.5,true],[2,true],[4,true],[8,true]],"messages":{},"strings":["byte","bit","hnsw"],"words":["auto","std","rd"]},{"ints":[[-1,true],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,true],[0,true],[0.001,true],[0.025,true],[0.5,true],[0.999,true],[1,true],[1.5,true],[2,true],[4,true],[8,true]],"messages":{},"strings":["byte","bit","hnsw"],"words":["auto","std","rd"]},{"ints":[[-1,true],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,true],[0,true],[0.001,true],[0.025,true],[0.5,true],[0.999,true],[1,true],[1.5,true],[2,true],[4,true],[8,true]],"messages":{},"strings":["byte","bit","hnsw"],"words":["auto","std","rd"]}]}}, {"name":"ngrambf_v1","syntax":"INDEX name expr TYPE ngrambf_v1(n, size_in_bytes, num_hash_functions, seed) GRANULARITY g","description":"A Bloom filter over all n-grams of the index expression's string values, for speeding up LIKE, IN, equality and similar searches on substrings.","rule":{"min":4,"max":4,"tooMany":"`ngrambf` index must have exactly 4 arguments.","tooFew":"`ngrambf` index must have exactly 4 arguments.","parameters":[{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"All parameters to *bf_v1 index must be unsigned integers.","int":"ngram length must be at least 1.","float":"Bad get: has Float64, requested UInt64.","string":"Bad get: has String, requested UInt64.","word":"Bad get: has String, requested UInt64."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"All parameters to *bf_v1 index must be unsigned integers.","int":"The size of bloom filter cannot be zero.","float":"All parameters to *bf_v1 index must be unsigned integers.","string":"All parameters to *bf_v1 index must be unsigned integers.","word":"All parameters to *bf_v1 index must be unsigned integers."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"All parameters to *bf_v1 index must be unsigned integers.","int":"The number of hash functions for bloom filter cannot be zero.","float":"All parameters to *bf_v1 index must be unsigned integers.","string":"All parameters to *bf_v1 index must be unsigned integers.","word":"All parameters to *bf_v1 index must be unsigned integers."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"All parameters to *bf_v1 index must be unsigned integers.","float":"All parameters to *bf_v1 index must be unsigned integers.","string":"All parameters to *bf_v1 index must be unsigned integers.","word":"All parameters to *bf_v1 index must be unsigned integers."},"stringsRefused":true,"wordsRefused":true}]}}, {"name":"set","syntax":"INDEX name expr TYPE set(max_rows) GRANULARITY n","description":"Stores up to `max_rows` distinct values of the index expression per granule (0 means unlimited), allowing granules to be skipped for equality and IN conditions.","rule":{"min":1,"max":1,"tooMany":"Set index must have exactly one argument.","tooFew":"Set index must have exactly one argument.","parameters":[{"ints":[[-1,false],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Set index argument must be positive integer.","float":"Set index argument must be positive integer.","string":"Set index argument must be positive integer.","word":"Set index argument must be positive integer."},"stringsRefused":true,"wordsRefused":true}]}}, {"name":"sparse_grams","syntax":"INDEX name expr TYPE sparse_grams(min_ngram_length, max_ngram_length[, min_cutoff_length], size_in_bytes, num_hash_functions, seed) GRANULARITY g","description":"A Bloom filter over the sparse n-grams of the index expression's string values, for speeding up substring searches.","rule":{"min":5,"max":5,"tooMany":"The number of hash functions for bloom filter cannot be zero.","tooFew":"`sparseGrams` index must have exactly 5 or 6 arguments.","parameters":[{"ints":[[-1,false],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"All parameters to *bf_v1 index must be unsigned integers.","float":"All parameters to *bf_v1 index must be unsigned integers.","string":"All parameters to *bf_v1 index must be unsigned integers.","word":"All parameters to *bf_v1 index must be unsigned integers."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"All parameters to *bf_v1 index must be unsigned integers.","float":"All parameters to *bf_v1 index must be unsigned integers.","string":"All parameters to *bf_v1 index must be unsigned integers.","word":"All parameters to *bf_v1 index must be unsigned integers."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"All parameters to *bf_v1 index must be unsigned integers.","int":"The size of bloom filter cannot be zero.","float":"All parameters to *bf_v1 index must be unsigned integers.","string":"All parameters to *bf_v1 index must be unsigned integers.","word":"All parameters to *bf_v1 index must be unsigned integers."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"All parameters to *bf_v1 index must be unsigned integers.","int":"The number of hash functions for bloom filter cannot be zero.","float":"All parameters to *bf_v1 index must be unsigned integers.","string":"All parameters to *bf_v1 index must be unsigned integers.","word":"All parameters to *bf_v1 index must be unsigned integers."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"All parameters to *bf_v1 index must be unsigned integers.","float":"All parameters to *bf_v1 index must be unsigned integers.","string":"All parameters to *bf_v1 index must be unsigned integers.","word":"All parameters to *bf_v1 index must be unsigned integers."},"stringsRefused":true,"wordsRefused":true}]}}, - {"name":"text","syntax":"INDEX name expr TYPE text(tokenizer = splitByNonAlpha) GRANULARITY g","description":"A full-text (inverted) index over the tokens of a string column, for speeding up text search functions such as `hasToken`, `hasAnyTokens`, `hasAllTokens`, and `hasPhrase`."}, + {"name":"text","syntax":"INDEX name expr TYPE text(tokenizer = splitByNonAlpha) GRANULARITY g","description":"A full-text (inverted) index over the tokens of a string column, for speeding up text search functions such as `hasToken`, `hasAnyTokens`, `hasAllTokens`, and `hasPhrase`.","named":{"required":["tokenizer"],"missing":"Text index argument '{}' is required.","positional":"Cannot mix key-value pair and single argument as text index arguments.","unknown":"Unexpected text index arguments: {}.","duplicate":"Text index '{}' argument is specified more than once.","arguments":{"preprocessor":null,"postprocessor":null,"dictionary_block_size":{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Text index argument 'dictionary_block_size' expected to be UInt64, but got Int64.","int":"Text index argument 'dictionary_block_size' must be greater than 0, but got 0.","float":"Text index argument 'dictionary_block_size' expected to be UInt64, but got Float64.","string":"Text index argument 'dictionary_block_size' expected to be UInt64, but got String.","word":"Text index argument 'dictionary_block_size' expected to be UInt64, but got String."},"stringsRefused":true,"wordsRefused":true},"dictionary_block_frontcoding_compression":{"ints":[[-1,false],[0,true],[1,true],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Text index argument 'dictionary_block_frontcoding_compression' expected to be UInt64, but got Int64.","int":"Text index argument 'dictionary_block_frontcoding_compression' must be 0 or 1, but got 2.","float":"Text index argument 'dictionary_block_frontcoding_compression' expected to be UInt64, but got Float64.","string":"Text index argument 'dictionary_block_frontcoding_compression' expected to be UInt64, but got String.","word":"Text index argument 'dictionary_block_frontcoding_compression' expected to be UInt64, but got String."},"stringsRefused":true,"wordsRefused":true},"posting_list_block_size":{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Text index argument 'posting_list_block_size' expected to be UInt64, but got Int64.","int":"Text index argument 'posting_list_block_size' must be greater than 0, but got 0.","float":"Text index argument 'posting_list_block_size' expected to be UInt64, but got Float64.","string":"Text index argument 'posting_list_block_size' expected to be UInt64, but got String.","word":"Text index argument 'posting_list_block_size' expected to be UInt64, but got String."},"stringsRefused":true,"wordsRefused":true},"posting_list_codec":null,"support_phrase_search":{"ints":[[-1,false],[0,true],[1,true],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Text index argument 'support_phrase_search' expected to be UInt64, but got Int64.","int":"Text index argument 'support_phrase_search' must be 0 or 1, but got 2.","float":"Text index argument 'support_phrase_search' expected to be UInt64, but got Float64.","string":"Text index argument 'support_phrase_search' expected to be UInt64, but got String.","word":"Text index argument 'support_phrase_search' expected to be UInt64, but got String."},"stringsRefused":true,"wordsRefused":true}},"tokenizers":["array","asciiCJK","chinese","icu","japanese","keyValuePairs","keyword","ngrambf_v1","ngrams","sparseGrams","sparse_grams","splitByNonAlpha","splitByRegexp","splitByString","tokenbf_v1","unicodeWord","unicode_word"],"unknownTokenizer":"Unknown tokenizer: '{}'."}}, {"name":"tokenbf_v1","syntax":"INDEX name expr TYPE tokenbf_v1(size_in_bytes, num_hash_functions, seed) GRANULARITY g","description":"A Bloom filter over the tokens (words) of the index expression's string values, for speeding up searches of whole tokens.","rule":{"min":3,"max":3,"tooMany":"`tokenbf` index must have exactly 3 arguments.","tooFew":"`tokenbf` index must have exactly 3 arguments.","parameters":[{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"All parameters to *bf_v1 index must be unsigned integers.","int":"The size of bloom filter cannot be zero.","float":"All parameters to *bf_v1 index must be unsigned integers.","string":"All parameters to *bf_v1 index must be unsigned integers.","word":"All parameters to *bf_v1 index must be unsigned integers."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"All parameters to *bf_v1 index must be unsigned integers.","int":"The number of hash functions for bloom filter cannot be zero.","float":"All parameters to *bf_v1 index must be unsigned integers.","string":"All parameters to *bf_v1 index must be unsigned integers.","word":"All parameters to *bf_v1 index must be unsigned integers."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"All parameters to *bf_v1 index must be unsigned integers.","float":"All parameters to *bf_v1 index must be unsigned integers.","string":"All parameters to *bf_v1 index must be unsigned integers.","word":"All parameters to *bf_v1 index must be unsigned integers."},"stringsRefused":true,"wordsRefused":true}]}}, - {"name":"vector_similarity","syntax":"INDEX name vector TYPE vector_similarity('hnsw', 'distance_function', dimensions[, quantization, hnsw_max_connections_per_layer, hnsw_candidate_list_size_for_construction]) GRANULARITY g","description":"An approximate nearest-neighbour index over a vector column (built using HNSW), for speeding up `ORDER BY (vector, reference) LIMIT n` queries."}, + {"name":"vector_similarity","syntax":"INDEX name vector TYPE vector_similarity('hnsw', 'distance_function', dimensions[, quantization, hnsw_max_connections_per_layer, hnsw_candidate_list_size_for_construction]) GRANULARITY g","description":"An approximate nearest-neighbour index over a vector column (built using HNSW), for speeding up `ORDER BY (vector, reference) LIMIT n` queries.","rule":{"min":3,"max":6,"tooMany":"Vector similarity index must have three or six arguments.","tooFew":"Vector similarity index must have three or six arguments.","gaps":[[4,"Vector similarity index must have three or six arguments."],[5,"Vector similarity index must have three or six arguments."]],"parameters":[{"ints":[[-1,false],[0,false],[1,false],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"First argument of vector similarity index (method) must be of type String.","int":"First argument of vector similarity index (method) must be of type String.","float":"First argument of vector similarity index (method) must be of type String.","string":"First argument (method) of vector similarity index is not supported. Supported methods are: hnsw.","word":"First argument (method) of vector similarity index is not supported. Supported methods are: hnsw."},"strings":["hnsw"],"stringsRefused":true},{"ints":[[-1,false],[0,false],[1,false],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Second argument of vector similarity index (metric) must be of type String.","int":"Second argument of vector similarity index (metric) must be of type String.","float":"Second argument of vector similarity index (metric) must be of type String.","string":"Second argument (distance function) of vector similarity index is not supported. Supported distance function are: cosineDistance, dotProduct, L2Distance.","word":"Second argument (distance function) of vector similarity index is not supported. Supported distance function are: cosineDistance, dotProduct, L2Distance."},"strings":["l2distance"]},{"ints":[[-1,false],[0,false],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Third argument of vector similarity index (dimensions) must be of type UInt64.","int":"Third argument (dimensions) of vector similarity index must be > 0.","float":"Third argument of vector similarity index (dimensions) must be of type UInt64.","string":"Third argument of vector similarity index (dimensions) must be of type UInt64.","word":"Third argument of vector similarity index (dimensions) must be of type UInt64."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,false],[1,false],[2,false],[3,false],[4,false],[5,false],[7,false],[8,false],[9,false],[10,false],[11,false],[12,false],[13,false],[15,false],[16,false],[20,false],[21,false],[22,false],[23,false],[24,false],[27,false],[28,false],[29,false],[30,false],[31,false],[32,false],[33,false],[63,false],[64,false],[65,false],[100,false],[127,false],[128,false],[255,false],[256,false],[257,false],[1000,false],[4096,false],[65536,false],[1000000,false]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Fourth argument of vector similarity index (quantization) must be of type String.","int":"Fourth argument of vector similarity index (quantization) must be of type String.","float":"Fourth argument of vector similarity index (quantization) must be of type String.","string":"Fourth argument (quantization) of vector similarity index is not supported. Supported quantizations are: f64, f16, i8, b1, bf16, f32.","word":"Fourth argument (quantization) of vector similarity index is not supported. Supported quantizations are: f64, f16, i8, b1, bf16, f32."},"strings":["bf16"]},{"ints":[[-1,false],[0,true],[1,false],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Fifth argument of vector similarity index (hnsw_max_connections_per_layer) must be of type UInt64.","int":"Invalid parameters passed to vector similarity index. Error: Connectivity must be at least 2, otherwise the index degenerates into ropes.","float":"Fifth argument of vector similarity index (hnsw_max_connections_per_layer) must be of type UInt64.","string":"Fifth argument of vector similarity index (hnsw_max_connections_per_layer) must be of type UInt64.","word":"Fifth argument of vector similarity index (hnsw_max_connections_per_layer) must be of type UInt64."},"stringsRefused":true,"wordsRefused":true},{"ints":[[-1,false],[0,true],[1,true],[2,true],[3,true],[4,true],[5,true],[7,true],[8,true],[9,true],[10,true],[11,true],[12,true],[13,true],[15,true],[16,true],[20,true],[21,true],[22,true],[23,true],[24,true],[27,true],[28,true],[29,true],[30,true],[31,true],[32,true],[33,true],[63,true],[64,true],[65,true],[100,true],[127,true],[128,true],[255,true],[256,true],[257,true],[1000,true],[4096,true],[65536,true],[1000000,true]],"floats":[[-0.5,false],[0,false],[0.001,false],[0.025,false],[0.5,false],[0.999,false],[1,false],[1.5,false],[2,false],[4,false],[8,false]],"messages":{"negative":"Sixth argument of vector similarity index (hnsw_candidate_list_size_for_construction) must be of type UInt64.","float":"Sixth argument of vector similarity index (hnsw_candidate_list_size_for_construction) must be of type UInt64.","string":"Sixth argument of vector similarity index (hnsw_candidate_list_size_for_construction) must be of type UInt64.","word":"Sixth argument of vector similarity index (hnsw_candidate_list_size_for_construction) must be of type UInt64."},"stringsRefused":true,"wordsRefused":true}]}}, ]; export const STATISTICS_TYPES: readonly NamedInfo[] = [ diff --git a/packages/core/src/generated/statements.ts b/packages/core/src/generated/statements.ts new file mode 100644 index 0000000..636e5a2 --- /dev/null +++ b/packages/core/src/generated/statements.ts @@ -0,0 +1,123 @@ +// Generated by scripts/generate-statements.ts from docs/reference/statements of ClickHouse; do not edit. +export interface KeywordDoc { + /** The keywords it documents, such as FLUSH LOGS after SYSTEM, or VALID UNTIL in CREATE USER. */ + keywords: readonly string[]; + markdown: string; +} + +export interface AccessStatementDoc { + /** Such as CREATE USER. */ + statement: string; + markdown: string; + clauses: readonly KeywordDoc[]; +} + +export const SYSTEM_DOCS: readonly KeywordDoc[] = [ + {"keywords":["RELOAD EMBEDDED DICTIONARIES"],"markdown":"**SYSTEM RELOAD EMBEDDED DICTIONARIES**\n\nReload all Internal dictionaries.\nBy default, internal dictionaries are disabled.\nAlways returns `Ok.` regardless of the result of the internal dictionary update."}, + {"keywords":["RELOAD DICTIONARIES"],"markdown":"**SYSTEM RELOAD DICTIONARIES**\n\nThe `SYSTEM RELOAD DICTIONARIES` query reloads dictionaries with a status of `LOADED` (see the `status` column of `system.dictionaries`), i.e dictionaries that have been successfully loaded before.\nBy default, dictionaries are loaded lazily (see dictionaries_lazy_load), so instead of being loaded automatically at startup, they are initialized on first access through use of the `dictGet` function or use of `SELECT` from tables with `ENGINE = Dictionary`.\n\n**Syntax**\n\n```sql\nSYSTEM RELOAD DICTIONARIES [ON CLUSTER cluster_name]\n```"}, + {"keywords":["RELOAD DICTIONARY"],"markdown":"**SYSTEM RELOAD DICTIONARY**\n\nCompletely reloads a dictionary `dictionary_name`, regardless of the state of the dictionary (LOADED / NOT_LOADED / FAILED).\nAlways returns `Ok.` regardless of the result of updating the dictionary.\n\n```sql\nSYSTEM RELOAD DICTIONARY [ON CLUSTER cluster_name] dictionary_name\n```\n\nThe status of the dictionary can be checked by querying the `system.dictionaries` table.\n\n```sql\nSELECT name, status FROM system.dictionaries;\n```"}, + {"keywords":["UNLOAD DICTIONARY"],"markdown":"**SYSTEM UNLOAD DICTIONARY**\n\nUnloads a dictionary `dictionary_name` to release its memory, if the dictionary status is `LOADED`.\nThe dictionary is lazy-reloaded when necessary again.\n\n```sql\nSYSTEM UNLOAD DICTIONARY dictionary_name\n```\n\nThe status of the dictionary can be checked by querying the `system.dictionaries` table.\n\n```sql\nSELECT name, status FROM system.dictionaries;\n```"}, + {"keywords":["UNLOAD DICTIONARIES"],"markdown":"**SYSTEM UNLOAD DICTIONARIES**\n\nThe `SYSTEM UNLOAD DICTIONARIES` query unloads all dictionaries with a `LOADED` status (see the `status` column of `system.dictionaries`), i.e dictionaries that have been successfully loaded before.\n\n```sql\nSYSTEM UNLOAD DICTIONARIES\n```"}, + {"keywords":["RELOAD FUNCTIONS","RELOAD FUNCTION"],"markdown":"**SYSTEM RELOAD FUNCTIONS**\n\nReloads all registered executable user defined functions or one of them from a configuration file.\n\n**Syntax**\n\n```sql\nSYSTEM RELOAD FUNCTIONS [ON CLUSTER cluster_name]\nSYSTEM RELOAD FUNCTION [ON CLUSTER cluster_name] function_name\n```"}, + {"keywords":["RELOAD ASYNCHRONOUS METRICS"],"markdown":"**SYSTEM RELOAD ASYNCHRONOUS METRICS**\n\nRe-calculates all asynchronous metrics. Since asynchronous metrics are periodically updated based on setting asynchronous_metrics_update_period_s, updating them manually using this statement is typically not necessary.\n\n```sql\nSYSTEM RELOAD ASYNCHRONOUS METRICS [ON CLUSTER cluster_name]\n```"}, + {"keywords":["CLEAR DNS CACHE","DROP DNS CACHE"],"markdown":"**SYSTEM CLEAR|DROP DNS CACHE**\n\nClears ClickHouse's internal DNS cache. Sometimes (for old ClickHouse versions) it is necessary to use this command when changing the infrastructure (changing the IP address of another ClickHouse server or the server used by dictionaries).\n\nFor more convenient (automatic) cache management, see `disable_internal_dns_cache`, `dns_cache_max_entries`, `dns_cache_update_period` parameters."}, + {"keywords":["CLEAR MARK CACHE","DROP MARK CACHE"],"markdown":"**SYSTEM CLEAR|DROP MARK CACHE**\n\nClears the mark cache."}, + {"keywords":["CLEAR PRIMARY INDEX CACHE","DROP PRIMARY INDEX CACHE"],"markdown":"**SYSTEM CLEAR|DROP PRIMARY INDEX CACHE**\n\nClears the primary index cache, which holds the primary keys of `MergeTree` tables in memory.\nIts size is configured with the server-level setting `primary_index_cache_size`."}, + {"keywords":["CLEAR STATISTICS CACHE","DROP STATISTICS CACHE"],"markdown":"**SYSTEM CLEAR|DROP STATISTICS CACHE**\n\nClears the statistics cache, which holds the column statistics of the data parts of `MergeTree` tables in memory, one entry per data part and column.\nIts size is configured with the server-level setting `statistics_cache_size`."}, + {"keywords":["CLEAR ICEBERG METADATA CACHE","DROP ICEBERG METADATA CACHE"],"markdown":"**SYSTEM CLEAR|DROP ICEBERG METADATA CACHE**\n\nClears the iceberg metadata cache."}, + {"keywords":["CLEAR AVRO SCHEMA CACHE","DROP AVRO SCHEMA CACHE"],"markdown":"**SYSTEM CLEAR|DROP AVRO SCHEMA CACHE**\n\nClears the per-URL Confluent Schema Registry caches used by the `AvroConfluent` format. This drops both the schema-fetch cache (id → schema) and the schema-registration cache (subject + schema → id), so subsequent reads and writes fall back to the registry server. Useful when a schema was deleted or rewritten on the registry side, or to verify the registry's idempotency in tests."}, + {"keywords":["DROP PARQUET METADATA CACHE"],"markdown":"**SYSTEM DROP PARQUET METADATA CACHE**\n\nClears the parquet metadata cache."}, + {"keywords":["CLEAR PAIMON METADATA CACHE","DROP PAIMON METADATA CACHE"],"markdown":"**SYSTEM CLEAR|DROP PAIMON METADATA CACHE**\n\nClears the in-memory cache of parsed Paimon metadata files (manifest lists and manifests)."}, + {"keywords":["CLEAR POINT IN POLYGON CACHE","DROP POINT IN POLYGON CACHE"],"markdown":"**SYSTEM CLEAR|DROP POINT IN POLYGON CACHE**\n\nClears the cache of preprocessed constant polygons used by the function `pointInPolygon`. The configured size limit (the `point_in_polygon_cache_size` server setting) is left unchanged, so the cache keeps accepting entries afterwards. To disable the cache instead, set `point_in_polygon_cache_size` to `0`."}, + {"keywords":["CLEAR TEXT INDEX CACHES","DROP TEXT INDEX CACHES"],"markdown":"**SYSTEM CLEAR|DROP TEXT INDEX CACHES**\n\nClears the text index's tokens, header and postings caches.\n\nIf you like to drop one of these caches individually, you can run\n\n- `SYSTEM CLEAR TEXT INDEX TOKENS CACHE`,\n- `SYSTEM CLEAR TEXT INDEX HEADER CACHE`, or\n- `SYSTEM CLEAR TEXT INDEX POSTINGS CACHE`"}, + {"keywords":["CLEAR INDEX MARK CACHE","DROP INDEX MARK CACHE"],"markdown":"**SYSTEM CLEAR|DROP INDEX MARK CACHE**\n\nClears the cache of marks for secondary (data-skipping) indexes."}, + {"keywords":["CLEAR INDEX UNCOMPRESSED CACHE","DROP INDEX UNCOMPRESSED CACHE"],"markdown":"**SYSTEM CLEAR|DROP INDEX UNCOMPRESSED CACHE**\n\nClears the cache of uncompressed blocks for secondary (data-skipping) indexes."}, + {"keywords":["CLEAR MMAP CACHE","DROP MMAP CACHE"],"markdown":"**SYSTEM CLEAR|DROP MMAP CACHE**\n\nClears the cache of memory-mapped files."}, + {"keywords":["CLEAR PAGE CACHE","DROP PAGE CACHE"],"markdown":"**SYSTEM CLEAR|DROP PAGE CACHE**\n\nClears the userspace page cache, ClickHouse's own in-memory cache of data read from the underlying storage."}, + {"keywords":["CLEAR VECTOR SIMILARITY INDEX CACHE","DROP VECTOR SIMILARITY INDEX CACHE"],"markdown":"**SYSTEM CLEAR|DROP VECTOR SIMILARITY INDEX CACHE**\n\nClears the vector similarity index cache."}, + {"keywords":["CLEAR CONNECTIONS CACHE","DROP CONNECTIONS CACHE"],"markdown":"**SYSTEM CLEAR|DROP CONNECTIONS CACHE**\n\nClears the cache of HTTP connection pools used for outgoing connections."}, + {"keywords":["CLEAR S3 CLIENT CACHE","DROP S3 CLIENT CACHE"],"markdown":"**SYSTEM CLEAR|DROP S3 CLIENT CACHE**\n\nClears the cache of S3 clients."}, + {"keywords":["PREWARM MARK CACHE"],"markdown":"**SYSTEM PREWARM MARK CACHE**\n\nLoads the marks of a table into the mark cache. Secondary-index marks are also loaded into the index mark cache.\n\n```sql\nSYSTEM PREWARM MARK CACHE [ON CLUSTER cluster_name] [db.]table\n```"}, + {"keywords":["PREWARM PRIMARY INDEX CACHE"],"markdown":"**SYSTEM PREWARM PRIMARY INDEX CACHE**\n\nLoads the primary indexes of a `MergeTree` table into the primary index cache.\n\n```sql\nSYSTEM PREWARM PRIMARY INDEX CACHE [ON CLUSTER cluster_name] [db.]table\n```"}, + {"keywords":["CLEAR DISK METADATA CACHE","DROP DISK METADATA CACHE"],"markdown":"**SYSTEM CLEAR|DROP DISK METADATA CACHE**\n\nClears the metadata cache of the specified disk.\n\n```sql\nSYSTEM DROP DISK METADATA CACHE \n```"}, + {"keywords":["CLEAR TIME SERIES CACHES","DROP TIME SERIES CACHES"],"markdown":"**SYSTEM CLEAR|DROP TIME SERIES CACHES**\n\nClears the caches of a `TimeSeries` table which let inserts skip the rows already present in its target tables. There are two such caches: the deduplication cache of the metric families table and the deduplication cache of the tags table, see the `metric_families_deduplication_cache_expiration_seconds` and `tags_deduplication_cache_expiration_seconds` settings of the table. The caches are local to the server and they don't notice changes made to the target tables directly. Such a difference disappears by itself when the entries expire; use the statement to make the next insert write all its rows without waiting (for example, after truncating the metric families table). `TRUNCATE TABLE` of a `TimeSeries` table clears the caches of the executing server only, so if the target tables are replicated, run the statement `ON CLUSTER` afterwards to clear the caches of the other replicas.\n\n```sql\nSYSTEM DROP TIME SERIES CACHES [ON CLUSTER cluster_name] [db.]table\n```"}, + {"keywords":["SYNC FILESYSTEM CACHE"],"markdown":"**SYSTEM SYNC FILESYSTEM CACHE**\n\nReconciles ClickHouse's in-memory state of the filesystem cache with the cache files actually present on disk, and returns the `cache_name`, `path` and downloaded `size` of each cached file segment. An optional cache name limits the operation to a single cache.\n\n```sql\nSYSTEM SYNC FILESYSTEM CACHE ['']\n```"}, + {"keywords":["CLEAR DISTRIBUTED CACHE","DROP DISTRIBUTED CACHE"],"markdown":"**SYSTEM CLEAR|DROP DISTRIBUTED CACHE**\n\n**Note**\n`SYSTEM CLEAR|DROP DISTRIBUTED CACHE` is available only in ClickHouse Cloud.\n\nDrops the distributed cache. Use `CONNECTIONS` to drop only the cached connections to the distributed cache servers, or pass a server identifier to target a single server.\n\n```sql\nSYSTEM DROP DISTRIBUTED CACHE [CONNECTIONS | 'server_id']\n```"}, + {"keywords":["DROP REPLICA"],"markdown":"**SYSTEM DROP REPLICA**\n\nDead replicas of `ReplicatedMergeTree` tables can be dropped using following syntax:\n\n```sql\nSYSTEM DROP REPLICA 'replica_name' FROM TABLE database.table;\nSYSTEM DROP REPLICA 'replica_name' FROM DATABASE database;\nSYSTEM DROP REPLICA 'replica_name';\nSYSTEM DROP REPLICA 'replica_name' FROM ZKPATH '/path/to/table/in/zk';\n```\n\nQueries will remove the `ReplicatedMergeTree` replica path in ZooKeeper. It is useful when the replica is dead and its metadata cannot be removed from ZooKeeper by `DROP TABLE` because there is no such table anymore. It will only drop the inactive/stale replica, and it cannot drop local replica, please use `DROP TABLE` for that. `DROP REPLICA` does not drop any tables and does not remove any data or metadata from disk.\n\nThe first one removes metadata of `'replica_name'` replica of `database.table` table.\nThe second one does the same for all replicated tables in the database.\nThe third one does the same for all replicated tables on the local server.\nThe fourth one is useful to remove metadata of dead replica when all other replicas of a table were dropped. It requires the table path to be specified explicitly. It must be the same path as was passed to the first argument of `ReplicatedMergeTree` engine on table creation."}, + {"keywords":["DROP DATABASE REPLICA"],"markdown":"**SYSTEM DROP DATABASE REPLICA**\n\nDead replicas of `Replicated` databases can be dropped using following syntax:\n\n```sql\nSYSTEM DROP DATABASE REPLICA 'replica_name' [FROM SHARD 'shard_name'] FROM DATABASE database;\nSYSTEM DROP DATABASE REPLICA 'replica_name' [FROM SHARD 'shard_name'];\nSYSTEM DROP DATABASE REPLICA 'replica_name' [FROM SHARD 'shard_name'] FROM ZKPATH '/path/to/table/in/zk';\n```\n\nSimilar to `SYSTEM DROP REPLICA`, but removes the `Replicated` database replica path from ZooKeeper when there's no database to run `DROP DATABASE`. Please note that it does not remove `ReplicatedMergeTree` replicas (so you may need `SYSTEM DROP REPLICA` as well). Shard and replica names are the names that were specified in `Replicated` engine arguments when creating the database. Also, these names can be obtained from `database_shard_name` and `database_replica_name` columns in `system.clusters`. If the `FROM SHARD` clause is missing, then `replica_name` must be a full replica name in `shard_name|replica_name` format."}, + {"keywords":["CLEAR UNCOMPRESSED CACHE","DROP UNCOMPRESSED CACHE"],"markdown":"**SYSTEM CLEAR|DROP UNCOMPRESSED CACHE**\n\nClears the uncompressed data cache.\nThe uncompressed data cache is enabled/disabled with the query/user/profile-level setting `use_uncompressed_cache`.\nIts size can be configured using the server-level setting `uncompressed_cache_size`."}, + {"keywords":["CLEAR COLUMNS CACHE","DROP COLUMNS CACHE"],"markdown":"**SYSTEM CLEAR|DROP COLUMNS CACHE**\n\nClears the columns cache (deserialized columns kept in memory).\nReads that are already in progress when the cache is cleared cannot put their data back into the cache, also from the parts of the data they reach only after the clear, so the cache stays cleared even under load.\nThe columns cache is enabled/disabled with the query/user/profile-level setting `use_columns_cache`.\nIts size can be configured using the server-level setting `columns_cache_size`.\nUse `system.columns_cache` to introspect cache contents."}, + {"keywords":["CLEAR COMPILED EXPRESSION CACHE","DROP COMPILED EXPRESSION CACHE"],"markdown":"**SYSTEM CLEAR|DROP COMPILED EXPRESSION CACHE**\n\nClears the compiled expression cache.\nThe compiled expression cache is enabled/disabled with the query/user/profile-level setting `compile_expressions`."}, + {"keywords":["CLEAR QUERY CONDITION CACHE","DROP QUERY CONDITION CACHE"],"markdown":"**SYSTEM CLEAR|DROP QUERY CONDITION CACHE**\n\nClears the query condition cache."}, + {"keywords":["CLEAR ENCRYPTION HEADERS CACHE","DROP ENCRYPTION HEADERS CACHE"],"markdown":"**SYSTEM CLEAR|DROP ENCRYPTION HEADERS CACHE**\n\nClears the encryption headers cache. This cache holds the encryption headers read from the front of encrypted files and is used by the experimental `use_reader_executor` read path to avoid re-reading them; its size is configured by the `encryption_header_cache_size` server setting."}, + {"keywords":["CLEAR QUERY CACHE","DROP QUERY CACHE","CLEAR QUERY","CLEAR QUERY CACHE TAG"],"markdown":"**SYSTEM CLEAR|DROP QUERY CACHE**\n\n```sql\nSYSTEM CLEAR QUERY CACHE;\nSYSTEM CLEAR QUERY CACHE TAG ''\n````\n\nClears the query cache.\nIf a tag is specified, only query cache entries with the specified tag are deleted."}, + {"keywords":["CLEAR FORMAT SCHEMA CACHE","DROP FORMAT SCHEMA CACHE"],"markdown":"**SYSTEM CLEAR|DROP FORMAT SCHEMA CACHE**\n\nClears cache for schemas loaded from `format_schema_path`.\n\nSupported targets:\n- Protobuf: Removes imported Protobuf message definitions from memory.\n- Files: Deletes cached schema files stored locally in the `format_schema_path`, generated when `format_schema_source` is set to `query`.\nNote: If no target is specified, both caches are cleared.\n\n```sql\nSYSTEM CLEAR|DROP FORMAT SCHEMA CACHE [FOR Protobuf/Files]\n```"}, + {"keywords":["FLUSH LOGS"],"markdown":"**SYSTEM FLUSH LOGS**\n\nFlushes buffered log messages to system tables, e.g. system.query_log. Mainly useful for debugging since most system tables have a default flush interval of 7.5 seconds.\nThis will also create system tables even if message queue is empty.\n\n```sql\nSYSTEM FLUSH LOGS [ON CLUSTER cluster_name] [log_name|[database.table]] [, ...]\n```\n\nIf you don't want to flush everything, you can flush one or more individual logs by passing either their name or their target table:\n\n```sql\nSYSTEM FLUSH LOGS query_log, system.query_views_log;\n```"}, + {"keywords":["RELOAD CONFIG"],"markdown":"**SYSTEM RELOAD CONFIG**\n\nReloads ClickHouse configuration. Used when configuration is stored in ZooKeeper. Note that `SYSTEM RELOAD CONFIG` does not reload `USER` configuration stored in ZooKeeper, it only reloads `USER` configuration that is stored in `users.xml`. To reload all `USER` config use `SYSTEM RELOAD USERS`\n\n```sql\nSYSTEM RELOAD CONFIG [ON CLUSTER cluster_name]\n```"}, + {"keywords":["RELOAD USERS"],"markdown":"**SYSTEM RELOAD USERS**\n\nReloads all access storages, including: users.xml, local disk access storage, replicated (in ZooKeeper) access storage.\n\n```sql\nSYSTEM RELOAD USERS [ON CLUSTER cluster_name]\n```"}, + {"keywords":["SHUTDOWN"],"markdown":"**SYSTEM SHUTDOWN**\n\nNormally shuts down ClickHouse (like `service clickhouse-server stop` / `kill {$pid_clickhouse-server}`)"}, + {"keywords":["KILL"],"markdown":"**SYSTEM KILL**\n\nAborts ClickHouse process (like `kill -9 {$ pid_clickhouse-server}`)"}, + {"keywords":["INSTRUMENT"],"markdown":"**SYSTEM INSTRUMENT**\n\nManages instrumentation points using LLVM's XRay feature which is available when ClickHouse is built using `ENABLE_XRAY=1`.\nThis enables to debug and profile in production without modifying the source code and with minimal overhead.\nWhen no instrumentation point is added, the performance penalty is negligible because it only adds an extra jump to a nearby\naddress at the prolog and epilog of those functions that are longer than 200 instructions."}, + {"keywords":["INSTRUMENT ADD","INSTRUMENT ADD FUNCTION HANDLER","INSTRUMENT ADD FUNCTION HANDLER ARGUMENTS"],"markdown":"**SYSTEM INSTRUMENT ADD**\n\nAdds a new instrumentation point. Functions instrumented can be inspected in the `system.instrumentation` system table. More than one handler can be added for the same function, and they will be executed in the same order the instrumentation is added.\nThe functions to be instrumented can be collected from `system.symbols` system table.\n\nThere are three different kind of handlers to add to functions:\n\n**Syntax**\n```sql\nSYSTEM INSTRUMENT ADD FUNCTION HANDLER [ARGUMENTS]\n```\n\nwhere `FUNCTION` is any function or substring of a function such as `QueryMetricLog::startQuery`, and the handler one of the following"}, + {"keywords":["INSTRUMENT REMOVE","INSTRUMENT REMOVE ID","INSTRUMENT REMOVE ALL"],"markdown":"**SYSTEM INSTRUMENT REMOVE**\n\nRemoves either a single instrumentation point with:\n\n```sql\nSYSTEM INSTRUMENT REMOVE ID\n```\n\nall of them using the `ALL` keyword:\n\n```sql\nSYSTEM INSTRUMENT REMOVE ALL\n```\n\na set of IDs from a subquery:\n\n```sql\nSYSTEM INSTRUMENT REMOVE (SELECT id FROM system.instrumentation WHERE handler = 'log')\n```\n\nor all instrumentation points that match a given function_name:\n\n```sql\nSYSTEM INSTRUMENT REMOVE 'QueryMetricLog::startQuery'\n```\n\nThe instrumentation point information can be collected from `system.instrumentation` system table."}, + {"keywords":["STOP DISTRIBUTED SENDS"],"markdown":"**SYSTEM STOP DISTRIBUTED SENDS**\n\nDisables background data distribution when inserting data into distributed tables.\n\n```sql\nSYSTEM STOP DISTRIBUTED SENDS [db.] [ON CLUSTER cluster_name]\n```\n\n**Note**\nIn case of `prefer_localhost_replica` is enabled (the default), the data to local shard will be inserted anyway."}, + {"keywords":["FLUSH DISTRIBUTED"],"markdown":"**SYSTEM FLUSH DISTRIBUTED**\n\nForces ClickHouse to send data to cluster nodes synchronously. If any nodes are unavailable, ClickHouse throws an exception and stops query execution. You can retry the query until it succeeds, which will happen when all nodes are back online.\n\nYou can also override some settings via `SETTINGS` clause, this can be useful to avoid some temporary limitations, like `max_concurrent_queries_for_all_users` or `max_memory_usage`.\n\n```sql\nSYSTEM FLUSH DISTRIBUTED [db.] [ON CLUSTER cluster_name] [SETTINGS ...]\n```\n\n**Note**\nEach pending block is stored in disk with settings from the initial INSERT query, so that is why sometimes you may want to override settings."}, + {"keywords":["START DISTRIBUTED SENDS"],"markdown":"**SYSTEM START DISTRIBUTED SENDS**\n\nEnables background data distribution when inserting data into distributed tables.\n\n```sql\nSYSTEM START DISTRIBUTED SENDS [db.] [ON CLUSTER cluster_name]\n```"}, + {"keywords":["STOP LISTEN"],"markdown":"**SYSTEM STOP LISTEN**\n\nCloses the socket and gracefully terminates the existing connections to the server on the specified port with the specified protocol.\n\nHowever, if the corresponding protocol settings were not specified in the clickhouse-server configuration, this command will have no effect.\n\n```sql\nSYSTEM STOP LISTEN [ON CLUSTER cluster_name] [QUERIES ALL | QUERIES DEFAULT | QUERIES CUSTOM | TCP | TCP WITH PROXY | TCP SECURE | HTTP | HTTPS | MYSQL | GRPC | POSTGRESQL | PROMETHEUS | CUSTOM 'protocol']\n```\n\n- If `CUSTOM 'protocol'` modifier is specified, the custom protocol with the specified name defined in the protocols section of the server configuration will be stopped.\n- If `QUERIES ALL [EXCEPT .. [,..]]` modifier is specified, all protocols are stopped, unless specified with `EXCEPT` clause.\n- If `QUERIES DEFAULT [EXCEPT .. [,..]]` modifier is specified, all default protocols are stopped, unless specified with `EXCEPT` clause.\n- If `QUERIES CUSTOM [EXCEPT .. [,..]]` modifier is specified, all custom protocols are stopped, unless specified with `EXCEPT` clause."}, + {"keywords":["START LISTEN"],"markdown":"**SYSTEM START LISTEN**\n\nAllows new connections to be established on the specified protocols.\n\nHowever, if the server on the specified port and protocol was not stopped using the SYSTEM STOP LISTEN command, this command will have no effect.\n\n```sql\nSYSTEM START LISTEN [ON CLUSTER cluster_name] [QUERIES ALL | QUERIES DEFAULT | QUERIES CUSTOM | TCP | TCP WITH PROXY | TCP SECURE | HTTP | HTTPS | MYSQL | GRPC | POSTGRESQL | PROMETHEUS | CUSTOM 'protocol']\n```"}, + {"keywords":["STOP MERGES"],"markdown":"**SYSTEM STOP MERGES**\n\nProvides possibility to stop background merges for tables in the MergeTree family:\n\n```sql\nSYSTEM STOP MERGES [ON CLUSTER cluster_name] [ON VOLUME | [db.]merge_tree_family_table_name]\n```\n\n**Note**\n`DETACH / ATTACH` table will start background merges for the table even in case when merges have been stopped for all MergeTree tables before.\n\nWithout `ON VOLUME`, merges are stopped only for the tables that exist when the statement runs, and only until the server restarts:\n\n- Without a table name (and without `ON VOLUME`), merges are stopped for every MergeTree table that exists at that moment. Tables created afterwards still merge.\n- With a table name, the statement returns `Ok.` even if the table does not exist, and it has no effect on a table created later with that name.\n- The stopped state is kept in memory. After a server restart, merges run again for all tables.\n\nWhile merges are stopped for a non-replicated MergeTree table by `SYSTEM STOP MERGES [db.]merge_tree_family_table_name`, `OPTIMIZE TABLE` on that table also fails, with or without `FINAL`, with the error `Cancelled merging parts. (ABORTED)`. Run `SYSTEM START MERGES` for the table first."}, + {"keywords":["START MERGES"],"markdown":"**SYSTEM START MERGES**\n\nProvides possibility to start background merges for tables in the MergeTree family:\n\n```sql\nSYSTEM START MERGES [ON CLUSTER cluster_name] [ON VOLUME | [db.]merge_tree_family_table_name]\n```"}, + {"keywords":["STOP TTL MERGES"],"markdown":"**SYSTEM STOP TTL MERGES**\n\nProvides possibility to stop background delete old data according to TTL expression for tables in the MergeTree family:\nReturns `Ok.` even if table does not exist or table has not MergeTree engine. Returns error when database does not exist:\n\n```sql\nSYSTEM STOP TTL MERGES [ON CLUSTER cluster_name] [[db.]merge_tree_family_table_name]\n```"}, + {"keywords":["START TTL MERGES"],"markdown":"**SYSTEM START TTL MERGES**\n\nProvides possibility to start background delete old data according to TTL expression for tables in the MergeTree family:\nReturns `Ok.` even if table does not exist. Returns error when database does not exist:\n\n```sql\nSYSTEM START TTL MERGES [ON CLUSTER cluster_name] [[db.]merge_tree_family_table_name]\n```"}, + {"keywords":["STOP MOVES"],"markdown":"**SYSTEM STOP MOVES**\n\nProvides possibility to stop background move data according to TTL table expression with TO VOLUME or TO DISK clause for tables in the MergeTree family:\nReturns `Ok.` even if table does not exist. Returns error when database does not exist:\n\n```sql\nSYSTEM STOP MOVES [ON CLUSTER cluster_name] [[db.]merge_tree_family_table_name]\n```"}, + {"keywords":["START MOVES"],"markdown":"**SYSTEM START MOVES**\n\nProvides possibility to start background move data according to TTL table expression with TO VOLUME and TO DISK clause for tables in the MergeTree family:\nReturns `Ok.` even if table does not exist. Returns error when database does not exist:\n\n```sql\nSYSTEM START MOVES [ON CLUSTER cluster_name] [[db.]merge_tree_family_table_name]\n```"}, + {"keywords":["UNFREEZE","UNFREEZE WITH NAME"],"markdown":"**SYSTEM UNFREEZE**\n\nClears a frozen backup with the specified name from all the disks. See more about unfreezing separate parts in ALTER TABLE table_name UNFREEZE WITH NAME \n\n```sql\nSYSTEM UNFREEZE WITH NAME \n```"}, + {"keywords":["WAIT LOADING PARTS"],"markdown":"**SYSTEM WAIT LOADING PARTS**\n\nWait until all asynchronously loading data parts of a table (outdated data parts) will became loaded.\n\n```sql\nSYSTEM WAIT LOADING PARTS [ON CLUSTER cluster_name] [db.]merge_tree_family_table_name\n```"}, + {"keywords":["STOP FETCHES"],"markdown":"**SYSTEM STOP FETCHES**\n\nProvides possibility to stop background fetches for inserted parts for tables in the `ReplicatedMergeTree` family:\nAlways returns `Ok.` regardless of the table engine and even if table or database does not exist.\n\n```sql\nSYSTEM STOP FETCHES [ON CLUSTER cluster_name] [[db.]replicated_merge_tree_family_table_name]\n```"}, + {"keywords":["START FETCHES"],"markdown":"**SYSTEM START FETCHES**\n\nProvides possibility to start background fetches for inserted parts for tables in the `ReplicatedMergeTree` family:\nAlways returns `Ok.` regardless of the table engine and even if table or database does not exist.\n\n```sql\nSYSTEM START FETCHES [ON CLUSTER cluster_name] [[db.]replicated_merge_tree_family_table_name]\n```"}, + {"keywords":["STOP REPLICATED SENDS"],"markdown":"**SYSTEM STOP REPLICATED SENDS**\n\nProvides possibility to stop background sends to other replicas in cluster for new inserted parts for tables in the `ReplicatedMergeTree` family:\n\n```sql\nSYSTEM STOP REPLICATED SENDS [ON CLUSTER cluster_name] [[db.]replicated_merge_tree_family_table_name]\n```"}, + {"keywords":["START REPLICATED SENDS"],"markdown":"**SYSTEM START REPLICATED SENDS**\n\nProvides possibility to start background sends to other replicas in cluster for new inserted parts for tables in the `ReplicatedMergeTree` family:\n\n```sql\nSYSTEM START REPLICATED SENDS [ON CLUSTER cluster_name] [[db.]replicated_merge_tree_family_table_name]\n```"}, + {"keywords":["STOP REPLICATION QUEUES"],"markdown":"**SYSTEM STOP REPLICATION QUEUES**\n\nProvides possibility to stop background fetch tasks from replication queues which stored in Zookeeper for tables in the `ReplicatedMergeTree` family. Possible background tasks types - merges, fetches, mutation, DDL statements with ON CLUSTER clause:\n\n```sql\nSYSTEM STOP REPLICATION QUEUES [ON CLUSTER cluster_name] [[db.]replicated_merge_tree_family_table_name]\n```"}, + {"keywords":["START REPLICATION QUEUES"],"markdown":"**SYSTEM START REPLICATION QUEUES**\n\nProvides possibility to start background fetch tasks from replication queues which stored in Zookeeper for tables in the `ReplicatedMergeTree` family. Possible background tasks types - merges, fetches, mutation, DDL statements with ON CLUSTER clause:\n\n```sql\nSYSTEM START REPLICATION QUEUES [ON CLUSTER cluster_name] [[db.]replicated_merge_tree_family_table_name]\n```"}, + {"keywords":["STOP REPLICATED DDL QUERIES"],"markdown":"**SYSTEM STOP REPLICATED DDL QUERIES**\n\nTemporarily prohibits DDL queries in `Replicated` databases and distributed `ON CLUSTER` DDL queries on the server where it is executed:\n\n```sql\nSYSTEM STOP REPLICATED DDL QUERIES [ON CLUSTER cluster_name]\n```\n\nWhile DDL queries are stopped:\n\n- DDL queries in `Replicated` databases fail with the `QUERY_IS_PROHIBITED` error. This includes DDL queries that were initiated on this server and are already queued, but not yet committed.\n- All `ON CLUSTER` DDL queries except `SYSTEM` queries fail with the `QUERY_IS_PROHIBITED` error when initiated on this server, regardless of the engine of the target database (not only for `Replicated` databases).\n- DDL queries initiated on other replicas are still applied on this server: the statement prevents this server from initiating DDL, it does not stop the execution of the DDL queue.\n\nWithout `ON CLUSTER`, the statement affects only the local server. With `ON CLUSTER`, it is executed on every host of the cluster (`SYSTEM` queries are not rejected while DDL queries are stopped, so `SYSTEM START REPLICATED DDL QUERIES ON CLUSTER` still works).\n\nThe prohibition is not persistent: it is stored in memory and in ephemeral nodes in Keeper, and is lifted when the server restarts. Do not rely on it as a durable maintenance fence; execute it again after a restart if needed.\n\nThe statement requires the global `SYSTEM REPLICATION QUEUES` privilege."}, + {"keywords":["START REPLICATED DDL QUERIES"],"markdown":"**SYSTEM START REPLICATED DDL QUERIES**\n\nCancels `SYSTEM STOP REPLICATED DDL QUERIES` and allows DDL queries in `Replicated` databases and distributed `ON CLUSTER` DDL queries again:\n\n```sql\nSYSTEM START REPLICATED DDL QUERIES [ON CLUSTER cluster_name]\n```\n\nThe statement requires the global `SYSTEM REPLICATION QUEUES` privilege."}, + {"keywords":["STOP PULLING REPLICATION LOG"],"markdown":"**SYSTEM STOP PULLING REPLICATION LOG**\n\nStops loading new entries from replication log to replication queue in a `ReplicatedMergeTree` table.\n\n```sql\nSYSTEM STOP PULLING REPLICATION LOG [ON CLUSTER cluster_name] [[db.]replicated_merge_tree_family_table_name]\n```"}, + {"keywords":["START PULLING REPLICATION LOG"],"markdown":"**SYSTEM START PULLING REPLICATION LOG**\n\nCancels `SYSTEM STOP PULLING REPLICATION LOG`.\n\n```sql\nSYSTEM START PULLING REPLICATION LOG [ON CLUSTER cluster_name] [[db.]replicated_merge_tree_family_table_name]\n```"}, + {"keywords":["SYNC REPLICA"],"markdown":"**SYSTEM SYNC REPLICA**\n\nWait until a `ReplicatedMergeTree` table will be synced with other replicas in a cluster, but no more than `receive_timeout` seconds.\n\n```sql\nSYSTEM SYNC REPLICA [ON CLUSTER cluster_name] [db.]replicated_merge_tree_family_table_name [IF EXISTS] [STRICT | LIGHTWEIGHT [FROM 'srcReplica1'[, 'srcReplica2'[, ...]]] | PULL]\n```\n\nAfter running this statement the `[db.]replicated_merge_tree_family_table_name` fetches commands from the common replicated log into its own replication queue, and then the query waits till the replica processes all of the fetched commands. The following modifiers are supported:\n\n- With `IF EXISTS` (available since 25.6) the query won't throw an error if the table does not exists. This is useful when adding a new replica to a cluster, when it's already part of the cluster configuration but it is still in the process of creating and synchronizing the table.\n- If a `STRICT` modifier was specified then the query waits for the replication queue to become empty. The `STRICT` version may never succeed if new entries constantly appear in the replication queue.\n- If a `LIGHTWEIGHT` modifier was specified then the query waits only for `GET_PART`, `ATTACH_PART`, `DROP_RANGE`, `REPLACE_RANGE` and `DROP_PART` entries to be processed.\n Additionally, the LIGHTWEIGHT modifier supports an optional FROM 'srcReplicas' clause, where 'srcReplicas' is a comma-separated list of source replica names. This extension allows for more targeted synchronization by focusing only on replication tasks originating from the specified source replicas.\n- If a `PULL` modifier was specified then the query pulls new replication queue entries from ZooKeeper, but does not wait for anything to be processed."}, + {"keywords":["SYNC DATABASE REPLICA"],"markdown":"**SYNC DATABASE REPLICA**\n\nWaits until the specified replicated database applies all schema changes from the DDL queue of that database.\n\n**Syntax**\n```sql\nSYSTEM SYNC DATABASE REPLICA replicated_database_name;\n```"}, + {"keywords":["RESTART REPLICA"],"markdown":"**SYSTEM RESTART REPLICA**\n\nProvides possibility to reinitialize Zookeeper session's state for `ReplicatedMergeTree` table, will compare current state with Zookeeper as source of truth and add tasks to Zookeeper queue if needed.\nInitialization of replication queue based on ZooKeeper data happens in the same way as for `ATTACH TABLE` statement. For a short time, the table will be unavailable for any operations.\n\n```sql\nSYSTEM RESTART REPLICA [ON CLUSTER cluster_name] [db.]replicated_merge_tree_family_table_name\n```"}, + {"keywords":["RESTORE REPLICA"],"markdown":"**SYSTEM RESTORE REPLICA**\n\nRestores a replica if data is [possibly] present but Zookeeper metadata is lost.\n\nWorks only on readonly `ReplicatedMergeTree` tables.\n\nOne may execute query after:\n\n- ZooKeeper root `/` loss.\n- Replicas path `/replicas` loss.\n- Individual replica path `/replicas/replica_name/` loss.\n\nReplica attaches locally found parts and sends info about them to Zookeeper.\nParts present on a replica before metadata loss are not re-fetched from other ones if not being outdated (so replica restoration does not mean re-downloading all data over the network).\n\n**Note**\nParts in all states are moved to `detached/` folder. Parts active before data loss (committed) are attached."}, + {"keywords":["RESTORE DATABASE REPLICA"],"markdown":"**SYSTEM RESTORE DATABASE REPLICA**\n\nRestores a replica if data is [possibly] present but Zookeeper metadata is lost.\n\n**Syntax**\n\n```sql\nSYSTEM RESTORE DATABASE REPLICA repl_db [ON CLUSTER cluster]\n```\n\n**Example**\n\n```sql\nCREATE DATABASE repl_db\nENGINE=Replicated(\"/clickhouse/repl_db\", shard1, replica1);\n\nCREATE TABLE repl_db.test_table (n UInt32)\nENGINE = ReplicatedMergeTree\nORDER BY n PARTITION BY n % 10;\n\n-- zookeeper_delete_path(\"/clickhouse/repl_db\", recursive=True) <- root loss.\n\nSYSTEM RESTORE DATABASE REPLICA repl_db;\n```\n\n**Syntax**\n\n```sql\nSYSTEM RESTORE REPLICA [db.]replicated_merge_tree_family_table_name [ON CLUSTER cluster_name]\n```\n\nAlternative syntax:\n\n```sql\nSYSTEM RESTORE REPLICA [ON CLUSTER cluster_name] [db.]replicated_merge_tree_family_table_name\n```\n\n**Example**\n\nCreating a table on multiple servers. After the replica's metadata in ZooKeeper is lost, the table will attach as read-only as metadata is missing. The last query needs to execute on every replica.\n\n```sql\nCREATE TABLE test(n UInt32)\nENGINE = ReplicatedMergeTree('/clickhouse/tables/test/', '{replica}')\nORDER BY n PARTITION BY n % 10;\n\nINSERT INTO test SELECT * FROM numbers(1000);\n\n-- zookeeper_delete_path(\"/clickhouse/tables/test\", recursive=True) <- root loss.\n\nSYSTEM RESTART REPLICA test;\nSYSTEM RESTORE REPLICA test;\n```\n\nAnother way:\n\n```sql\nSYSTEM RESTORE REPLICA test ON CLUSTER cluster;\n```"}, + {"keywords":["RESTART REPLICAS"],"markdown":"**SYSTEM RESTART REPLICAS**\n\nProvides possibility to reinitialize Zookeeper sessions state for all `ReplicatedMergeTree` tables, will compare current state with Zookeeper as source of true and add tasks to Zookeeper queue if needed"}, + {"keywords":["CLEAR FILESYSTEM CACHE","DROP FILESYSTEM CACHE"],"markdown":"**SYSTEM CLEAR|DROP FILESYSTEM CACHE**\n\nAllows to drop filesystem cache.\n\n```sql\nSYSTEM CLEAR FILESYSTEM CACHE [ON CLUSTER cluster_name]\n```"}, + {"keywords":["SYNC FILE CACHE"],"markdown":"**SYSTEM SYNC FILE CACHE**\n\n**Note**\nIt's too heavy and has potential for misuse.\n\nWill do sync syscall.\n\n```sql\nSYSTEM SYNC FILE CACHE [ON CLUSTER cluster_name]\n```"}, + {"keywords":["LOAD PRIMARY KEY"],"markdown":"**SYSTEM LOAD PRIMARY KEY**\n\nLoad the primary keys for the given table or for all tables.\n\n```sql\nSYSTEM LOAD PRIMARY KEY [db.]name\n```\n\n```sql\nSYSTEM LOAD PRIMARY KEY\n```"}, + {"keywords":["UNLOAD PRIMARY KEY"],"markdown":"**SYSTEM UNLOAD PRIMARY KEY**\n\nUnload the primary keys for the given table or for all tables.\n\n```sql\nSYSTEM UNLOAD PRIMARY KEY [db.]name\n```\n\n```sql\nSYSTEM UNLOAD PRIMARY KEY\n```"}, + {"keywords":["STOP VIEW","STOP REPLICATED VIEW","STOP VIEWS"],"markdown":"**SYSTEM STOP [REPLICATED] VIEW, STOP VIEWS**\n\nDisable periodic refreshing of the given view or all refreshable views. If a refresh is in progress, cancel it too.\n\nIf the view is in a Replicated or Shared database, `STOP VIEW` only affects the current replica, while `STOP REPLICATED VIEW` affects all replicas.\n\n**Note**\nThe stopped state does not persist across server restarts. After a restart, views will resume their configured refresh schedules.\nIn Replicated or Shared databases, `SYSTEM STOP VIEW` only affects the current replica. Use `SYSTEM STOP REPLICATED VIEW` to stop refreshes on all replicas.\n\n```sql\nSYSTEM STOP VIEW [db.]name\n```\n```sql\nSYSTEM STOP VIEWS\n```"}, + {"keywords":["START VIEW","START REPLICATED VIEW","START VIEWS"],"markdown":"**SYSTEM START [REPLICATED] VIEW, START VIEWS**\n\nEnable periodic refreshing for the given view or all refreshable views. No immediate refresh is triggered.\n\nIf the view is in a Replicated or Shared database, `START VIEW` undoes the effect of `STOP VIEW`, and `START REPLICATED VIEW` undoes the effect of `STOP REPLICATED VIEW`. `START VIEW` also undoes the effect of `PAUSE VIEW`.\n\n```sql\nSYSTEM START VIEW [db.]name\n```\n```sql\nSYSTEM START VIEWS\n```"}, + {"keywords":["PAUSE VIEW","PAUSE VIEWS"],"markdown":"**SYSTEM PAUSE VIEW, PAUSE VIEWS**\n\nDisable periodic refreshing of the given view or all refreshable views.\nUnlike `SYSTEM STOP VIEW`, `SYSTEM PAUSE VIEW` does not interrupt a refresh that is already in progress: the running refresh is allowed to finish, and only subsequent refreshes are prevented.\n\nUndo with `SYSTEM START VIEW` or `SYSTEM START VIEWS`.\n\n**Note**\nThe paused state does not persist across server restarts. After a restart, views will resume their configured refresh schedules.\nIn Replicated or Shared databases, `SYSTEM PAUSE VIEW` only affects the current replica.\n\n```sql\nSYSTEM PAUSE VIEW [db.]name\n```\n```sql\nSYSTEM PAUSE VIEWS\n```"}, + {"keywords":["REFRESH VIEW"],"markdown":"**SYSTEM REFRESH VIEW**\n\nTrigger an immediate out-of-schedule refresh of a given view.\n\nIf the view is in a Replicated or Shared database, the request is shared with all replicas: the refresh may run on another replica, e.g. if refreshing is stopped on the current one.\n\n```sql\nSYSTEM REFRESH VIEW [db.]name\n```"}, + {"keywords":["WAIT VIEW"],"markdown":"**SYSTEM WAIT VIEW**\n\nWaits for the running refresh to complete. If no refresh is running, returns immediately. If the latest refresh attempt failed, reports an error.\n\nCan be used right after creating a new refreshable materialized view (without EMPTY keyword) to wait for the initial refresh to complete.\n\nIf the view is in a Replicated or Shared database, also waits for a refresh that is running or requested on another replica.\n\n```sql\nSYSTEM WAIT VIEW [db.]name\n```"}, + {"keywords":["CANCEL VIEW"],"markdown":"**SYSTEM CANCEL VIEW**\n\nIf there's a refresh in progress for the given view on the current replica, interrupt and cancel it. Otherwise do nothing.\n\n```sql\nSYSTEM CANCEL VIEW [db.]name\n```"}, + {"keywords":["STOP","STOP ALL BACKGROUND"],"markdown":"**SYSTEM STOP**\n\nStop the background activity and keep it stopped: interrupt what is running now, and run nothing further until `SYSTEM START`. Equivalent to `PAUSE` + `CANCEL`.\n\n```sql\nSYSTEM STOP [db.]table\nSYSTEM STOP ALL BACKGROUND\n```"}, + {"keywords":["START","START ALL BACKGROUND"],"markdown":"**SYSTEM START**\n\nResume activity, undoing a previous `SYSTEM STOP` or `SYSTEM PAUSE`. No activity is interrupted.\n\n```sql\nSYSTEM START [db.]table\nSYSTEM START ALL BACKGROUND\n```"}, + {"keywords":["PAUSE","PAUSE ALL BACKGROUND"],"markdown":"**SYSTEM PAUSE**\n\nPrevent further background activity, but let whatever is running right now finish first.\n\n```sql\nSYSTEM PAUSE [db.]table\nSYSTEM PAUSE ALL BACKGROUND\n```"}, + {"keywords":["CANCEL","CANCEL ALL BACKGROUND"],"markdown":"**SYSTEM CANCEL**\n\nInterrupt the activity running right now only, without blocking future activity — the table keeps refreshing or consuming on its schedule. Does nothing if no activity is in progress.\n\n```sql\nSYSTEM CANCEL [db.]table\nSYSTEM CANCEL ALL BACKGROUND\n```"}, + {"keywords":["REFRESH","REFRESH ALL BACKGROUND"],"markdown":"**SYSTEM REFRESH**\n\nRun one extra cycle out of schedule. On a streaming table it runs immediately and once, even while the table is stopped or paused. On a refreshable materialized view it behaves like `SYSTEM REFRESH VIEW`: if the view is stopped, the refresh is remembered and runs once `SYSTEM START` releases it or another replica takes it over.\n\n```sql\nSYSTEM REFRESH [db.]table\nSYSTEM REFRESH ALL BACKGROUND\n```"}, + {"keywords":["FLUSH OBJECT STORAGE QUEUE"],"markdown":"**SYSTEM FLUSH OBJECT STORAGE QUEUE**\n\nBlocks until the given file has been processed or permanently failed by the given S3Queue or AzureQueue table. Returns immediately if the file was already processed. Raises an error if the file has permanently failed (all retries exhausted).\n\n```sql\nSYSTEM FLUSH OBJECT STORAGE QUEUE [db.]table_name PATH 'path'\n```"}, + {"keywords":["RESET FILELOG"],"markdown":"**SYSTEM RESET FILELOG**\n\nChanges where a FileLog table continues reading, without recreating the table.\n\n```sql\nSYSTEM RESET FILELOG [ON CLUSTER cluster_name] [db.]table_name [FILE 'file_name' [OFFSET n | TO END]]\n```\n\nWithout `FILE`, every file of the table is read again from the beginning. With `FILE`, only the named file is affected (the name as in the `_filename` virtual column): it is read again from the beginning, from byte `n` with `OFFSET n` (the start of a line, such as an `_offset` value), or, with `TO END`, only from data appended after the command, so that what the file contains now is skipped.\n\nRows that are read again are inserted into the materialized views again. The new position is saved immediately and kept after a server restart. The command waits for a read that is running in the background and fails if a `SELECT` query is reading from the table. It requires the `SYSTEM RESET FILELOG` privilege on the table."}, + {"keywords":["ENABLE FAILPOINT","DISABLE FAILPOINT","DISABLE ALL FAILPOINTS","WAIT FAILPOINT","NOTIFY FAILPOINT"],"markdown":"**SYSTEM ENABLE|DISABLE FAILPOINT**\n\nFail points are named places in the server code where a fault can be injected on demand - an error, a delay, or a pause of the executing thread - for testing. They are listed in the `system.fail_points` table together with their current state.\n\n```sql\nSYSTEM ENABLE FAILPOINT name\nSYSTEM DISABLE FAILPOINT name\nSYSTEM DISABLE ALL FAILPOINTS\nSYSTEM WAIT FAILPOINT name [PAUSE|RESUME]\nSYSTEM NOTIFY FAILPOINT name\n```\n\n`SYSTEM ENABLE FAILPOINT` arms a single fail point; `SYSTEM DISABLE FAILPOINT` disarms it and resumes any thread blocked on it, and is a no-op if it was not enabled.\n\n`SYSTEM DISABLE ALL FAILPOINTS` disables every fail point at once and resumes every thread blocked on a pauseable one. It takes no name and is idempotent, so a test harness can use it to return the server to a state that injects nothing without knowing which fail points the previous test enabled. On a build without fail point support the statement succeeds and does nothing.\n\n`SYSTEM WAIT FAILPOINT ... PAUSE` blocks until a thread pauses on the given pauseable fail point (or the fail point is disabled), `... RESUME` blocks until the paused thread is resumed, and `SYSTEM NOTIFY FAILPOINT` resumes the paused threads without disabling the fail point.\n\nFail points are node-local state, so none of these statements accept `ON CLUSTER`. All of them require the `SYSTEM FAILPOINT` privilege."}, +]; + +export const ACCESS_DOCS: readonly AccessStatementDoc[] = [ + {"statement":"CREATE USER","markdown":"**CREATE USER**\n\nCreates user accounts.\n\nSyntax:\n\n```sql\nCREATE USER [IF NOT EXISTS | OR REPLACE] name1 [, name2 [,...]] [ON CLUSTER cluster_name]\n [{VALID UNTIL datetime | VALID FOR interval}]\n [NOT IDENTIFIED | IDENTIFIED {[WITH {plaintext_password | sha256_password | sha256_hash | double_sha1_password | double_sha1_hash}] BY {'password' | 'hash'}} | WITH NO_PASSWORD | {WITH ldap SERVER 'server_name'} | {WITH kerberos [REALM 'realm']} | {WITH ssl_certificate CN 'common_name' | SAN 'TYPE:subject_alt_name'} | {WITH ssh_key BY KEY 'public_key' TYPE 'ssh-rsa|...'} | {WITH http SERVER 'server_name' [SCHEME 'Basic']} [{VALID UNTIL datetime | VALID FOR interval}] [GRANTS (privilege ON object [,...])]\n [, {[{plaintext_password | sha256_password | sha256_hash | ...}] BY {'password' | 'hash'}} | {ldap SERVER 'server_name'} | {...} | ... [,...]]]\n [HOST {LOCAL | NAME 'name' | REGEXP 'name_regexp' | IP 'address' | LIKE 'pattern'} [,...] | ANY | NONE]\n [IN access_storage_type]\n [ROLE role [,...]]\n [DEFAULT ROLE role [,...]]\n [DEFAULT DATABASE database | NONE]\n [GRANTEES {user | role | ANY | NONE} [,...] [EXCEPT {user | role} [,...]]]\n [SETTINGS variable [= value] [MIN [=] min_value] [MAX [=] max_value] [READONLY | WRITABLE] | PROFILE 'profile_name'] [,...]\n```\n\n`ON CLUSTER` clause allows creating users on a cluster, see Distributed DDL.\n\n`CREATE USER` requires the CREATE USER privilege. `OR REPLACE` overwrites an existing user of the same name - its authentication methods, its granted roles and its settings - so it additionally requires the DROP USER privilege. The `DROP USER` privilege is required on every name listed in the statement whether or not that user already exists, so the statement cannot be used to find out which users exist.","clauses":[{"keywords":["ON CLUSTER"],"markdown":"**CREATE USER**\n\n`ON CLUSTER` clause allows creating users on a cluster, see Distributed DDL."},{"keywords":["IDENTIFIED"],"markdown":"**CREATE USER: Identification**\n\nThere are multiple ways of user identification:\n\n- `IDENTIFIED WITH no_password`\n- `IDENTIFIED WITH plaintext_password BY 'qwerty'`\n- `IDENTIFIED WITH sha256_password BY 'qwerty'` or `IDENTIFIED BY 'password'`\n- `IDENTIFIED WITH sha256_hash BY 'hash'` or `IDENTIFIED WITH sha256_hash BY 'hash' SALT 'salt'`\n- `IDENTIFIED WITH double_sha1_password BY 'qwerty'`\n- `IDENTIFIED WITH double_sha1_hash BY 'hash'`\n- `IDENTIFIED WITH bcrypt_password BY 'qwerty'`\n- `IDENTIFIED WITH bcrypt_hash BY 'hash'`\n- `IDENTIFIED WITH ldap SERVER 'server_name'`\n- `IDENTIFIED WITH kerberos` or `IDENTIFIED WITH kerberos REALM 'realm'`\n- `IDENTIFIED WITH ssl_certificate CN 'mysite.com:user'`\n- `IDENTIFIED WITH ssh_key BY KEY 'public_key' TYPE 'ssh-rsa', KEY 'another_public_key' TYPE 'ssh-ed25519'`\n- `IDENTIFIED WITH http SERVER 'http_server'` or `IDENTIFIED WITH http SERVER 'http_server' SCHEME 'basic'`\n- `IDENTIFIED BY 'qwerty'`\n\nPassword complexity requirements can be edited in config.xml. Below is an example configuration that requires passwords to be at least 12 characters long and contain 1 number. Each password complexity rule requires a regex to match against passwords and a description of the rule.\n\n```xml\n\n \n \n .{12}\n be at least 12 characters long\n \n \n \\p{N}\n contain at least 1 numeric character\n \n \n\n```\n\n**Note**\nIn ClickHouse Cloud, by default, passwords must meet the following complexity requirements:\n- Be at least 12 characters long\n- Contain at least 1 numeric character\n- Contain at least 1 uppercase character\n- Contain at least 1 lowercase character\n- Contain at least 1 special character"},{"keywords":["HOST"],"markdown":"**CREATE USER: User Host**\n\nUser host is a host from which a connection to ClickHouse server could be established. The host can be specified in the `HOST` query section in the following ways:\n\n- `HOST IP 'ip_address_or_subnetwork'` — User can connect to ClickHouse server only from the specified IP address or a [subnetwork](https://en.wikipedia.org/wiki/Subnetwork). Examples: `HOST IP '192.168.0.0/16'`, `HOST IP '2001:DB8::/32'`. For use in production, only specify `HOST IP` elements (IP addresses and their masks), since using `host` and `host_regexp` might cause extra latency.\n- `HOST ANY` — User can connect from any location. This is a default option.\n- `HOST LOCAL` — User can connect only locally.\n- `HOST NAME 'fqdn'` — User host can be specified as FQDN. For example, `HOST NAME 'mysite.com'`.\n- `HOST REGEXP 'regexp'` — You can use [pcre](http://www.pcre.org/) regular expressions when specifying user hosts. For example, `HOST REGEXP '.*\\.mysite\\.com'`.\n- `HOST LIKE 'template'` — Allows you to use the LIKE operator to filter the user hosts. For example, `HOST LIKE '%'` is equivalent to `HOST ANY`, `HOST LIKE '%.mysite.com'` filters all the hosts in the `mysite.com` domain.\n\nAnother way of specifying host is to use `@` syntax following the username. Examples:\n\n- `CREATE USER mira@'127.0.0.1'` — Equivalent to the `HOST IP` syntax.\n- `CREATE USER mira@'localhost'` — Equivalent to the `HOST LOCAL` syntax.\n- `CREATE USER mira@'192.168.%.%'` — Equivalent to the `HOST LIKE` syntax.\n\n**Tip**\nClickHouse treats `user_name@'address'` as a username as a whole. Thus, technically you can create multiple users with the same `user_name` and different constructions after `@`. However, we do not recommend to do so."},{"keywords":["VALID UNTIL"],"markdown":"**CREATE USER: VALID UNTIL Clause**\n\nAllows you to specify the expiration date and, optionally, the time for an authentication method. It accepts a string as a parameter. It is recommended to use the `YYYY-MM-DD [hh:mm:ss] [timezone]` format for datetime, where `[timezone]` must be a numeric offset such as `+09:00` or one of `UTC`, `GMT`, `Z`, `MSK`, `MSD`; named IANA zones like `Asia/Tokyo` are not recognized (see the note below). By default, this parameter equals `'infinity'`. The accepted deadline range is `1900-01-01 00:00:00 UTC` through `9999-12-31 09:59:59 UTC` — the latest instant that stays within year 9999 in every time zone, so the stored instant is never clamped when it is rendered. A deadline in the past means the credentials are already expired. Deadlines before `1970-01-01 00:00:01 UTC` are accepted only as an \"already expired\" marker: they are canonicalized to the smallest expired instant, one second after the Unix epoch (`1970-01-01 00:00:01 UTC`), so `SHOW CREATE USER` reports that instant instead of the deadline you wrote. Deadlines from that instant onward are stored exactly.\n\nA deadline is stored as an absolute instant, but `SHOW CREATE USER` and `system.users` render it in the server or session time zone, so the same stored instant appears as different wall-clock text on differently configured servers: the canonicalized expired instant above, for example, renders as `1970-01-01 00:00:01` on a server in `UTC` and as `1970-01-01 14:00:01` on a server in `Pacific/Kiritimati`. Enforcement always uses the stored instant, not its rendering.\n\nThe placement of the clause determines which authentication methods it applies to:\n\n- Before the `IDENTIFIED` clause (or when the query specifies no authentication method at all): the deadline is a user-level deadline that applies to every authentication method of the user.\n- After an authentication method: the deadline applies to that method only. A clause written after the whole `IDENTIFIED` list therefore binds to the last method only, leaving the earlier methods non-expiring.\n\nExamples:\n\n- `CREATE USER name1 VALID UNTIL '2025-01-01'`\n- `CREATE USER name1 VALID UNTIL '2025-01-01 12:00:00 UTC'`\n- `CREATE USER name1 VALID UNTIL '2025-01-01 12:00:00 +09:00'`\n- `CREATE USER name1 VALID UNTIL 'infinity'`\n- `CREATE USER name1 VALID UNTIL '2025-01-01' IDENTIFIED WITH plaintext_password BY 'password_1', bcrypt_password BY 'password_2'` — the user-level deadline applies to both methods.\n- `CREATE USER name1 IDENTIFIED WITH plaintext_password BY 'no_expiration', bcrypt_password BY 'expiration_set' VALID UNTIL '2025-01-01'` — the deadline applies only to the `bcrypt_password` method; `plaintext_password` never expires.\n\n**Note**\nThe datetime string is parsed by `parseDateTimeBestEffort`, which only recognizes the timezone tokens `UTC`, `GMT`, `Z`, `MSK`, `MSD`, and numeric offsets such as `+09:00` or `-05:00`. Named IANA timezones like `Asia/Tokyo` or `Europe/London` are not supported, and a fixed offset is not equivalent to an IANA zone for regions that observe daylight saving time, so you must compute the correct offset for the specific date you are encoding."},{"keywords":["VALID FOR"],"markdown":"**CREATE USER: VALID FOR Clause**\n\nThe `VALID FOR` clause is a convenience shorthand for `VALID UNTIL`. Instead of an absolute date and time, it accepts an interval, and the expiration deadline is computed as the current time plus that interval at the moment the query is executed. The result is then stored in the `VALID UNTIL` form, so `SHOW CREATE USER` always displays the resolved absolute deadline. It can be used everywhere `VALID UNTIL` can, and it follows the same placement rules: before `IDENTIFIED` (or with no authentication method) it is a user-level deadline that applies to every method, while after an authentication method it applies to that method only. The deadline is stored and enforced with second precision, so sub-second intervals (`NANOSECOND`, `MICROSECOND`, `MILLISECOND`) are rejected; the smallest accepted unit is `SECOND`. A negative interval is accepted as a way to mark the credentials as already expired; if the resulting deadline falls before `1970-01-01 00:00:01 UTC`, it is canonicalized to that smallest expired instant, which is what `SHOW CREATE USER` then reports — rendered in the server or session time zone, as described for `VALID UNTIL`.\n\nExamples:\n\n- `CREATE USER name1 VALID FOR INTERVAL 1 DAY`\n- `CREATE USER name1 VALID FOR INTERVAL 3 MONTH`\n- `CREATE USER name1 VALID FOR INTERVAL 1 DAY + INTERVAL 12 HOUR`\n- `CREATE USER name1 VALID FOR INTERVAL 30 DAY IDENTIFIED WITH plaintext_password BY 'password_1', bcrypt_password BY 'password_2'` — the user-level deadline applies to both methods.\n- `CREATE USER name1 IDENTIFIED WITH plaintext_password BY 'no_expiration', bcrypt_password BY 'expiration_set' VALID FOR INTERVAL 30 DAY` — the deadline applies only to the `bcrypt_password` method; `plaintext_password` never expires."},{"keywords":["GRANTS"],"markdown":"**CREATE USER: GRANTS Clause**\n\nAllows you to limit the access rights available to a session authenticated with a particular authentication method. It accepts a list of privileges in the same form as the GRANT statement, in parentheses. The clause is specified after an authentication method (after its `VALID UNTIL` clause, if any) and applies only to that method.\n\nWhen a user logs in with such an authentication method, the access rights of the session are the intersection of the user's access rights (including the rights from granted roles) with the privileges listed in the clause. The clause never adds any access rights: if a listed privilege is not granted to the user, the session does not have it. Sessions authenticated with such a method also cannot grant privileges (the `GRANT OPTION` never survives the intersection) or administer roles. Administering roles includes not only creating, altering, dropping, granting and revoking roles, but also changing which roles are activated by default for a user (`SET DEFAULT ROLE` and `ALTER USER ... DEFAULT ROLE`), which is rejected as well.\n\n`EXECUTE AS` switches the principal of the session, so a statement running under impersonation is limited by the intersection of the **target** user's access rights with the listed privileges, rather than by the rights of the user who logged in. The limit itself is never shed, and impersonating requires `IMPERSONATE ON target` to be both granted to the user and listed in the clause, so a limited credential can never reach further than the same user's unlimited credential.\n\nThis provides a convenient way to create tokens for applications: an additional credential with an expiration date and a limited set of privileges, which is tied to the user - it is displayed in `system.query_log` and `system.processes` as the user, it stops working if the user is deleted, and it loses access rights when the user loses them.\n\n**Warning**\n**Enforcement is initiator-only.** The authentication-method `GRANTS` limit and its `VALID UNTIL` expiration are enforced only on the node that receives the query (the initiator). They are **not** propagated to other nodes of a cluster, so do not rely on the clause to constrain execution cluster-wide. Remote nodes retain their usual role scoping. The clause is also not available in `users.xml`. The query result cache is shared by all authentication methods of a user: it isolates entries by user and roles, and a cache hit is not re-checked against the `GRANTS` of the method the session logged in with.\n\nExamples:\n\n- `CREATE USER name1 IDENTIFIED BY 'qwerty' GRANTS (SELECT ON db.*)`\n- `ALTER USER name1 ADD IDENTIFIED WITH plaintext_password BY 'app_token' VALID UNTIL '2026-12-31' GRANTS (SELECT ON db.table, INSERT ON db.table)`\n\nNote that the limit is a property of the authentication method, captured at the moment of the login: changing the clause with `ALTER USER` affects new sessions, not the already established ones.\n\nFiltered source grants such as `READ ON S3('s3://bucket/.*')` are not supported in the clause yet: the intersection compares a source filter as an opaque string and cannot narrow one filter to another, so such a grant is rejected rather than silently granting no access.\n\nThe clause is supported only for authentication methods whose credentials are verified purely locally by the server. For methods whose verification contacts (or, in the case of `jwt`, may contact — for example to fetch the signing keys) an external system (`ldap`, `kerberos`, `http`, `jwt`) the clause is rejected: when several authentication methods accept the same credential, the limit is enforced by re-checking the credential against the other methods, and an extra probe of an external system is unsafe, so another method accepting the same credential could bypass the limit.\n\nWhen the same effective credential is accepted by more than one authentication method, the login is limited fail-close by all of them: the session gets the intersection of the `GRANTS` of all matching methods and expires at the earliest of their `VALID UNTIL`. The earliest `VALID UNTIL` wins even when it has already passed — the login is rejected, exactly as if the single matched method had expired, so the expiry of a token never silently hands the shared credential the rights or lifetime of a broader method.\n\nThis combination is only checked among authentication methods verified locally by the server, for the same reason the clause itself is rejected on an externally verified method above: re-checking the credential there would require an unsafe extra probe of the external system. So if the same credential also happens to be accepted by an externally verified method (`ldap`, `kerberos`, `http`, `jwt`) on the same user, that method's own `VALID UNTIL` is not part of the combination, and an earlier expiry configured on it does not shorten the session obtained through the locally verified method."},{"keywords":["EXECUTE AS"],"markdown":"**CREATE USER**\n\n`EXECUTE AS` switches the principal of the session, so a statement running under impersonation is limited by the intersection of the **target** user's access rights with the listed privileges, rather than by the rights of the user who logged in. The limit itself is never shed, and impersonating requires `IMPERSONATE ON target` to be both granted to the user and listed in the clause, so a limited credential can never reach further than the same user's unlimited credential."},{"keywords":["GRANTEES"],"markdown":"**CREATE USER: GRANTEES Clause**\n\nSpecifies users or roles which are allowed to receive privileges from this user on the condition this user has also all required access granted with GRANT OPTION. Options of the `GRANTEES` clause:\n\n- `user` — Specifies a user this user can grant privileges to.\n- `role` — Specifies a role this user can grant privileges to.\n- `ANY` — This user can grant privileges to anyone. It's the default setting.\n- `NONE` — This user can grant privileges to none.\n\nYou can exclude any user or role by using the `EXCEPT` expression. For example, `CREATE USER user1 GRANTEES ANY EXCEPT user2`. It means if `user1` has some privileges granted with `GRANT OPTION` it will be able to grant those privileges to anyone except `user2`."},{"keywords":["DEFAULT DATABASE NONE"],"markdown":"**CREATE USER**\n\n`DEFAULT DATABASE NONE` leaves the default database unset. To use a database named `NONE`, quote its name with backticks:"}]}, + {"statement":"CREATE ROLE","markdown":"**CREATE ROLE**\n\nCreates new roles. Role is a set of privileges. A user assigned a role gets all the privileges of this role.\n\nSyntax:\n\n```sql\nCREATE ROLE [IF NOT EXISTS | OR REPLACE] name1 [, name2 [,...]] [ON CLUSTER cluster_name]\n [IN access_storage_type]\n [SETTINGS variable [= value] [MIN [=] min_value] [MAX [=] max_value] [CONST|READONLY|WRITABLE|CHANGEABLE_IN_READONLY] | PROFILE 'profile_name'] [,...]\n```\n\n`CREATE ROLE` requires the CREATE ROLE privilege. `OR REPLACE` throws away an existing role of the same name, along with the privileges granted to it, so it additionally requires the DROP ROLE privilege. The `DROP ROLE` privilege is required on every name listed in the statement whether or not that role already exists, so the statement cannot be used to find out which roles exist.","clauses":[]}, + {"statement":"CREATE QUOTA","markdown":"**CREATE QUOTA**\n\nCreates a quota that can be assigned to a user or a role.\n\nSyntax:\n\n```sql\nCREATE QUOTA [IF NOT EXISTS | OR REPLACE] name [ON CLUSTER cluster_name]\n [IN access_storage_type]\n [KEYED BY {user_name | ip_address | forwarded_ip_address | client_key | client_key,user_name | client_key,ip_address | normalized_query_hash} | NOT KEYED]\n [IPV4_PREFIX_BITS number]\n [IPV6_PREFIX_BITS number]\n [FOR [RANDOMIZED] INTERVAL number {second | minute | hour | day | week | month | quarter | year}\n {MAX { {queries | query_selects | query_inserts | errors | result_rows | result_bytes | read_rows | read_bytes | execution_time | written_bytes | written_rows | failed_sequential_authentications | queries_per_normalized_hash | profile_event_name} = number } [,...] | NO LIMITS | TRACKING ONLY} [,...]]\n [TO {role [,...] | ALL | ALL EXCEPT role [,...]}]\n```\n\nKeys `user_name`, `ip_address`, `forwarded_ip_address`, `client_key`, `client_key, user_name`, `client_key, ip_address`, and `normalized_query_hash` correspond to the fields in the system.quotas table.\n\n`IPV4_PREFIX_BITS` and `IPV6_PREFIX_BITS` options can only be used when `KEYED BY` is `ip_address` or `forwarded_ip_address`. They correspond to the field in the system.quotas table.\n\nParameters `queries`, `query_selects`, `query_inserts`, `errors`, `result_rows`, `result_bytes`, `read_rows`, `read_bytes`, `execution_time`, `written_bytes`, `written_rows`, `failed_sequential_authentications`, `queries_per_normalized_hash` correspond to the fields in the system.quotas_usage table.\n\nIn addition to the predefined resource types above, a limit can be set on any profile event (see the system.events table) by its name, for example `MAX S3GetObject = 1000`. Profile events are accounted once per query, at the moment the query finishes (successfully or with an error): a query that crosses the limit is still allowed to finish, and the subsequent queries are rejected until the interval ends. The current consumption is reported in the `profile_events` column of the system.quotas_usage table.\n\n`ON CLUSTER` clause allows creating quotas on a cluster, see Distributed DDL.\n\n`CREATE QUOTA` requires the CREATE QUOTA privilege. `OR REPLACE` throws away an existing quota of the same name, including which roles it applies to, so it additionally requires the DROP QUOTA privilege. The `DROP QUOTA` privilege is required whether or not the quota already exists, so the statement cannot be used to find out which quotas exist.","clauses":[{"keywords":["IPV4_PREFIX_BITS"],"markdown":"**CREATE QUOTA**\n\n`IPV4_PREFIX_BITS` and `IPV6_PREFIX_BITS` options can only be used when `KEYED BY` is `ip_address` or `forwarded_ip_address`. They correspond to the field in the system.quotas table."},{"keywords":["IPV6_PREFIX_BITS"],"markdown":"**CREATE QUOTA**\n\n`IPV4_PREFIX_BITS` and `IPV6_PREFIX_BITS` options can only be used when `KEYED BY` is `ip_address` or `forwarded_ip_address`. They correspond to the field in the system.quotas table."},{"keywords":["ON CLUSTER"],"markdown":"**CREATE QUOTA**\n\n`ON CLUSTER` clause allows creating quotas on a cluster, see Distributed DDL."}]}, + {"statement":"CREATE ROW POLICY","markdown":"**CREATE ROW POLICY**\n\nCreates a row policy, i.e. a filter used to determine which rows a user can read from a table.\n\n**Tip**\nRow policies make sense only for users with readonly access. If a user can modify a table or copy partitions between tables, it defeats the restrictions of row policies.\n\nSyntax:\n\n```sql\n-- Multiple names on one table target\nCREATE [ROW] POLICY [IF NOT EXISTS | OR REPLACE] policy_name [, ...]\n [ON CLUSTER cluster_name]\n ON { [db.]table | db.* }\n [IN access_storage_type]\n [[FOR SELECT] USING {condition | NONE}]\n [AS {PERMISSIVE | RESTRICTIVE}]\n [TO {role1 [, role2 ...] | ALL | ALL EXCEPT role1 [, role2 ...]}]\n\n-- One name on multiple table targets\nCREATE [ROW] POLICY [IF NOT EXISTS | OR REPLACE] policy_name\n [ON CLUSTER cluster_name]\n ON { [db.]table | db.* } [, ...]\n [IN access_storage_type]\n [[FOR SELECT] USING {condition | NONE}]\n [AS {PERMISSIVE | RESTRICTIVE}]\n [TO {role1 [, role2 ...] | ALL | ALL EXCEPT role1 [, role2 ...]}]\n\n-- Mixed packing: each name paired with its own table target\nCREATE [ROW] POLICY [IF NOT EXISTS | OR REPLACE]\n policy_name ON { [db.]table | db.* } [, policy_name ON { [db.]table | db.* } ...]\n [ON CLUSTER cluster_name]\n [IN access_storage_type]\n [[FOR SELECT] USING {condition | NONE}]\n [AS {PERMISSIVE | RESTRICTIVE}]\n [TO {role1 [, role2 ...] | ALL | ALL EXCEPT role1 [, role2 ...]}]\n```\n\n`ParserRowPolicyNames` accepts **three** packing forms (not a full Cartesian product):\n\n1. **Multiple names, one target** — `pol1, pol2 ON table1` creates each listed name on that single table (or `db.*`).\n2. **One name, multiple targets** — `pol1 ON table1, table2` creates the same short name on each listed target.\n3. **Mixed pairs** — `p1 ON t1, p2 ON t2` creates each name only on its paired target.\n\nA multi-name list **cannot** be combined with a multi-table `ON` list in one group: `p1, p2 ON t1, t2` is rejected. After a multi-name group, you also cannot append another comma-separated `name ON target` group in the same statement.\n\nOptional `ON CLUSTER` applies to the whole statement (one cluster name). ClickHouse does **not** accept a different `ON CLUSTER` per policy name packed into a single create — run separate `CREATE ROW POLICY` statements when policies must be created on different clusters.\n\n`CREATE ROW POLICY` requires the CREATE ROW POLICY privilege on the table the policy is created on. `OR REPLACE` throws away an existing policy of the same name, including which roles it applies to, so it additionally requires the DROP ROW POLICY privilege on that table. The `DROP ROW POLICY` privilege is required whether or not the policy already exists, so the statement cannot be used to find out which policies exist.","clauses":[{"keywords":["USING"],"markdown":"**CREATE ROW POLICY: USING clause**\n\nDefines a filter condition for a table. A user can only see rows for which the condition is true (evaluates to a non-zero value). This is similar to adding an extra `WHERE` condition to every query the user runs against the table.\n\nFor example, the following policy limits `analyst_role` to rows from the EU:\n\n```sql\nCREATE ROW POLICY region_filter ON db.orders\nUSING region = 'EU'\nTO analyst_role;\n```\n\nWith this policy, `SELECT * FROM db.orders` returns the same rows as `SELECT * FROM db.orders WHERE region = 'EU'` would."},{"keywords":["TO"],"markdown":"**CREATE ROW POLICY: TO Clause**\n\nIn the `TO` section you can provide a list of users and roles this policy should work for. For example, `CREATE ROW POLICY ... TO accountant, john@localhost`.\n\nKeyword `ALL` means all the ClickHouse users, including current user. Keyword `ALL EXCEPT` allows excluding some users from the all users list, for example, `CREATE ROW POLICY ... TO ALL EXCEPT accountant, john@localhost`\n\nRoles named in the `TO` section, including those after `ALL EXCEPT`, are matched against the current user's enabled roles (`system.enabled_roles`), not against every role granted to the user, so `SET ROLE` can change which policies apply."},{"keywords":["AS"],"markdown":"**CREATE ROW POLICY: AS Clause**\n\nIt's allowed to have more than one policy enabled on the same table for the same user at one time. So we need a way to combine the conditions from multiple policies.\n\nBy default, policies are combined using the boolean `OR` operator. For example, the following policies:\n\n```sql\nCREATE ROW POLICY pol1 ON mydb.table1 USING b=1 TO mira, peter\nCREATE ROW POLICY pol2 ON mydb.table1 USING c=2 TO peter, antonio\n```\n\nenable the user `peter` to see rows with either `b=1` or `c=2`.\n\nThe `AS` clause specifies how policies should be combined with other policies. Policies can be either permissive or restrictive. By default, policies are permissive, which means they are combined using the boolean `OR` operator.\n\nA policy can be defined as restrictive as an alternative. Restrictive policies are combined using the boolean `AND` operator.\n\nHere is the general formula:\n\n```text\nrow_is_visible = (one or more of the conditions from the permissive policies that apply to the current user and their enabled roles are non-zero) AND\n (all of the conditions from the restrictive policies that apply to the current user and their enabled roles are non-zero)\n```\n\nIf no permissive condition applies, the first condition has no effect and only the restrictive policies decide, because `access_control_improvements.users_without_row_policies_can_read_rows` is enabled by default. A user to whom no condition applies therefore sees every row, and `access_control_improvements.throw_on_unmatched_row_policies`, disabled by default, raises an exception instead when the table does have conditions and none of them apply.\n\nFor example, the following policies:\n\n```sql\nCREATE ROW POLICY pol1 ON mydb.table1 USING b=1 TO mira, peter\nCREATE ROW POLICY pol2 ON mydb.table1 USING c=2 AS RESTRICTIVE TO peter, antonio\n```\n\nenable the user `peter` to see rows only if both `b=1` AND `c=2`.\n\nDatabase policies are combined with table policies.\n\nFor example, the following policies:\n\n```sql\nCREATE ROW POLICY pol1 ON mydb.* USING b=1 TO mira, peter\nCREATE ROW POLICY pol2 ON mydb.table1 USING c=2 AS RESTRICTIVE TO peter, antonio\n```\n\nenable the user `peter` to see table1 rows only if both `b=1` AND `c=2`, although\nany other table in mydb would have only `b=1` policy applied for the user."},{"keywords":["ON CLUSTER"],"markdown":"**CREATE ROW POLICY: ON CLUSTER Clause**\n\nAllows creating row policies on a cluster, see Distributed DDL. This is also the convenient way to create the policy on the local tables of every server of the cluster."}]}, + {"statement":"CREATE SETTINGS PROFILE","markdown":"**CREATE SETTINGS PROFILE**\n\nCreates settings profiles that can be assigned to a user or a role.\n\nSyntax:\n\n```sql\nCREATE SETTINGS PROFILE [IF NOT EXISTS | OR REPLACE] name1 [, name2 [,...]]\n [ON CLUSTER cluster_name]\n [IN access_storage_type]\n [SETTINGS variable [= value] [MIN [=] min_value] [MAX [=] max_value] [CONST|READONLY|WRITABLE|CHANGEABLE_IN_READONLY] | INHERIT 'profile_name'] [,...]\n [TO {{role1 | user1 [, role2 | user2 ...]} | NONE | ALL | ALL EXCEPT {role1 | user1 [, role2 | user2 ...]}}]\n```\n\n`ON CLUSTER` clause allows creating settings profiles on a cluster, see Distributed DDL.\n\n`CREATE SETTINGS PROFILE` requires the CREATE SETTINGS PROFILE privilege. `OR REPLACE` throws away an existing profile of the same name, including which roles it applies to, so it additionally requires the DROP SETTINGS PROFILE privilege. The `DROP SETTINGS PROFILE` privilege is required whether or not the profile already exists, so the statement cannot be used to find out which profiles exist.","clauses":[{"keywords":["ON CLUSTER"],"markdown":"**CREATE SETTINGS PROFILE**\n\n`ON CLUSTER` clause allows creating settings profiles on a cluster, see Distributed DDL."}]}, + {"statement":"ALTER USER","markdown":"**ALTER USER**\n\nChanges ClickHouse user accounts.\n\nSyntax:\n\n```sql\nALTER USER [IF EXISTS] name1 [RENAME TO new_name |, name2 [,...]]\n [ON CLUSTER cluster_name]\n [{VALID UNTIL datetime | VALID FOR interval}]\n [NOT IDENTIFIED | RESET AUTHENTICATION METHODS TO NEW | {IDENTIFIED | ADD IDENTIFIED} {[WITH {plaintext_password | sha256_password | sha256_hash | double_sha1_password | double_sha1_hash}] BY {'password' | 'hash'}} | WITH NO_PASSWORD | {WITH ldap SERVER 'server_name'} | {WITH kerberos [REALM 'realm']} | {WITH ssl_certificate CN 'common_name' | SAN 'TYPE:subject_alt_name'} | {WITH ssh_key BY KEY 'public_key' TYPE 'ssh-rsa|...'} | {WITH http SERVER 'server_name' [SCHEME 'Basic']} [{VALID UNTIL datetime | VALID FOR interval}] [GRANTS (privilege ON object [,...])]\n [, {[{plaintext_password | sha256_password | sha256_hash | ...}] BY {'password' | 'hash'}} | {ldap SERVER 'server_name'} | {...} | ... [,...]]]\n [[ADD | DROP] HOST {LOCAL | NAME 'name' | REGEXP 'name_regexp' | IP 'address' | LIKE 'pattern'} [,...] | ANY | NONE]\n [IN access_storage_type]\n [DEFAULT ROLE role [,...] | ALL | ALL EXCEPT role [,...] ]\n [DEFAULT DATABASE database | NONE]\n [GRANTEES {user | role | ANY | NONE} [,...] [EXCEPT {user | role} [,...]]]\n [DROP ALL PROFILES]\n [DROP ALL SETTINGS]\n [DROP SETTINGS variable [,...] ]\n [DROP PROFILES 'profile_name' [,...] ]\n [ADD|MODIFY SETTINGS variable [=value] [MIN [=] min_value] [MAX [=] max_value] [READONLY|WRITABLE|CONST|CHANGEABLE_IN_READONLY] [,...] ]\n [SET variable [=value] [MIN [=] min_value] [MAX [=] max_value] [READONLY|WRITABLE|CONST|CHANGEABLE_IN_READONLY] [,...] ]\n [ADD PROFILES 'profile_name' [,...] ]\n```\n\nTo use `ALTER USER` you must have the ALTER USER privilege.\n\n`SET variable = value` is an alias for `MODIFY SETTING variable = value`: it changes a single setting in place while keeping the rest. Prefer it (or `MODIFY SETTING`) over the bare `SETTINGS` clause, which replaces the whole settings list and also removes all inherited (parent) profiles.","clauses":[{"keywords":["SET"],"markdown":"**ALTER USER**\n\n`SET variable = value` is an alias for `MODIFY SETTING variable = value`: it changes a single setting in place while keeping the rest. Prefer it (or `MODIFY SETTING`) over the bare `SETTINGS` clause, which replaces the whole settings list and also removes all inherited (parent) profiles."},{"keywords":["GRANTEES"],"markdown":"**ALTER USER: GRANTEES Clause**\n\nSpecifies users or roles which are allowed to receive privileges from this user on the condition this user has also all required access granted with GRANT OPTION. Options of the `GRANTEES` clause:\n\n- `user` — Specifies a user this user can grant privileges to.\n- `role` — Specifies a role this user can grant privileges to.\n- `ANY` — This user can grant privileges to anyone. It's the default setting.\n- `NONE` — This user can grant privileges to none.\n\nYou can exclude any user or role by using the `EXCEPT` expression. For example, `ALTER USER user1 GRANTEES ANY EXCEPT user2`. It means if `user1` has some privileges granted with `GRANT OPTION` it will be able to grant those privileges to anyone except `user2`."},{"keywords":["DEFAULT DATABASE NONE"],"markdown":"**ALTER USER**\n\n`DEFAULT DATABASE NONE` clears the user's default database. To use a database named `NONE`, quote its name with backticks:"},{"keywords":["VALID UNTIL"],"markdown":"**ALTER USER: VALID UNTIL Clause**\n\nAllows you to specify the expiration date and, optionally, the time for an authentication method. It accepts a string as a parameter. It is recommended to use the `YYYY-MM-DD [hh:mm:ss] [timezone]` format for datetime. By default, this parameter equals `'infinity'`. The accepted deadline range is `1900-01-01 00:00:00 UTC` through `9999-12-31 09:59:59 UTC` — the latest instant that stays within year 9999 in every time zone, so the stored instant is never clamped when it is rendered. A deadline in the past means the credentials are already expired. Deadlines before `1970-01-01 00:00:01 UTC` are accepted only as an \"already expired\" marker: they are canonicalized to the smallest expired instant, one second after the Unix epoch (`1970-01-01 00:00:01 UTC`), so `SHOW CREATE USER` reports that instant instead of the deadline you wrote. Deadlines from that instant onward are stored exactly.\n\nA deadline is stored as an absolute instant, but `SHOW CREATE USER` and `system.users` render it in the server or session time zone, so the same stored instant appears as different wall-clock text on differently configured servers: the canonicalized expired instant above, for example, renders as `1970-01-01 00:00:01` on a server in `UTC` and as `1970-01-01 14:00:01` on a server in `Pacific/Kiritimati`. Enforcement always uses the stored instant, not its rendering.\n\nThe placement of the clause determines which authentication methods it applies to:\n\n- Before the `IDENTIFIED` clause (or when the query specifies no authentication method at all): the deadline is a user-level deadline that applies to every authentication method of the user.\n- After an authentication method: the deadline applies to that method only. A clause written after the whole `IDENTIFIED` list therefore binds to the last method only, leaving the earlier methods non-expiring.\n\nExamples:\n\n- `ALTER USER name1 VALID UNTIL '2025-01-01'`\n- `ALTER USER name1 VALID UNTIL '2025-01-01 12:00:00 UTC'`\n- `ALTER USER name1 VALID UNTIL 'infinity'`\n- `ALTER USER name1 VALID UNTIL '2025-01-01' IDENTIFIED WITH plaintext_password BY 'password_1', bcrypt_password BY 'password_2'` — the user-level deadline applies to both methods.\n- `ALTER USER name1 IDENTIFIED WITH plaintext_password BY 'no_expiration', bcrypt_password BY 'expiration_set' VALID UNTIL '2025-01-01'` — the deadline applies only to the `bcrypt_password` method; `plaintext_password` never expires.\n\nExpired authentication methods are removed automatically: every `ALTER USER` on a user drops the methods whose deadline has already passed - whether or not the statement itself mentions authentication - while the others, including the ones that never expire, are kept. An expired method can never accept a credential again, so keeping it would only occupy a slot in `max_authentication_methods_per_user`. This is what makes rotating short-lived credentials work: `ALTER USER ... ADD IDENTIFIED WITH ... VALID UNTIL ...` can be issued indefinitely without the dead credentials piling up against that limit, and without arranging a window in which only one credential is still valid so that `IDENTIFIED WITH` or `RESET AUTHENTICATION METHODS TO NEW` would not drop a credential still in use.\n\nTwo things are deliberately left alone:\n\n- A method that the same statement adds. Writing an already expired credential (`VALID UNTIL` a past date, or `VALID FOR` a negative interval) stays possible.\n- A user whose every method is expired. Such a user keeps them and simply cannot authenticate, which is the state it is already in; the alternative would be a user with no authentication method at all, which is read back as `no_password` and would turn a lapsed credential into an unauthenticated one. Give it a new credential with `ALTER USER ... IDENTIFIED WITH ...` (or `ADD IDENTIFIED WITH ...`, which drops the expired ones in the same statement).\n\nSince the user-level clause is applied before the expired methods are dropped, extending the deadline of a user whose credentials have already lapsed keeps working, and only the methods that are still expired afterwards are removed."},{"keywords":["VALID FOR"],"markdown":"**ALTER USER: VALID FOR Clause**\n\nThe `VALID FOR` clause is a convenience shorthand for `VALID UNTIL`. Instead of an absolute date and time it accepts an interval, and the expiration deadline is computed as the current time plus that interval at the moment the query is executed. The result is stored in the `VALID UNTIL` form, so `SHOW CREATE USER` always displays the resolved absolute deadline. It follows the same placement rules as `VALID UNTIL`: before `IDENTIFIED` (or with no authentication method) it is a user-level deadline that applies to every method, while after an authentication method it applies to that method only. The deadline is stored and enforced with second precision, so sub-second intervals (`NANOSECOND`, `MICROSECOND`, `MILLISECOND`) are rejected; the smallest accepted unit is `SECOND`. A negative interval is accepted as a way to mark the credentials as already expired; if the resulting deadline falls before `1970-01-01 00:00:01 UTC`, it is canonicalized to that smallest expired instant, which is what `SHOW CREATE USER` then reports — rendered in the server or session time zone, as described for `VALID UNTIL`.\n\nExamples:\n\n- `ALTER USER name1 VALID FOR INTERVAL 1 DAY`\n- `ALTER USER name1 VALID FOR INTERVAL 3 MONTH`\n- `ALTER USER name1 VALID FOR INTERVAL 30 DAY IDENTIFIED WITH plaintext_password BY 'password_1', bcrypt_password BY 'password_2'` — the user-level deadline applies to both methods.\n- `ALTER USER name1 IDENTIFIED WITH plaintext_password BY 'no_expiration', bcrypt_password BY 'expiration_set' VALID FOR INTERVAL 30 DAY` — the deadline applies only to the `bcrypt_password` method; `plaintext_password` never expires."},{"keywords":["GRANTS"],"markdown":"**ALTER USER: GRANTS Clause**\n\nAllows you to limit the access rights available to a session authenticated with a particular authentication method. See the GRANTS clause of CREATE USER for details.\n\nTogether with `ADD IDENTIFIED`, this provides a convenient way to create tokens for applications: an additional credential with an expiration date and a limited set of privileges.\n\nExample:\n\n- `ALTER USER name1 ADD IDENTIFIED WITH plaintext_password BY 'app_token' VALID UNTIL '2026-12-31' GRANTS (SELECT ON db.table, INSERT ON db.table)`"}]}, + {"statement":"ALTER ROLE","markdown":"**ALTER ROLE**\n\nChanges roles.\n\nSyntax:\n\n```sql\nALTER ROLE [IF EXISTS] name1 [RENAME TO new_name |, name2 [,...]]\n [ON CLUSTER cluster_name]\n [DROP ALL PROFILES]\n [DROP ALL SETTINGS]\n [DROP PROFILES 'profile_name' [,...] ]\n [DROP SETTINGS variable [,...] ]\n [ADD|MODIFY SETTINGS variable [= value] [MIN [=] min_value] [MAX [=] max_value] [CONST|READONLY|WRITABLE|CHANGEABLE_IN_READONLY] | PROFILE 'profile_name'] [,...]\n [SET variable [= value] [MIN [=] min_value] [MAX [=] max_value] [CONST|READONLY|WRITABLE|CHANGEABLE_IN_READONLY] [,...] ]\n [ADD PROFILES 'profile_name' [,...] ]\n```\n\n`SET variable = value` is an alias for `MODIFY SETTING variable = value`: it changes a single setting in place while keeping the rest, unlike the bare `SETTINGS` clause which replaces the whole settings list and also removes all inherited (parent) profiles.\n\n**Part of:** `ALTER`\n\n**Related:** `CREATE ROLE`, `ALTER`, `SET ROLE`, `GRANT`","clauses":[{"keywords":["SET"],"markdown":"**ALTER ROLE**\n\n`SET variable = value` is an alias for `MODIFY SETTING variable = value`: it changes a single setting in place while keeping the rest, unlike the bare `SETTINGS` clause which replaces the whole settings list and also removes all inherited (parent) profiles."}]}, + {"statement":"ALTER QUOTA","markdown":"**ALTER QUOTA**\n\nChanges quotas.\n\nSyntax:\n\n```sql\nALTER QUOTA [IF EXISTS] name [ON CLUSTER cluster_name]\n [RENAME TO new_name]\n [KEYED BY {user_name | ip_address | forwarded_ip_address | client_key | client_key,user_name | client_key,ip_address | normalized_query_hash} | NOT KEYED]\n [IPV4_PREFIX_BITS number]\n [IPV6_PREFIX_BITS number]\n [FOR [RANDOMIZED] INTERVAL number {second | minute | hour | day | week | month | quarter | year}\n {MAX { {queries | query_selects | query_inserts | errors | result_rows | result_bytes | read_rows | read_bytes | execution_time | written_bytes | written_rows | failed_sequential_authentications | queries_per_normalized_hash | profile_event_name} = number } [,...] | NO LIMITS | TRACKING ONLY} [,...]]\n [TO {role [,...] | ALL | ALL EXCEPT role [,...]}]\n```\nKeys `user_name`, `ip_address`, `forwarded_ip_address`, `client_key`, `client_key, user_name`, `client_key, ip_address`, and `normalized_query_hash` correspond to the fields in the system.quotas table.\n\n`IPV4_PREFIX_BITS` and `IPV6_PREFIX_BITS` options can only be used when `KEYED BY` is `ip_address` or `forwarded_ip_address`. They correspond to the field in the system.quotas table.\n\nParameters `queries`, `query_selects`, `query_inserts`, `errors`, `result_rows`, `result_bytes`, `read_rows`, `read_bytes`, `execution_time`, `written_bytes`, `written_rows`, `failed_sequential_authentications`, `queries_per_normalized_hash` correspond to the fields in the system.quotas_usage table.\n\nA limit can also be set on any profile event (see the system.events table) by its name, for example `MAX S3GetObject = 1000`.\n\n`ON CLUSTER` clause allows creating quotas on a cluster, see Distributed DDL.","clauses":[{"keywords":["IPV4_PREFIX_BITS"],"markdown":"**ALTER QUOTA**\n\n`IPV4_PREFIX_BITS` and `IPV6_PREFIX_BITS` options can only be used when `KEYED BY` is `ip_address` or `forwarded_ip_address`. They correspond to the field in the system.quotas table."},{"keywords":["IPV6_PREFIX_BITS"],"markdown":"**ALTER QUOTA**\n\n`IPV4_PREFIX_BITS` and `IPV6_PREFIX_BITS` options can only be used when `KEYED BY` is `ip_address` or `forwarded_ip_address`. They correspond to the field in the system.quotas table."},{"keywords":["ON CLUSTER"],"markdown":"**ALTER QUOTA**\n\n`ON CLUSTER` clause allows creating quotas on a cluster, see Distributed DDL."}]}, + {"statement":"ALTER ROW POLICY","markdown":"**ALTER ROW POLICY**\n\nChanges row policy.\n\nSyntax:\n\n```sql\n-- Rename: exactly one fully qualified policy (one name on one target).\n-- RENAME TO may be combined with the same optional alteration clauses as below.\nALTER [ROW] POLICY [IF EXISTS] name\n ON { [database.]table | database.* }\n RENAME TO new_name\n [ON CLUSTER cluster_name]\n [AS {PERMISSIVE | RESTRICTIVE}]\n [FOR SELECT]\n [USING {condition | NONE}][,...]\n [TO {role [,...] | ALL | ALL EXCEPT role [,...]}]\n\n-- Multiple names on one table target (no RENAME)\nALTER [ROW] POLICY [IF EXISTS] name [, ...]\n [ON CLUSTER cluster_name]\n ON { [database.]table | database.* }\n [AS {PERMISSIVE | RESTRICTIVE}]\n [FOR SELECT]\n [USING {condition | NONE}][,...]\n [TO {role [,...] | ALL | ALL EXCEPT role [,...]}]\n\n-- One name on multiple table targets (no RENAME)\nALTER [ROW] POLICY [IF EXISTS] name\n [ON CLUSTER cluster_name]\n ON { [database.]table | database.* } [, ...]\n [AS {PERMISSIVE | RESTRICTIVE}]\n [FOR SELECT]\n [USING {condition | NONE}][,...]\n [TO {role [,...] | ALL | ALL EXCEPT role [,...]}]\n\n-- Mixed packing: each name paired with its own table target (no RENAME)\nALTER [ROW] POLICY [IF EXISTS]\n name ON { [database.]table | database.* } [, name ON { [database.]table | database.* } ...]\n [ON CLUSTER cluster_name]\n [AS {PERMISSIVE | RESTRICTIVE}]\n [FOR SELECT]\n [USING {condition | NONE}][,...]\n [TO {role [,...] | ALL | ALL EXCEPT role [,...]}]\n```\n\n`RENAME TO` is only accepted when the statement names **exactly one** policy on **one** table target. Packed multi-name or multi-table lists cannot include `RENAME TO` — rename those policies in separate statements. On that single-policy form, `RENAME TO` can still be combined with other alterations such as `AS`, `USING`, and `TO` in the same statement.\n\nWithout `RENAME TO`, packing matches `ParserRowPolicyNames`: multiple names on **one** target, one name on **multiple** targets, or mixed `name ON target` pairs. Multi-name × multi-table (`p1, p2 ON t1, t2`) is **not** accepted. Optional `ON CLUSTER` applies once to the whole statement; run separate `ALTER ROW POLICY` statements when different clusters are required.","clauses":[{"keywords":["RENAME TO"],"markdown":"**ALTER ROW POLICY**\n\n`RENAME TO` is only accepted when the statement names **exactly one** policy on **one** table target. Packed multi-name or multi-table lists cannot include `RENAME TO` — rename those policies in separate statements. On that single-policy form, `RENAME TO` can still be combined with other alterations such as `AS`, `USING`, and `TO` in the same statement."}]}, + {"statement":"ALTER SETTINGS PROFILE","markdown":"**ALTER SETTINGS PROFILE**\n\nChanges settings profiles.\n\nSyntax:\n\n```sql\nALTER SETTINGS PROFILE [IF EXISTS] name1 [RENAME TO new_name |, name2 [,...]]\n [ON CLUSTER cluster_name]\n [SETTINGS variable [= value] [MIN [=] min_value] [MAX [=] max_value] [CONST|READONLY|WRITABLE|CHANGEABLE_IN_READONLY] | INHERIT 'profile_name'] [,...]\n [ADD|MODIFY SETTINGS variable [= value] [MIN [=] min_value] [MAX [=] max_value] [CONST|READONLY|WRITABLE|CHANGEABLE_IN_READONLY] [,...]\n [SET variable [= value] [MIN [=] min_value] [MAX [=] max_value] [CONST|READONLY|WRITABLE|CHANGEABLE_IN_READONLY] [,...] ]\n [DROP SETTINGS variable [,...] ]\n [ADD PROFILES 'profile_name' [,...] ]\n [DROP PROFILES 'profile_name' [,...] ]\n [DROP ALL SETTINGS]\n [DROP ALL PROFILES]\n [TO {{role1 | user1 [, role2 | user2 ...]} | NONE | ALL | ALL EXCEPT {role1 | user1 [, role2 | user2 ...]}}]\n```\n\n`ON CLUSTER` clause allows altering settings profiles on a cluster, see Distributed DDL.","clauses":[{"keywords":["ON CLUSTER"],"markdown":"**ALTER SETTINGS PROFILE**\n\n`ON CLUSTER` clause allows altering settings profiles on a cluster, see Distributed DDL."},{"keywords":["SETTINGS"],"markdown":"**ALTER SETTINGS PROFILE: Replacing form: bare `SETTINGS` / `INHERIT`**\n\nA bare `SETTINGS` clause (without `ADD`, `MODIFY` or `DROP`) **replaces the entire settings list and all parent profiles** of the profile with exactly what you list. Anything previously present but not listed is silently dropped — there is no warning.\n\n```sql\nCREATE SETTINGS PROFILE OR REPLACE p\n SETTINGS max_execution_time = 10, enable_lazy_columns_replication = 1;\n\nALTER SETTINGS PROFILE p SETTINGS max_memory_usage = 16106127360;\n\nSHOW CREATE SETTINGS PROFILE p;\n-- → CREATE SETTINGS PROFILE p SETTINGS max_memory_usage = 16106127360\n-- max_execution_time and enable_lazy_columns_replication are gone.\n```\n\n**Warning**\nBecause the bare `SETTINGS` form is a full replace, using it to \"override one setting\" on top of a populated base profile will drop every other setting (and every parent profile) on that profile. If you only want to change a single setting while keeping the rest, use the incremental `MODIFY`/`ADD`/`DROP` form described below.\n\nThis is the same behavior as `SETTINGS` in `CREATE SETTINGS PROFILE`: the clause defines the complete settings list."},{"keywords":["INHERIT"],"markdown":"**ALTER SETTINGS PROFILE: Replacing form: bare `SETTINGS` / `INHERIT`**\n\nA bare `SETTINGS` clause (without `ADD`, `MODIFY` or `DROP`) **replaces the entire settings list and all parent profiles** of the profile with exactly what you list. Anything previously present but not listed is silently dropped — there is no warning.\n\n```sql\nCREATE SETTINGS PROFILE OR REPLACE p\n SETTINGS max_execution_time = 10, enable_lazy_columns_replication = 1;\n\nALTER SETTINGS PROFILE p SETTINGS max_memory_usage = 16106127360;\n\nSHOW CREATE SETTINGS PROFILE p;\n-- → CREATE SETTINGS PROFILE p SETTINGS max_memory_usage = 16106127360\n-- max_execution_time and enable_lazy_columns_replication are gone.\n```\n\n**Warning**\nBecause the bare `SETTINGS` form is a full replace, using it to \"override one setting\" on top of a populated base profile will drop every other setting (and every parent profile) on that profile. If you only want to change a single setting while keeping the rest, use the incremental `MODIFY`/`ADD`/`DROP` form described below.\n\nThis is the same behavior as `SETTINGS` in `CREATE SETTINGS PROFILE`: the clause defines the complete settings list."},{"keywords":["ADD"],"markdown":"**ALTER SETTINGS PROFILE: Incremental form: `ADD` / `MODIFY` / `DROP`**\n\nThe `ADD`, `MODIFY` and `DROP` keywords change individual entries while leaving everything else on the profile untouched:\n\n- `ADD SETTINGS variable = value [constraints]` — adds a setting that is not yet present.\n- `MODIFY SETTINGS variable = value [constraints]` — replaces a single setting's entry. The whole entry (value and constraints) is overwritten, so re-specify `MIN`/`MAX`/`READONLY`/etc. if you want to keep them.\n- `DROP SETTINGS variable [,...]` — removes the listed settings.\n- `ADD PROFILES 'profile_name' [,...]` / `DROP PROFILES 'profile_name' [,...]` — add or remove parent (inherited) profiles.\n- `DROP ALL SETTINGS` / `DROP ALL PROFILES` — remove all settings or all parent profiles.\n\nSeveral of these clauses can be combined in a single statement, for example `DROP SETTINGS a ADD SETTINGS b = 1`.\n\n`SET variable = value` is an alias for `MODIFY SETTINGS variable = value`. It is offered because `SET` feels natural and because typing the replacing `SETTINGS` clause when an incremental change was intended is a common mistake."},{"keywords":["MODIFY"],"markdown":"**ALTER SETTINGS PROFILE: Incremental form: `ADD` / `MODIFY` / `DROP`**\n\nThe `ADD`, `MODIFY` and `DROP` keywords change individual entries while leaving everything else on the profile untouched:\n\n- `ADD SETTINGS variable = value [constraints]` — adds a setting that is not yet present.\n- `MODIFY SETTINGS variable = value [constraints]` — replaces a single setting's entry. The whole entry (value and constraints) is overwritten, so re-specify `MIN`/`MAX`/`READONLY`/etc. if you want to keep them.\n- `DROP SETTINGS variable [,...]` — removes the listed settings.\n- `ADD PROFILES 'profile_name' [,...]` / `DROP PROFILES 'profile_name' [,...]` — add or remove parent (inherited) profiles.\n- `DROP ALL SETTINGS` / `DROP ALL PROFILES` — remove all settings or all parent profiles.\n\nSeveral of these clauses can be combined in a single statement, for example `DROP SETTINGS a ADD SETTINGS b = 1`.\n\n`SET variable = value` is an alias for `MODIFY SETTINGS variable = value`. It is offered because `SET` feels natural and because typing the replacing `SETTINGS` clause when an incremental change was intended is a common mistake."},{"keywords":["DROP"],"markdown":"**ALTER SETTINGS PROFILE: Incremental form: `ADD` / `MODIFY` / `DROP`**\n\nThe `ADD`, `MODIFY` and `DROP` keywords change individual entries while leaving everything else on the profile untouched:\n\n- `ADD SETTINGS variable = value [constraints]` — adds a setting that is not yet present.\n- `MODIFY SETTINGS variable = value [constraints]` — replaces a single setting's entry. The whole entry (value and constraints) is overwritten, so re-specify `MIN`/`MAX`/`READONLY`/etc. if you want to keep them.\n- `DROP SETTINGS variable [,...]` — removes the listed settings.\n- `ADD PROFILES 'profile_name' [,...]` / `DROP PROFILES 'profile_name' [,...]` — add or remove parent (inherited) profiles.\n- `DROP ALL SETTINGS` / `DROP ALL PROFILES` — remove all settings or all parent profiles.\n\nSeveral of these clauses can be combined in a single statement, for example `DROP SETTINGS a ADD SETTINGS b = 1`.\n\n`SET variable = value` is an alias for `MODIFY SETTINGS variable = value`. It is offered because `SET` feels natural and because typing the replacing `SETTINGS` clause when an incremental change was intended is a common mistake."},{"keywords":["SET"],"markdown":"**ALTER SETTINGS PROFILE**\n\n`SET variable = value` is an alias for `MODIFY SETTINGS variable = value`. It is offered because `SET` feels natural and because typing the replacing `SETTINGS` clause when an incremental change was intended is a common mistake."}]}, +]; diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 6d89e6e..e91bcdf 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -1,27 +1,28 @@ import { check } from "./parser/index.js"; import { checkOptions } from "./options/index.js"; -import { complete, describe, engineArgument, functionArgument } from "./complete/index.js"; +import { complete, describe, engineArgument, functionArgument, keywordDocumentation, parameterHints } from "./complete/index.js"; import { format, split } from "./format/index.js"; import { highlight } from "./highlight/index.js"; import { tokenize } from "./lexer/index.js"; import { foldingRanges, outline, selectionRanges } from "./structure/index.js"; -import { definition, prepareRename, references, rename } from "./names/index.js"; +import { definition, objectDefinitions, objectReferences, objects, prepareRename, references, rename } from "./names/index.js"; import type { ClickHouseSqlSyntax } from "./types.js"; export * from "./types.js"; export { DATA_TYPES, type DataTypeName } from "./generated/data-types.js"; export { KEYWORDS } from "./generated/keywords.js"; export { SYSTEM_COMMANDS } from "./highlight/index.js"; +export { messageFixes, type NameHint, nameHints, replaceFixes } from "./options/index.js"; export { checkWithRoles, ParseCache, roles, type Role, type RoleSpan } from "./parser/index.js"; export type { Unit } from "./format/index.js"; export { - check, checkOptions, complete, definition, describe, engineArgument, foldingRanges, format, functionArgument, highlight, outline, prepareRename, - references, rename, selectionRanges, split, tokenize, + check, checkOptions, complete, definition, describe, engineArgument, foldingRanges, format, functionArgument, highlight, keywordDocumentation, + objectDefinitions, objectReferences, objects, outline, parameterHints, prepareRename, references, rename, selectionRanges, split, tokenize, }; // Typed against the interface, so an implementation that drifts from the API fails to compile. const api: ClickHouseSqlSyntax = { tokenize, highlight, check, checkOptions, format, complete, describe, foldingRanges, outline, selectionRanges, definition, references, - prepareRename, rename, + prepareRename, rename, objects, objectReferences, objectDefinitions, }; export default api; diff --git a/packages/core/src/names/index.ts b/packages/core/src/names/index.ts index a773f5a..597675c 100644 --- a/packages/core/src/names/index.ts +++ b/packages/core/src/names/index.ts @@ -9,6 +9,8 @@ import { needsQuotes, quote, written } from "./quote.js"; import { type Def, isDef, type Resolution, resolveStatement } from "./resolve.js"; import { buildStatement, type Statement } from "./tree.js"; +export { objectDefinitions, objectReferences, objects } from "./objects.js"; + /** Settings that change how ClickHouse resolves names, other than prefer_column_name_to_alias. */ const SCOPING_SETTINGS = new Set([ "enable_global_with_statement", "enable_scopes_for_with_statement", "analyzer_compatibility_join_using_top_level_identifier", diff --git a/packages/core/src/names/objects.ts b/packages/core/src/names/objects.ts new file mode 100644 index 0000000..bbd123f --- /dev/null +++ b/packages/core/src/names/objects.ts @@ -0,0 +1,154 @@ +/** + * The tables, views, dictionaries, functions and databases that CREATE statements make, the names of tables and + * databases in statements, and which objects a name may mean, with the current database that USE sets. + */ +import { split } from "../format/index.js"; +import { tokenize } from "../lexer/index.js"; +import { parsedStatements, quotedName } from "../parser/index.js"; +import { isSignificant, type NameOptions, type ObjectKind, type ObjectReference, type SqlObject, type Token, TokenKind as T } from "../types.js"; + +const KINDS: Partial> = { TABLE: "table", VIEW: "view", DICTIONARY: "dictionary", FUNCTION: "function", DATABASE: "database" }; +const TABLES = new Set(["table", "view", "materialized view", "dictionary"]); + +/** A part of a name: a word, a quoted name or a {parameter}, and the token after it. */ +interface Part { + text: string; + param: boolean; + start: number; + end: number; + next: number; +} + +function part(sql: string, sig: Token[], i: number): Part | undefined { + const token = sig[i]; + if (!token) return undefined; + const text = sql.slice(token.start, token.end); + if (token.kind === T.BareWord) return { text, param: false, start: token.start, end: token.end, next: i + 1 }; + if (token.kind === T.QuotedIdentifier) { + const name = quotedName(text); + return name === undefined ? undefined : { text: name, param: false, start: token.start, end: token.end, next: i + 1 }; + } + if (token.kind !== T.OpeningCurlyBrace) return undefined; + // {name:Type} is five tokens. + const close = sig[i + 4]; + if (close?.kind !== T.ClosingCurlyBrace) return undefined; + return { text: sql.slice(token.start, close.end), param: true, start: token.start, end: close.end, next: i + 5 }; +} + +/** [database.]name at token `i`. */ +function qualifiedName(sql: string, sig: Token[], i: number): { database?: Part; name: Part } | undefined { + const first = part(sql, sig, i); + if (!first) return undefined; + if (sig[first.next]?.kind !== T.Dot) return { name: first }; + const second = part(sql, sig, first.next + 1); + return second ? { database: first, name: second } : undefined; +} + +function word(sql: string, sig: Token[], i: number): string { + const token = sig[i]; + return token?.kind === T.BareWord ? sql.slice(token.start, token.end).toUpperCase() : ""; +} + +/** The database of `USE [DATABASE] name`. */ +function useTarget(sql: string, sig: Token[]): Part | undefined { + if (word(sql, sig, 0) !== "USE") return undefined; + const named = word(sql, sig, 1) === "DATABASE" ? part(sql, sig, 2) : undefined; + return named ?? part(sql, sig, 1); +} + +export function objects(sql: string): SqlObject[] { + const found: SqlObject[] = []; + let current: string | undefined; + for (const unit of split(sql)) { + if (!unit.statement) continue; + const sig = unit.tokens.filter((token) => isSignificant(token.kind)); + const use = useTarget(sql, sig); + if (use) current = use.text; + const w = (i: number) => word(sql, sig, i); + let i = w(0) === "REPLACE" ? 1 : w(0) !== "CREATE" ? -1 : w(1) === "OR" && w(2) === "REPLACE" ? 3 : 1; + if (i < 0) continue; + const temporary = w(i) === "TEMPORARY"; + if (temporary) i++; + let kind: ObjectKind | undefined; + if (w(i + 1) === "VIEW" && (w(i) === "MATERIALIZED" || w(i) === "LIVE" || w(i) === "WINDOW")) { + kind = w(i) === "MATERIALIZED" ? "materialized view" : "view"; + i += 2; + } else { + kind = KINDS[w(i++)]; + } + if (!kind) continue; + if (w(i) === "IF" && w(i + 1) === "NOT" && w(i + 2) === "EXISTS") i += 3; + const name = qualifiedName(sql, sig, i); + if (!name || name.name.param || (name.database && !TABLES.has(kind))) continue; + const database = temporary || !TABLES.has(kind) ? undefined : (name.database?.text ?? current); + found.push({ + kind, name: name.name.text, database, temporary, start: unit.statement.start, end: unit.statement.end, + nameStart: (name.database ?? name.name).start, nameEnd: name.name.end, + }); + } + return found; +} + +export function objectReferences(sql: string, options: NameOptions = {}): ObjectReference[] { + const found: ObjectReference[] = []; + let current: string | undefined; + for (const statement of parsedStatements(sql, options.cache)) { + const sigOf = (start: number, end: number) => + tokenize(sql.slice(start, end)) + .filter((token) => isSignificant(token.kind) && token.kind !== T.EndOfStream) + .map((token) => ({ ...token, start: token.start + start, end: token.end + start })); + if (/^use\b/i.test(sql.slice(statement.begin, statement.begin + 4))) { + const use = useTarget(sql, sigOf(statement.begin, statement.end)); + if (use) { + current = use.text; + if (!use.param) found.push({ kind: "database", name: use.text, qualified: false, start: use.start, end: use.end }); + } + continue; + } + // A CTE takes the place of a table of the same name. + const ctes = new Set(); + for (const mark of statement.names) { + if (mark.kind !== "with" || !mark.name || (mark.flag !== "cte" && mark.flag !== "materialized")) continue; + const cte = part(sql, sigOf(statement.begin + mark.name[0], statement.begin + mark.name[1]), 0); + if (cte) ctes.add(cte.text); + } + for (const mark of statement.names) { + if (mark.kind !== "object") continue; + const sig = sigOf(statement.begin + mark.start, statement.begin + mark.end); + const name = qualifiedName(sql, sig, 0); + if (!name || name.name.param || name.name.next !== sig.length) continue; + if (!name.database && ctes.has(name.name.text)) continue; + const database = name.database?.text ?? current; + if (name.database && !name.database.param) found.push({ kind: "database", name: name.database.text, qualified: false, start: name.database.start, end: name.database.end }); + found.push({ kind: "table", name: name.name.text, database, qualified: !!name.database, start: (name.database ?? name.name).start, end: name.name.end }); + } + } + return found.sort((a, b) => a.start - b.start || a.end - b.end); +} + +const isParam = (database: string | undefined) => database?.startsWith("{") ?? false; + +export function objectDefinitions(reference: ObjectReference, file: F, files: ReadonlyMap): { file: F; object: SqlObject }[] { + const found: { file: F; object: SqlObject }[] = []; + for (const [path, list] of files) { + for (const object of list) { + if (object.name !== reference.name) continue; + if (reference.kind === "database" ? object.kind === "database" : TABLES.has(object.kind)) found.push({ file: path, object }); + } + } + // In its own file, the last CREATE before the name is the one that it names. + const lastBefore = (list: typeof found) => list.filter((c) => c.file === file && c.object.nameStart <= reference.start).at(-1); + const narrow = (list: typeof found) => { + const before = lastBefore(list); + return before ? [before] : list; + }; + if (reference.kind === "database") return narrow(found); + // A temporary table lives in its session, and a name without a database means it before any table. + const temporary = reference.qualified ? undefined : lastBefore(found.filter((c) => c.object.temporary)); + if (temporary) return [temporary]; + const tables = found.filter((c) => !c.object.temporary); + const database = reference.database; + const certain = tables.filter((c) => c.object.database === database && (database === undefined ? c.file === file : !isParam(database))); + if (certain.length) return narrow(certain); + return tables.filter((c) => database === undefined || c.object.database === undefined || isParam(database) || isParam(c.object.database) || c.object.database === database); +} diff --git a/packages/core/src/names/tree.ts b/packages/core/src/names/tree.ts index 7eca99c..d61329e 100644 --- a/packages/core/src/names/tree.ts +++ b/packages/core/src/names/tree.ts @@ -117,7 +117,7 @@ export type Node = Union | Query | With | Table | Join | ArrayJoin | Alias | Lam /** The order of marks with the same range, outermost first. */ const PRIORITY: Record = { union: 0, query: 1, with: 2, join: 3, table: 4, arrayjoin: 5, interpolate: 6, opaque: 7, matcher: 8, alias: 9, - lambda: 10, call: 11, ref: 12, settings: 13, + lambda: 10, call: 11, ref: 12, settings: 13, object: 14, }; /** A query parameter {name:Type}: the offsets of its name. */ @@ -261,7 +261,7 @@ class Builder { // The alias of a table expression is a table alias, which the table mark carries. const tableAliases = new Set(marks.flatMap((mark) => (mark.kind === "table" && mark.alias ? [mark.alias[0]] : []))); const sorted = marks - .filter((mark) => mark.kind !== "settings" && !(mark.kind === "alias" && tableAliases.has(mark.name![0]))) + .filter((mark) => mark.kind !== "settings" && mark.kind !== "object" && !(mark.kind === "alias" && tableAliases.has(mark.name![0]))) .sort((a, b) => a.start - b.start || b.end - a.end || PRIORITY[a.kind] - PRIORITY[b.kind]); const stack: Node[] = []; for (const mark of sorted) { @@ -334,6 +334,7 @@ class Builder { return { ...base, kind: "interpolate" }; case "opaque": case "settings": + case "object": return { ...base, kind: "opaque" }; } } diff --git a/packages/core/src/options/check.ts b/packages/core/src/options/check.ts index ce8f9c8..389c915 100644 --- a/packages/core/src/options/check.ts +++ b/packages/core/src/options/check.ts @@ -2,12 +2,18 @@ * checkOptions(): what ClickHouse refuses in the options of statements that its parser takes: settings, codecs, * index and statistics types, formats and compression methods. It looks only at statements without syntax errors. */ +import { DATA_TYPES } from "../generated/data-types.js"; import { FORMATS } from "../generated/formats.js"; -import { CODECS, type CallRule, INDEX_TYPES, type ParameterRule, STATISTICS_MESSAGE, STATISTICS_TYPES, UNKNOWN_INDEX_TYPE } from "../generated/options.js"; +import { + type CallRule, CODECS, COMPRESSION_LEVEL_MESSAGE, COMPRESSION_LEVELS, INDEX_TYPES, type NamedArguments, type ParameterRule, STATISTICS_MESSAGE, STATISTICS_TYPES, + type TypeRefusal, UNKNOWN_INDEX_TYPE, +} from "../generated/options.js"; import { tokenize } from "../lexer/index.js"; import { Lines, type ParseCache, parsedStatements, quotedName, type Role, stringLiteralValue } from "../parser/index.js"; -import { type Diagnostic, isSignificant, type Severity, type Token, TokenKind as T } from "../types.js"; +import { type Diagnostic, type Fix, isSignificant, type Severity, type Token, TokenKind as T } from "../types.js"; import { explainSettings, querySetting, type SettingsContext, settingsScope } from "./catalog.js"; +import { ENGINE_REQUIRED, missingEngine, setsNoEngine } from "./engine.js"; +import { nameHints, replaceFixes, valueFixes } from "./fixes.js"; import { readValue, valueError } from "./values.js"; import { COMPRESSION_METHODS, EXPLAIN_KINDS } from "./words.js"; @@ -27,9 +33,15 @@ const nameOf = (s: Statement, token: Token): string | undefined => token.kind === T.BareWord ? textOf(s, token) : token.kind === T.QuotedIdentifier ? quotedName(textOf(s, token)) : undefined; const isKeyword = (s: Statement, token: Token | undefined, keyword: string) => word(s, token) === keyword && s.roles.get(token!.start) === "keyword"; -function report(s: Statement, from: Token, to: Token, message: string, severity: Severity = "error"): void { +function report(s: Statement, from: Token, to: Token, message: string, severity: Severity = "error", fixes?: Fix[]): void { const { line, col } = s.lines.at(from.start); - s.out.push({ message, start: from.start, end: to.end, line, col, severity }); + s.out.push({ message, start: from.start, end: to.end, line, col, severity, ...(fixes?.length && { fixes }) }); +} + +/** Fixes that write one of `names` in place of the tokens from `from` to `to`; with `closest`, only the closest ones. */ +function nameFixes(from: Token, to: Token, name: string, names: Iterable, closest = false): Fix[] { + const hints = nameHints(name, names); + return replaceFixes(from.start, to.end, closest ? hints.filter((h) => h.distance === hints[0]!.distance) : hints); } /** Errors of options in the statements of `sql` that parse, with the same cache as checkWithRoles(). */ @@ -40,6 +52,7 @@ export function checkOptions(sql: string, cache?: ParseCache): Diagnostic[] { const all = tokenize(sql).filter((t) => isSignificant(t.kind) && t.kind !== T.EndOfStream); const lines = new Lines(sql); let at = 0; + let noEngine = false; for (const parsed of statements) { while (at < all.length && all[at]!.start < parsed.begin) at++; let end = at; @@ -51,6 +64,9 @@ export function checkOptions(sql: string, cache?: ParseCache): Diagnostic[] { at = end; continue; } + noEngine = setsNoEngine(sql, tokens) ?? noEngine; + const missing = noEngine ? missingEngine(sql, tokens, (t) => s.roles.get(t.start) === "keyword") : undefined; + if (missing) report(s, missing.name, missing.name, ENGINE_REQUIRED, "error", [missing.fix]); for (const mark of parsed.names) if (mark.kind === "settings") checkSettingsList(s, parsed.begin + mark.start, parsed.begin + mark.end, mark.flag as SettingsContext, mark.name && sql.slice(parsed.begin + mark.name[0], parsed.begin + mark.name[1])); checkCalls(s); @@ -129,7 +145,8 @@ function checkSettingsList(s: Statement, start: number, end: number, context: Se } } const isDefault = valueTokens?.length === 1 && word(s, valueTokens[0]) === "default"; - const isParameter = /^\{.*\}$/.test(valueTokens?.map((t) => textOf(s, t)).join("") ?? ""); + // A query parameter is {name:Type}; a map starts with a string or is empty. + const isParameter = valueTokens?.[0]?.kind === T.OpeningCurlyBrace && valueTokens[1] !== undefined && valueTokens[1].kind !== T.StringLiteral && valueTokens[1].kind !== T.ClosingCurlyBrace; // SET profile = 'p' applies a profile, which no list of settings has. if (name.startsWith("param_") || isDefault || (name === "profile" && settingsScopeIsQuery(scopeContext))) continue; // Settings with the prefix that servers allow for custom settings by default. @@ -145,7 +162,12 @@ function checkSettingsList(s: Statement, start: number, end: number, context: Se } const setting = scope.find(name) ?? (queue && name.startsWith("s3queue_") ? scope.find(name.slice(8)) : undefined); if (!setting) { - if (scope.unknown && (context !== "explain" || explainSettings(kind!))) report(s, piece[0]!, nameEnd, scope.unknown.replace("{}", name)); + if (scope.unknown && (context !== "explain" || explainSettings(kind!))) { + // Only the closest settings that take the value written, as the checks of values are not complete. + const value = valueTokens && !isParameter && context !== "explain" ? readValue(s.sql, valueTokens) : undefined; + const names = scope.offered.filter((o) => !value || !valueError(o.info, value)).map((o) => o.info.name); + report(s, piece[0]!, nameEnd, scope.unknown.replace("{}", name), "error", nameFixes(piece[0]!, nameEnd, name, names, true)); + } continue; } if (!valueTokens || isParameter) continue; @@ -155,7 +177,7 @@ function checkSettingsList(s: Statement, start: number, end: number, context: Se } const value = readValue(s.sql, valueTokens); const problem = value && valueError(setting, value); - if (problem) report(s, valueTokens[0]!, valueTokens[valueTokens.length - 1]!, problem); + if (problem) report(s, valueTokens[0]!, valueTokens[valueTokens.length - 1]!, problem, "error", valueFixes(s.sql, valueTokens, setting, value!)); if (context === "profile") checkConstraints(s, piece, setting); } } @@ -261,24 +283,41 @@ function checkCalls(s: Statement): void { case "format": { const next = tokens[i + 1]; const name = next && nameOf(s, next); - if (name !== undefined && s.roles.get(next!.start) !== "keyword" && !formatNames.has(name.toLowerCase()) && /^\w+$/.test(name)) - report(s, next!, next!, `Unknown format ${name}.`); + if (name !== undefined && s.roles.get(next!.start) !== "keyword" && !formatNames.has(name.toLowerCase()) && /^\w+$/.test(name)) { + const input = word(s, tokens[0]) === "insert"; + report(s, next!, next!, `Unknown format ${name}.`, "error", nameFixes(next!, next!, name, FORMATS.filter((f) => (input ? f.input : f.output)).map((f) => f.name))); + } break; } case "compression": { const next = tokens[i + 1]; const method = next?.kind === T.StringLiteral ? stringLiteralValue(textOf(s, next)) : undefined; if (method !== undefined && !COMPRESSION_METHODS.has(method.toLowerCase())) - report(s, next!, next!, `Unknown compression method '${method}'. Only 'auto', 'none', 'gzip', 'deflate', 'br', 'xz', 'zstd', 'lz4', 'bz2', 'snappy' are supported as compression methods.`, "warning"); + report(s, next!, next!, `Unknown compression method '${method}'. Only 'auto', 'none', 'gzip', 'deflate', 'br', 'xz', 'zstd', 'lz4', 'bz2', 'snappy' are supported as compression methods.`, "warning", + replaceFixes(next!.start, next!.end, nameHints(method, [...COMPRESSION_METHODS].filter(Boolean)), (m) => `'${m}'`)); + else if (method !== undefined && isKeyword(s, tokens[i + 2], "level")) checkLevel(s, method, tokens.slice(i + 3, i + 5)); break; } } } } +/** The LEVEL of INTO OUTFILE, which the client reads as an unsigned integer in the method's range. */ +function checkLevel(s: Statement, method: string, after: Token[]): void { + const range = COMPRESSION_LEVELS[method.toLowerCase()]; + const negative = after[0]?.kind === T.Minus; + const level = negative ? after[1] : after[0]; + if (!range || level?.kind !== T.Number) return; + const text = textOf(s, level).replace(/_/g, ""); + const value = /^(\d+|0x[0-9a-f]+|0b[01]+)$/i.test(text) ? Number(BigInt(text)) : undefined; + if (negative || value === undefined || value < range[0] || value > range[1]) + report(s, after[0]!, level, COMPRESSION_LEVEL_MESSAGE.replace("{min}", String(range[0])).replace("{max}", String(range[1]))); +} + function checkStatistics(s: Statement, name: Token): void { const text = nameOf(s, name); - if (text !== undefined && !statisticsNames.has(text.toLowerCase())) report(s, name, name, STATISTICS_MESSAGE.replace("{}", text)); + if (text !== undefined && !statisticsNames.has(text.toLowerCase())) + report(s, name, name, STATISTICS_MESSAGE.replace("{}", text), "error", nameFixes(name, name, text, STATISTICS_TYPES.map((t) => t.name))); } /** TYPE after INDEX name expr, or after STATISTICS columns. */ @@ -316,7 +355,8 @@ function checkType(s: Statement, at: number): void { if (!name || text === undefined) return; const info = indexTypesByName.get(text.toLowerCase()); if (!info) { - report(s, name, name, UNKNOWN_INDEX_TYPE.replace("{}", text)); + const count = tokens[at + 2]?.kind === T.OpeningRoundBracket ? splitList(tokens.slice(at + 3, matching(tokens, at + 2))).length : 0; + report(s, name, name, UNKNOWN_INDEX_TYPE.replace("{}", text), "error", nameFixes(name, name, text, INDEX_TYPES.filter((t) => !t.refused && takes(t.rule, count)).map((t) => t.name))); return; } if (info.refused) report(s, name, name, info.refused); @@ -327,9 +367,12 @@ function checkType(s: Statement, at: number): void { // Index types whose syntax has optional parameters past the ones asked about are only checked at the probed count. const syntaxCount = (/\((.*)\)/.exec(info.syntax ?? "")?.[1] ?? "").split(",").filter((p) => p.trim()).length; if (info.rule) checkRule(s, name, tokens[close] ?? name, args, info.rule, syntaxCount > info.rule.max, true); + if (info.named) checkNamed(s, name, tokens[close] ?? name, args, info.named); next = close + 1; } else if (info.rule && info.rule.min > 0 && info.rule.tooFew) { report(s, name, name, info.rule.tooFew); + } else if (info.named) { + checkNamed(s, name, name, [], info.named); } for (let j = next; j < tokens.length && j < next + 3; j++) if (isKeyword(s, tokens[j], "granularity")) { @@ -341,6 +384,7 @@ function checkType(s: Statement, at: number): void { function checkCodecs(s: Statement, open: number): void { const { calls } = bracketCalls(s.tokens, open); + const before = s.out.length; const known: { name: string; info: (typeof CODECS)[number]; text: string }[] = []; for (const call of calls) { const text = nameOf(s, call.name); @@ -349,7 +393,8 @@ function checkCodecs(s: Statement, open: number): void { if (text.toLowerCase() === "default") return; const info = codecsByName.get(text.toLowerCase()); if (!info) { - report(s, call.name, call.name, `Unknown codec family: ${text}.`); + const names = CODECS.filter((c) => !c.refused && !c.setting && !c.encryption && takes(c.rule, call.args?.length ?? 0)).map((c) => c.name); + report(s, call.name, call.name, `Unknown codec family: ${text}.`, "error", nameFixes(call.name, call.name, text, names)); return; } if (info.refused) { @@ -360,9 +405,82 @@ function checkCodecs(s: Statement, open: number): void { else if (info.rule && !call.args && info.rule.min > 0 && info.rule.tooFew) report(s, call.name, call.name, info.rule.tooFew); known.push({ name: info.name, info, text: s.sql.slice(call.name.start, (call.close ?? call.name).end) }); } + if (s.out.length === before) checkCodecTypes(s, open - 1, calls); codecWarnings(s, known, calls[0]!.name, calls[calls.length - 1]!.close ?? calls[calls.length - 1]!.name); } +const typeNames = new Map(DATA_TYPES.map((t) => [t.name, t])); +const caseInsensitiveTypes = new Map(DATA_TYPES.filter((t) => t.caseInsensitive).map((t) => [t.name.toLowerCase(), t])); + +/** The name a data type is known by, through its aliases, or undefined for one not known. */ +function canonicalType(name: string): string | undefined { + let type = typeNames.get(name) ?? caseInsensitiveTypes.get(name.toLowerCase()); + for (let hops = 0; type?.aliasTo && hops < 5; hops++) type = typeNames.get(type.aliasTo); + return type?.name; +} + +/** The tokens of the type of the column whose CODEC is at `at`, or undefined when it has no type written. */ +function columnType(s: Statement, at: number): Token[] | undefined { + const tokens = s.tokens; + let depth = 0; + let start = 0; + for (let j = at - 1; j >= 0; j--) { + const t = tokens[j]!; + if (t.kind === T.ClosingRoundBracket || t.kind === T.ClosingSquareBracket || t.kind === T.ClosingCurlyBrace) depth++; + else if (t.kind === T.OpeningRoundBracket || t.kind === T.OpeningSquareBracket || t.kind === T.OpeningCurlyBrace) { + if (depth-- === 0) { + start = j + 1; + break; + } + } else if (depth === 0 && (t.kind === T.Comma || isKeyword(s, t, "column") || isKeyword(s, t, "exists"))) { + start = j + 1; + break; + } + } + let i = start; + if (!tokens[i] || nameOf(s, tokens[i]!) === undefined || s.roles.get(tokens[i]!.start) === "keyword") return undefined; + for (i++; tokens[i]?.kind === T.Dot; i += 2); + if (!tokens[i] || s.roles.get(tokens[i]!.start) !== "type") return undefined; + const end = tokens[i + 1]?.kind === T.OpeningRoundBracket ? matching(tokens, i + 1) : i; + // A type of several words, as INT UNSIGNED, is not read. + if (end + 1 !== at && s.roles.get(tokens[end + 1]?.start ?? -1) !== "keyword") return undefined; + return tokens.slice(i, end + 1); +} + +/** The types of the values a type holds, through Nullable, LowCardinality, Array, Map and Tuple; undefined for one not known. */ +function typeLeaves(s: Statement, tokens: Token[]): (string | undefined)[] { + const name = canonicalType(textOf(s, tokens[0]!)); + if (tokens.length === 1) return [name]; + if (tokens[1]!.kind !== T.OpeningRoundBracket || tokens[tokens.length - 1]!.kind !== T.ClosingRoundBracket) return [undefined]; + const args = splitList(tokens.slice(2, -1)); + const count = { Nullable: 1, LowCardinality: 1, Array: 1, Map: 2 }[name as string] as number | undefined; + if (count === args.length) return args.flatMap((arg) => typeLeaves(s, arg)); + if (name === "Tuple" && args.length) + return args.flatMap((arg) => typeLeaves(s, arg.length > 1 && s.roles.get(arg[0]!.start) === "identifier" && s.roles.get(arg[1]!.start) === "type" ? arg.slice(1) : arg)); + return [undefined]; +} + +/** The first codec that refuses a type the column holds, by the types ClickHouse refused for it. */ +function checkCodecTypes(s: Statement, at: number, calls: ReturnType["calls"]): void { + const type = columnType(s, at); + if (!type) return; + const leaves = typeLeaves(s, type); + for (const call of calls) { + const info = codecsByName.get(nameOf(s, call.name)?.toLowerCase() ?? ""); + const refusals: readonly TypeRefusal[] | undefined = call.args ? info?.typesWithParameters?.filter((r) => r.parameters === undefined || r.parameters === call.args!.length) : info?.types; + for (const leaf of leaves) { + const refusal = leaf && refusals?.find((r) => r.types.includes(leaf)); + if (refusal) { + report(s, call.name, call.close ?? call.name, refusal.message.replace("{}", leaf)); + return; + } + } + } +} + +/** Whether a codec or index type takes `count` parameters. */ +const takes = (rule: CallRule | undefined, count: number) => rule !== undefined && count >= rule.min && count <= rule.max; + const SUSPICIOUS_NOTE = "(Note: you can enable setting 'allow_suspicious_codecs' to skip this check)."; /** What ClickHouse refuses unless allow_suspicious_codecs or a codec's own setting is on: a warning, since a profile may turn them on. */ @@ -426,20 +544,69 @@ function checkRule(s: Statement, nameToken: Token, close: Token, args: Token[][] if (rule.tooFew) report(s, nameToken, close, rule.tooFew); return; } + const gap = rule.gaps?.find(([n]) => n === args.length); + if (gap) { + report(s, nameToken, close, gap[1]); + return; + } // An index type was asked about one parameter of a call with all of them, so only such a call is checked. if (countsOnly || (fullCallsOnly && args.length !== rule.parameters.length)) return; args.forEach((tokens, i) => { const param: ParameterRule | undefined = rule.parameters[i]; - const value = param && argument(s, tokens); - if (!value || !param) return; - let ok: boolean | undefined; - if (value.kind === "int") ok = numberVerdict(param.ints, value.value as number); - else if (value.kind === "float") ok = numberVerdict(param.floats, value.value as number); - else if (value.kind === "string") ok = param.strings?.includes(String(value.value).toLowerCase()) || (param.stringsRefused ? false : undefined); - else ok = param.words?.includes(String(value.value).toLowerCase()) || (param.wordsRefused ? false : undefined); - if (ok !== false) return; - const key = value.kind === "int" && (value.value as number) < 0 && param.messages.negative ? "negative" : value.kind; - const message = param.messages[key] ?? param.messages.int; - if (message) report(s, tokens[0]!, tokens[tokens.length - 1]!, message.replace("{}", value.text)); + if (param) checkValue(s, tokens, param); }); } + +/** One value against what ClickHouse took at its position. */ +function checkValue(s: Statement, tokens: Token[], param: ParameterRule): void { + const value = argument(s, tokens); + if (!value) return; + let ok: boolean | undefined; + if (value.kind === "int") ok = numberVerdict(param.ints, value.value as number); + else if (value.kind === "float") ok = numberVerdict(param.floats, value.value as number); + else if (value.kind === "string") ok = param.strings?.includes(String(value.value).toLowerCase()) || (param.stringsRefused ? false : undefined); + else ok = param.words?.includes(String(value.value).toLowerCase()) || (param.wordsRefused ? false : undefined); + if (ok !== false) return; + const key = value.kind === "int" && (value.value as number) < 0 && param.messages.negative ? "negative" : value.kind; + const message = param.messages[key] ?? param.messages.int; + if (message) report(s, tokens[0]!, tokens[tokens.length - 1]!, message.replace("{}", value.text)); +} + +/** The name = value arguments of an index type, checked in the order ClickHouse reads them. */ +function checkNamed(s: Statement, nameToken: Token, close: Token, args: Token[][], named: NamedArguments): void { + const values = new Map(); + for (const arg of args) { + const key = arg.length > 2 && arg[1]!.kind === T.Equals ? nameOf(s, arg[0]!) : undefined; + if (key === undefined) { + if (named.positional) report(s, arg[0]!, arg[arg.length - 1]!, named.positional); + return; + } + if (values.has(key)) { + if (named.duplicate) report(s, arg[0]!, arg[0]!, named.duplicate.replace("{}", key)); + return; + } + values.set(key, arg.slice(2)); + } + const missing = named.required.find((r) => !values.has(r)); + if (missing) { + if (named.missing) report(s, nameToken, close, named.missing.replace("{}", missing)); + return; + } + const tokenizer = values.get("tokenizer"); + if (tokenizer && named.unknownTokenizer) { + const first = tokenizer[0]!; + const word = first.kind === T.StringLiteral ? stringLiteralValue(textOf(s, first)) : nameOf(s, first); + const whole = tokenizer.length === 1 || (tokenizer[1]!.kind === T.OpeningRoundBracket && first.kind !== T.StringLiteral && matching(tokenizer, 1) === tokenizer.length - 1); + if (word !== undefined && whole && !named.tokenizers.includes(word)) { + report(s, first, first, named.unknownTokenizer.replace("{}", word)); + return; + } + } + const before = s.out.length; + for (const [key, tokens] of values) { + const param = named.arguments[key]; + if (param) checkValue(s, tokens, param); + } + const unknown = [...values.keys()].filter((key) => !(key in named.arguments) && !named.required.includes(key)).sort(); + if (s.out.length === before && unknown.length && named.unknown) report(s, nameToken, close, named.unknown.replace("{}", unknown.join(", "))); +} diff --git a/packages/core/src/options/engine.ts b/packages/core/src/options/engine.ts new file mode 100644 index 0000000..7860f67 --- /dev/null +++ b/packages/core/src/options/engine.ts @@ -0,0 +1,67 @@ +/** + * CREATE TABLE without ENGINE after SET default_table_engine = 'None', which ClickHouse refuses with ENGINE_REQUIRED. + */ +import { stringLiteralValue } from "../parser/index.js"; +import { type Fix, type Token, TokenKind as T } from "../types.js"; + +export const ENGINE_REQUIRED = "Table engine is not specified in CREATE query."; + +const wordOf = (sql: string, token: Token | undefined) => (token?.kind === T.BareWord ? sql.slice(token.start, token.end).toLowerCase() : undefined); + +/** For a SET statement that sets default_table_engine: whether it sets it to None; undefined for any other statement. */ +export function setsNoEngine(sql: string, tokens: readonly Token[]): boolean | undefined { + if (wordOf(sql, tokens[0]) !== "set") return undefined; + let found: boolean | undefined; + for (let i = 1; i + 2 < tokens.length; i++) + if (wordOf(sql, tokens[i]) === "default_table_engine" && tokens[i + 1]!.kind === T.Equals) { + const value = tokens[i + 2]!; + found = value.kind === T.StringLiteral && stringLiteralValue(sql.slice(value.start, value.end)) === "None"; + } + return found; +} + +// The clauses of CREATE TABLE that come after its engine. +const AFTER_ENGINE = new Set(["partition", "primary", "order", "sample", "ttl", "settings", "comment", "empty", "as"]); + +/** The table name of a CREATE TABLE that needs an engine and has none, with the fix that adds ENGINE = MergeTree. */ +export function missingEngine(sql: string, tokens: readonly Token[], isKeyword: (token: Token) => boolean): { name: Token; fix: Fix } | undefined { + let i = 0; + const w = (j: number) => wordOf(sql, tokens[j]); + if (w(0) === "create" && w(1) === "or" && w(2) === "replace") i = 3; + else if (w(0) === "create" || w(0) === "replace") i = 1; + else return undefined; + if (w(i) !== "table") return undefined; + i++; + if (w(i) === "if" && w(i + 1) === "not" && w(i + 2) === "exists") i += 3; + const name = tokens[i]; + if (!name) return undefined; + let depth = 0; + let at: Token | undefined; + let columns = false; + let ordered = false; + for (let j = i + 1; j < tokens.length; j++) { + const token = tokens[j]!; + if (token.kind === T.OpeningRoundBracket) { + if (depth++ === 0 && !at) columns = true; + continue; + } + if (token.kind === T.ClosingRoundBracket) depth--; + if (depth > 0 || !isKeyword(token)) continue; + const keyword = w(j); + if (keyword === "engine") return undefined; + // AS another table or table function takes its engine, and CLONE AS too. + if (keyword === "as" && !at && !["select", "with"].includes(w(j + 1) ?? "") && tokens[j + 1]?.kind !== T.OpeningRoundBracket) return undefined; + if (keyword === "order" || keyword === "primary") ordered = true; + if (keyword && AFTER_ENGINE.has(keyword)) at ??= token; + // The SELECT after AS makes the columns, and its clauses are not the table's. + if (keyword === "as") { + columns = true; + break; + } + } + if (!columns) return undefined; + const engine = `ENGINE = MergeTree${ordered ? "" : " ORDER BY tuple()"}`; + const last = tokens[tokens.length - 1]!; + const edit = at ? { start: at.start, end: at.start, newText: `${engine} ` } : { start: last.end, end: last.end, newText: ` ${engine}` }; + return { name, fix: { title: `Add ${engine}`, edits: [edit], preferred: true } }; +} diff --git a/packages/core/src/options/fixes.ts b/packages/core/src/options/fixes.ts new file mode 100644 index 0000000..d46e76e --- /dev/null +++ b/packages/core/src/options/fixes.ts @@ -0,0 +1,81 @@ +/** + * Quick fixes: the names and values that ClickHouse would take in place of a wrong one, with the edits that write them. + */ +import type { SettingInfo } from "../generated/settings.js"; +import { levenshtein } from "../parser/index.js"; +import type { Fix, Token } from "../types.js"; +import { type Value, valueError } from "./values.js"; + +/** A name that may be meant, and how many typos away it is. */ +export interface NameHint { + name: string; + distance: number; +} + +/** NamePrompter::getHints of ClickHouse: the names within a few typos of `name`, closest first. */ +export function nameHints(name: string, names: Iterable, max = 3): NameHint[] { + const found: NameHint[] = []; + const mistakes = Math.floor((name.length + 2) / 3); + for (const prompt of names) { + const least = Math.abs(name.length - prompt.length); + if (least > 0 && Math.floor(name.length / least) < 3) continue; + if (prompt.length > name.length + mistakes || prompt.length + mistakes < name.length) continue; + const distance = levenshtein(prompt, name); + if (distance <= mistakes) found.push({ name: prompt, distance }); + } + // A stable sort keeps the order of `names` among equally close ones, as ClickHouse does. + return found.sort((a, b) => a.distance - b.distance).slice(0, max); +} + +/** A fix for each hint, written by `write` from `start` to `end`; the closest one is preferred when no other is as close. */ +export function replaceFixes(start: number, end: number, hints: readonly NameHint[], write: (name: string) => string = (name) => name): Fix[] { + return hints.map((hint, i) => { + const newText = write(hint.name); + const fix: Fix = { title: `Change to ${newText}`, edits: [{ start, end, newText }] }; + if (i === 0 && hints[1]?.distance !== hint.distance) fix.preferred = true; + return fix; + }); +} + +const quoteString = (text: string) => `'${text.replace(/\\/g, "\\\\").replace(/'/g, "\\'")}'`; + +/** Fixes of a setting's value: the enum value it is likely a typo of, or a number in quotes for a setting of one character. */ +export function valueFixes(sql: string, tokens: readonly Token[], setting: SettingInfo, value: Value): Fix[] | undefined { + const start = tokens[0]!.start; + const end = tokens[tokens.length - 1]!.end; + if (value.t === "String" && setting.values?.length) { + const parts = setting.multi ? value.v.split(/[\s,]+/).filter((part) => part !== "") : [value.v]; + const wrong = parts.filter((part) => !setting.values!.includes(part)); + if (wrong.length !== 1) return undefined; + const hints = nameHints(wrong[0]!, setting.values); + // A list keeps its other values: the wrong one is replaced where it is written. + if (!hints.length || !/^\w+$/.test(wrong[0]!)) return undefined; + const at = new RegExp(`(? m[1]!) : one ? [one[1]!] : []; + const hints = names + .filter((name) => PLAIN_NAME.test(name) && name !== written) + .map((name) => ({ name, distance: levenshtein(name, written) })) + .sort((a, b) => a.distance - b.distance); + return replaceFixes(start, end, hints); +} diff --git a/packages/core/src/options/index.ts b/packages/core/src/options/index.ts index a3377ac..30b6678 100644 --- a/packages/core/src/options/index.ts +++ b/packages/core/src/options/index.ts @@ -1,4 +1,5 @@ export { checkOptions } from "./check.js"; +export { messageFixes, type NameHint, nameHints, replaceFixes } from "./fixes.js"; export { querySetting, type SettingsContext, type SettingsScope, settingsScope } from "./catalog.js"; export { readValue, valueError } from "./values.js"; export { COMPRESSION_METHODS, EXPLAIN_KINDS } from "./words.js"; diff --git a/packages/core/src/options/values.ts b/packages/core/src/options/values.ts index ac38d5f..54e345f 100644 --- a/packages/core/src/options/values.ts +++ b/packages/core/src/options/values.ts @@ -13,7 +13,8 @@ export type Value = | { t: "String"; v: string; text: string } | { t: "Bool"; v: boolean; text: string } | { t: "Null"; text: string } - | { t: "Map" | "Array" | "Tuple"; text: string }; + | { t: "Map"; text: string; /** Whether it holds string pairs only, so that text is as ClickHouse writes it. */ pairs?: boolean } + | { t: "Array" | "Tuple"; text: string }; const UINT64_MAX = 2n ** 64n - 1n; const INT64_MIN = -(2n ** 63n); @@ -38,11 +39,29 @@ export function readValue(sql: string, tokens: readonly Token[]): Value | undefi return undefined; } const last = tokens[tokens.length - 1]!.kind; - if (first.kind === T.OpeningCurlyBrace && last === T.ClosingCurlyBrace) return { t: "Map", text }; + if (first.kind === T.OpeningCurlyBrace && last === T.ClosingCurlyBrace) return mapValue(sql, tokens, text); if (first.kind === T.OpeningSquareBracket && last === T.ClosingSquareBracket) return { t: "Array", text }; return undefined; } +/** A map of string pairs, with the text that ClickHouse writes for it in messages, as [('a', 'b')]. */ +function mapValue(sql: string, tokens: readonly Token[], source: string): Value { + const pairs: string[] = []; + for (let i = 1; i < tokens.length - 1; i += 4) { + const [key, colon, value, comma] = [tokens[i], tokens[i + 1], tokens[i + 2], tokens[i + 3]]; + const k = key?.kind === T.StringLiteral ? stringLiteralValue(sql.slice(key.start, key.end)) : undefined; + const v = value?.kind === T.StringLiteral ? stringLiteralValue(sql.slice(value.start, value.end)) : undefined; + if (k === undefined || v === undefined || colon?.kind !== T.Colon || (comma !== tokens[tokens.length - 1] && comma?.kind !== T.Comma)) + return { t: "Map", text: source }; + pairs.push(`(${quoted(k)}, ${quoted(v)})`); + } + return { t: "Map", text: `[${pairs.join(", ")}]`, pairs: true }; +} + +const ESCAPES: Record = { "\\": "\\\\", "'": "\\'", "\n": "\\n", "\t": "\\t", "\r": "\\r", "\0": "\\0", "\b": "\\b", "\f": "\\f" }; +/** A string in quotes as ClickHouse writes a Field of one. */ +const quoted = (v: string) => `'${v.replace(/[\\'\n\t\r\0\b\f]/g, (c) => ESCAPES[c]!)}'`; + /** A number as ClickHouse's parser reads it: an integer that fits 64 bits, or a Float64. */ function number(raw: string, negative: boolean, text: string): Value | undefined { const digits = raw.replace(/_/g, ""); @@ -92,12 +111,15 @@ function integerOf(type: string): { base: string; suffix: boolean; nonZero?: boo /** A number in a string as parseWithSizeSuffix() reads it: digits, then optional k, M, G, T, P, E with an optional i. */ const INTEGER_STRING = /^[+-]?\d+([kKMGTPE]i?)?$/; const PLAIN_INTEGER_STRING = /^[+-]?\d+$/; -const FLOAT_STRING = /^[+-]?(\d+\.?\d*|\.\d+)([eE][+-]?\d+)?([kKMGTPE]i?)?$|^[+-]?(inf|nan)$/i; +const FLOAT_PREFIX = /^[+-]?(\d+\.?\d*|\.\d+)([eE][+-]?\d+)?/; const fieldName = (value: Value) => (value.t === "Map" || value.t === "Array" || value.t === "Tuple" ? value.t : value.t); /** ClickHouse's message for a value that a setting does not take, or undefined when it takes it or the check is not known. */ export function valueError(setting: SettingInfo, value: Value): string | undefined { + // Settings.cpp reads the compatibility setting as a version, and Context.cpp words the error otherwise. + if (setting.name === "compatibility" && value.t === "String" && !isVersion(value.v)) + return `Cannot parse ClickHouse version here: ${value.v}: in attempt to set the value of setting 'compatibility' to ${quoted(value.v)}.`; const problem = valueProblem(setting, value); return problem && `${problem}: while setting '${setting.name}' to value ${value.text}.`; } @@ -112,16 +134,14 @@ function valueProblem(setting: SettingInfo, value: Value): string | undefined { return boolProblem(value); case "BoolAuto": return value.t === "String" && value.v.toLowerCase() === "auto" ? undefined : boolProblem(value); + case "BoolForce": + return value.t === "String" && value.v.toLowerCase() === "force" ? undefined : boolProblem(value); case "Float": case "Double": case "Seconds": case "FloatAuto": case "DoubleAuto": - if (value.t === "String") { - if (type.endsWith("Auto") && value.v.toLowerCase() === "auto") return undefined; - return FLOAT_STRING.test(value.v) || value.v === "" ? undefined : `Cannot read floating point value here: ${value.v}`; - } - return value.t === "Null" ? `Invalid value NULL of the setting, which needs ${type === "Float" ? "float" : "double"}` : undefined; + return floatProblem(type, value); case "String": return value.t === "String" ? undefined : badGet(value, "String"); case "Char": @@ -132,6 +152,10 @@ function valueProblem(setting: SettingInfo, value: Value): string | undefined { if (value.t === "Map") return undefined; if (value.t !== "String") return badGet(value, "Map"); return value.v === "" || value.v.trimStart().startsWith("{") ? undefined : `Cannot parse input: expected '{' before: '${value.v}'`; + case "URI": + if (value.t !== "String") return badGet(value, "String"); + // Poco::URI refuses spaces, control characters and bytes past ASCII before anything else. + return /[\0- \x7f-\uffff]/.test(value.v) ? "Cannot parse URI: Bad URI syntax: URI contains invalid characters" : undefined; case "Timezone": if (value.t !== "String") return badGet(value, "String"); // Fixed offsets such as Fixed/UTC+03:00:00 and +03:00 are time zones too. @@ -142,6 +166,9 @@ function valueProblem(setting: SettingInfo, value: Value): string | undefined { } } +/** A version as ClickHouseVersion reads it: unsigned integers between dots, or nothing. */ +const isVersion = (v: string) => v === "" || v.split(".").every((part) => /^\+?\d+$/.test(part) && BigInt(part) <= UINT64_MAX); + const TIME_ZONE_NAMES = new Set(TIME_ZONES.map((z) => z.name.toLowerCase())); function badGet(value: Value, requested: string): string | undefined { @@ -159,6 +186,45 @@ function enumProblem(setting: SettingInfo, value: Value): string | undefined { return `Unexpected value of ${setting.type}: '${wrong}'. Must be one of [${values.map((v) => `'${v}'`).join(", ")}]`; } +/** A float setting: a number in a string without size suffixes, finite, and an integer that the float type holds exactly. */ +function floatProblem(type: string, value: Value): string | undefined { + const cpp = type.startsWith("Float") ? "float" : "double"; + let v: number | undefined; + switch (value.t) { + case "String": { + if (type.endsWith("Auto") && value.v.toLowerCase() === "auto") return undefined; + const infinite = /^([+-]?)(inf|infinity)$/i.exec(value.v); + if (infinite) return `Float setting value must be finite, got ${infinite[1] === "-" ? "-" : ""}inf`; + if (/^\+?nan$/i.test(value.v)) return "Float setting value must be finite, got nan"; + if (/^[+-]?(inf|nan)/i.test(value.v)) return undefined; + const prefix = FLOAT_PREFIX.exec(value.v)?.[0]; + if (!prefix) return `Cannot read floating point value here: ${value.v}`; + if (prefix.length < value.v.length) return `Cannot parse input: expected 'eof' before: '${value.v.slice(prefix.length)}'`; + v = Number(prefix); + break; + } + case "UInt64": + case "Int64": { + const n = Number(value.big!); + if (BigInt(n) !== value.big || (cpp === "float" && Math.fround(n) !== n)) return `Field value ${value.text} is out of range of ${cpp} type`; + v = n; + break; + } + case "Float64": + v = value.v; + break; + case "Null": + return `Invalid value NULL of the setting, which needs ${cpp}`; + case "Map": + return value.pairs ? `Invalid value ${value.text} of the setting, which needs ${cpp}` : undefined; + default: + return undefined; + } + // Seconds are kept as microseconds in an Int64. + if (type === "Seconds" && Math.abs(v) * 1e6 > 2 ** 63) return `Cannot convert seconds to microseconds: the setting's value in seconds is too big: ${Math.abs(v) >= 1e16 ? v.toExponential() : String(v)}`; + return undefined; +} + function boolProblem(value: Value): string | undefined { switch (value.t) { case "Bool": @@ -174,6 +240,8 @@ function boolProblem(value: Value): string | undefined { } case "Null": return "Invalid value NULL of the setting, which needs bool"; + case "Map": + return value.pairs ? `Invalid value ${value.text} of the setting, which needs bool` : undefined; default: return undefined; } @@ -200,6 +268,7 @@ function integerProblem(integer: NonNullable>, valu if (integer.maxThreads && /^'?auto/.test(value.v)) return undefined; if (integer.auto && value.v.toLowerCase() === "auto") return undefined; if (value.v === "") return "Attempt to read after eof"; + if (range.min === 0n && value.v.startsWith("-")) return "Unsigned type must not contain '-' symbol"; if (!(integer.suffix ? INTEGER_STRING : PLAIN_INTEGER_STRING).test(value.v)) { const at = /^[+-]?\d*/.exec(value.v)![0].length; return /^[+-]?\d/.test(value.v) || at === 0 ? `Cannot parse input: expected 'eof' before: '${value.v.slice(at)}'` : undefined; @@ -208,7 +277,9 @@ function integerProblem(integer: NonNullable>, valu break; } case "Null": - return `Invalid value NULL of the setting, which needs ${range.cpp}`; + case "Map": + if (value.t === "Map" && !value.pairs) return undefined; + return `Invalid value ${value.text} of the setting, which needs ${range.cpp}`; default: return undefined; } diff --git a/packages/core/src/parser/check.ts b/packages/core/src/parser/check.ts index 32186f9..50490f9 100644 --- a/packages/core/src/parser/check.ts +++ b/packages/core/src/parser/check.ts @@ -5,6 +5,7 @@ import { tokenize } from "../lexer/index.js"; import { type Diagnostic, + type Fix, isErrorKind, isSignificant, LEXER_ERROR_MESSAGES, @@ -59,7 +60,8 @@ function utf8Length(text: string): number { return length; } -function levenshtein(a: string, b: string): number { +/** The edit distance of two words, ignoring case, as levenshteinDistanceCaseInsensitive. */ +export function levenshtein(a: string, b: string): number { const x = a.toLowerCase(); const y = b.toLowerCase(); let previous = Array.from({ length: y.length + 1 }, (_, j) => j); @@ -90,17 +92,43 @@ function keywordTypoHint(word: string, variants: string[]): string { return hint; } -/** The "Expected ..." part of a syntax error. */ -function expectedMessage(token: Token, sql: string, expected: Expected): string { +/** The "Expected ..." part of a syntax error, and the keyword it hints at as a fix. */ +function expectedMessage(token: Token, sql: string, expected: Expected): { message: string; fixes?: Fix[] } { const variants = expected.variants; - if (!variants.length) return "Syntax error"; + if (!variants.length) return { message: "Syntax error" }; let message = ""; + let fixes: Fix[] | undefined; if (token.kind === T.BareWord) { - const hint = keywordTypoHint(sql.slice(token.start, token.end), variants); - if (hint) message = `Maybe you meant: ${hint}. `; + const word = sql.slice(token.start, token.end); + const hint = keywordTypoHint(word, variants); + if (hint) { + message = `Maybe you meant: ${hint}. `; + // In the case the word was written in. + const newText = /^[a-z_]+$/.test(word) ? hint.toLowerCase() : hint; + fixes = [{ title: `Change to ${newText}`, edits: [{ start: token.start, end: token.end, newText }], preferred: true }]; + } } message += variants.length === 1 ? `Expected ${variants[0]}` : `Expected one of: ${variants.join(", ")}`; - return message; + return { message, fixes }; +} + +let verifying = false; + +/** Whether the statement from `begin` to `end` passes check() with the fix, which keeps only fixes that ClickHouse parses. */ +function parsesWith(fix: Fix, sql: string, begin: number, end: number): boolean { + if (verifying) return false; + const edit = fix.edits[0]!; + verifying = true; + try { + return check(sql.slice(begin, edit.start) + edit.newText + sql.slice(edit.end, end)).length === 0; + } finally { + verifying = false; + } +} + +/** The fixes with their edits moved by `delta`. */ +function shiftFixes(fixes: Fix[] | undefined, delta: number): Fix[] | undefined { + return fixes?.map((fix) => ({ ...fix, edits: fix.edits.map((e) => ({ ...e, start: e.start + delta, end: e.end + delta })) })); } /** What checking one statement gave, with offsets from where it begins; or that it needs more than `over` backtracks. */ @@ -161,13 +189,13 @@ class Checker { this.lines = new Lines(sql); } - private report(start: number, end: number, message: string, severity: Severity = "error"): void { + private report(start: number, end: number, message: string, severity: Severity = "error", fixes?: Fix[]): void { const { line, col } = this.lines.at(start); - this.diagnostics.push({ message, start, end, line, col, severity }); + this.diagnostics.push({ message, start, end, line, col, severity, ...(fixes && { fixes }) }); } - reportToken(token: Token, message: string, severity?: Severity): void { - this.report(token.start, token.kind === T.EndOfStream ? token.start : token.end, message, severity); + reportToken(token: Token, message: string, severity?: Severity, fixes?: Fix[]): void { + this.report(token.start, token.kind === T.EndOfStream ? token.start : token.end, message, severity, fixes); } /** Byte position from the query begin, with line and column if the statement has several lines. */ @@ -253,7 +281,7 @@ class Checker { if (hit && hit.over === undefined) { this.budget.backtracks -= hit.backtracks; if (this.budget.backtracks < 0) throw new OutOfBudget(); - for (const d of hit.diagnostics) this.report(begin + d.start, begin + d.end, d.message, d.severity); + for (const d of hit.diagnostics) this.report(begin + d.start, begin + d.end, d.message, d.severity, shiftFixes(d.fixes, begin)); this.spans = hit.spans.map((span) => ({ ...span, start: begin + span.start, end: begin + span.end })); this.names = hit.names; return begin + hit.next; @@ -270,7 +298,7 @@ class Checker { const p = this.parsed; if (this.keep && next <= keyEnd && Math.max(this.lookedAt, p?.maxIndex ?? 0) <= endIndex) { cache.set(key, { - diagnostics: this.diagnostics.slice(reported).map((d) => ({ message: d.message, start: d.start - begin, end: d.end - begin, severity: d.severity })), + diagnostics: this.diagnostics.slice(reported).map((d) => ({ message: d.message, start: d.start - begin, end: d.end - begin, severity: d.severity, fixes: shiftFixes(d.fixes, -begin) })), spans: this.spans.map((span) => ({ ...span, start: span.start - begin, end: span.end - begin })), names: this.names, next: next - begin, @@ -359,13 +387,15 @@ class Checker { } if (!node) { - this.reportToken(this.clamp(toks, lastIndex, endIndex), expectedMessage(last, sql, p.e)); + const { message, fixes } = expectedMessage(last, sql, p.e); + this.reportToken(this.clamp(toks, lastIndex, endIndex), message, undefined, fixes && parsesWith(fixes[0]!, sql, begin, afterStatement) ? fixes : undefined); return afterStatement; } const stop = toks[Math.min(p.i, toks.length - 1)]!; if (stop.kind !== T.EndOfStream && stop.kind !== T.Semicolon) { p.e.add(lastIndex, "end of query"); - this.reportToken(this.clamp(toks, lastIndex, endIndex), expectedMessage(last, sql, p.e)); + const { message, fixes } = expectedMessage(last, sql, p.e); + this.reportToken(this.clamp(toks, lastIndex, endIndex), message, undefined, fixes && parsesWith(fixes[0]!, sql, begin, afterStatement) ? fixes : undefined); return afterStatement; } // A query may run past a ';', like COPY to an unsupported target; the next one starts after it. diff --git a/packages/core/src/parser/common.ts b/packages/core/src/parser/common.ts index dfa0ac3..baea819 100644 --- a/packages/core/src/parser/common.ts +++ b/packages/core/src/parser/common.ts @@ -44,19 +44,24 @@ export function parseOnCluster(p: P): boolean { /** parseDatabaseAndTableName: [db.]table with plain identifiers. It does not move back on failure. */ export function parseDatabaseAndTableName(p: P): { database: string; table: string } | null { + const begin = p.i; const first = parseIdentifier(p); if (!first) return null; if (tokQuiet(p, T.Dot)) { const second = parseIdentifier(p); + if (second) p.names?.add("object", begin, p.i - 1); return second ? { database: first.name, table: second.name } : null; } + p.names?.add("object", begin, p.i - 1); return { database: "", table: first.name }; } /** parseDatabaseAndTableAsAST: [db.]table where both may be {name:Identifier} parameters. */ export function parseDatabaseAndTableAsAST(p: P): boolean { + const begin = p.i; if (!parseIdentifier(p, true)) return false; - if (tokQuiet(p, T.Dot)) return parseIdentifier(p, true) !== null; + if (tokQuiet(p, T.Dot) && !parseIdentifier(p, true)) return false; + p.names?.add("object", begin, p.i - 1); return true; } diff --git a/packages/core/src/parser/create.ts b/packages/core/src/parser/create.ts index 5481a2e..5ec1aa8 100644 --- a/packages/core/src/parser/create.ts +++ b/packages/core/src/parser/create.ts @@ -695,8 +695,10 @@ export function parseCreateTableQuery(p: P): Ast | null { select = parseSelectWithUnionQuery(p); if (!select && !storage) asTableFunction = parseFunction(p, true, true); if (!select && !asTableFunction) { + const begin = p.i; if (!parseIdentifier(p)) return null; if (tok(p, T.Dot) && !parseIdentifier(p)) return null; + p.names?.add("object", begin, p.i - 1); asTable = true; if (!storage) parseStorageAndTargets(); } diff --git a/packages/core/src/parser/elements.ts b/packages/core/src/parser/elements.ts index 902772c..1b55c2b 100644 --- a/packages/core/src/parser/elements.ts +++ b/packages/core/src/parser/elements.ts @@ -127,6 +127,7 @@ const UUID_TEXT = /^(?:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{ /** ParserCompoundIdentifier: a.b.c, json.a.:Type, or db.table UUID '...'. */ export function parseCompoundIdentifier(p: P, tableWithUuid = false, allowParam = false): Ast | null { return base(p, "compound identifier", () => { + const first = p.i; const parts: string[] = []; let hasParam = false; let special = ""; @@ -163,6 +164,7 @@ export function parseCompoundIdentifier(p: P, tableWithUuid = false, allowParam if (tableWithUuid) { if (parts.length > 2) return null; + p.names?.add("object", first, p.i - 1); let uuid: Ast | null = null; if (kw(p, "UUID")) { uuid = parseStringLiteral(p); diff --git a/packages/core/src/parser/index.ts b/packages/core/src/parser/index.ts index 7b25d70..e382199 100644 --- a/packages/core/src/parser/index.ts +++ b/packages/core/src/parser/index.ts @@ -1,4 +1,4 @@ -export { check, checkWithRoles, Lines, ParseCache, type ParsedStatement, parsedStatements, roles } from "./check.js"; +export { check, checkWithRoles, levenshtein, Lines, ParseCache, type ParsedStatement, parsedStatements, roles } from "./check.js"; export type { NameMark, NameMarkKind } from "./names.js"; export type { Role, RoleSpan } from "./roles.js"; export { expectedAt, keywordFilter } from "./expected.js"; diff --git a/packages/core/src/parser/insert.ts b/packages/core/src/parser/insert.ts index e01f53c..75cb7eb 100644 --- a/packages/core/src/parser/insert.ts +++ b/packages/core/src/parser/insert.ts @@ -63,8 +63,10 @@ export function parseInsertQuery(p: P, allowSettingsAfterFormat = false): Ast | if (!keep(node, "table_function", parseFunction(p, false, true))) return null; if (kw(p, "PARTITION BY") && !keep(node, "partition_by", parseExpressionWithOptionalAlias(p, false))) return null; } else { + const begin = p.i; if (!parseIdentifier(p, true)) return null; if (tok(p, T.Dot) && !parseIdentifier(p, true)) return null; + p.names?.add("object", begin, p.i - 1); } const beforeBracket = p.i; diff --git a/packages/core/src/parser/misc.ts b/packages/core/src/parser/misc.ts index 8033137..1e8a352 100644 --- a/packages/core/src/parser/misc.ts +++ b/packages/core/src/parser/misc.ts @@ -128,8 +128,10 @@ export function parseUndropQuery(p: P): Ast | null { return base(p, "UNDROP query", () => { if (!kw(p, "UNDROP")) return null; if (!kw(p, "TABLE")) return null; + const begin = p.i; if (!parseIdentifier(p, true)) return null; if (tok(p, T.Dot) && !parseIdentifier(p, true)) return null; + p.names?.add("object", begin, p.i - 1); if (!parseUuidClause(p)) return null; if (!parseOnCluster(p)) return null; return new Ast("UndropQuery"); @@ -186,8 +188,10 @@ function parseOptimizeColumns(p: P): Ast | null { export function parseOptimizeQuery(p: P): Ast | null { return base(p, "OPTIMIZE query", () => { if (!kw(p, "OPTIMIZE TABLE")) return null; + const begin = p.i; if (!parseIdentifier(p, true)) return null; if (tok(p, T.Dot) && !parseIdentifier(p, true)) return null; + p.names?.add("object", begin, p.i - 1); if (kw(p, "ON") && !parseCluster(p)) return null; const node = new Ast("OptimizeQuery"); if (kw(p, "PARTITION") && !keep(node, "partition", parsePartition(p))) return null; diff --git a/packages/core/src/parser/names.ts b/packages/core/src/parser/names.ts index 1b195bb..7edafc3 100644 --- a/packages/core/src/parser/names.ts +++ b/packages/core/src/parser/names.ts @@ -33,7 +33,9 @@ export type NameMarkKind = /** Something names analysis does not follow, such as a subquery of EXPLAIN. */ | "opaque" /** A list of settings; the flag tells what it belongs to, `name` is the engine of a storage. Names analysis skips it. */ - | "settings"; + | "settings" + /** The [db.]name of a table, view or dictionary. Names analysis skips it. */ + | "object"; /** A mark with the offsets of its first and last token, from where its statement begins; ranges are [start, end). */ export interface NameMark { diff --git a/packages/core/src/parser/show.ts b/packages/core/src/parser/show.ts index d67e242..68881dc 100644 --- a/packages/core/src/parser/show.ts +++ b/packages/core/src/parser/show.ts @@ -147,8 +147,10 @@ export function parseTablePropertiesQuery(p: P): Ast | null { if (!temporary) kw(p, "TEMPORARY"); if (!kw(p, "TABLE")) kw(p, "DICTIONARY"); } + const begin = p.i; if (!parseIdentifier(p, true)) return null; if (tok(p, T.Dot) && !parseIdentifier(p, true)) return null; + p.names?.add("object", begin, p.i - 1); return new Ast("ShowQuery"); }); } diff --git a/packages/core/src/types.ts b/packages/core/src/types.ts index 78f76e4..d7f543d 100644 --- a/packages/core/src/types.ts +++ b/packages/core/src/types.ts @@ -137,6 +137,16 @@ export interface Diagnostic { line: number; col: number; severity: Severity; + /** Edits that fix the problem, for a quick fix. */ + fixes?: Fix[]; +} + +/** A change of the input that fixes a diagnostic. */ +export interface Fix { + title: string; + edits: TextEdit[]; + /** The fix to apply first, when it is the one closest to what was written. */ + preferred?: boolean; } /** @@ -209,6 +219,12 @@ export interface ClickHouseSqlSyntax { prepareRename(sql: string, offset: number, options?: NameOptions): PrepareRenameResult; /** The edits that give the name at an offset a new name, quoted where needed. */ rename(sql: string, offset: number, newName: string, options?: NameOptions): RenameResult; + /** The tables, views, dictionaries, functions and databases that the CREATE statements make, without parsing them. */ + objects(sql: string): SqlObject[]; + /** The names of tables and databases in the statements that parse, with the database that each one means. */ + objectReferences(sql: string, options?: NameOptions): ObjectReference[]; + /** The objects of `files` that a reference in `file` may name, as ClickHouse runs each file in a session of its own. */ + objectDefinitions(reference: ObjectReference, file: F, files: ReadonlyMap): { file: F; object: SqlObject }[]; } /** Options of definition(), references(), prepareRename() and rename(). */ @@ -224,6 +240,39 @@ export interface NameLocation { definition: boolean; } +/** What a CREATE statement makes. */ +export type ObjectKind = "table" | "view" | "materialized view" | "dictionary" | "function" | "database"; + +/** An object that a CREATE statement of the input makes, as objects() finds it. */ +export interface SqlObject { + kind: ObjectKind; + name: string; + /** + * The database written before the name, or set by an earlier USE; none for the session's database, a function, a + * temporary table and a database. A query parameter such as `{db:Identifier}` is kept as written. + */ + database?: string; + temporary: boolean; + /** The statement. */ + start: number; + end: number; + /** The name with its database, as written. */ + nameStart: number; + nameEnd: number; +} + +/** A name of a table, view or dictionary, or of a database, in a statement that parses. */ +export interface ObjectReference { + kind: "table" | "database"; + name: string; + /** For a table, the database written before it or set by an earlier USE, as in SqlObject. */ + database?: string; + /** Whether the database is written before the name. */ + qualified: boolean; + start: number; + end: number; +} + /** Text from `start` to `end` to replace with `newText`. */ export interface TextEdit { start: number; @@ -393,3 +442,15 @@ export interface TextRange { start: number; end: number; } + +/** The documented parameter that an argument of a call is, for an inlay hint before it. */ +export interface ParameterHint { + /** Where the argument starts. */ + offset: number; + name: string; +} + +/** The documentation of a SYSTEM command or of a clause of an access statement, and the keywords it covers. */ +export interface KeywordDocumentation extends TextRange { + documentation: string; +} diff --git a/packages/core/test/complete/hints.test.ts b/packages/core/test/complete/hints.test.ts new file mode 100644 index 0000000..9d49cb1 --- /dev/null +++ b/packages/core/test/complete/hints.test.ts @@ -0,0 +1,43 @@ +import assert from "node:assert/strict"; +import { describe, test } from "node:test"; + +import { parameterHints } from "../../src/index.js"; + +/** Each hint as the parameter's name and the text of the argument from where it starts. */ +function hints(sql: string): string[] { + return parameterHints(sql).map((h) => `${h.name}: ${sql.slice(h.offset).split(/[,)]/)[0]}`); +} + +describe("parameterHints()", () => { + test("names the arguments of a function by its documented syntax", () => { + assert.deepEqual(hints("SELECT dateDiff('day', a, b)"), ["unit: 'day'", "startdate: a", "enddate: b"]); + assert.deepEqual(hints("SELECT JSONExtract(j, 'a', 'String')"), ["json: j", "indices_or_keys: 'a'", "return_type: 'String'"]); + }); + + test("leaves out an argument that is a name equal to the parameter", () => { + assert.deepEqual(hints("SELECT dateDiff(unit, t.startdate, b)"), ["enddate: b"]); + assert.deepEqual(hints("SELECT dateDiff(UNIT, `startdate`, b)"), ["enddate: b"]); + }); + + test("leaves out calls with one argument, and the arguments of a repeated parameter", () => { + assert.deepEqual(hints("SELECT toDate(x), quantile(0.5)(x), tuple(1, 2), concat(a, b)"), []); + }); + + test("names the arguments of table functions and engines", () => { + assert.deepEqual(hints("SELECT * FROM numbers(1, 10)"), ["offset: 1", "count: 10"]); + assert.deepEqual(hints("CREATE TABLE t (a Int, v UInt64, d UInt8) ENGINE = ReplacingMergeTree(v, d) ORDER BY a"), [ + "ver: v", "is_deleted: d", + ]); + }); + + test("only where the parser reads a function or an engine", () => { + assert.deepEqual(hints("CREATE TABLE file (x Int, y Decimal(10, 2)) ENGINE = Memory"), []); + assert.deepEqual(hints("SELECT 1 IN (1, 2), (a, b)"), []); + assert.deepEqual(hints("SELECT dateDiff('day', a, b) +"), []); + }); + + test("in each statement, with offsets in the whole text", () => { + const sql = "SELECT 1;\nSELECT round(v, 2);\nINSERT INTO t VALUES (round(1, 2))"; + assert.deepEqual(parameterHints(sql), [{ offset: sql.indexOf("v, 2"), name: "x" }, { offset: sql.indexOf("2)"), name: "N" }]); + }); +}); diff --git a/packages/core/test/complete/statements.test.ts b/packages/core/test/complete/statements.test.ts new file mode 100644 index 0000000..3ebd849 --- /dev/null +++ b/packages/core/test/complete/statements.test.ts @@ -0,0 +1,67 @@ +import assert from "node:assert/strict"; +import { describe, test } from "node:test"; + +import { keywordDocumentation, SYSTEM_COMMANDS } from "../../src/index.js"; +import { ACCESS_DOCS, SYSTEM_DOCS } from "../../src/generated/statements.js"; + +/** The words the documentation covers and its first line, for the word `word` of the SQL. */ +function at(sql: string, word: string): [string, string] | undefined { + const found = keywordDocumentation(sql, sql.indexOf(word) + 1); + return found && [sql.slice(found.start, found.end), found.documentation.split("\n")[0]!]; +} + +describe("keywordDocumentation()", () => { + test("documents a SYSTEM command on each of its words", () => { + assert.deepEqual(at("SYSTEM FLUSH LOGS query_log", "LOGS"), ["SYSTEM FLUSH LOGS", "**SYSTEM FLUSH LOGS**"]); + assert.deepEqual(at("system flush logs", "system"), ["system flush logs", "**SYSTEM FLUSH LOGS**"]); + assert.deepEqual(at("SYSTEM DROP DNS CACHE", "CACHE"), ["SYSTEM DROP DNS CACHE", "**SYSTEM CLEAR|DROP DNS CACHE**"]); + assert.deepEqual(at("SYSTEM RELOAD DICTIONARY ON CLUSTER c d", "RELOAD"), ["SYSTEM RELOAD DICTIONARY", "**SYSTEM RELOAD DICTIONARY**"]); + assert.equal(at("SYSTEM STOP MERGES db.t", "db"), undefined); + }); + + test("takes the longest command, and the forms that only the syntax of a section names", () => { + assert.deepEqual(at("SYSTEM STOP TTL MERGES", "STOP"), ["SYSTEM STOP TTL MERGES", "**SYSTEM STOP TTL MERGES**"]); + assert.deepEqual(at("SYSTEM STOP db.t", "STOP"), ["SYSTEM STOP", "**SYSTEM STOP**"]); + assert.deepEqual(at("SYSTEM DISABLE ALL FAILPOINTS", "ALL"), ["SYSTEM DISABLE ALL FAILPOINTS", "**SYSTEM ENABLE|DISABLE FAILPOINT**"]); + }); + + test("documents the clauses of access statements, and the statement for the other keywords", () => { + assert.deepEqual(at("CREATE USER u VALID UNTIL '2030-01-01'", "UNTIL"), ["VALID UNTIL", "**CREATE USER: VALID UNTIL Clause**"]); + assert.deepEqual(at("CREATE USER u HOST ANY", "HOST"), ["HOST", "**CREATE USER: User Host**"]); + assert.deepEqual(at("CREATE USER u IDENTIFIED BY 'p'", "IDENTIFIED"), ["IDENTIFIED", "**CREATE USER: Identification**"]); + assert.deepEqual(at("CREATE POLICY p ON t USING 1 TO r", "TO"), ["TO", "**CREATE ROW POLICY: TO Clause**"]); + assert.deepEqual(at("ALTER ROW POLICY p ON t RENAME TO q", "TO"), ["RENAME TO", "**ALTER ROW POLICY**"]); + assert.deepEqual(at("CREATE QUOTA q ON CLUSTER c", "CLUSTER"), ["ON CLUSTER", "**CREATE QUOTA**"]); + assert.deepEqual(at("ALTER PROFILE p ADD SETTINGS max_threads = 1", "ADD"), ["ADD", "**ALTER SETTINGS PROFILE: Incremental form: `ADD` / `MODIFY` / `DROP`**"]); + assert.deepEqual(at("CREATE ROLE r SETTINGS max_threads = 1", "SETTINGS"), ["SETTINGS", "**CREATE ROLE**"]); + assert.deepEqual(at("CREATE QUOTA q KEYED BY user_name", "KEYED"), ["KEYED", "**CREATE QUOTA**"]); + }); + + test("only for the words that the parser reads as keywords", () => { + assert.equal(at("CREATE USER host HOST ANY", "host"), undefined); + assert.equal(at("CREATE ROW POLICY p ON t USING a = 1", "a ="), undefined); + assert.equal(at("SELECT 1", "SELECT"), undefined); + assert.equal(at("CREATE TABLE t (a Int) ENGINE = Memory", "TABLE"), undefined); + }); +}); + +describe("the generated documentation", () => { + test("has the pages of the five kinds of access entities, CREATE and ALTER", () => { + assert.deepEqual(ACCESS_DOCS.map((d) => d.statement).sort(), [ + "ALTER QUOTA", "ALTER ROLE", "ALTER ROW POLICY", "ALTER SETTINGS PROFILE", "ALTER USER", + "CREATE QUOTA", "CREATE ROLE", "CREATE ROW POLICY", "CREATE SETTINGS PROFILE", "CREATE USER", + ]); + for (const d of ACCESS_DOCS) assert.match(d.markdown, /```sql\n(?:CREATE|ALTER|-- )/, d.statement); + }); + + test("names SYSTEM commands that ClickHouse's parser has, or the first words of them", () => { + const commands = SYSTEM_COMMANDS.map((c) => c.replace(/_/g, " ")); + // Words after a command that its syntax takes, and the per-table forms of Managing Background Activity. + const unknown = SYSTEM_DOCS.flatMap((d) => d.keywords).filter((k) => !commands.some((c) => c === k || c.startsWith(`${k} `))); + assert.deepEqual(unknown, [ + "CLEAR QUERY CACHE TAG", "INSTRUMENT ADD FUNCTION HANDLER", "INSTRUMENT ADD FUNCTION HANDLER ARGUMENTS", "INSTRUMENT REMOVE ID", + "INSTRUMENT REMOVE ALL", "UNFREEZE WITH NAME", "REFRESH ALL BACKGROUND", + ]); + assert.ok(SYSTEM_DOCS.flatMap((d) => d.keywords).length > 100); + }); +}); diff --git a/packages/core/test/contract.test.ts b/packages/core/test/contract.test.ts index 5ddc0d4..8234f36 100644 --- a/packages/core/test/contract.test.ts +++ b/packages/core/test/contract.test.ts @@ -5,8 +5,8 @@ import api, { DATA_TYPES, KEYWORDS, LEXER_ERROR_MESSAGES, TokenKind, isErrorKind test("the default export carries the whole API", () => { assert.deepEqual(Object.keys(api).sort(), [ - "check", "checkOptions", "complete", "definition", "describe", "foldingRanges", "format", "highlight", "outline", "prepareRename", - "references", "rename", "selectionRanges", "tokenize", + "check", "checkOptions", "complete", "definition", "describe", "foldingRanges", "format", "highlight", "objectDefinitions", "objectReferences", + "objects", "outline", "prepareRename", "references", "rename", "selectionRanges", "tokenize", ]); }); diff --git a/packages/core/test/names/objects.test.ts b/packages/core/test/names/objects.test.ts new file mode 100644 index 0000000..1959000 --- /dev/null +++ b/packages/core/test/names/objects.test.ts @@ -0,0 +1,153 @@ +import assert from "node:assert/strict"; +import { describe, test } from "node:test"; + +import { objectDefinitions, objectReferences, objects, ParseCache } from "../../src/index.js"; + +/** The objects of `sql` as `kind database.name`, `kind name` or `temporary kind name`. */ +function made(sql: string): string[] { + return objects(sql).map((o) => `${o.temporary ? "temporary " : ""}${o.kind} ${o.database === undefined ? "" : `${o.database}.`}${o.name}`); +} + +/** The references of `sql` as `kind database.name@text`, with the text they cover. */ +function named(sql: string): string[] { + return objectReferences(sql).map((r) => `${r.kind} ${r.database === undefined ? "" : `${r.database}.`}${r.name}@${sql.slice(r.start, r.end)}`); +} + +describe("objects", () => { + test("finds what CREATE makes, by kind", () => { + const sql = [ + "CREATE TABLE IF NOT EXISTS db.t (x UInt8) ENGINE = Memory", + "CREATE OR REPLACE VIEW v AS SELECT 1", + "CREATE MATERIALIZED VIEW db.mv TO db.t AS SELECT 1 AS x", + "CREATE DICTIONARY d (k UInt64, v String) PRIMARY KEY k SOURCE(NULL()) LAYOUT(FLAT()) LIFETIME(0)", + "CREATE FUNCTION plus_one AS (x) -> x + 1", + "CREATE DATABASE IF NOT EXISTS logs", + "REPLACE TABLE `my table` (x UInt8) ENGINE = Memory", + "CREATE TEMPORARY TABLE tmp (x UInt8)", + "SELECT 1", + ].join(";\n"); + assert.deepEqual(made(sql), [ + "table db.t", "view v", "materialized view db.mv", "dictionary d", "function plus_one", "database logs", "table my table", "temporary table tmp", + ]); + const [first] = objects(sql); + assert.equal(sql.slice(first!.nameStart, first!.nameEnd), "db.t"); + assert.equal(first!.start, 0); + }); + + test("an earlier USE gives the database of a name without one", () => { + const sql = "CREATE TABLE a (x UInt8) ENGINE = Memory; USE db1; CREATE TABLE b (x UInt8) ENGINE = Memory; USE DATABASE db2; CREATE VIEW c AS SELECT 1; CREATE TABLE other.d (x UInt8) ENGINE = Memory; CREATE TEMPORARY TABLE e (x UInt8); CREATE FUNCTION f AS () -> 1"; + assert.deepEqual(made(sql), ["table a", "table db1.b", "view db2.c", "table other.d", "temporary table e", "function f"]); + }); + + test("a parameter for the database is kept as written, and a parameter for the name makes nothing", () => { + assert.deepEqual(made("CREATE TABLE {CLICKHOUSE_DATABASE:Identifier}.t (x UInt8) ENGINE = Memory; CREATE TABLE {name:Identifier} (x UInt8) ENGINE = Memory"), [ + "table {CLICKHOUSE_DATABASE:Identifier}.t", + ]); + }); + + test("inline data and comments do not make objects", () => { + const sql = "INSERT INTO t FORMAT CSV\n1,'CREATE TABLE fake (x UInt8) ENGINE = Memory;\n\n-- CREATE TABLE commented (x UInt8)\nCREATE TABLE real (x UInt8) ENGINE = Memory"; + assert.deepEqual(made(sql), ["table real"]); + }); +}); + +describe("objectReferences", () => { + test("finds the tables that statements name", () => { + const sql = [ + "SELECT t.x, extract(DAY FROM d) FROM db.t JOIN u ON t.x = u.x", + "INSERT INTO db.t VALUES (1)", + "INSERT INTO TABLE u SELECT * FROM v", + "ALTER TABLE t DELETE WHERE 1", + "DROP TABLE IF EXISTS a, b", + "TRUNCATE c", + "OPTIMIZE TABLE e FINAL", + "RENAME TABLE f TO g", + "EXCHANGE TABLES h AND i", + "DESCRIBE j", + "SHOW CREATE TABLE k", + "EXISTS TABLE l", + "CREATE TABLE m AS n", + "CREATE MATERIALIZED VIEW o TO p AS SELECT 1 FROM q", + ].join(";\n"); + assert.deepEqual(named(sql), [ + "database db@db", "table db.t@db.t", "table u@u", + "database db@db", "table db.t@db.t", + "table u@u", "table v@v", + "table t@t", + "table a@a", "table b@b", + "table c@c", + "table e@e", + "table f@f", "table g@g", + "table h@h", "table i@i", + "table j@j", + "table k@k", + "table l@l", + "table m@m", "table n@n", + "table o@o", "table p@p", "table q@q", + ]); + }); + + test("an earlier USE gives the database, and USE names a database", () => { + assert.deepEqual(named("SELECT * FROM a; USE db1; SELECT * FROM b; USE DATABASE db2; SELECT * FROM other.c"), [ + "table a@a", "database db1@db1", "table db1.b@b", "database db2@db2", "database other@other", "table other.c@other.c", + ]); + }); + + test("leaves out a name that a CTE of the statement takes, and statements that do not parse", () => { + assert.deepEqual(named("WITH t AS (SELECT 1) SELECT * FROM t JOIN db.t USING (x); SELECT * FROM u WHERE; SELECT * FROM `v w`"), [ + "database db@db", "table db.t@db.t", "table v w@`v w`", + ]); + }); + + test("takes the statements from the cache", () => { + const cache = new ParseCache(); + assert.deepEqual(objectReferences("SELECT * FROM t", { cache }), objectReferences("SELECT * FROM t")); + }); +}); + +describe("objectDefinitions", () => { + const texts = new Map([ + ["a.sql", "USE db1; CREATE TABLE t (x UInt8) ENGINE = Memory; CREATE TABLE t2 (x UInt8) ENGINE = Memory; CREATE DATABASE db1"], + ["b.sql", "CREATE TABLE db1.t (x UInt8) ENGINE = Memory; CREATE TABLE db2.t (x UInt8) ENGINE = Memory; CREATE TEMPORARY TABLE t2 (x UInt8); SELECT * FROM t2; USE db2; SELECT * FROM t2; SELECT * FROM db1.t2"], + ["c.sql", "CREATE TABLE t (x UInt8) ENGINE = Memory; CREATE TABLE t (y UInt8) ENGINE = Memory; CREATE FUNCTION t AS () -> 1; SELECT * FROM t"], + ["d.sql", "SELECT * FROM t; SELECT * FROM db2.t; SELECT * FROM t2; SELECT * FROM db1.t; USE db1; SELECT * FROM t"], + ]); + const files = new Map([...texts].map(([file, sql]) => [file, objects(sql)])); + /** Where the object is created, as file:database.name@offset. */ + const where = (file: string, written: string, nth = 0) => { + const sql = texts.get(file)!; + const object = objects(sql).filter((o) => sql.slice(o.nameStart, o.nameEnd) === written)[nth]!; + return `${file}:${object.database ?? ""}.${object.name}@${object.nameStart}`; + }; + /** The definitions of the `nth` reference to `written` in `file`. */ + const of = (file: string, written: string, nth = 0) => { + const sql = texts.get(file)!; + const reference = objectReferences(sql).filter((r) => sql.slice(r.start, r.end) === written)[nth]; + assert.ok(reference, `${written} is referenced ${nth + 1} times in ${file}`); + return objectDefinitions(reference, file, files).map(({ file, object }) => `${file}:${object.database ?? ""}.${object.name}@${object.nameStart}`); + }; + + test("a database written or set by USE finds the tables of that database", () => { + assert.deepEqual(of("d.sql", "db2.t"), [where("b.sql", "db2.t")]); + assert.deepEqual(of("d.sql", "t", 1), [where("a.sql", "t"), where("b.sql", "db1.t")]); + }); + + test("a name in the session's database may be any table of that name, but not a function", () => { + assert.deepEqual(of("d.sql", "t"), [where("a.sql", "t"), where("b.sql", "db1.t"), where("b.sql", "db2.t"), where("c.sql", "t"), where("c.sql", "t", 1)]); + }); + + test("in the same file, the last CREATE before the name", () => { + assert.deepEqual(of("c.sql", "t", 2), [where("c.sql", "t", 1)]); + }); + + test("a temporary table is found only from its own file, before the tables of the database", () => { + assert.deepEqual(of("b.sql", "t2"), [where("b.sql", "t2")]); + assert.deepEqual(of("b.sql", "t2", 1), [where("b.sql", "t2")]); + assert.deepEqual(of("b.sql", "db1.t2"), [where("a.sql", "t2")]); + assert.deepEqual(of("d.sql", "t2"), [where("a.sql", "t2")]); + }); + + test("a database name finds CREATE DATABASE", () => { + assert.deepEqual(of("d.sql", "db1"), [where("a.sql", "db1")]); + }); +}); diff --git a/packages/core/test/options/check.test.ts b/packages/core/test/options/check.test.ts index c22e792..dcc45ef 100644 --- a/packages/core/test/options/check.test.ts +++ b/packages/core/test/options/check.test.ts @@ -55,6 +55,54 @@ describe("checkOptions", () => { assert.deepEqual(problems("ALTER TABLE t MODIFY SETTING index_granularity = 8192, s3queue_processing_threads_num = 2"), []); }); + test("checks maps given to scalar settings, URIs and the compatibility version", () => { + assert.deepEqual(problems("SET max_threads = {'a':'b', 'it''s':'x'}"), [ + "error: Invalid value [('a', 'b'), ('it\\'s', 'x')] of the setting, which needs unsigned long long: while setting 'max_threads' to value [('a', 'b'), ('it\\'s', 'x')].", + ]); + assert.deepEqual(problems("SET use_uncompressed_cache = {}"), ["error: Invalid value [] of the setting, which needs bool: while setting 'use_uncompressed_cache' to value []."]); + assert.deepEqual(problems("SELECT 1 SETTINGS max_execution_time = {'a':'b'}"), [ + "error: Invalid value [('a', 'b')] of the setting, which needs double: while setting 'max_execution_time' to value [('a', 'b')].", + ]); + assert.deepEqual(problems("SET opentelemetry_start_keeper_trace_probability = NULL"), [ + "error: Invalid value NULL of the setting, which needs float: while setting 'opentelemetry_start_keeper_trace_probability' to value NULL.", + ]); + assert.deepEqual(problems("SET log_comment = {'a':'b'}"), ["error: Bad get: has Map, requested String: while setting 'log_comment' to value [('a', 'b')]."]); + assert.deepEqual(problems("CREATE TABLE t (x UInt64) ENGINE = MergeTree ORDER BY x SETTINGS index_granularity = {'a':'b'}"), [ + "error: Invalid value [('a', 'b')] of the setting, which needs unsigned long long: while setting 'index_granularity' to value [('a', 'b')].", + ]); + assert.deepEqual(problems("SET format_avro_schema_registry_url = 1"), [ + "error: Bad get: has UInt64, requested String: while setting 'format_avro_schema_registry_url' to value 1.", + ]); + assert.deepEqual(problems("SET format_avro_schema_registry_url = 'a b'"), [ + "error: Cannot parse URI: Bad URI syntax: URI contains invalid characters: while setting 'format_avro_schema_registry_url' to value 'a b'.", + ]); + assert.deepEqual(problems("SET compatibility = 'abc'"), [ + "error: Cannot parse ClickHouse version here: abc: in attempt to set the value of setting 'compatibility' to 'abc'.", + ]); + assert.deepEqual(problems("SET totals_auto_threshold = '10K'"), [ + "error: Cannot parse input: expected 'eof' before: 'K': while setting 'totals_auto_threshold' to value '10K'.", + ]); + assert.deepEqual(problems("SET max_block_size = '-1'"), ["error: Unsigned type must not contain '-' symbol: while setting 'max_block_size' to value '-1'."]); + assert.deepEqual(problems("SET max_threads = '-1'"), ["error: Unsigned type must not contain '-' symbol: while setting 'max_threads' to value '-1'."]); + assert.deepEqual(problems("SET connect_timeout = ''"), ["error: Cannot read floating point value here: : while setting 'connect_timeout' to value ''."]); + assert.deepEqual(problems("SET limit = '-inf'"), ["error: Float setting value must be finite, got -inf: while setting 'limit' to value '-inf'."]); + assert.deepEqual(problems("SET totals_auto_threshold = 16777217"), [ + "error: Field value 16777217 is out of range of float type: while setting 'totals_auto_threshold' to value 16777217.", + ]); + assert.deepEqual(problems("SET connect_timeout = 9223372036855"), [ + "error: Cannot convert seconds to microseconds: the setting's value in seconds is too big: 9223372036855: while setting 'connect_timeout' to value 9223372036855.", + ]); + assert.deepEqual(problems("SET query_plan_read_in_order_through_join = 2"), [ + "error: Field value 2 is out of range of bool type: while setting 'query_plan_read_in_order_through_join' to value 2.", + ]); + for (const ok of ["SET limit = 16777217", "SET connect_timeout = '1e3'", "SET connect_timeout = 9223372036854", "SET query_plan_read_in_order_through_join = 'force'", + "SET promql_evaluation_time = 'auto'", "SET os_thread_priority = '-1'", "SET max_block_size = '+5'", "SET totals_auto_threshold = '.5'"]) + assert.deepEqual(problems(ok), [], ok); + for (const ok of ["SET compatibility = '24.8'", "SET compatibility = ''", "SET compatibility = '+1.2.3.4.5'", "SET format_avro_schema_registry_url = 'http://a:8081'", + "SET format_avro_schema_registry_url = ''", "SET additional_table_filters = {}"]) + assert.deepEqual(problems(ok), [], ok); + }); + test("checks EXPLAIN, BACKUP and RESTORE settings", () => { assert.match(problems("EXPLAIN PLAN zz = 1 SELECT 1")[0]!, /^error: Unknown setting "zz" for EXPLAIN PLAN query\. Supported settings: actions, /); assert.deepEqual(problems("EXPLAIN PLAN actions = 2 SELECT 1"), ['error: Invalid value 2 for setting "actions". Expected boolean type.']); @@ -80,6 +128,56 @@ describe("checkOptions", () => { assert.deepEqual(problems(`ATTACH TABLE t (x UInt64 CODEC(Zzz)) ENGINE = MergeTree ORDER BY x`), []); }); + test("checks that each codec takes the types the column holds", () => { + const errors = (sql: string) => problems(sql).filter((p) => p.startsWith("error")); + const table = (column: string) => `CREATE TABLE t (${column}) ENGINE = MergeTree ORDER BY tuple()`; + assert.deepEqual(errors(table("x String CODEC(Delta, LZ4)")), ["error: Codec Delta is not applicable for String because the data type is not of fixed size."]); + assert.deepEqual(errors(table("x LowCardinality(String) CODEC(T64, LZ4)")), ["error: T64 codec is not supported for specified type String."]); + assert.deepEqual(errors(table("x Map(String, UInt64) CODEC(DoubleDelta)")), ["error: Codec DoubleDelta is not applicable for String because the data type is not numeric."]); + assert.deepEqual(errors(table("x Tuple(a UInt64, b Float64) CODEC(T64, LZ4)")), ["error: T64 codec is not supported for specified type Float64."]); + assert.deepEqual(errors(table("x UInt64 CODEC(FPC)")), ["error: Codec 'FPC' is not applicable for UInt64 because the data type is not Float*."]); + assert.deepEqual(errors(table("x DOUBLE CODEC(T64, LZ4)")), ["error: T64 codec is not supported for specified type Float64."]); + assert.deepEqual(errors("ALTER TABLE t MODIFY COLUMN x String CODEC(Gorilla, LZ4)"), ["error: Codec Gorilla is not applicable for String because the data type is not of fixed size."]); + // A codec given its width does not look at the type, and a column without a written type is left alone. + for (const ok of ["x String CODEC(Delta(8), LZ4)", "x Nullable(UInt64) CODEC(T64, LZ4)", "x Array(Float32) CODEC(FPC, LZ4)", "x DEFAULT 'a' CODEC(Delta, LZ4)", + "x INT UNSIGNED CODEC(Delta, LZ4)", "x UInt64 DEFAULT y::String CODEC(Delta, LZ4)", "x DateTime64(3) CODEC(Delta, LZ4)", "x String CODEC(ZSTD)"]) + assert.deepEqual(errors(table(ok)), [], ok); + assert.deepEqual(errors("ALTER TABLE t MODIFY COLUMN x CODEC(Delta, LZ4)"), []); + }); + + test("checks the parameters of Quantized and SZ3", () => { + const errors = (codec: string) => problems(`CREATE TABLE t (x Array(Float32) CODEC(${codec})) ENGINE = MergeTree ORDER BY tuple()`).filter((p) => p.startsWith("error")); + assert.deepEqual(errors("Quantized(1)"), ["error: Codec Quantized requires 2 to 4 parameters: Quantized(method, dimensions[, bits[, m]]) ."]); + assert.deepEqual(errors("Quantized(1, 2)"), ["error: First argument of codec Quantized (method) must be a string literal."]); + assert.deepEqual(errors("Quantized('int8', 1.5)"), ["error: Second argument of codec Quantized (dimensions) must be an unsigned integer."]); + for (const ok of ["Quantized('product', 64, 8, 8)", "Quantized('rabitq', 64)", "Quantized('int8', 8, 4)"]) assert.deepEqual(errors(ok), [], ok); + const sz3 = (codec: string) => problems(`CREATE TABLE t (x Float64 CODEC(${codec})) ENGINE = MergeTree ORDER BY tuple()`).filter((p) => p.startsWith("error")); + assert.deepEqual(sz3("SZ3(1)"), ["error: Codec SZ3 must have 0 or 3 arguments but 1 arguments are given."]); + for (const ok of ["SZ3", "SZ3('ALGO_INTERP', 'NORM', 0.1)"]) assert.deepEqual(sz3(ok), [], ok); + }); + + test("checks the arguments of the text and vector_similarity indexes", () => { + const text = (args: string) => problems(`CREATE TABLE t (x String, INDEX i x TYPE text${args} GRANULARITY 1) ENGINE = MergeTree ORDER BY tuple()`); + assert.deepEqual(text(""), ["error: Text index argument 'tokenizer' is required."]); + assert.deepEqual(text("(1)"), ["error: Cannot mix key-value pair and single argument as text index arguments."]); + assert.deepEqual(text("(tokenizer = zz)"), ["error: Unknown tokenizer: 'zz'."]); + assert.deepEqual(text("(tokenizer = array, zz = 1, aa = 2)"), ["error: Unexpected text index arguments: aa, zz."]); + assert.deepEqual(text("(tokenizer = array, tokenizer = ngrams)"), ["error: Text index 'tokenizer' argument is specified more than once."]); + assert.deepEqual(text("(tokenizer = array, posting_list_block_size = 0)"), ["error: Text index argument 'posting_list_block_size' must be greater than 0, but got 0."]); + for (const ok of ["(tokenizer = ngrams(3))", "(tokenizer = 'splitByNonAlpha', preprocessor = lower(x))", "(tokenizer = array, dictionary_block_size = 512)"]) + assert.deepEqual(text(ok), [], ok); + const vector = (args: string) => problems(`CREATE TABLE t (x Array(Float32), INDEX i x TYPE vector_similarity${args} GRANULARITY 1) ENGINE = MergeTree ORDER BY tuple()`); + assert.deepEqual(vector("('hnsw', 'L2Distance', 3, 'bf16')"), ["error: Vector similarity index must have three or six arguments."]); + for (const ok of ["('hnsw', 'L2Distance', 3)", "('hnsw', 'cosineDistance', 3, 'i8', 16, 100)"]) assert.deepEqual(vector(ok), [], ok); + }); + + test("checks the compression level of INTO OUTFILE", () => { + assert.deepEqual(problems("SELECT 1 INTO OUTFILE 'f' COMPRESSION 'zstd' LEVEL 23"), ["error: Invalid compression level, must be positive integer in range 1-22."]); + assert.deepEqual(problems("SELECT 1 INTO OUTFILE 'f' COMPRESSION 'xz' LEVEL 0"), ["error: Invalid compression level, must be positive integer in range 1-9."]); + for (const ok of ["SELECT 1 INTO OUTFILE 'f' COMPRESSION 'gzip' LEVEL 12", "SELECT 1 INTO OUTFILE 'f' COMPRESSION 'lz4' LEVEL 0x05", "SELECT 1 INTO OUTFILE 'f.gz' LEVEL 3"]) + assert.deepEqual(problems(ok), [], ok); + }); + test("checks skipping index and statistics types", () => { const table = (index: string, type = "String") => `CREATE TABLE t (x ${type}, INDEX i x TYPE ${index}) ENGINE = MergeTree ORDER BY tuple()`; assert.match(problems(table("zz GRANULARITY 1"))[0]!, /^error: Unknown Index type 'zz'\. Available index types: /); diff --git a/packages/core/test/options/fixes.test.ts b/packages/core/test/options/fixes.test.ts new file mode 100644 index 0000000..dc04e32 --- /dev/null +++ b/packages/core/test/options/fixes.test.ts @@ -0,0 +1,97 @@ +import assert from "node:assert/strict"; +import { describe, test } from "node:test"; + +import { check, checkOptions, checkWithRoles, type Diagnostic, messageFixes, nameHints, ParseCache } from "../../src/index.js"; + +/** Each fix of the diagnostics of `sql` as the text it gives, with a * when it is preferred. */ +function fixed(sql: string, diagnostics: Diagnostic[] = [...check(sql), ...checkOptions(sql)]): string[] { + return diagnostics.flatMap((d) => + (d.fixes ?? []).map((fix) => { + let text = sql; + for (const edit of [...fix.edits].sort((a, b) => b.start - a.start)) text = text.slice(0, edit.start) + edit.newText + text.slice(edit.end); + return `${fix.preferred ? "*" : ""}${text}`; + }), + ); +} + +describe("quick fixes", () => { + test("write the keyword that a syntax error hints at, in the case of the word", () => { + assert.deepEqual(fixed("SELEC 1"), ["*SELECT 1"]); + assert.deepEqual(fixed("selec 1"), ["*select 1"]); + assert.deepEqual(fixed("SELECT 1 FROM t WHER x"), []); + }); + + test("keep their offsets through the parse cache", () => { + const cache = new ParseCache(); + const sql = "SELECT 1;\nSELECT 2 SETTINGS max_thread = 1;\nSELEC 3"; + checkWithRoles(sql, cache); + const again = [...checkWithRoles(sql, cache).diagnostics, ...checkOptions(sql, cache)]; + assert.deepEqual(fixed(sql, again), ["*SELECT 1;\nSELECT 2 SETTINGS max_thread = 1;\nSELECT 3", "*SELECT 1;\nSELECT 2 SETTINGS max_threads = 1;\nSELEC 3"]); + }); + + test("rename unknown settings to the closest ones of their place", () => { + assert.deepEqual(fixed("SELECT 1 SETTINGS max_thread = 1"), ["*SELECT 1 SETTINGS max_threads = 1"]); + assert.deepEqual(fixed("SET Max_Threads = 1"), ["*SET max_threads = 1"]); + assert.deepEqual(fixed("CREATE TABLE t (x UInt64) ENGINE = MergeTree ORDER BY x SETTINGS index_granularit = 1"), [ + "*CREATE TABLE t (x UInt64) ENGINE = MergeTree ORDER BY x SETTINGS index_granularity = 1", + ]); + assert.deepEqual(fixed("SET zz_nothing_like_it = 1"), []); + }); + + test("rename unknown codecs, index types, statistics, formats and compression methods", () => { + assert.deepEqual(fixed("CREATE TABLE t (x UInt64 CODEC(ZST(3))) ENGINE = MergeTree ORDER BY x"), ["*CREATE TABLE t (x UInt64 CODEC(ZSTD(3))) ENGINE = MergeTree ORDER BY x"]); + // Only index types that take the parameters written. + assert.deepEqual(fixed("CREATE TABLE t (x UInt64, INDEX i x TYPE minmx) ENGINE = MergeTree ORDER BY x"), ["*CREATE TABLE t (x UInt64, INDEX i x TYPE minmax) ENGINE = MergeTree ORDER BY x"]); + assert.deepEqual(fixed("CREATE TABLE t (s String, INDEX i s TYPE tokenbf_v2(256, 2, 0)) ENGINE = MergeTree ORDER BY s"), [ + "*CREATE TABLE t (s String, INDEX i s TYPE tokenbf_v1(256, 2, 0)) ENGINE = MergeTree ORDER BY s", + ]); + assert.deepEqual(fixed("CREATE TABLE t (x UInt64 STATISTICS(tdigst)) ENGINE = MergeTree ORDER BY x"), ["*CREATE TABLE t (x UInt64 STATISTICS(tdigest)) ENGINE = MergeTree ORDER BY x"]); + assert.deepEqual(fixed("SELECT 1 FORMAT JSONEachRw"), ["*SELECT 1 FORMAT JSONEachRow", "SELECT 1 FORMAT BSONEachRow"]); + // An INSERT takes the formats ClickHouse reads. + assert.ok(!fixed("INSERT INTO t FORMAT Prety\n").some((text) => text.includes("Pretty"))); + assert.deepEqual(fixed("SELECT 1 INTO OUTFILE 'f' COMPRESSION 'zstdd'"), ["*SELECT 1 INTO OUTFILE 'f' COMPRESSION 'zstd'"]); + }); + + test("fix values: an enum value's typo, and a value that the setting takes as a string", () => { + assert.deepEqual(fixed("SET join_algorithm = 'hsh'"), ["*SET join_algorithm = 'hash'"]); + assert.deepEqual(fixed("SET join_algorithm = 'hash,grce_hash'"), ["*SET join_algorithm = 'hash,grace_hash'"]); + assert.deepEqual(fixed("SET join_algorithm = 'has,hash'"), ["*SET join_algorithm = 'hash,hash'"]); + assert.deepEqual(fixed("SET format_csv_delimiter = 1"), ["*SET format_csv_delimiter = '1'"]); + // Quoted, these are still wrong, or may be: settings of strings such as format check their values further. + assert.deepEqual(fixed("SET join_algorithm = 1"), []); + assert.deepEqual(fixed("SET format_csv_delimiter = 12"), []); + assert.deepEqual(fixed("SET format = 1"), []); + assert.deepEqual(fixed("SET log_comment = NULL"), []); + }); + + test("add ENGINE where default_table_engine is None", () => { + const none = "SET default_table_engine = 'None';\n"; + assert.deepEqual(checkOptions("CREATE TABLE t (x UInt64)"), []); + assert.deepEqual(checkOptions(`${none}CREATE TABLE t (x UInt64)`).map((d) => d.message), ["Table engine is not specified in CREATE query."]); + assert.deepEqual(fixed(`${none}CREATE TABLE t (x UInt64)`), [`*${none}CREATE TABLE t (x UInt64) ENGINE = MergeTree ORDER BY tuple()`]); + assert.deepEqual(fixed(`${none}CREATE TABLE IF NOT EXISTS t (x UInt64) PRIMARY KEY x`), [`*${none}CREATE TABLE IF NOT EXISTS t (x UInt64) ENGINE = MergeTree PRIMARY KEY x`]); + assert.deepEqual(fixed(`${none}CREATE OR REPLACE TABLE t AS SELECT 1 AS x ORDER BY x`), [`*${none}CREATE OR REPLACE TABLE t ENGINE = MergeTree ORDER BY tuple() AS SELECT 1 AS x ORDER BY x`]); + // These do not need one: a temporary table, the engine of another table or of a table function, and after the setting is reset. + for (const sql of ["CREATE TEMPORARY TABLE t (x UInt64)", "CREATE TABLE t AS other", "CREATE TABLE t AS numbers(1)", "CREATE TABLE t (x UInt64) ENGINE = Memory", + "SET default_table_engine = DEFAULT; CREATE TABLE t (x UInt64)", "SET default_table_engine = 'Memory'; CREATE TABLE t (x UInt64)"]) + assert.deepEqual(checkOptions(`${none}${sql}`), [], sql); + }); + + test("nameHints follows ClickHouse's NamePrompter", () => { + assert.deepEqual(nameHints("max_thread", ["max_threads", "max_insert_threads", "min_threads"]), [{ name: "max_threads", distance: 1 }, { name: "min_threads", distance: 3 }]); + // A name much shorter or longer is no hint, however few edits apart. + assert.deepEqual(nameHints("ab", ["abc"]), []); + }); + + test("messageFixes reads the hints of ClickHouse's messages", () => { + const sql = "SELECT arrayMp(x -> x, [1]) FROM sytem.numbers"; + const at = (name: string) => [sql.indexOf(name), sql.indexOf(name) + name.length] as const; + const titles = (start: number, end: number, message: string) => messageFixes(sql, start, end, message).map((f) => `${f.preferred ? "*" : ""}${f.title}`); + assert.deepEqual(titles(...at("arrayMp"), "Function with name `arrayMp` does not exist. Maybe you meant: ['arrayMax','arrayMap']. In scope ..."), ["*Change to arrayMap", "Change to arrayMax"]); + assert.deepEqual(titles(...at("arrayMp"), "Maybe you meant: ['arrayMax','arrayMan']"), ["Change to arrayMax", "Change to arrayMan"]); + assert.deepEqual(titles(...at("sytem"), "Database sytem does not exist. Maybe you meant system?."), ["*Change to system"]); + // Not a name: the whole query, when the message names nothing in it. + assert.deepEqual(titles(0, sql.length, "Unknown data type family: Strng. Maybe you meant: ['String']"), []); + assert.deepEqual(titles(...at("arrayMp"), "Function with name `arrayMp` does not exist."), []); + }); +}); diff --git a/packages/vscode/CHANGELOG.md b/packages/vscode/CHANGELOG.md index 28abadc..e250548 100644 --- a/packages/vscode/CHANGELOG.md +++ b/packages/vscode/CHANGELOG.md @@ -1,5 +1,22 @@ # Changelog +## 0.5.0 + +- More values that ClickHouse refuses are marked: maps given to numbers and booleans, size suffixes and `inf` in float + settings, URIs, `compatibility` versions, codecs on column types they do not take, the parameters of `Quantized` + and `SZ3`, the arguments of the `text` and `vector_similarity` indexes, and the `LEVEL` of `INTO OUTFILE`. +- Quick fixes for errors that know the answer: the closest setting, codec, index or statistics type, format, + compression method, enum value, keyword or test hint error name; the names in "Maybe you meant" of the server check; + a number in quotes for a setting of one character; and `ENGINE = MergeTree` after + `SET default_table_engine = 'None'`, which is now marked as ClickHouse refuses it. +- Inlay hints, off by default (`clickhouseSql.inlayHints.enabled`): the parameter names of the arguments of function, + table function and engine calls, and the type of a query parameter after the `param_name` that sets it. +- Hover over SYSTEM commands and over the clauses of `CREATE` and `ALTER` `USER`, `ROLE`, `QUOTA`, `ROW POLICY` and + `SETTINGS PROFILE` shows ClickHouse's documentation of them. +- Go to Symbol in Workspace finds the tables, views, materialized views, dictionaries, functions and databases that + the CREATE statements of the workspace's ClickHouse SQL files make. Go to definition and Find All References of a + table in a query reach the CREATE statement and the uses in every file, with the database that `USE` sets. + ## 0.4.0 - Suggestions and signature help for the arguments of every documented function. diff --git a/packages/vscode/README.md b/packages/vscode/README.md index f8fe70e..9029d74 100644 --- a/packages/vscode/README.md +++ b/packages/vscode/README.md @@ -6,7 +6,8 @@ Support for [ClickHouse](https://github.com/ClickHouse/ClickHouse) SQL that read - **Highlighting.** Words are coloured by how the parser reads them, so a column called `timestamp` or `key` looks like a name, not a keyword. - **Syntax errors as you type**, in ClickHouse's words, such as `Expected one of: BY, end of query`. -- **Options.** Settings of queries, MergeTree, Kafka and the other engines, databases, columns, EXPLAIN and BACKUP, with their values; codecs and their parameters, index and statistics types, database engines, formats and compression methods: autocomplete, hover, and an error where ClickHouse would refuse one, such as `max_threads = 'abc'` or `CODEC(ZSTD(23))`. +- **Options.** Settings of queries, MergeTree, Kafka and the other engines, databases, columns, EXPLAIN and BACKUP, with their values; codecs and their parameters, index and statistics types, database engines, formats and compression methods: autocomplete, hover, and an error where ClickHouse would refuse one, such as `max_threads = 'abc'`, `CODEC(ZSTD(23))`, a codec on a type it does not take, as `String CODEC(Delta)`, the arguments of the `text` and `vector_similarity` indexes, or `COMPRESSION 'zstd' LEVEL 23`. +- **Quick fixes** for errors that know the answer: the setting, codec, index or statistics type, format, compression method, enum value or keyword that a typo is closest to, the function, type, table or database that the server's "Maybe you meant" names, the error name of a test hint, a number in quotes for a setting of one character, and `ENGINE = MergeTree` after `SET default_table_engine = 'None'`. - **Formatting.** Format Document and Format Selection lay out queries like `clickhouse format`, changing only whitespace and the case of keywords. - **Autocomplete** of the keywords that fit at the cursor and of ClickHouse's functions, types, settings, table engines and formats. - **Engine arguments.** In `ENGINE = Name(...)`, autocomplete offers what the argument at the cursor takes, such as a ZooKeeper path, a column of the table of a type that fits, a format or a cluster, and signature help shows the engine's parameters. @@ -15,8 +16,10 @@ Support for [ClickHouse](https://github.com/ClickHouse/ClickHouse) SQL that read - **Folding** of statements, subqueries and comments of several lines, and of `-- #region` ... `-- #endregion` blocks. - **Expand Selection** grows from the word at the cursor to its expression, call, clause, subquery and statement, and Shrink Selection goes back. - **Table function arguments.** Calls such as `file`, `s3` and `cluster` show their arguments and suggest formats, compression methods and names from the server. Aggregate combinators such as `sumIf` and window functions also have signature help. -- **Hover and signature help** show ClickHouse's documentation of functions, settings, engines and formats, and the arguments of the function you are typing. +- **Hover and signature help** show ClickHouse's documentation of functions, settings, engines and formats, of SYSTEM commands such as `SYSTEM FLUSH LOGS`, and of the clauses of `CREATE` and `ALTER` `USER`, `ROLE`, `QUOTA`, `ROW POLICY` and `SETTINGS PROFILE`, and the arguments of the function you are typing. +- **Inlay hints** (off by default). The names of the parameters before the arguments of calls, as in `dateDiff(unit: 'day', startdate: a, enddate: b)`, and the type of a query parameter after the `param_name` that sets it. - **Go to definition, references and rename** of aliases, CTEs, table aliases and lambda and query parameters, scoped as ClickHouse scopes them; a name that ClickHouse could read otherwise is left alone. +- **Workspace symbols and tables across files.** Go to Symbol in Workspace (Ctrl+T) finds the tables, views, materialized views, dictionaries, functions and databases that the CREATE statements of the folder's ClickHouse SQL files make. From a table in `FROM db.t`, `JOIN t`, `INSERT INTO t`, `ALTER TABLE t` and the other places that name one, go to definition and Find All References reach its CREATE statement and its uses in any of those files. A name without a database means the one that an earlier `USE db` sets, as in ClickHouse; where that is not known, every table of that name is offered. - **Query parameters.** Autocomplete offers the types after `{name:`, only `Identifier` where a name goes, as in `FROM {table:Identifier}`, the parameters of the file after `{`, and `param_name` in `SET` and `SETTINGS`. Hover over `{name:Type}` or `param_name` says how ClickHouse reads the value and where it can come from: `SET param_name = 'value'`, `SETTINGS param_name = 'value'`, `--param_name='value'` of `clickhouse-client` and `clickhouse local`, or `param_name=value` in the URL of the HTTP interface. - **Test hints and tags** of ClickHouse's own tests. In `-- { serverError UNKNOWN_TABLE }` and the other hints of `clickhouse-client` (`clientError`, `error`, `echo`, `echoOn`, `echoOff`, `retry N until|while`), autocomplete offers the commands and error names, hover gives the code of each error, and a hint that would make `clickhouse-client` stop, such as an unknown error name, is marked in its words. A `serverError` hint on a statement that the client rejects is marked too. In `-- Tags:` lines, autocomplete and hover know the tags that `clickhouse-test` reads, and mark `no-stress`, which it refuses, and tags it ignores; `-- Random settings limits:` lines are checked as it reads them. - **Other dialects.** After `SET dialect = 'kusto'`, `'prql'` or another dialect, statements get no syntax errors until `SET dialect = 'clickhouse'`, as `clickhouse-client` parses them in that dialect. @@ -55,11 +58,17 @@ For example, `"lower"` formats `SELECT a FROM t` as `select a from t`. It can di - `"lower"`: `select`, `from`, `where` - `"preserve"`: Keywords keep the case they are written in +### `clickhouseSql.inlayHints.enabled` + +Default `false`; boolean; user or workspace settings, also per language. + +Show the name of the parameter before each argument of a function, table function or engine call with two arguments or more, as in `dateDiff(unit: 'day', startdate: a, enddate: b)`, and the type after each `param_name` that a `{name:Type}` of the file uses. An argument that is already a name equal to the parameter gets no name. VS Code shows them as `editor.inlayHints.enabled` says. + ### `clickhouseSql.maxFileSize` Default `5000000`; integer, at least 0; user settings only. -Files longer than this many characters get no syntax check, semantic highlighting, formatting, outline, folding, Expand Selection, completion, hover, signature help, go to definition, references, rename or server check, which would slow VS Code down; a message says so once per file. The TextMate highlighting still works. +Files longer than this many characters get no syntax check, semantic highlighting, formatting, outline, folding, Expand Selection, completion, hover, signature help, go to definition, references, rename, workspace symbols or server check, which would slow VS Code down; a message says so once per file. The TextMate highlighting still works. For example, `20000000` lets files of up to 20 million characters in. diff --git a/packages/vscode/package.json b/packages/vscode/package.json index 1442d67..a37ab38 100644 --- a/packages/vscode/package.json +++ b/packages/vscode/package.json @@ -2,7 +2,7 @@ "name": "clickhouse-sql", "displayName": "ClickHouse SQL", "description": "Syntax highlighting and syntax checking for ClickHouse SQL", - "version": "0.4.0", + "version": "0.5.0", "publisher": "valerypetrov", "private": true, "license": "SEE LICENSE IN LICENSE", @@ -31,6 +31,10 @@ "Linters" ], "main": "./dist/extension.js", + "activationEvents": [ + "workspaceContains:**/*.chsql", + "workspaceContains:**/*.clickhouse.sql" + ], "capabilities": { "untrustedWorkspaces": { "supported": true @@ -117,12 +121,18 @@ "scope": "language-overridable", "markdownDescription": "The case of keywords when **Format Document** and **Format Selection** lay out ClickHouse SQL. Only words that the parser reads as keywords change; names, strings and comments stay as written.\n\nFor example, `\"lower\"` formats `SELECT a FROM t` as `select a from t`. It can differ per language, as in `\"[clickhouse-sql]\": { \"clickhouseSql.format.keywordCase\": \"lower\" }`." }, + "clickhouseSql.inlayHints.enabled": { + "type": "boolean", + "default": false, + "scope": "language-overridable", + "markdownDescription": "Show the name of the parameter before each argument of a function, table function or engine call with two arguments or more, as in `dateDiff(unit: 'day', startdate: a, enddate: b)`, and the type after each `param_name` that a `{name:Type}` of the file uses. An argument that is already a name equal to the parameter gets no name. VS Code shows them as `editor.inlayHints.enabled` says." + }, "clickhouseSql.maxFileSize": { "type": "integer", "default": 5000000, "minimum": 0, "scope": "application", - "markdownDescription": "Files longer than this many characters get no syntax check, semantic highlighting, formatting, outline, folding, Expand Selection, completion, hover, signature help, go to definition, references, rename or server check, which would slow VS Code down; a message says so once per file. The TextMate highlighting still works.\n\nFor example, `20000000` lets files of up to 20 million characters in." + "markdownDescription": "Files longer than this many characters get no syntax check, semantic highlighting, formatting, outline, folding, Expand Selection, completion, hover, signature help, go to definition, references, rename, workspace symbols or server check, which would slow VS Code down; a message says so once per file. The TextMate highlighting still works.\n\nFor example, `20000000` lets files of up to 20 million characters in." }, "clickhouseSql.server.enabled": { "type": "boolean", @@ -177,7 +187,7 @@ "scripts": { "generate": "tsx scripts/generate-grammar.ts", "build": "esbuild src/extension.ts --bundle --platform=node --format=cjs --target=node20 --external:vscode --outfile=dist/extension.js", - "build:e2e": "esbuild test/e2e/run.ts test/e2e/index.ts test/e2e/extension.test.ts test/e2e/format.test.ts test/e2e/complete.test.ts test/e2e/server.test.ts test/e2e/semantic.test.ts test/e2e/folding.test.ts test/e2e/outline.test.ts test/e2e/selection.test.ts test/e2e/names.test.ts test/e2e/options.test.ts --bundle --platform=node --format=cjs --target=node20 --external:vscode --external:@vscode/test-electron --log-override:require-resolve-not-external=silent --outdir=out/e2e", + "build:e2e": "esbuild test/e2e/run.ts test/e2e/index.ts test/e2e/extension.test.ts test/e2e/format.test.ts test/e2e/complete.test.ts test/e2e/server.test.ts test/e2e/semantic.test.ts test/e2e/folding.test.ts test/e2e/outline.test.ts test/e2e/selection.test.ts test/e2e/names.test.ts test/e2e/options.test.ts test/e2e/fixes.test.ts test/e2e/inlay.test.ts test/e2e/symbols.test.ts --bundle --platform=node --format=cjs --target=node20 --external:vscode --external:@vscode/test-electron --log-override:require-resolve-not-external=silent --outdir=out/e2e", "typecheck": "tsc -p tsconfig.json", "test": "npm run typecheck && tsx --test \"test/grammar/*.test.ts\" \"test/server/*.test.ts\" \"test/complete/*.test.ts\" \"test/manifest/*.test.ts\" \"test/options/*.test.ts\"", "test:e2e": "npm run build && npm run build:e2e && node out/e2e/run.js", @@ -193,7 +203,7 @@ "@types/vscode": "~1.90.0", "@vscode/test-electron": "^3.1.0", "@vscode/vsce": "^4.1.0", - "clickhouse-sql-syntax": "0.4.0", + "clickhouse-sql-syntax": "0.5.0", "esbuild": "^0.28.2", "mocha": "^12.0.3", "vscode-oniguruma": "^2.0.1", diff --git a/packages/vscode/src/extension.ts b/packages/vscode/src/extension.ts index 1c0ed58..b55d830 100644 --- a/packages/vscode/src/extension.ts +++ b/packages/vscode/src/extension.ts @@ -3,15 +3,18 @@ import { checkOptions, checkWithRoles, type Diagnostic as SyntaxError, ParseCach import { registerCompletion } from "./complete.js"; import { otherDialectSpans } from "./dialect.js"; import { registerFolding } from "./folding.js"; +import { type Fixable, registerQuickFixes } from "./fixes.js"; import { registerFormatter } from "./format.js"; import { checkHints, type Problem } from "./hints.js"; import { registerHover } from "./hover.js"; +import { registerInlayHints } from "./inlay.js"; import { registerOutline } from "./outline.js"; import { registerSelectionRanges } from "./selection.js"; import { registerNames } from "./names.js"; import { registerSemanticTokens } from "./semantic.js"; import { registerServer, type ServerApi } from "./server/index.js"; import { registerSignatureHelp } from "./signature.js"; +import { registerWorkspaceSymbols, WorkspaceObjects } from "./symbols.js"; import { tooLarge } from "./size.js"; import { checkTags } from "./tags.js"; @@ -24,9 +27,12 @@ export function activate(context: vscode.ExtensionContext): ServerApi { // The test hints and tags that clickhouse-client and clickhouse-test read from comments. const testDiagnostics = vscode.languages.createDiagnosticCollection(`${LANGUAGE_ID}-test`); const server = registerServer(context, LANGUAGE_ID); + const objects = new WorkspaceObjects(LANGUAGE_ID); const pending = new Map>(); // The statements of each document parsed once, for its diagnostics, semantic tokens, hover, outline, selections and names. const caches = new Map(); + // The problems with fixes of each document's last check, for its quick fixes. + const fixable = new Map(); const cacheOf = (document: vscode.TextDocument): ParseCache => { const key = document.uri.toString(); let cache = caches.get(key); @@ -45,6 +51,7 @@ export function activate(context: vscode.ExtensionContext): ServerApi { if (tooLarge(document)) { diagnostics.delete(document.uri); testDiagnostics.delete(document.uri); + fixable.delete(document.uri.toString()); return; } const text = document.getText(); @@ -55,6 +62,7 @@ export function activate(context: vscode.ExtensionContext): ServerApi { diagnostics.set(document.uri, errors.map((error) => toDiagnostic(document, error))); const problems = [...checkTags(text), ...checkHints(text, errors.map((e) => e.start))].sort((a, b) => a.start - b.start); testDiagnostics.set(document.uri, problems.map((problem) => toTestDiagnostic(document, problem))); + fixable.set(document.uri.toString(), [...errors, ...problems].filter((p) => p.fixes)); }; const schedule = (document: vscode.TextDocument): void => { @@ -71,8 +79,12 @@ export function activate(context: vscode.ExtensionContext): ServerApi { registerOutline(LANGUAGE_ID, cacheOf), registerSelectionRanges(LANGUAGE_ID, cacheOf), registerSignatureHelp(LANGUAGE_ID), + registerInlayHints(LANGUAGE_ID, cacheOf), registerSemanticTokens(LANGUAGE_ID, cacheOf), - registerNames(LANGUAGE_ID, cacheOf), + registerNames(LANGUAGE_ID, cacheOf, objects), + registerQuickFixes(LANGUAGE_ID, (document) => fixable.get(document.uri.toString()) ?? []), + registerWorkspaceSymbols(objects), + objects, diagnostics, testDiagnostics, vscode.workspace.onDidOpenTextDocument(schedule), @@ -82,6 +94,7 @@ export function activate(context: vscode.ExtensionContext): ServerApi { diagnostics.delete(document.uri); testDiagnostics.delete(document.uri); caches.delete(document.uri.toString()); + fixable.delete(document.uri.toString()); }), { dispose: () => [...pending.keys()].forEach(cancel) }, ); diff --git a/packages/vscode/src/fixes.ts b/packages/vscode/src/fixes.ts new file mode 100644 index 0000000..b682908 --- /dev/null +++ b/packages/vscode/src/fixes.ts @@ -0,0 +1,42 @@ +import * as vscode from "vscode"; +import { type Fix, messageFixes } from "clickhouse-sql-syntax"; + +/** A problem of the last check of a document, by offsets, with its fixes. */ +export interface Fixable { + start: number; + end: number; + message: string; + fixes?: Fix[]; +} + +/** Offers the fixes of the diagnostics under the cursor: those of the last check, and the hints in the server's errors. */ +export function registerQuickFixes(languageId: string, lastCheck: (document: vscode.TextDocument) => readonly Fixable[]): vscode.Disposable { + return vscode.languages.registerCodeActionsProvider( + languageId, + { + provideCodeActions(document, _range, context) { + const actions: vscode.CodeAction[] = []; + for (const diagnostic of context.diagnostics) { + const start = document.offsetAt(diagnostic.range.start); + const end = document.offsetAt(diagnostic.range.end); + const fixes = + diagnostic.source === "ClickHouse server" + ? messageFixes(document.getText(), start, end, diagnostic.message) + : lastCheck(document).find((p) => p.start === start && p.end === end && p.message === diagnostic.message)?.fixes ?? []; + for (const fix of fixes) actions.push(toAction(document, diagnostic, fix)); + } + return actions; + }, + }, + { providedCodeActionKinds: [vscode.CodeActionKind.QuickFix] }, + ); +} + +function toAction(document: vscode.TextDocument, diagnostic: vscode.Diagnostic, fix: Fix): vscode.CodeAction { + const action = new vscode.CodeAction(fix.title, vscode.CodeActionKind.QuickFix); + action.edit = new vscode.WorkspaceEdit(); + for (const edit of fix.edits) action.edit.replace(document.uri, new vscode.Range(document.positionAt(edit.start), document.positionAt(edit.end)), edit.newText); + action.diagnostics = [diagnostic]; + action.isPreferred = fix.preferred; + return action; +} diff --git a/packages/vscode/src/hints.ts b/packages/vscode/src/hints.ts index 04f82b9..453d1f5 100644 --- a/packages/vscode/src/hints.ts +++ b/packages/vscode/src/hints.ts @@ -1,5 +1,5 @@ // Test hints such as "-- { serverError UNKNOWN_TABLE }", read as TestHint in src/Client/TestHint.cpp of ClickHouse does. -import { isSignificant, split, type Token, TokenKind as T, tokenize } from "clickhouse-sql-syntax"; +import { type Fix, isSignificant, nameHints, replaceFixes, split, type Token, TokenKind as T, tokenize } from "clickhouse-sql-syntax"; import { ERROR_CODES } from "./generated/errors.js"; /** A problem in the file, by offsets. */ @@ -8,6 +8,7 @@ export interface Problem { end: number; message: string; severity: "error" | "warning" | "information"; + fixes?: Fix[]; } /** A suggestion inside a hint. */ @@ -77,7 +78,10 @@ function parse(text: string, hint: Hint): Parsed { const result: Parsed = { serverErrors: [], clientErrors: [] }; const tokens = hint.tokens; const word = (token: Token) => text.slice(token.start, token.end); - const fail = (token: Token, message: string): Parsed => ({ ...result, problem: { start: token.start, end: Math.max(token.end, token.start + 1), message, severity: "error" } }); + const fail = (token: Token, message: string, fixes?: Fix[]): Parsed => ({ + ...result, + problem: { start: token.start, end: Math.max(token.end, token.start + 1), message, severity: "error", ...(fixes?.length && { fixes }) }, + }); let retries = 0; let retryToken: Token | undefined; for (let i = 0; i < tokens.length && tokens[i]!.kind !== T.EndOfStream; i++) { @@ -101,7 +105,7 @@ function parse(text: string, hint: Hint): Parsed { codes.push(Number.parseInt(word(code), 10)); } else if (code.kind === T.BareWord) { const known = Object.hasOwn(ERROR_CODES, word(code)) ? ERROR_CODES[word(code)] : undefined; - if (known === undefined) return fail(code, `No error code with name: '${word(code)}'`); + if (known === undefined) return fail(code, `No error code with name: '${word(code)}'`, replaceFixes(code.start, code.end, nameHints(word(code), Object.keys(ERROR_CODES)))); codes.push(known); } else return fail(code, `Could not parse error code in ${code.kind}: ${word(code)}`); const next = tokens[i + 1]!; diff --git a/packages/vscode/src/hover.ts b/packages/vscode/src/hover.ts index b32aa72..8c73a40 100644 --- a/packages/vscode/src/hover.ts +++ b/packages/vscode/src/hover.ts @@ -1,5 +1,5 @@ import * as vscode from "vscode"; -import { checkWithRoles, type CompletionKind, complete, describe, type ParseCache, type Role } from "clickhouse-sql-syntax"; +import { checkWithRoles, type CompletionKind, complete, describe, keywordDocumentation, type ParseCache, type Role } from "clickhouse-sql-syntax"; import { hintHover } from "./hints.js"; import { parameterHover } from "./parameters.js"; import { tooLarge } from "./size.js"; @@ -12,7 +12,7 @@ const KINDS: Partial> = { setting: ["setting"], }; -/** The documentation of the function, setting, engine, format or table function under the mouse. */ +/** The documentation of the function, setting, engine, format, table function, SYSTEM command or access clause under the mouse. */ export function registerHover(languageId: string, cacheOf: (document: vscode.TextDocument) => ParseCache): vscode.Disposable { return vscode.languages.registerHoverProvider(languageId, { provideHover(document, position) { @@ -27,6 +27,12 @@ export function registerHover(languageId: string, cacheOf: (document: vscode.Tex const span = new vscode.Range(document.positionAt(parameter.start), document.positionAt(parameter.end)); return new vscode.Hover(new vscode.MarkdownString(parameter.markdown), span); } + // A SYSTEM command, or a clause of CREATE or ALTER USER, ROLE, QUOTA, ROW POLICY or SETTINGS PROFILE. + const keyword = keywordDocumentation(text, offset, cacheOf(document)); + if (keyword) { + const span = new vscode.Range(document.positionAt(keyword.start), document.positionAt(keyword.end)); + return new vscode.Hover(new vscode.MarkdownString(keyword.documentation), span); + } // What complete() offers for the whole word tells what the word is at that place. const lower = word.toLowerCase(); const found = complete(text, document.offsetAt(range.end)).find((c) => c.documentation && c.label.toLowerCase() === lower); diff --git a/packages/vscode/src/inlay.ts b/packages/vscode/src/inlay.ts new file mode 100644 index 0000000..c03584b --- /dev/null +++ b/packages/vscode/src/inlay.ts @@ -0,0 +1,48 @@ +import * as vscode from "vscode"; +import { parameterHints, type ParseCache } from "clickhouse-sql-syntax"; +import { parameters } from "./parameters.js"; +import { pieces } from "./semantic.js"; +import { editorSetting, setting } from "./settings.js"; +import { tooLarge } from "./size.js"; + +/** The parameter names of call arguments, and the type of the query parameter that each param_name sets. */ +export function registerInlayHints(languageId: string, cacheOf: (document: vscode.TextDocument) => ParseCache): vscode.Disposable { + const changed = new vscode.EventEmitter(); + const provider: vscode.InlayHintsProvider = { + onDidChangeInlayHints: changed.event, + provideInlayHints(document, range) { + if (!setting("clickhouseSql.inlayHints.enabled", document) || tooLarge(document)) return []; + if (editorSetting("editor.inlayHints.enabled", document) === "off") return []; + const text = document.getText(); + const from = document.offsetAt(range.start); + const to = document.offsetAt(range.end); + const hints: vscode.InlayHint[] = []; + for (const [start, end] of pieces(text)) { + if (end <= from || start >= to) continue; + for (const { offset, name } of parameterHints(text.slice(start, end), cacheOf(document))) { + if (start + offset < from || start + offset > to) continue; + const hint = new vscode.InlayHint(document.positionAt(start + offset), `${name}:`, vscode.InlayHintKind.Parameter); + hint.paddingRight = true; + hints.push(hint); + } + } + // Only a screen that writes param_ costs a pass over the whole text. + if (text.slice(from, to).includes("param_")) { + const { substitutions, settings } = parameters(text); + for (const s of settings) { + if (s.start < from || s.end > to) continue; + const types = new Set(substitutions.filter((u) => u.name === s.name).map((u) => u.type)); + if (types.size === 1) hints.push(new vscode.InlayHint(document.positionAt(s.end), `: ${[...types][0]}`, vscode.InlayHintKind.Type)); + } + } + return hints; + }, + }; + return vscode.Disposable.from( + changed, + vscode.languages.registerInlayHintsProvider(languageId, provider), + vscode.workspace.onDidChangeConfiguration((event) => { + if (event.affectsConfiguration("clickhouseSql.inlayHints.enabled") || event.affectsConfiguration("editor.inlayHints.enabled")) changed.fire(); + }), + ); +} diff --git a/packages/vscode/src/names.ts b/packages/vscode/src/names.ts index 1947b13..2fde156 100644 --- a/packages/vscode/src/names.ts +++ b/packages/vscode/src/names.ts @@ -1,9 +1,23 @@ import * as vscode from "vscode"; -import { definition, type NameLocation, type ParseCache, prepareRename, references, rename } from "clickhouse-sql-syntax"; +import { + definition, type NameLocation, objectDefinitions, objectReferences, type ParseCache, prepareRename, references, rename, type SqlObject, +} from "clickhouse-sql-syntax"; import { tooLarge } from "./size.js"; +import { type FileObjects, positions, type WorkspaceObjects } from "./symbols.js"; -/** Go to definition, references, highlights and rename of the names a statement defines, through the library. */ -export function registerNames(languageId: string, cacheOf: (document: vscode.TextDocument) => ParseCache): vscode.Disposable { +/** A CREATE statement's object in a file of the workspace. */ +interface Found { + uri: vscode.Uri; + key: string; + object: SqlObject; + range: vscode.Range; +} + +/** + * Go to definition, references, highlights and rename of the names a statement defines, through the library, and + * definition and references of the tables and databases that the files of the workspace create. + */ +export function registerNames(languageId: string, cacheOf: (document: vscode.TextDocument) => ParseCache, index: WorkspaceObjects): vscode.Disposable { const range = (document: vscode.TextDocument, place: { start: number; end: number }) => new vscode.Range(document.positionAt(place.start), document.positionAt(place.end)); const places = (document: vscode.TextDocument, position: vscode.Position): NameLocation[] => @@ -13,20 +27,53 @@ export function registerNames(languageId: string, cacheOf: (document: vscode.Tex if (tooLarge(document)) throw new Error("This file is longer than clickhouseSql.maxFileSize."); }; + /** The table or database named at `offset`, and what it may be among the files' objects. */ + const objectsAt = async (document: vscode.TextDocument, offset: number) => { + const here = objectReferences(document.getText(), { cache: cacheOf(document) }).filter((r) => r.start <= offset && offset <= r.end); + // The database of db.t when the cursor is on it. + const reference = here.sort((a, b) => a.end - a.start - (b.end - b.start))[0]; + if (!reference) return undefined; + const files = await index.all(document); + const objects = new Map([...files].map(([key, file]) => [key, file.objects])); + const found = objectDefinitions(reference, document.uri.toString(), objects).map(({ file, object }) => toFound(files.get(file)!, file, object)); + return { reference, files, objects, found }; + }; + return vscode.Disposable.from( vscode.languages.registerDefinitionProvider(languageId, { - provideDefinition(document, position) { + async provideDefinition(document, position) { if (tooLarge(document)) return undefined; - return definition(document.getText(), document.offsetAt(position), { cache: cacheOf(document) }).map( - (place) => new vscode.Location(document.uri, range(document, place)), - ); + const offset = document.offsetAt(position); + const local = definition(document.getText(), offset, { cache: cacheOf(document) }); + if (local.length) return local.map((place) => new vscode.Location(document.uri, range(document, place))); + return (await objectsAt(document, offset))?.found.map((found) => new vscode.Location(found.uri, found.range)) ?? []; }, }), vscode.languages.registerReferenceProvider(languageId, { - provideReferences(document, position, context) { - return places(document, position) - .filter((place) => context.includeDeclaration || !place.definition) - .map((place) => new vscode.Location(document.uri, range(document, place))); + async provideReferences(document, position, context, token) { + const local = places(document, position); + if (local.length || tooLarge(document)) { + return local + .filter((place) => context.includeDeclaration || !place.definition) + .map((place) => new vscode.Location(document.uri, range(document, place))); + } + const at = await objectsAt(document, document.offsetAt(position)); + if (!at?.found.length) return []; + const { reference, objects, found } = at; + const targets = new Set(found.map((f) => f.object)); + const declarations = new Set(found.map((f) => `${f.key} ${f.object.nameStart}`)); + const result = context.includeDeclaration ? found.map((f) => new vscode.Location(f.uri, f.range)) : []; + for await (const { uri, text } of index.texts(token)) { + if (!text.includes(reference.name)) continue; + const key = uri.toString(); + const position = positions(text); + for (const other of objectReferences(text, key === document.uri.toString() ? { cache: cacheOf(document) } : {})) { + if (other.kind !== reference.kind || other.name !== reference.name || declarations.has(`${key} ${other.start}`)) continue; + if (!objectDefinitions(other, key, objects).some((d) => targets.has(d.object))) continue; + result.push(new vscode.Location(uri, new vscode.Range(position(other.start), position(other.end)))); + } + } + return result; }, }), vscode.languages.registerDocumentHighlightProvider(languageId, { @@ -53,3 +100,7 @@ export function registerNames(languageId: string, cacheOf: (document: vscode.Tex }), ); } + +function toFound(file: FileObjects, key: string, object: SqlObject): Found { + return { uri: file.uri, key, object, range: file.ranges[file.objects.indexOf(object)]! }; +} diff --git a/packages/vscode/src/semantic.ts b/packages/vscode/src/semantic.ts index 028f283..4a886b2 100644 --- a/packages/vscode/src/semantic.ts +++ b/packages/vscode/src/semantic.ts @@ -136,7 +136,7 @@ export function registerSemanticTokens(languageId: string, cacheOf: (document: v * The text split after each ';', as offsets; the parser reads each piece alone. A ';' in inline * INSERT data ends a piece too, so the rest of that data is read as SQL, as the grammar reads it. */ -function pieces(text: string): [number, number][] { +export function pieces(text: string): [number, number][] { const result: [number, number][] = []; let start = 0; for (const token of tokenize(text)) { diff --git a/packages/vscode/src/settings.ts b/packages/vscode/src/settings.ts index dea3d68..0cab52f 100644 --- a/packages/vscode/src/settings.ts +++ b/packages/vscode/src/settings.ts @@ -25,3 +25,13 @@ export function userSetting(key: string): { value: T; problem?: string } { const problem = invalidValue(key, value); return problem ? { value: fallback, problem } : { value }; } + +/** The value of a setting of VS Code itself for `scope`, such as `editor.inlayHints.enabled`. */ +export function editorSetting(key: string, scope?: vscode.ConfigurationScope): T | undefined { + return vscode.workspace.getConfiguration(undefined, scope).get(key); +} + +/** The files.associations of VS Code, which say what language a file opens as. */ +export function fileAssociations(): Record { + return vscode.workspace.getConfiguration("files").get>("associations") ?? {}; +} diff --git a/packages/vscode/src/symbols.ts b/packages/vscode/src/symbols.ts new file mode 100644 index 0000000..90f95ab --- /dev/null +++ b/packages/vscode/src/symbols.ts @@ -0,0 +1,209 @@ +import * as vscode from "vscode"; +import { type ObjectKind, objects, type SqlObject } from "clickhouse-sql-syntax"; +import { fileAssociations, setting } from "./settings.js"; + +const KINDS: Record = { + table: vscode.SymbolKind.Struct, + view: vscode.SymbolKind.Interface, + "materialized view": vscode.SymbolKind.Interface, + dictionary: vscode.SymbolKind.Key, + function: vscode.SymbolKind.Function, + database: vscode.SymbolKind.Namespace, +}; +// Files read at once while the index is built. +const BATCH = 64; + +/** The objects that one file makes, with the range of each name. */ +export interface FileObjects { + uri: vscode.Uri; + objects: SqlObject[]; + ranges: vscode.Range[]; +} + +/** The objects that the ClickHouse SQL files of the workspace make, read when first asked for and kept up to date. */ +export class WorkspaceObjects implements vscode.Disposable { + private files?: Map; + /** The files that the glob finds on disk. */ + private onDisk = new Map(); + /** The files to read again. */ + private dirty = new Map(); + /** The build or refresh under way. */ + private pending?: Promise; + /** Counts the resets, so that a build that a reset overtook stops. */ + private generation = 0; + private watcher?: vscode.FileSystemWatcher; + private readonly subscriptions: vscode.Disposable[]; + + constructor(private readonly languageId: string) { + const touch = (document: vscode.TextDocument) => { + if (this.files) this.dirty.set(document.uri.toString(), document.uri); + }; + this.subscriptions = [ + vscode.workspace.onDidOpenTextDocument(touch), + vscode.workspace.onDidChangeTextDocument((event) => touch(event.document)), + vscode.workspace.onDidCloseTextDocument(touch), + vscode.workspace.onDidChangeConfiguration((event) => { + if (event.affectsConfiguration("files.associations") || event.affectsConfiguration("clickhouseSql.maxFileSize")) this.reset(); + }), + ]; + } + + /** Every file's objects, `document` read from its current text. */ + async all(document?: vscode.TextDocument): Promise> { + while (this.pending || !this.files || this.dirty.size) { + this.pending ??= (this.files ? this.refresh() : this.build()).finally(() => (this.pending = undefined)); + await this.pending; + } + const files = this.files; + if (!document) return files; + const current = new Map(files); + const entry = read(document.uri, document.getText()); + if (entry) current.set(document.uri.toString(), entry); + else current.delete(document.uri.toString()); + return current; + } + + /** The files of the index, each with its text, the open ones as they are now. */ + async *texts(token: vscode.CancellationToken): AsyncGenerator<{ uri: vscode.Uri; text: string }> { + for (const { uri } of (await this.all()).values()) { + if (token.isCancellationRequested) return; + const text = openDocument(uri)?.getText() ?? (await readText(uri)); + if (text !== undefined) yield { uri, text }; + } + } + + private async build(): Promise { + const generation = this.generation; + const glob = `{${patterns(this.languageId).join(",")}}`; + const uris = await vscode.workspace.findFiles(glob); + if (generation !== this.generation) return; + this.watcher = vscode.workspace.createFileSystemWatcher(glob); + const found = (uri: vscode.Uri) => { + this.onDisk.set(uri.toString(), uri); + this.dirty.set(uri.toString(), uri); + }; + this.watcher.onDidCreate(found); + this.watcher.onDidChange(found); + this.watcher.onDidDelete((uri) => { + this.onDisk.delete(uri.toString()); + this.dirty.set(uri.toString(), uri); + }); + this.files = new Map(); + for (const uri of uris) found(uri); + for (const document of vscode.workspace.textDocuments) if (document.languageId === this.languageId) this.dirty.set(document.uri.toString(), document.uri); + } + + private async refresh(): Promise { + const files = this.files!; + const uris = [...this.dirty.values()]; + this.dirty.clear(); + for (let i = 0; i < uris.length; i += BATCH) { + const read = await Promise.all(uris.slice(i, i + BATCH).map((uri) => this.read(uri))); + // A reset while reading drops what was read. + if (this.files !== files) return; + read.forEach((entry, j) => { + const key = uris[i + j]!.toString(); + if (entry) files.set(key, entry); + else files.delete(key); + }); + } + } + + /** The objects of a file: an open ClickHouse document as it is now, else the file on disk that the glob finds. */ + private async read(uri: vscode.Uri): Promise { + const document = openDocument(uri); + if (document?.languageId === this.languageId) return read(uri, document.getText()); + if (!this.onDisk.has(uri.toString())) return undefined; + const text = await readText(uri); + return text === undefined ? undefined : read(uri, text); + } + + private reset(): void { + this.generation++; + this.watcher?.dispose(); + this.watcher = undefined; + this.files = undefined; + this.onDisk.clear(); + this.dirty.clear(); + } + + dispose(): void { + this.reset(); + this.subscriptions.forEach((subscription) => subscription.dispose()); + } +} + +/** Go to Symbol in Workspace: the tables, views, dictionaries, functions and databases that CREATE statements make. */ +export function registerWorkspaceSymbols(index: WorkspaceObjects): vscode.Disposable { + return vscode.languages.registerWorkspaceSymbolProvider({ + async provideWorkspaceSymbols(query) { + const symbols: vscode.SymbolInformation[] = []; + for (const file of (await index.all()).values()) { + file.objects.forEach((object, i) => { + if (!matches(object.name, query)) return; + const location = new vscode.Location(file.uri, file.ranges[i]!); + symbols.push(new vscode.SymbolInformation(object.name, KINDS[object.kind], object.database ?? "", location)); + }); + } + return symbols; + }, + }); +} + +/** The globs of the files that open as ClickHouse SQL: its extensions and the files.associations that name it. */ +function patterns(languageId: string): string[] { + const opted = Object.entries(fileAssociations()).flatMap(([glob, language]) => (language !== languageId ? [] : [glob.includes("/") ? glob : `**/${glob}`])); + return ["**/*.chsql", "**/*.clickhouse.sql", ...opted]; +} + +function openDocument(uri: vscode.Uri): vscode.TextDocument | undefined { + const key = uri.toString(); + return vscode.workspace.textDocuments.find((document) => document.uri.toString() === key); +} + +/** The text of a file, or nothing when it is gone or longer than clickhouseSql.maxFileSize. */ +async function readText(uri: vscode.Uri): Promise { + const max = setting("clickhouseSql.maxFileSize"); + try { + // A character takes at most 3 bytes in UTF-8, so a larger file is too long without reading it. + if ((await vscode.workspace.fs.stat(uri)).size > 3 * max) return undefined; + const text = new TextDecoder().decode(await vscode.workspace.fs.readFile(uri)); + return text.length > max ? undefined : text; + } catch { + return undefined; + } +} + +function read(uri: vscode.Uri, text: string): FileObjects | undefined { + if (text.length > setting("clickhouseSql.maxFileSize")) return undefined; + const found = objects(text); + const position = positions(text); + return { uri, objects: found, ranges: found.map((object) => new vscode.Range(position(object.nameStart), position(object.nameEnd))) }; +} + +/** Offsets to positions; a line ends at \n, \r\n or a lone \r, as in VS Code. */ +export function positions(text: string): (offset: number) => vscode.Position { + const starts = [0]; + for (const match of text.matchAll(/\r\n?|\n/g)) starts.push(match.index + match[0].length); + return (offset) => { + let lo = 0; + let hi = starts.length - 1; + while (lo < hi) { + const mid = (lo + hi + 1) >> 1; + if (starts[mid]! <= offset) lo = mid; + else hi = mid - 1; + } + return new vscode.Position(lo, offset - starts[lo]!); + }; +} + +/** Whether the characters of `query` appear in `name` in order, ignoring case, as VS Code asks of the provider. */ +function matches(name: string, query: string): boolean { + const lower = name.toLowerCase(); + let at = 0; + for (const c of query.toLowerCase()) { + at = lower.indexOf(c, at) + 1; + if (at === 0) return false; + } + return true; +} diff --git a/packages/vscode/test/e2e/complete.test.ts b/packages/vscode/test/e2e/complete.test.ts index c77f7f2..900c3b9 100644 --- a/packages/vscode/test/e2e/complete.test.ts +++ b/packages/vscode/test/e2e/complete.test.ts @@ -278,6 +278,32 @@ describe("Completion, hover and signature help", () => { assert.equal(await hover(text.indexOf("FROM") + 1), ""); }); + it("shows the documentation of SYSTEM commands and of the clauses of access statements on hover", async () => { + const { editor } = await open( + ["SYSTEM FLUSH LOGS query_log;", "SYSTEM DROP DNS CACHE;", "CREATE USER host HOST ANY VALID UNTIL '2030-01-01';", "ALTER QUOTA q KEYED BY user_name;|"].join("\n"), + ); + const text = editor.document.getText(); + const hover = async (word: string) => { + const position = editor.document.positionAt(text.indexOf(word) + 1); + const hovers = await vscode.commands.executeCommand("vscode.executeHoverProvider", editor.document.uri, position); + return hovers.map((h) => ({ + text: h.contents.map((c) => (c as vscode.MarkdownString).value).join("\n"), + range: h.range && editor.document.getText(h.range), + })); + }; + const [logs] = await hover("LOGS"); + assert.match(logs!.text, /^\*\*SYSTEM FLUSH LOGS\*\*\n\nFlushes buffered log messages to system tables/); + assert.equal(logs!.range, "SYSTEM FLUSH LOGS"); + assert.match((await hover("DNS"))[0]!.text, /^\*\*SYSTEM CLEAR\|DROP DNS CACHE\*\*/); + assert.deepEqual(await hover("query_log"), []); + assert.match((await hover("HOST ANY"))[0]!.text, /User host is a host from which a connection to ClickHouse server could be established/); + assert.deepEqual(await hover("host"), []); + const [until] = await hover("UNTIL"); + assert.match(until!.text, /^\*\*CREATE USER: VALID UNTIL Clause\*\*/); + assert.equal(until!.range, "VALID UNTIL"); + assert.match((await hover("KEYED"))[0]!.text, /^\*\*ALTER QUOTA\*\*\n\nChanges quotas/); + }); + it("completes and documents the options of a statement", async () => { const { editor } = await open( "CREATE TABLE t (x UInt64 CODEC(Delta(4), ZSTD(3)), s String, INDEX i s TYPE bloom_filter(0.01) GRANULARITY 1) " + diff --git a/packages/vscode/test/e2e/extension.test.ts b/packages/vscode/test/e2e/extension.test.ts index ef9bd06..524e6e6 100644 --- a/packages/vscode/test/e2e/extension.test.ts +++ b/packages/vscode/test/e2e/extension.test.ts @@ -149,6 +149,21 @@ describe("ClickHouse SQL extension", () => { ); }); + it("shows a codec on a type it does not take, a map given to a number and a wrong compression level", async () => { + const editor = await untitled( + "SET max_threads = {'a':'b'};\nCREATE TABLE t (x String CODEC(Delta, LZ4)) ENGINE = MergeTree ORDER BY x;\nSELECT 1 INTO OUTFILE 'f' COMPRESSION 'zstd' LEVEL 23;\n", + ); + const diagnostics = await waitForCheck(editor.document); + assert.deepEqual( + diagnostics.map((d) => [d.severity, editor.document.getText(d.range), d.message.slice(0, 40)]), + [ + [vscode.DiagnosticSeverity.Error, "{'a':'b'}", "Invalid value [('a', 'b')] of the settin"], + [vscode.DiagnosticSeverity.Error, "Delta", "Codec Delta is not applicable for String"], + [vscode.DiagnosticSeverity.Error, "23", "Invalid compression level, must be posit"], + ], + ); + }); + it("updates diagnostics as the text changes", async () => { const editor = await untitled("SELECT 1"); assert.deepEqual(await waitForCheck(editor.document), []); diff --git a/packages/vscode/test/e2e/fixes.test.ts b/packages/vscode/test/e2e/fixes.test.ts new file mode 100644 index 0000000..364a309 --- /dev/null +++ b/packages/vscode/test/e2e/fixes.test.ts @@ -0,0 +1,57 @@ +import * as assert from "node:assert/strict"; +import * as vscode from "vscode"; + +const WAIT_MS = 10_000; + +async function until(what: string, poll: () => T | undefined | Promise): Promise { + const deadline = Date.now() + WAIT_MS; + for (;;) { + const value = await poll(); + if (value !== undefined) return value; + if (Date.now() > deadline) throw new Error(`Timed out waiting for ${what}`); + await new Promise((resolve) => setTimeout(resolve, 50)); + } +} + +/** The quick fixes at the diagnostic of `source` on `word` in a new document of `content`, once it is shown. */ +async function quickFixes(content: string, word: string, source: string): Promise<{ editor: vscode.TextEditor; actions: vscode.CodeAction[] }> { + const document = await vscode.workspace.openTextDocument({ language: "clickhouse-sql", content }); + const editor = await vscode.window.showTextDocument(document); + const at = document.positionAt(content.indexOf(word) + 1); + await until(`a diagnostic on ${word}`, () => vscode.languages.getDiagnostics(document.uri).find((d) => d.source === source && d.range.contains(at))); + const actions = await vscode.commands.executeCommand("vscode.executeCodeActionProvider", document.uri, new vscode.Range(at, at), vscode.CodeActionKind.QuickFix.value); + return { editor, actions: actions ?? [] }; +} + +/** The titles of the actions, with a * for the preferred one. */ +const titles = (actions: vscode.CodeAction[]) => actions.map((a) => `${a.isPreferred ? "*" : ""}${a.title}`); + +describe("quick fixes", () => { + afterEach(() => vscode.commands.executeCommand("workbench.action.closeAllEditors")); + + it("renames an unknown setting and the diagnostic goes", async () => { + const { editor, actions } = await quickFixes("SELECT 1 SETTINGS max_thread = 1", "max_thread", "ClickHouse SQL"); + assert.deepEqual(titles(actions), ["*Change to max_threads"]); + assert.equal(actions[0]!.kind?.value, vscode.CodeActionKind.QuickFix.value); + assert.ok(await vscode.workspace.applyEdit(actions[0]!.edit!)); + assert.equal(editor.document.getText(), "SELECT 1 SETTINGS max_threads = 1"); + await until("the diagnostic to go", () => (vscode.languages.getDiagnostics(editor.document.uri).length === 0 ? true : undefined)); + }); + + it("writes the keyword that a syntax error hints at", async () => { + const { actions } = await quickFixes("SELEC 1", "SELEC", "ClickHouse SQL"); + assert.deepEqual(titles(actions), ["*Change to SELECT"]); + }); + + it("adds an ENGINE where default_table_engine is None", async () => { + const { editor, actions } = await quickFixes("SET default_table_engine = 'None';\nCREATE TABLE t (x UInt64)", "t (", "ClickHouse SQL"); + assert.deepEqual(titles(actions), ["*Add ENGINE = MergeTree ORDER BY tuple()"]); + assert.ok(await vscode.workspace.applyEdit(actions[0]!.edit!)); + assert.equal(editor.document.getText(), "SET default_table_engine = 'None';\nCREATE TABLE t (x UInt64) ENGINE = MergeTree ORDER BY tuple()"); + }); + + it("fixes an unknown error name of a test hint", async () => { + const { actions } = await quickFixes("SELECT 1; -- { serverError SYNTAX_EROR }", "SYNTAX_EROR", "ClickHouse test"); + assert.deepEqual(titles(actions), ["*Change to SYNTAX_ERROR", "Change to SYSTEM_ERROR"]); + }); +}); diff --git a/packages/vscode/test/e2e/index.ts b/packages/vscode/test/e2e/index.ts index a308fc8..7f1ebc4 100644 --- a/packages/vscode/test/e2e/index.ts +++ b/packages/vscode/test/e2e/index.ts @@ -14,6 +14,9 @@ export function run(): Promise { mocha.addFile(resolve(__dirname, "selection.test.js")); mocha.addFile(resolve(__dirname, "names.test.js")); mocha.addFile(resolve(__dirname, "options.test.js")); + mocha.addFile(resolve(__dirname, "fixes.test.js")); + mocha.addFile(resolve(__dirname, "inlay.test.js")); + mocha.addFile(resolve(__dirname, "symbols.test.js")); return new Promise((done, fail) => { mocha.run((failures) => (failures > 0 ? fail(new Error(`${failures} test(s) failed`)) : done())); }); diff --git a/packages/vscode/test/e2e/inlay.test.ts b/packages/vscode/test/e2e/inlay.test.ts new file mode 100644 index 0000000..a8d774d --- /dev/null +++ b/packages/vscode/test/e2e/inlay.test.ts @@ -0,0 +1,49 @@ +import * as assert from "node:assert/strict"; +import * as vscode from "vscode"; + +const LANGUAGE_ID = "clickhouse-sql"; +const global = vscode.ConfigurationTarget.Global; + +async function hints(text: string): Promise { + const document = await vscode.workspace.openTextDocument({ language: LANGUAGE_ID, content: text }); + await vscode.window.showTextDocument(document); + const range = new vscode.Range(new vscode.Position(0, 0), document.positionAt(text.length)); + const found = await vscode.commands.executeCommand("vscode.executeInlayHintProvider", document.uri, range); + return found.map((hint) => `${document.offsetAt(hint.position)} ${typeof hint.label === "string" ? hint.label : hint.label.map((p) => p.value).join("")}`); +} + +describe("Inlay hints", () => { + const SQL = "SELECT dateDiff('day', a, b), toDate(x), dateDiff(unit, s, e)"; + + afterEach(async () => { + await vscode.workspace.getConfiguration("clickhouseSql.inlayHints").update("enabled", undefined, global); + await vscode.workspace.getConfiguration("editor.inlayHints").update("enabled", undefined, global); + await vscode.commands.executeCommand("workbench.action.closeAllEditors"); + }); + + it("are off by default", async () => { + assert.deepEqual(await hints(SQL), []); + }); + + it("name the parameters of call arguments, but not of an argument that says it", async () => { + await vscode.workspace.getConfiguration("clickhouseSql.inlayHints").update("enabled", true, global); + const at = (part: string) => SQL.indexOf(part); + assert.deepEqual(await hints(SQL), [ + `${at("'day'")} unit:`, `${at("a, b")} startdate:`, `${at("b)")} enddate:`, `${at("s, e")} startdate:`, `${at("e)")} enddate:`, + ]); + const engine = "CREATE TABLE t (a Int, v UInt64) ENGINE = ReplacingMergeTree(v, 1) ORDER BY a"; + assert.deepEqual(await hints(engine), [`${engine.indexOf("v, 1")} ver:`, `${engine.indexOf("1)")} is_deleted:`]); + }); + + it("give the type of a query parameter after the param_name that sets it", async () => { + await vscode.workspace.getConfiguration("clickhouseSql.inlayHints").update("enabled", true, global); + const sql = "SET param_n = 5;\nSELECT {n:UInt32} SETTINGS param_n = 6, param_other = 1"; + assert.deepEqual(await hints(sql), [`${"SET param_n".length} : UInt32`, `${sql.lastIndexOf("param_n") + 7} : UInt32`]); + }); + + it("are off where editor.inlayHints.enabled is off", async () => { + await vscode.workspace.getConfiguration("clickhouseSql.inlayHints").update("enabled", true, global); + await vscode.workspace.getConfiguration("editor.inlayHints").update("enabled", "off", global); + assert.deepEqual(await hints(SQL), []); + }); +}); diff --git a/packages/vscode/test/e2e/server.test.ts b/packages/vscode/test/e2e/server.test.ts index 09074bb..f00c4a5 100644 --- a/packages/vscode/test/e2e/server.test.ts +++ b/packages/vscode/test/e2e/server.test.ts @@ -213,6 +213,19 @@ describe("ClickHouse server check", function () { await until("the diagnostic to go", () => serverDiagnostics(editor.document).length === 0 || undefined); }); + it("offers the names that the server's error hints at as quick fixes", async () => { + const editor = await open("SELECT lenght('abc')"); + const [diagnostic] = await until("a server diagnostic", () => { + const diagnostics = serverDiagnostics(editor.document); + return diagnostics.length ? diagnostics : undefined; + }); + const actions = await vscode.commands.executeCommand("vscode.executeCodeActionProvider", editor.document.uri, diagnostic!.range, vscode.CodeActionKind.QuickFix.value); + assert.deepEqual(actions.map((a) => `${a.isPreferred ? "*" : ""}${a.title}`), ["*Change to length"]); + assert.ok(await vscode.workspace.applyEdit(actions[0]!.edit!)); + assert.equal(editor.document.getText(), "SELECT length('abc')"); + await until("the diagnostic to go", () => serverDiagnostics(editor.document).length === 0 || undefined); + }); + // EXPLAIN would write its own output in the format, which GeoJSON and Template cannot do. it("explains a SELECT without its FORMAT, and readonly=1 still refuses its SETTINGS", async () => { const before = sent.length; diff --git a/packages/vscode/test/e2e/stress.test.ts b/packages/vscode/test/e2e/stress.test.ts index 6eb593c..7ecbba6 100644 --- a/packages/vscode/test/e2e/stress.test.ts +++ b/packages/vscode/test/e2e/stress.test.ts @@ -162,6 +162,28 @@ describe("A file of 10,000 statements, about 1 MB", function () { assert.ok(ofScreen < TOOL_MS, `of a screen: ${ofScreen.toFixed(0)} ms`); }); + it(`gives the inlay hints of a screen within ${TOOL_MS} ms after an edit`, async () => { + const enabled = vscode.workspace.getConfiguration("clickhouseSql.inlayHints"); + await enabled.update("enabled", true, vscode.ConfigurationTarget.Global); + try { + const editor = await open(sql); + const middle = editor.document.positionAt(sql.length / 2); + const screen = new vscode.Range(middle.line - 30, 0, middle.line + 30, 0); + const hints = () => vscode.commands.executeCommand("vscode.executeInlayHintProvider", editor.document.uri, screen); + const first = await timed(hints); + let afterEdit = Infinity; + for (let i = 0; i < 5; i++) { + assert.ok(await editor.edit((edit) => edit.insert(middle, " "))); + afterEdit = Math.min(afterEdit, await timed(hints)); + } + assert.ok((await hints()).length > 0); + console.log(` inlay hints of a screen: first ${first.toFixed(0)} ms, after an edit ${afterEdit.toFixed(0)} ms`); + assert.ok(afterEdit < TOOL_MS, `after an edit: ${afterEdit.toFixed(0)} ms`); + } finally { + await enabled.update("enabled", undefined, vscode.ConfigurationTarget.Global); + } + }); + it(`completes within ${TOOL_MS} ms at its end and in its middle`, async () => { const half = sql.indexOf(";\n\n", sql.length / 2) + 3; const editor = await open(`${sql.slice(0, half)}${TYPING};\n${sql.slice(half)}${TYPING}`); diff --git a/packages/vscode/test/e2e/symbols.test.ts b/packages/vscode/test/e2e/symbols.test.ts new file mode 100644 index 0000000..030c415 --- /dev/null +++ b/packages/vscode/test/e2e/symbols.test.ts @@ -0,0 +1,145 @@ +import * as assert from "node:assert/strict"; +import * as vscode from "vscode"; + +const folder = () => vscode.workspace.workspaceFolders![0]!.uri; +const fixture = (path: string) => vscode.Uri.joinPath(folder(), path); + +async function symbols(query = ""): Promise { + return (await vscode.commands.executeCommand("vscode.executeWorkspaceSymbolProvider", query)) ?? []; +} + +/** Each symbol as `kind container.name path:line`, sorted. */ +async function listed(query = ""): Promise { + return (await symbols(query)) + .map((s) => `${vscode.SymbolKind[s.kind]} ${s.containerName ? `${s.containerName}.` : ""}${s.name} ${vscode.workspace.asRelativePath(s.location.uri)}:${s.location.range.start.line + 1}`) + .sort(); +} + +/** Waits until `check` holds for the symbols, as the file watcher reports changes a little later. */ +async function until(check: (names: string[]) => boolean): Promise { + let names: string[] = []; + for (let i = 0; i < 100; i++) { + names = (await symbols()).map((s) => s.name); + if (check(names)) return names; + await new Promise((resolve) => setTimeout(resolve, 100)); + } + return names; +} + +/** The position of the `nth` occurrence of `text` in the document, `skip` characters into it. */ +function at(document: vscode.TextDocument, text: string, nth = 0, skip = 0): vscode.Position { + let offset = -1; + for (let i = 0; i <= nth; i++) offset = document.getText().indexOf(text, offset + 1); + assert.ok(offset >= 0, `${text} is in the document ${nth + 1} times`); + return document.positionAt(offset + skip); +} + +/** Locations as `path:line:column`, sorted. */ +function places(locations: (vscode.Location | vscode.LocationLink)[]): string[] { + return locations + .map((l) => ("targetUri" in l ? { uri: l.targetUri, range: l.targetRange } : l)) + .map((l) => `${vscode.workspace.asRelativePath(l.uri)}:${l.range.start.line + 1}:${l.range.start.character + 1}`) + .sort(); +} + +describe("Workspace symbols", () => { + afterEach(() => vscode.commands.executeCommand("workbench.action.closeAllEditors")); + + it("Go to Symbol in Workspace finds what the CREATE statements of ClickHouse files make", async () => { + const found = await listed(); + for (const symbol of [ + "Namespace analytics objects/schema.chsql:1", + "Struct analytics.events objects/schema.chsql:3", + "Interface analytics.events_by_kind objects/schema.chsql:4", + "Key analytics.kinds objects/schema.chsql:5", + "Function kind_title objects/schema.chsql:6", + "Struct staging.events objects/staging.chsql:1", + "Interface staging.recent objects/staging.chsql:2", + "Struct hits sample.chsql:2", + "Interface logs.mv keywords-as-names.chsql:1", + ]) { + assert.ok(found.includes(symbol), `${symbol} in ${found.join(", ")}`); + } + assert.ok(!found.some((symbol) => symbol.includes("ignored_table")), "a plain .sql file is not ClickHouse SQL"); + const narrowed = (await symbols("kndtitle")).map((s) => s.name); + assert.deepEqual(narrowed, ["kind_title"]); + }); + + it("follows files as they are created, changed and deleted", async function () { + this.timeout(60_000); + await symbols(); + const uri = fixture("objects/new.chsql"); + try { + await vscode.workspace.fs.writeFile(uri, Buffer.from("CREATE TABLE fresh_table (x UInt8) ENGINE = Memory;\n")); + assert.ok((await until((names) => names.includes("fresh_table"))).includes("fresh_table")); + await vscode.workspace.fs.writeFile(uri, Buffer.from("CREATE TABLE renamed_table (x UInt8) ENGINE = Memory;\n")); + const renamed = await until((names) => names.includes("renamed_table") && !names.includes("fresh_table")); + assert.ok(renamed.includes("renamed_table") && !renamed.includes("fresh_table")); + // A file longer than clickhouseSql.maxFileSize, 5,000,000 characters by default, is left out. + await vscode.workspace.fs.writeFile(uri, Buffer.from(`CREATE TABLE big_table (x UInt8) ENGINE = Memory;\n${"-- a comment\n".repeat(400_000)}`)); + const big = await until((names) => !names.includes("renamed_table")); + assert.ok(!big.includes("renamed_table") && !big.includes("big_table")); + } finally { + await vscode.workspace.fs.delete(uri); + } + assert.ok(!(await until((names) => !names.includes("big_table"))).includes("big_table")); + }); + + it("an open file counts as it is edited", async () => { + const document = await vscode.workspace.openTextDocument(fixture("objects/staging.chsql")); + const editor = await vscode.window.showTextDocument(document); + await editor.edit((edit) => edit.insert(new vscode.Position(0, 0), "CREATE TABLE unsaved_table (x UInt8) ENGINE = Memory;\n")); + try { + assert.ok((await symbols()).some((s) => s.name === "unsaved_table")); + } finally { + await vscode.commands.executeCommand("workbench.action.files.revert"); + } + assert.ok(!(await until((names) => !names.includes("unsaved_table"))).includes("unsaved_table")); + }); +}); + +describe("Tables across files", () => { + afterEach(() => vscode.commands.executeCommand("workbench.action.closeAllEditors")); + + const definitionsAt = async (document: vscode.TextDocument, position: vscode.Position) => + places(await vscode.commands.executeCommand("vscode.executeDefinitionProvider", document.uri, position)); + + it("go from a table name to the CREATE statement in another file, by the database it means", async () => { + const document = await vscode.workspace.openTextDocument(fixture("objects/queries.chsql")); + await vscode.window.showTextDocument(document); + const analytics = "objects/schema.chsql:3:14"; + const staging = "objects/staging.chsql:1:14"; + assert.deepEqual(await definitionsAt(document, at(document, "analytics.events", 0, 12)), [analytics]); + assert.deepEqual(await definitionsAt(document, at(document, "staging.events", 0, 10)), [staging]); + // Before USE, the database of the session is not known, so both tables may be meant. + assert.deepEqual(await definitionsAt(document, at(document, "FROM events", 0, 6)), [analytics, staging]); + assert.deepEqual(await definitionsAt(document, at(document, "FROM events", 1, 6)), [analytics]); + assert.deepEqual(await definitionsAt(document, at(document, "kinds")), ["objects/schema.chsql:5:19"]); + assert.deepEqual(await definitionsAt(document, at(document, "analytics.events")), ["objects/schema.chsql:1:17"]); + // A CTE of the same name takes the place of the table. + assert.deepEqual(await definitionsAt(document, at(document, "FROM events", 2, 6)), ["objects/queries.chsql:6:6"]); + }); + + it("go from an opted-in .sql file to the table of a .chsql file", async () => { + const document = await vscode.workspace.openTextDocument(fixture("opt-in/report.sql")); + await vscode.window.showTextDocument(document); + assert.deepEqual(await definitionsAt(document, at(document, "hits")), ["sample.chsql:2:28"]); + }); + + it("find the references of a table in every file", async () => { + const document = await vscode.workspace.openTextDocument(fixture("objects/schema.chsql")); + await vscode.window.showTextDocument(document); + const position = at(document, "events ("); + const references = await vscode.commands.executeCommand("vscode.executeReferenceProvider", document.uri, position); + assert.deepEqual(places(references), [ + "objects/queries.chsql:1:21", + "objects/queries.chsql:2:43", + // This one may also be staging.events. + "objects/queries.chsql:3:15", + "objects/queries.chsql:5:15", + "objects/schema.chsql:3:14", + "objects/schema.chsql:4:115", + "objects/staging.chsql:2:45", + ]); + }); +}); diff --git a/packages/vscode/test/fixtures/workspace/objects/not-clickhouse.sql b/packages/vscode/test/fixtures/workspace/objects/not-clickhouse.sql new file mode 100644 index 0000000..f1f2b6e --- /dev/null +++ b/packages/vscode/test/fixtures/workspace/objects/not-clickhouse.sql @@ -0,0 +1,2 @@ +-- A .sql file that stays with the built-in SQL language, so its tables are not ClickHouse's. +CREATE TABLE ignored_table (x UInt8) ENGINE = Memory; diff --git a/packages/vscode/test/fixtures/workspace/objects/queries.chsql b/packages/vscode/test/fixtures/workspace/objects/queries.chsql new file mode 100644 index 0000000..1910606 --- /dev/null +++ b/packages/vscode/test/fixtures/workspace/objects/queries.chsql @@ -0,0 +1,6 @@ +SELECT count() FROM analytics.events; +INSERT INTO staging.events SELECT id FROM analytics.events; +SELECT * FROM events; +USE analytics; +SELECT * FROM events JOIN kinds USING (kind); +WITH events AS (SELECT 1 AS id) SELECT id FROM events; diff --git a/packages/vscode/test/fixtures/workspace/objects/schema.chsql b/packages/vscode/test/fixtures/workspace/objects/schema.chsql new file mode 100644 index 0000000..a57fee2 --- /dev/null +++ b/packages/vscode/test/fixtures/workspace/objects/schema.chsql @@ -0,0 +1,6 @@ +CREATE DATABASE analytics; +USE analytics; +CREATE TABLE events (id UInt64, kind String, title String) ENGINE = MergeTree ORDER BY id; +CREATE MATERIALIZED VIEW events_by_kind ENGINE = SummingMergeTree ORDER BY kind AS SELECT kind, count() AS n FROM events GROUP BY kind; +CREATE DICTIONARY kinds (kind String, title String) PRIMARY KEY kind SOURCE(CLICKHOUSE(TABLE 'events')) LAYOUT(HASHED()) LIFETIME(300); +CREATE FUNCTION kind_title AS (k) -> dictGet('analytics.kinds', 'title', k); diff --git a/packages/vscode/test/fixtures/workspace/objects/staging.chsql b/packages/vscode/test/fixtures/workspace/objects/staging.chsql new file mode 100644 index 0000000..6de4371 --- /dev/null +++ b/packages/vscode/test/fixtures/workspace/objects/staging.chsql @@ -0,0 +1,2 @@ +CREATE TABLE staging.events (id UInt64) ENGINE = Memory; +CREATE VIEW staging.recent AS SELECT * FROM analytics.events; diff --git a/packages/vscode/test/options/hints.test.ts b/packages/vscode/test/options/hints.test.ts index 2342484..c5aeb2a 100644 --- a/packages/vscode/test/options/hints.test.ts +++ b/packages/vscode/test/options/hints.test.ts @@ -42,6 +42,17 @@ describe("test hints", () => { for (const [text, at, message] of cases) assert.deepEqual(problems(text), [[at, "error", message]], text); }); + test("fixes an unknown error name with the closest names", () => { + const fixes = (text: string) => checkHints(text).flatMap((p) => (p.fixes ?? []).map((f) => `${f.preferred ? "*" : ""}${text.slice(0, f.edits[0]!.start)}${f.edits[0]!.newText}${text.slice(f.edits[0]!.end)}`)); + assert.deepEqual(fixes("SELECT 1; -- { serverError UNKNOWN_TABL }"), [ + "*SELECT 1; -- { serverError UNKNOWN_TABLE }", + "SELECT 1; -- { serverError UNKNOWN_TYPE }", + "SELECT 1; -- { serverError UNKNOWN_USER }", + ]); + assert.deepEqual(fixes("SELECT 1; -- { serverError 60, SYNTAX_EROR }"), ["*SELECT 1; -- { serverError 60, SYNTAX_ERROR }", "SELECT 1; -- { serverError 60, SYSTEM_ERROR }"]); + assert.deepEqual(fixes("SELECT 1; -- { serverError NO_SUCH_THING_AT_ALL }"), []); + }); + test("warns when serverError expects the error of a statement that the client rejects", () => { const [problem] = problems("SELECT (; -- { serverError SYNTAX_ERROR }\nSELECT 1;"); assert.deepEqual(problem!.slice(0, 2), ["{ serverError SYNTAX_ERROR }", "warning"]); diff --git a/scripts/clickhouse-release.ts b/scripts/clickhouse-release.ts index d5d87f7..fa1adb0 100644 --- a/scripts/clickhouse-release.ts +++ b/scripts/clickhouse-release.ts @@ -16,7 +16,10 @@ const ROOT = resolve(import.meta.dirname, ".."); const VERSION_FILE = join(ROOT, "data", "clickhouse-version.json"); const GENERATED = join(ROOT, "packages", "core", "src", "generated"); const REPOSITORY = "ClickHouse/ClickHouse"; -const SPARSE_PATHS = ["/tests/queries/0_stateless/", "/src/Parsers/", "/src/Access/Common/AccessType.h"]; +const SPARSE_PATHS = [ + "/tests/queries/0_stateless/", "/src/Parsers/", "/src/Access/Common/AccessType.h", "/src/Storages/", "/src/Databases/", "/src/Backups/", + "/docs/reference/statements/", +]; const USAGE = `Usage: npx tsx scripts/clickhouse-release.ts [options] @@ -268,6 +271,9 @@ export function listChanges(before: Names, after: Names): ListChange[] { function regenerate(clickhouse: string, src: string): void { run("npm", ["run", "generate", "--", "--src", src, "--clickhouse", clickhouse]); run("npx", ["tsx", "scripts/generate-catalog.ts", "--clickhouse", clickhouse]); + const docs = join(src, "docs", "reference"); + run("npx", ["tsx", "scripts/generate-options.ts", "--clickhouse", clickhouse, "--src", src, "--docs", docs]); + run("npx", ["tsx", "scripts/generate-statements.ts", "--docs", docs]); // The grammar generator reads the lists from the built library. run("npm", ["run", "build", "-w", "packages/core"]); run("npm", ["run", "generate", "-w", "packages/vscode"]); diff --git a/scripts/generate-options.ts b/scripts/generate-options.ts index 6d30d3d..89104d5 100644 --- a/scripts/generate-options.ts +++ b/scripts/generate-options.ts @@ -7,7 +7,8 @@ // the binary by running statements in clickhouse local. import { execFileSync } from "node:child_process"; -import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; import { join } from "node:path"; import { parseArgs } from "node:util"; @@ -404,6 +405,8 @@ interface CallRule { /** The message for too many, with {} for the count. */ tooMany?: string; tooFew?: string; + /** Counts between min and max that ClickHouse refuses, with its message. */ + gaps?: [number, string][]; parameters: ParameterRule[]; } @@ -413,50 +416,79 @@ function template(message: string, value: string): string { return message.replace(new RegExp(`(given |: |')${escaped}(?=('|\\.?$|\\)|,))`), "$1{}"); } +const clean = (o: Outcome | undefined) => o?.message?.replace(/: When validating secondary index `\w+`/, ""); + +/** What ClickHouse takes at one position of a call, from the probes there and the values its messages list. */ +function parameterRule(at: (value: string) => string, given: string | undefined, setup: string[]): { rule: ParameterRule; taken: string } | undefined { + const kindOf = (text: string): Probe["kind"] => (/^'/.test(text) ? "string" : /^[A-Za-z]/.test(text) ? "word" : text.includes(".") ? "float" : "int"); + const tried = given === undefined || probes.some((p) => p.text === given) ? probes : [...probes, { kind: kindOf(given), text: given }]; + // A value that a setting would allow is taken, as the setting may be on where the statement runs. + const outcomes = probe(clickhouse, tried.map((p) => at(p.text)), { setup }).map((o) => (/\bEnable it with\b/.test(o.message ?? "") ? { ok: true } : o)); + if (!outcomes.some((o) => o.ok)) return undefined; + const rule: ParameterRule = { ints: [], floats: [], messages: {} }; + tried.forEach((p, i) => { + const ok = outcomes[i]!.ok; + if (p.kind === "int") rule.ints.push([Number(p.text), ok]); + else if (p.kind === "float") rule.floats.push([Number(p.text), ok]); + else if (ok && !p.text.includes("zz_probe")) (rule[p.kind === "string" ? "strings" : "words"] ??= []).push(p.text.replace(/'/g, "").toLowerCase()); + if (!ok && outcomes[i]!.message) { + const key = p.kind === "int" ? (Number(p.text) < 0 ? "negative" : "int") : p.kind; + const value = p.kind === "string" ? p.text.slice(1, -1) : p.text; + rule.messages[key] ??= template(clean(outcomes[i])!, value); + } + }); + rule.ints.sort((a, b) => a[0] - b[0]); + rule.floats.sort((a, b) => a[0] - b[0]); + // An unknown string or word is refused when the refusal is about its kind, or when the probes took one of + // their own values and the base call has no value they lack. + for (const kind of ["string", "word"] as const) { + const unknown = tried.findIndex((p) => p.kind === kind && p.text.includes("zz_probe")); + if (outcomes[unknown]!.ok) continue; + const message = clean(outcomes[unknown]) ?? ""; + const covered = !(given !== undefined && tried.length > probes.length && kindOf(given) === kind); + const ofKind = outcomes.some((o, i) => !o.ok && (tried[i]!.kind === "int" || tried[i]!.kind === "float") && clean(o) === message); + const takes = (k: Probe["kind"]) => tried.some((p, i) => p.kind === k && outcomes[i]!.ok); + // A bare word refused as the unknown string is, where strings are taken, is read as a string. + const asString = kind === "word" && takes("string") && message === clean(outcomes[tried.findIndex((p) => p.kind === "string" && p.text.includes("zz_probe"))]); + const noneOfKind = !message.includes("zz_probe") && !takes(kind) && !asString; + const refused = ofKind || noneOfKind || (covered && (rule[kind === "string" ? "strings" : "words"]?.length ?? 0) > 0); + if (refused) rule[kind === "string" ? "stringsRefused" : "wordsRefused"] = true; + } + const taken = tried.find((p, i) => outcomes[i]!.ok && p.kind === "int" && Number(p.text) > 0) ?? tried.find((_, i) => outcomes[i]!.ok)!; + return { rule, taken: taken.text }; +} + /** The parameters `call(params)` takes, asked of ClickHouse at each position with the others at a value it takes. */ function callRule(call: (params: string[] | undefined) => string, base: string[], setup: string[], limit = 4): CallRule | undefined { - const verdict = (params: string[] | undefined) => probe(clickhouse, [call(params)], { setup })[0]!; const parameters: ParameterRule[] = []; const valid = [...base]; for (let position = 0; position < limit; position++) { - const statements = probes.map((p) => call([...valid.slice(0, position), p.text, ...valid.slice(position + 1)])); - const outcomes = probe(clickhouse, statements, { setup }); - if (!outcomes.some((o) => o.ok)) break; - const rule: ParameterRule = { ints: [], floats: [], messages: {} }; - probes.forEach((p, i) => { - const ok = outcomes[i]!.ok; - if (p.kind === "int") rule.ints.push([Number(p.text), ok]); - else if (p.kind === "float") rule.floats.push([Number(p.text), ok]); - else if (p.text.includes("zz_probe")) rule[p.kind === "string" ? "stringsRefused" : "wordsRefused"] = !ok; - else if (ok) (rule[p.kind === "string" ? "strings" : "words"] ??= []).push(p.text.replace(/'/g, "").toLowerCase()); - if (!ok && outcomes[i]!.message) { - const key = p.kind === "int" ? (Number(p.text) < 0 ? "negative" : "int") : p.kind; - const message = outcomes[i]!.message!.replace(/: When validating secondary index `\w+`/, ""); - const value = p.kind === "string" ? p.text.slice(1, -1) : p.text; - rule.messages[key] ??= template(message, value); - } - }); - parameters.push(rule); - if (valid.length <= position) { - const ok = probes.find((p, i) => outcomes[i]!.ok && p.kind === "int" && Number(p.text) > 0) ?? probes.find((_, i) => outcomes[i]!.ok)!; - valid.push(ok.text); - } + const found = parameterRule((value) => call([...valid.slice(0, position), value, ...valid.slice(position + 1)]), valid[position], setup); + if (!found) break; + parameters.push(found.rule); + if (valid.length <= position) valid.push(found.taken); } - const counts = [0, 1, 2, 3, 4, 5, 6].map((n) => (n <= valid.length ? valid.slice(0, n) : [...valid, ...Array(n - valid.length).fill(valid[valid.length - 1] ?? "1")])); + const counts = [0, 1, 2, 3, 4, 5, 6, 7].map((n) => (n <= valid.length ? valid.slice(0, n) : [...valid, ...Array(n - valid.length).fill(valid[valid.length - 1] ?? "1")])); const countOutcomes = probe(clickhouse, [call(undefined), ...counts.slice(1).map((c) => call(c))], { setup }); const okCounts = countOutcomes.map((o, n) => (o.ok ? n : -1)).filter((n) => n >= 0); if (!okCounts.length) return undefined; const min = Math.min(...okCounts); - const max = Math.max(...okCounts); - const clean = (o: Outcome | undefined) => o?.message?.replace(/: When validating secondary index `\w+`/, ""); + let max = Math.max(...okCounts); + // A message such as "requires 2 to 4 parameters" tells the most it takes when the probes could not reach it. + const stated = countOutcomes.map((o) => /\b\d+ to (\d+) param/.exec(clean(o) ?? "")).find(Boolean); + if (stated && Number(stated[1]) > max) max = Number(stated[1]); // A codec that took no probe value may still take values the probes do not have, unless ClickHouse says it takes none. const tooManyMessage = clean(countOutcomes[max + 1]); if (max === 0 && !/\b(0 param|no param|not have param|cannot have arguments)/i.test(tooManyMessage ?? "")) return { min, max: 99, parameters: [] }; + // Counts between the least and the most that ClickHouse refuses with the message of a wrong count. + const countMessage = (n: number) => clean(countOutcomes[n])?.replace(new RegExp(`\\b${n}\\b`), "{}"); + const gaps = okCounts.length ? [...Array(max - min).keys()].map((k) => k + min).filter((n) => !countOutcomes[n]?.ok && n < countOutcomes.length - 1 && countMessage(n) === countMessage(max + 1)) : []; return { min, max, - tooMany: max + 1 < countOutcomes.length ? (clean(countOutcomes[max + 1]) && template(clean(countOutcomes[max + 1])!, String(max + 1))) : undefined, + tooMany: max + 1 < countOutcomes.length ? (tooManyMessage && template(tooManyMessage, String(max + 1))) : undefined, tooFew: min > 0 ? clean(countOutcomes[min - 1]) : undefined, + gaps: gaps.length ? gaps.map((n) => [n, clean(countOutcomes[n])!] as [number, string]) : undefined, parameters: parameters.slice(0, max), }; } @@ -479,13 +511,48 @@ codecRows.forEach((c, i) => { const gate = /You can enable it with the '(\w+)' setting/.exec(codecGates[i]!.message ?? ""); if (gate) codecSetup.push(`SET ${gate[1]} = 1`); }); +// A call that each codec with required parameters takes, to probe one parameter at a time. +const CODEC_BASE: Record = { + Quantized: { column: "Array(Float32)", params: ["'int8'", "8"] }, + SZ3: { column: "Float64", params: ["'ALGO_LORENZO_REG'", "'REL'", "0.01"] }, +}; +// The data types that a column can have as they are, to ask each codec about. +const columnTypes = rows<{ name: string }>("SELECT name FROM system.data_type_families WHERE alias_to = '' ORDER BY name").map((r) => r.name); +const typeOutcomes = probe(clickhouse, columnTypes.map((c) => `CREATE TABLE p_${id()} (x ${c}) ENGINE = Memory`)); +const usableTypes = columnTypes.filter((_, i) => typeOutcomes[i]!.ok); + +/** The types a codec refuses as `call`, grouped by ClickHouse's message with {} for the type; none when it refuses all of them. */ +function refusedTypes(call: string): { message: string; types: string[] }[] | undefined { + const outcomes = probe(clickhouse, usableTypes.map((t) => `CREATE TABLE p_${id()} (x ${t} CODEC(${call})) ENGINE = Memory`), { setup: codecSetup }); + if (!outcomes.some((o) => o.ok)) return undefined; + const groups = new Map(); + usableTypes.forEach((t, i) => { + const message = outcomes[i]!.message; + // Only a refusal that names the type is about it. + if (outcomes[i]!.ok || !message || !new RegExp(`\\b${t}\\b`).test(message)) return; + const key = message.replace(new RegExp(`\\b${t}\\b`), "{}"); + groups.set(key, [...(groups.get(key) ?? []), t]); + }); + return [...groups].map(([message, types]) => ({ message, types })); +} + const codecs = codecRows.map((c, i) => { const gate = /You can enable it with the '(\w+)' setting/.exec(codecGates[i]!.message ?? "")?.[1]; // The first column type that the codec takes without parameters, or with its first probe value. const type = ["UInt64", "Float64", "String", "Array(Float32)"].find((t) => probe(clickhouse, [`CREATE TABLE p_${id()} (x ${t} CODEC(${c.name})) ENGINE = Memory`], { setup: codecSetup })[0]!.ok); - const rule = c.name === "Multiple" ? undefined : callRule((params) => `CREATE TABLE p_${id()} (x ${type ?? "UInt64"} CODEC(${c.name}${params ? `(${params.join(", ")})` : ""})) ENGINE = Memory`, [], codecSetup, 3); + const base = CODEC_BASE[c.name]; + const rule = c.name === "Multiple" ? undefined : callRule((params) => `CREATE TABLE p_${id()} (x ${base?.column ?? type ?? "UInt64"} CODEC(${c.name}${params ? `(${params.join(", ")})` : ""})) ENGINE = Memory`, base?.params ?? [], codecSetup, 3); const refused = c.name === "Multiple" ? probe(clickhouse, [`CREATE TABLE p_${id()} (x UInt64 CODEC(Multiple)) ENGINE = Memory`])[0]!.message : undefined; - console.log(`codec ${c.name}: ${type ?? "no type"}, ${rule ? `${rule.min}..${rule.max} parameters` : "parameters not known"}`); + // The types it refuses written bare and with parameters, which some codecs use instead of the type. + const p0 = rule?.parameters[0]; + const first = p0?.ints.find(([v, ok]) => ok && v > 0)?.[0] ?? (p0?.strings?.[0] !== undefined ? `'${p0.strings[0]}'` : p0?.words?.[0]); + const params = base ? base.params : first !== undefined && rule!.min <= 1 && rule!.max >= 1 ? [String(first)] : undefined; + const types = c.name === "Multiple" ? undefined : refusedTypes(c.name); + const withParameters = params ? refusedTypes(`${c.name}(${params.join(", ")})`) : undefined; + // When the parameters change nothing, the refusals hold for any of them. + const same = JSON.stringify(withParameters) === JSON.stringify(types); + const typesWithParameters = withParameters?.map((r) => (same ? r : { ...r, parameters: params!.length })); + console.log(`codec ${c.name}: ${type ?? "no type"}, ${rule ? `${rule.min}..${rule.max} parameters` : "parameters not known"}, refuses ${types?.flatMap((g) => g.types).length ?? "?"} types bare, ${typesWithParameters?.flatMap((g) => g.types).length ?? "?"} with parameters`); return { name: c.name, description: c.description, @@ -498,6 +565,8 @@ const codecs = codecRows.map((c, i) => { type, refused, rule, + types, + typesWithParameters, }; }); @@ -515,9 +584,37 @@ const INDEX_BASE: Record = { ngrambf_v1: { column: "String", params: ["3", "256", "2", "0"] }, tokenbf_v1: { column: "String", params: ["256", "2", "0"] }, sparse_grams: { column: "String", params: ["3", "100", "256", "2", "0"] }, + vector_similarity: { column: "Array(Float32)", params: ["'hnsw'", "'L2Distance'", "3", "'bf16'", "16", "100"] }, }; const indexRows = rows("SELECT name, description, syntax FROM system.data_skipping_index_types ORDER BY name"); const unknownIndex = probe(clickhouse, [`CREATE TABLE p_${id()} (x UInt64, INDEX i x TYPE zz_probe GRANULARITY 1) ENGINE = MergeTree ORDER BY x`])[0]!; +// The named arguments of the text index: the names come from its source, and what each takes from ClickHouse. +function textArguments(): object | undefined { + const file = join(args.src!, "src/Storages/MergeTree/MergeTreeIndexText.cpp"); + if (!existsSync(file)) return undefined; + const statement = (list: string) => `CREATE TABLE p_${id()} (x String, INDEX i x TYPE text${list} GRANULARITY 1) ENGINE = MergeTree ORDER BY x`; + const [none, positional, unknown, duplicate, tokenizer] = probe(clickhouse, [ + statement(""), statement("(1)"), statement("(tokenizer = splitByNonAlpha, zz_probe = 1)"), + statement("(tokenizer = splitByNonAlpha, tokenizer = splitByNonAlpha)"), statement("(tokenizer = zz_probe)"), + ]).map(clean); + const required = [...(none ?? "").matchAll(/argument '(\w+)' is required/g)].map((m) => m[1]!); + const names = [...readFileSync(file, "utf8").matchAll(/static const String ARGUMENT_\w+ = "(\w+)";/g)].map((m) => m[1]!); + const base = "tokenizer = splitByNonAlpha"; + const taken = probe(clickhouse, names.map((m) => statement(`(${base}, ${m} = 1)`))); + const known = names.filter((n, i) => n === "tokenizer" || !/^Unexpected/.test(clean(taken[i]) ?? "")); + const values = Object.fromEntries(known.filter((n) => n !== "tokenizer").map((n) => [n, parameterRule((v) => statement(`(${base}, ${n} = ${v})`), undefined, [])?.rule ?? null])); + return { + required, + missing: none?.replace(/'(\w+)'/, "'{}'"), + positional, + unknown: unknown?.replace("zz_probe", "{}"), + duplicate: duplicate?.replace("'tokenizer'", "'{}'"), + arguments: values, + tokenizers: rows<{ name: string }>("SELECT name FROM system.tokenizers ORDER BY name").map((r) => r.name), + unknownTokenizer: tokenizer?.replace("zz_probe", "{}"), + }; +} + const indexTypes = indexRows.map((r) => { const base = INDEX_BASE[r.name]; const rule = base && @@ -529,6 +626,7 @@ const indexTypes = indexRows.map((r) => { syntax: r.syntax.trim(), description: summary(r.description, 300), rule, + named: r.name === "text" ? textArguments() : undefined, // Only for a type that no table can use any more, as hypothesis. refused: refused && /no longer supported/.test(refused) ? refused.replace(/: When validating secondary index `\w+`/, "") : undefined, }; @@ -593,6 +691,24 @@ const explain = EXPLAIN_KINDS.flatMap((kind, i) => { }]; }); +// ---------------------------------------------------------------- Compression levels of INTO OUTFILE + +const outDirectory = mkdtempSync(join(tmpdir(), "options-")); +const outfile = (method: string, level: string) => `SELECT 1 INTO OUTFILE '${join(outDirectory, "f")}' TRUNCATE COMPRESSION '${method}' LEVEL ${level}`; +const unknownMethod = probe(clickhouse, [outfile("zz_probe", "1")])[0]!.message ?? ""; +const methods = [...(/Only (.*) are supported/.exec(unknownMethod)?.[1] ?? "").matchAll(/'(\w+)'/g)].map((m) => m[1]!).filter((m) => m !== "auto"); +const levels = [...Array(30).keys()]; +const levelOutcomes = probe(clickhouse, methods.flatMap((m) => levels.map((l) => outfile(m, String(l))))); +rmSync(outDirectory, { recursive: true, force: true }); +let levelMessage = ""; +const compressionLevels = Object.fromEntries(methods.flatMap((m, i) => { + const taken = levels.filter((_, j) => levelOutcomes[i * levels.length + j]!.ok); + const refused = levelOutcomes.slice(i * levels.length, (i + 1) * levels.length).find((o) => !o.ok)?.message; + if (!taken.length || taken.some((l, j) => j > 0 && l !== taken[j - 1]! + 1)) return []; + levelMessage ||= refused?.replace(/\d+-\d+/, "{min}-{max}") ?? ""; + return [[m, [taken[0]!, taken[taken.length - 1]!]]]; +})); + write("options.ts", "system.codecs, system.data_skipping_index_types, system.database_engines and clickhouse local", [ "export interface ParameterRule {", " /** Numbers ClickHouse was asked about, with whether it took them; a value between two that agree gets their verdict. */", @@ -614,6 +730,8 @@ write("options.ts", "system.codecs, system.data_skipping_index_types, system.dat " /** ClickHouse's message for too many parameters, with {} for their count. */", " tooMany?: string;", " tooFew?: string;", + " /** Counts between min and max that ClickHouse refuses, with its message. */", + " gaps?: readonly (readonly [number, string])[];", " parameters: readonly ParameterRule[];", "}", "", @@ -634,6 +752,30 @@ write("options.ts", "system.codecs, system.data_skipping_index_types, system.dat " /** ClickHouse's message when it cannot be written in CODEC at all. */", " refused?: string;", " rule?: CallRule;", + " /** The column types it refuses, by ClickHouse's message with {} for the type, written without parameters and with them. */", + " types?: readonly TypeRefusal[];", + " typesWithParameters?: readonly TypeRefusal[];", + "}", + "", + "export interface TypeRefusal {", + " message: string;", + " types: readonly string[];", + " /** The number of parameters it was written with; none when any parameters give the same refusals. */", + " parameters?: number;", + "}", + "", + "/** The arguments of an index type written as name = value, as those of text. */", + "export interface NamedArguments {", + " required: readonly string[];", + " /** ClickHouse's messages, with {} for the name or names. */", + " missing?: string;", + " positional?: string;", + " unknown?: string;", + " duplicate?: string;", + " /** The names it takes, with what each value may be; null when its values are not known. */", + " arguments: Readonly>;", + " tokenizers: readonly string[];", + " unknownTokenizer?: string;", "}", "", "export interface IndexTypeInfo {", @@ -641,6 +783,7 @@ write("options.ts", "system.codecs, system.data_skipping_index_types, system.dat " syntax?: string;", " description?: string;", " rule?: CallRule;", + " named?: NamedArguments;", " /** ClickHouse's message for a type that no table can use any more. */", " refused?: string;", "}", @@ -668,6 +811,9 @@ write("options.ts", "system.codecs, system.data_skipping_index_types, system.dat `/** The settings that a column can have in SETTINGS (...), and ClickHouse's message for others, with {} for the name. */`, `export const COLUMN_SETTINGS: readonly string[] = ${JSON.stringify(columnSettings)};`, `export const COLUMN_SETTING_MESSAGE = ${JSON.stringify(columnUnknown)};`, + "/** The levels that INTO OUTFILE takes after COMPRESSION, by method, and ClickHouse's message for others. */", + `export const COMPRESSION_LEVELS: Readonly> = ${JSON.stringify(compressionLevels)};`, + `export const COMPRESSION_LEVEL_MESSAGE = ${JSON.stringify(levelMessage)};`, "", "export const CODECS: readonly CodecInfo[] = [", ...codecs.map((c) => ` ${compact(c)},`), diff --git a/scripts/generate-statements.ts b/scripts/generate-statements.ts new file mode 100644 index 0000000..620997c --- /dev/null +++ b/scripts/generate-statements.ts @@ -0,0 +1,152 @@ +// Regenerates the documentation that hover shows for SYSTEM commands and for the clauses of the statements that make +// and change users, roles, quotas, row policies and settings profiles. +// Usage: npx tsx scripts/generate-statements.ts --docs +// +// The text is the documentation of ClickHouse: statements/system.mdx, and the pages of those statements in +// statements/create and statements/alter. + +import { readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { parseArgs } from "node:util"; + +const { values: args } = parseArgs({ options: { docs: { type: "string" } } }); +if (!args.docs) { + console.error("Usage: npx tsx scripts/generate-statements.ts --docs "); + process.exit(2); +} +const statements = join(args.docs, "statements"); +const out = join(import.meta.dirname, "..", "packages", "core", "src", "generated", "statements.ts"); + +interface Section { + level: number; + heading: string; + body: string; +} + +/** The Markdown of a page as hover shows it: no MDX imports, components or anchors, and links to other pages as text. */ +function clean(markdown: string): string { + return markdown + .replace(/\{\/\*.*?\*\/\}/g, "") + .replace(/^import .*$/gm, "") + .replace(/<[A-Z]\w*\s*\/>/g, "") + .replace(/<(Note|Tip|Warning)>/g, "**$1**") + .replace(/<\/(?:Note|Tip|Warning)>/g, "") + .replace(/\[([^\]]*)\]\((?!https?:)[^)]*\)/g, "$1") + .replace(/\n{3,}/g, "\n\n") + .trim(); +} + +/** The page without its front matter, cut at its headings; the text before the first one has level 0. */ +function sections(file: string): { title: string; sections: Section[] } { + const text = readFileSync(join(statements, file), "utf8"); + const title = /^title:\s*'([^']*)'/m.exec(text)?.[1] ?? ""; + const lines = text.replace(/^---\n[\s\S]*?\n---\n/, "").split("\n"); + const found: Section[] = [{ level: 0, heading: "", body: "" }]; + let fenced = false; + for (const line of lines) { + if (line.startsWith("```")) fenced = !fenced; + const heading = fenced ? null : /^(#{2,4}) (.*?)\s*(?:\{#[^}]*\})?\s*$/.exec(line); + if (heading) found.push({ level: heading[1]!.length, heading: heading[2]!, body: "" }); + else found[found.length - 1]!.body += `${line}\n`; + } + return { title, sections: found.map((s) => ({ ...s, body: clean(s.body) })) }; +} + +/** The commands a line of SYSTEM syntax names: its words in capitals, with `A|B` and `[OPTIONAL]` spelled out. */ +function commands(line: string): string[][] { + let forms: string[][] = [[]]; + for (const word of line.trim().split(/\s+/)) { + const optional = /^\[([A-Z][A-Z0-9_]*)\]$/.exec(word); + if (optional) { + forms = forms.flatMap((f) => [f, [...f, optional[1]!]]); + continue; + } + if (!/^[A-Z][A-Z0-9_]*(?:\|[A-Z][A-Z0-9_]*)*$/.test(word)) break; + forms = forms.flatMap((f) => word.split("|").map((w) => [...f, w])); + } + return forms.filter((f) => f.length > 0); +} + +// ---------------------------------------------------------------- SYSTEM + +const system = sections("system.mdx").sections; +const systemDocs: { keywords: string[]; markdown: string }[] = []; +const claimed = new Set(); +const claim = (key: string) => { + if (claimed.has(key)) return false; + claimed.add(key); + return true; +}; +// The headings name the commands first, then the syntax of a section adds the forms that no heading names. +// A heading of a command starts with SYSTEM, as all but SYNC DATABASE REPLICA do; the ones below them are not commands. +const bySection = system.map((s) => { + if (s.level > 3 || !/^(?:SYSTEM )?[A-Z][A-Z0-9_]*(?:[ |,\[]|$)/.test(s.heading) || /[a-z]/.test(s.heading)) return []; + return s.heading.split(/,\s*/).flatMap((part) => commands(part.replace(/^SYSTEM /, ""))).map((words) => words.join(" ")).filter(claim); +}); +system.forEach((s, i) => { + if (!bySection[i]!.length) return; + for (const line of s.body.split("\n")) { + if (line.startsWith("SYSTEM ")) bySection[i]!.push(...commands(line.slice(7)).map((words) => words.join(" ")).filter(claim)); + } +}); +system.forEach((s, i) => { + if (bySection[i]!.length) systemDocs.push({ keywords: bySection[i]!, markdown: `**${s.heading}**\n\n${s.body}` }); +}); + +// ---------------------------------------------------------------- Access entities + +/** Sections that the pages name by their subject rather than by their keywords. */ +const NAMED_SECTIONS: Record = { Identification: "IDENTIFIED", "User Host": "HOST" }; +const PAGES = ["user", "role", "quota", "row-policy", "settings-profile"]; + +const accessDocs = ["create", "alter"].flatMap((kind) => + PAGES.map((page) => { + const { title, sections: all } = sections(`${kind}/${page}.mdx`); + // The statement: what comes before the first heading, up to its examples. + const intro = all[0]!.body.split(/^(?:\*\*Examples?\*\*|Examples?:)\s*$/m)[0]!.trim(); + const clauses = new Map(); + const add = (keyword: string, markdown: string) => clauses.set(keyword, [...(clauses.get(keyword) ?? []), markdown]); + for (const s of all) { + const clause = /^(.*) clause$/i.exec(s.heading)?.[1]; + const keywords = clause ? [clause.toUpperCase()] : NAMED_SECTIONS[s.heading] ? [NAMED_SECTIONS[s.heading]!] : [...s.heading.matchAll(/`([A-Z][A-Z_ ]*)`/g)].map((m) => m[1]!); + for (const keyword of keywords) add(keyword, `**${title}: ${s.heading}**\n\n${s.body}`); + // A paragraph that starts with clauses in back quotes is about them, as "`ON CLUSTER` clause allows ...". + for (const paragraph of s.body.split(/\n\n+/)) { + const lead = /^((?:`[^`]+`(?:,? and |, )?)+)/.exec(paragraph)?.[1]; + if (!lead) continue; + for (const m of lead.matchAll(/`([A-Z][A-Z0-9_]*(?: [A-Z][A-Z0-9_]*)*)[` ]/g)) if (!title.startsWith(m[1]!)) add(m[1]!, `**${title}**\n\n${paragraph}`); + } + } + return { statement: title, markdown: `**${title}**\n\n${intro}`, clauses: [...clauses].map(([keyword, parts]) => ({ keywords: [keyword], markdown: parts.join("\n\n") })) }; + }), +); + +const json = (value: unknown) => JSON.stringify(value); +writeFileSync( + out, + [ + "// Generated by scripts/generate-statements.ts from docs/reference/statements of ClickHouse; do not edit.", + "export interface KeywordDoc {", + " /** The keywords it documents, such as FLUSH LOGS after SYSTEM, or VALID UNTIL in CREATE USER. */", + " keywords: readonly string[];", + " markdown: string;", + "}", + "", + "export interface AccessStatementDoc {", + " /** Such as CREATE USER. */", + " statement: string;", + " markdown: string;", + " clauses: readonly KeywordDoc[];", + "}", + "", + "export const SYSTEM_DOCS: readonly KeywordDoc[] = [", + ...systemDocs.map((d) => ` ${json(d)},`), + "];", + "", + "export const ACCESS_DOCS: readonly AccessStatementDoc[] = [", + ...accessDocs.map((d) => ` ${json(d)},`), + "];", + "", + ].join("\n"), +); +console.log(`${claimed.size} SYSTEM commands, ${accessDocs.reduce((n, d) => n + d.clauses.length, 0)} clauses of ${accessDocs.length} statements`);