feat: native Spark add_months scalar function

CMakeLists.txt

@@ -22,6 +22,7 @@ set(EXTENSION_SOURCES
     src/core/parser_plan_helpers.cpp
     src/core/parser_sql_extractors.cpp
     src/core/sql_utils.cpp
+    src/core/time_travel_pins.cpp
     src/core/refresh_metadata.cpp
     src/core/incremental_checker.cpp
     src/core/plan_rewrite.cpp
@@ -78,7 +79,8 @@ set(EXTENSION_SOURCES
     ${LPTS_DIR}/src/lpts_ast_renderer.cpp
     ${LPTS_DIR}/src/lpts_ast_builder.cpp
     ${LPTS_DIR}/src/lpts_ast_flattener.cpp
-    ${LPTS_DIR}/src/dialect_function_map.cpp)
+    ${LPTS_DIR}/src/dialect_function_map.cpp
+    ${LPTS_DIR}/src/spark_scalar_functions.cpp)
 
 build_static_extension(${TARGET_NAME} ${EXTENSION_SOURCES})
 build_loadable_extension(${TARGET_NAME} " " ${EXTENSION_SOURCES})

README.md

@@ -97,6 +97,7 @@ MVs can be created using any SQL construct. Unsupported operators automatically
 | `openivm_regular_nterm` | BOOLEAN | `true` | Use N-term telescoping for eligible regular-table inner joins compiled for external engines | [Inner join](docs/operators/inner-join.md#regular-table-n-term-compilation) |
 | `openivm_profile_refresh` | BOOLEAN | `false` | Record per-step refresh timings in `openivm_refresh_profile` | [Automatic refresh](docs/refresh/automatic-refresh.md) |
 | `openivm_files_path` | VARCHAR | — | Directory for compiled SQL reference files | [Internals](docs/internals/delta-tables.md) |
+| `openivm_input_dialect` | VARCHAR | `duckdb` | Dialect of incoming `CREATE MATERIALIZED VIEW` bodies (e.g. `spark` for `VERSION AS OF`) | [Parser](docs/internals/parser.md#input-dialect) |
 
 
 ## Pragmas

docs/internals/parser.md

@@ -43,6 +43,38 @@ CREATE MATERIALIZED VIEW mv REFRESH EVERY '5 minutes' AS
 
 The parsed interval (300 seconds) is stored in the `refresh_interval` column of `openivm_views`. When omitted, `refresh_interval` is `NULL` (manual refresh only). See [Automatic Refresh](../refresh/automatic-refresh.md) for how the daemon uses this.
 
+## Input dialect
+
+`openivm_input_dialect` (default `duckdb`) declares the dialect the *incoming* `CREATE MATERIALIZED VIEW` body is written in. When it is not `duckdb`, the body is run through LPTS' `NormalizeInputSqlToDuckDB` before `Parser::ParseQuery`, so backtick identifiers, dialect casts, interval literals and time-travel clauses are translated into DuckDB syntax first. The setting is mirrored onto the parser extension's `ParserExtensionInfo` because `parser_override` is not given a `ClientContext`.
+
+```sql
+SET openivm_input_dialect='spark';
+CREATE MATERIALIZED VIEW mv AS
+    SELECT region, SUM(amount) FROM sales VERSION AS OF 366 GROUP BY region;
+```
+
+## Time-travel pins
+
+A Spark/Delta pin (`VERSION AS OF n`, `TIMESTAMP AS OF '...'`) normalizes to DuckDB's `AT (VERSION => n)` / `AT (TIMESTAMP => '...')` qualifier. OpenIVM registers the sources of a compiled view as plain in-memory stand-in tables whose catalog reports `SupportsTimeTravel() == false`, so a pinned scan cannot be bound directly.
+
+`src/core/time_travel_pins.cpp` handles this by *peeling* the qualifier off each `BaseTableRef` whose catalog cannot honour it, planning against the pin-less stand-in, and *restoring* the qualifier onto the matching `AstGetNode::table_name` after `LogicalPlanToAst`. Pins are keyed by relation, so aliases, repeated scans, CTEs and joins all keep their own pin, and two relations pinned to different snapshots stay distinct. A catalog that does support time travel (DuckLake) is never peeled and binds natively.
+
+Because pin restoration is keyed by relation, a view that pins the same relation ambiguously is refused with `NotImplementedException` rather than compiled with a guessed snapshot:
+
+- the same relation pinned to two different snapshots,
+- the same pin naming two differently qualified relations,
+- the same relation scanned both pinned and unpinned.
+
+The first is a real limitation rather than a policy choice. DuckDB resolves the `AT (...)` qualifier during *catalog lookup* — it selects which snapshot of the catalog entry to bind — so neither `LogicalGet` nor `AstGetNode` carries a per-scan pin. Two scans of one relation at two versions are indistinguishable in the bound plan except by `table_index`, and pairing those back to parse-tree references would rest on the binder's index-allocation order, whose failure mode is silently reading the wrong snapshot. Repeated scans that share a pin, including across CTEs, are supported normally.
+
+The pin is stored in the view SQL in `openivm_views.sql_string` so it survives a restart. Every site that binds or locally executes that SQL peels it first, and refresh source qualification drops it, since the local stand-in catalog holds no snapshots. Foreign-dialect output re-attaches it: Spark renders `VERSION AS OF n`, DuckDB keeps `AT (...)`, and any dialect LPTS has no verified time-travel syntax for raises `LPTS_UNSUPPORTED_TIME_TRAVEL` instead of silently reading the latest snapshot.
+
+### Alias association
+
+Spark writes the pin *between* a relation and its alias (`FROM t VERSION AS OF n p`) where DuckDB wants it after both (`FROM t AS p AT (VERSION => n)`), so normalization has to carry the alias across the rewrite. That is the one step where a pin could land on a neighbouring relation, attach to the wrong alias, or be dropped outright — each of which reads a different snapshot while still compiling cleanly.
+
+The association is therefore checked rather than trusted. `CollectSourceSnapshotBindings` reads every `{relation, alias, qualifier}` triple straight off the *source* text before normalization, and `VerifySnapshotBindings` re-checks them against the parse tree DuckDB produced, raising `NotImplementedException` on any pin that did not come through on the same relation and alias.
+
 ## IVM compatibility classification
 
 After rewriting, the parser plans the query and walks the logical plan to classify the view into a refresh type:

src/core/parser.cpp

@@ -13,6 +13,7 @@
 #include "core/ivm_view_classifier.hpp"
 #include "lpts_pipeline.hpp"
 #include "core/sql_utils.hpp"
+#include "core/time_travel_pins.hpp"
 #include "rules/column_hider.hpp"
 #include "upsert/refresh_compiler.hpp"
 #include "duckdb/common/printer.hpp"
@@ -405,6 +406,13 @@ MaterializedViewParserExtension::PlanFunction(ParserExtensionInfo *info, ClientC
 	// re-derives what it needs from the plan.
 	unordered_set<string> table_names;
 	auto table_names_start = create_profile_now();
+	// Peel time-travel pins the catalog cannot bind before anything plans this statement, keeping
+	// each pin keyed by its relation so it can be re-attached to the generated SQL below.
+	auto time_travel_pins = openivm::TimeTravelPins::Peel(context, *statement);
+	// SQL OpenIVM binds or executes itself runs against the very catalog that cannot honour the pin,
+	// so those copies drop it. The stored view SQL keeps it, and refresh re-attaches it when
+	// rendering for a foreign dialect.
+	auto local_view_query = time_travel_pins.StripFrom(original_view_query);
 	try {
 		table_names = con.GetTableNames(statement->query);
 	} catch (const std::exception &e) {
@@ -438,6 +446,7 @@ MaterializedViewParserExtension::PlanFunction(ParserExtensionInfo *info, ClientC
 		auto select_parse_plan_start = create_profile_now();
 		Parser select_parser;
 		select_parser.ParseQuery(original_view_query);
+		openivm::TimeTravelPins::PeelForLocalBinding(context, *select_parser.statements[0]);
 		Planner select_planner(context);
 		select_planner.CreatePlan(std::move(select_parser.statements[0]));
 		auto select_plan = std::move(select_planner.plan);
@@ -539,6 +548,7 @@ MaterializedViewParserExtension::PlanFunction(ParserExtensionInfo *info, ClientC
 			// Refresh-time target dialects are selected per CompileFacts.
 			SqlDialect dialect = SqlDialect::DUCKDB;
 			auto ast = LogicalPlanToAst(context, select_plan, dialect);
+			time_travel_pins.RestoreInto(*ast);
 			auto cte_list = AstToCteList(*ast, dialect);
 			view_query = cte_list->ToQuery(true, output_names);
 			if (!view_query.empty() && view_query.back() == ';') {
@@ -663,7 +673,7 @@ MaterializedViewParserExtension::PlanFunction(ParserExtensionInfo *info, ClientC
 	FilteredGroupCountExtract filtered_group_count_extract;
 	FilteredGroupCountAuxRequirement filtered_group_count_aux_candidate;
 	if (analysis.found_nested_aggregate &&
-	    ExtractFilteredGroupCount(original_view_query, output_names, filtered_group_count_extract)) {
+	    ExtractFilteredGroupCount(local_view_query, output_names, filtered_group_count_extract)) {
 		string aux_table = "openivm_filtered_group_count_" + view_name;
 		string group_q = KeywordHelper::WriteOptionallyQuoted(filtered_group_count_extract.group_col);
 		string sum_q = KeywordHelper::WriteOptionallyQuoted(filtered_group_count_extract.sum_col);
@@ -677,7 +687,7 @@ MaterializedViewParserExtension::PlanFunction(ParserExtensionInfo *info, ClientC
 	}
 
 	if (analysis.found_semi_anti_join && !analysis.found_aggregation) {
-		if (ExtractSemiAntiQuery(original_view_query, semi_anti_extract)) {
+		if (ExtractSemiAntiQuery(local_view_query, semi_anti_extract)) {
 			string left_table_name = SqlUtils::LastIdentifierPart(semi_anti_extract.left_table);
 			auto col_result = con.Query("SELECT column_name FROM information_schema.columns WHERE "
 			                            "lower(table_name) = lower('" +
@@ -803,7 +813,7 @@ MaterializedViewParserExtension::PlanFunction(ParserExtensionInfo *info, ClientC
 		if (aux_enabled && single_source) {
 			vector<string> dcols;
 			string d_input_sql, d_source, d_filter;
-			if (!ExtractInnerDistinct(original_view_query, dcols, d_input_sql, d_source, d_filter)) {
+			if (!ExtractInnerDistinct(local_view_query, dcols, d_input_sql, d_source, d_filter)) {
 				OPENIVM_DEBUG_PRINT("[CREATE MV] DISTINCT_INCREMENTAL extractor failed — demoting to "
 				                    "GROUP_RECOMPUTE\n");
 			} else {
@@ -891,7 +901,7 @@ MaterializedViewParserExtension::PlanFunction(ParserExtensionInfo *info, ClientC
 				candidate_group_columns.push_back(output_names[i]);
 			}
 			CountDistinctExtract cd_extract;
-			if (ExtractCountDistinctAggregate(original_view_query, candidate_group_columns, output_names, cd_extract)) {
+			if (ExtractCountDistinctAggregate(local_view_query, candidate_group_columns, output_names, cd_extract)) {
 				count_distinct_aux_candidate = {
 				    "openivm_aux_" + view_name, SqlUtils::LastIdentifierPart(cd_extract.source),
 				    candidate_group_columns,    cd_extract.group_exprs,
@@ -1354,7 +1364,8 @@ MaterializedViewParserExtension::PlanFunction(ParserExtensionInfo *info, ClientC
 		if (!current_catalog.empty() && current_catalog != default_db) {
 			con.Query("USE " + current_catalog_schema);
 		}
-		string initial_load_statement = "CREATE TABLE " + initial_load_target + " AS " + view_query;
+		string local_initial_load_query = time_travel_pins.StripFrom(view_query);
+		string initial_load_statement = "CREATE TABLE " + initial_load_target + " AS " + local_initial_load_query;
 		string diagnostic;
 		diagnostic += "\n[OpenIVM initial-load diagnostic]\n";
 		diagnostic += "view_name: " + view_name + "\n";
@@ -1364,8 +1375,8 @@ MaterializedViewParserExtension::PlanFunction(ParserExtensionInfo *info, ClientC
 		diagnostic += "initial_load_statement:\n" + initial_load_statement + "\n\n";
 		diagnostic += "original_view_query:\n" + original_view_query + "\n\n";
 		diagnostic += "generated_view_query:\n" + view_query + "\n\n";
-		diagnostic += ExplainInitialLoadQuery(con, "EXPLAIN original_view_query:", original_view_query);
-		diagnostic += ExplainInitialLoadQuery(con, "EXPLAIN generated_view_query:", view_query);
+		diagnostic += ExplainInitialLoadQuery(con, "EXPLAIN original_view_query:", local_view_query);
+		diagnostic += ExplainInitialLoadQuery(con, "EXPLAIN generated_view_query:", local_initial_load_query);
 		diagnostic += ExplainInitialLoadQuery(con, "EXPLAIN initial_load_statement:", initial_load_statement);
 		Printer::Print(diagnostic);
 
@@ -1398,7 +1409,7 @@ MaterializedViewParserExtension::PlanFunction(ParserExtensionInfo *info, ClientC
 		ddl.push_back(BuildSemiAntiInitialDataSQL(initial_load_target, aux_target, meta.join_type, meta.left_cols,
 		                                          meta.output_cols, meta.null_aware, meta.null_aware_left_col));
 	} else {
-		ddl.push_back("create table " + initial_load_target + " as " + view_query);
+		ddl.push_back("create table " + initial_load_target + " as " + time_travel_pins.StripFrom(view_query));
 	}
 	if (staged_cross_catalog_replace) {
 		// DuckDB cannot make the DuckLake objects and native metadata atomic
@@ -1640,7 +1651,7 @@ MaterializedViewParserExtension::PlanFunction(ParserExtensionInfo *info, ClientC
 
 string MaterializedViewLifecycleQuery(ClientContext &context, const FunctionParameters &parameters) {
 	auto query = StringValue::Get(parameters.values[0]);
-	auto parse_result = MaterializedViewParserExtension::ParseFunction(nullptr, query);
+	auto parse_result = ParseMaterializedViewStatement(query, OpenIvmInputDialect(context));
 	if (parse_result.type != ParserExtensionResultType::PARSE_SUCCESSFUL) {
 		throw ParserException("OpenIVM could not parse the materialized-view lifecycle statement");
 	}

src/core/parser_parse.cpp

@@ -3,11 +3,15 @@
 #include "core/openivm_constants.hpp"
 #include "core/openivm_debug.hpp"
 #include "core/sql_utils.hpp"
+#include "core/time_travel_pins.hpp"
+#include "duckdb/main/config.hpp"
+#include "duckdb/main/extension_callback_manager.hpp"
 #include "duckdb/parser/expression/constant_expression.hpp"
 #include "duckdb/parser/parser.hpp"
 #include "duckdb/parser/qualified_name.hpp"
 #include "duckdb/parser/statement/drop_statement.hpp"
 #include "duckdb/parser/statement/pragma_statement.hpp"
+#include "lpts_parser.hpp"
 
 #include <regex>
 
@@ -20,10 +24,37 @@ static unique_ptr<SQLStatement> BuildInternalPragma(const string &name, const st
 	return std::move(statement);
 }
 
+SqlDialect OpenIvmInputDialect(ClientContext &context) {
+	Value setting_value;
+	if (!context.TryGetCurrentSetting(OPENIVM_INPUT_DIALECT_SETTING, setting_value) || setting_value.IsNull()) {
+		return SqlDialect::DUCKDB;
+	}
+	return ParseSqlDialectSetting(setting_value.ToString(), OPENIVM_INPUT_DIALECT_SETTING);
+}
+
+void SetOpenIvmInputDialect(ClientContext &context, SetScope scope, Value &parameter) {
+	auto dialect = parameter.IsNull() ? SqlDialect::DUCKDB
+	                                  : ParseSqlDialectSetting(parameter.ToString(), OPENIVM_INPUT_DIALECT_SETTING);
+	// `parser_override` and `parse_function` are handed the extension info, not a ClientContext, so
+	// mirror the resolved dialect there. The info is owned by the DBConfig, so the mirror stays
+	// scoped to this database.
+	for (auto &extension : ExtensionCallbackManager::Get(context).ParserExtensions()) {
+		auto info = dynamic_cast<MaterializedViewParserExtensionInfo *>(extension.parser_info.get());
+		if (info) {
+			info->SetInputDialect(dialect);
+		}
+	}
+}
+
+static SqlDialect InputDialectFromInfo(ParserExtensionInfo *info) {
+	auto materialized_view_info = dynamic_cast<MaterializedViewParserExtensionInfo *>(info);
+	return materialized_view_info ? materialized_view_info->InputDialect() : SqlDialect::DUCKDB;
+}
+
 ParserOverrideResult MaterializedViewParserExtension::OverrideFunction(ParserExtensionInfo *info, const string &query,
                                                                        ParserOptions &options) {
 	try {
-		auto extension_result = ParseFunction(info, query);
+		auto extension_result = ParseMaterializedViewStatement(query, InputDialectFromInfo(info));
 		if (extension_result.type == ParserExtensionResultType::PARSE_SUCCESSFUL) {
 			vector<unique_ptr<SQLStatement>> statements;
 			statements.push_back(BuildInternalPragma("openivm_materialized_view_lifecycle", query));
@@ -55,6 +86,10 @@ ParserOverrideResult MaterializedViewParserExtension::OverrideFunction(ParserExt
 
 ParserExtensionParseResult MaterializedViewParserExtension::ParseFunction(ParserExtensionInfo *info,
                                                                           const string &query) {
+	return ParseMaterializedViewStatement(query, InputDialectFromInfo(info));
+}
+
+ParserExtensionParseResult ParseMaterializedViewStatement(const string &query, SqlDialect input_dialect) {
 	auto query_lower = SqlUtils::SQLToLowercase(StringUtil::Replace(query, ";", ""));
 	StringUtil::Trim(query_lower);
 	// Strip SQL line comments (-- to end of line) before whitespace normalization.
@@ -126,8 +161,25 @@ ParserExtensionParseResult MaterializedViewParserExtension::ParseFunction(Parser
 	// at the plan level in PlanFunction via PlanRewrite + LPTS.
 	OPENIVM_DEBUG_PRINT("[CREATE MV] After structural rewrite: %s\n", query_lower.c_str());
 
+	vector<openivm::SnapshotBinding> pin_bindings;
+	if (input_dialect != SqlDialect::DUCKDB) {
+		// The body is written in another dialect, so DuckDB's parser cannot read it as-is — a
+		// Spark/Delta temporal clause (`FROM t VERSION AS OF 366`) dies on `AS`. LPTS rewrites the
+		// source spelling into the semantically equivalent DuckDB one, keeping the pin
+		// (`AT (VERSION => 366)`) rather than dropping it, which would silently promote the scan to
+		// "read latest". Record what each pin is written against first, so the rewrite can be held
+		// to it below.
+		pin_bindings = openivm::CollectSourceSnapshotBindings(query_lower, input_dialect);
+		query_lower = NormalizeInputSqlToDuckDB(query_lower, input_dialect);
+		OPENIVM_DEBUG_PRINT("[CREATE MV] After %s input normalization: %s\n", SqlDialectToString(input_dialect).c_str(),
+		                    query_lower.c_str());
+	}
+
 	Parser p;
 	p.ParseQuery(query_lower);
+	// Rewriting the temporal clause moves it across the relation's alias, so re-check against the
+	// parse tree that every pin still names the relation and alias it was written against.
+	openivm::VerifySnapshotBindings(*p.statements[0], pin_bindings);
 
 	auto parse_data = make_uniq_base<ParserExtensionParseData, MaterializedViewParseData>(std::move(p.statements[0]),
 	                                                                                      refresh_interval);