Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -389,7 +389,7 @@ jobs:
}

build_and_push cmd sqrl-cli --build-arg BASE_TAG=$BASE_TAG
build_and_push sqrl-server sqrl-server/sqrl-server-vertx --build-arg DUCKDB_EXTENSIONS_TAG=$DUCKDB_EXTENSIONS_TAG
build_and_push sqrl-server sqrl-server/sqrl-server-spring --build-arg DUCKDB_EXTENSIONS_TAG=$DUCKDB_EXTENSIONS_TAG

docker-build-duckdb-extensions:
machine:
Expand Down
30 changes: 14 additions & 16 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ mvn clean install -P quickbuild
mvn -P dev initialize

# Server-specific builds
mvn clean package # Build fat JAR (vertx-server.jar)
mvn clean package # Build fat JAR (spring-server.jar)
mvn clean package -Pskip-shade-plugin # Build without fat JAR
mvn clean package -Pinstrument # Build with JaCoCo instrumentation
```
Expand Down Expand Up @@ -94,8 +94,7 @@ This is a multi-module Maven project with the following key components:

**sqrl-server/** - GraphQL API server implementation that translates GraphQL queries, mutations, and subscriptions into database calls:
- `sqrl-server-core/` - Core interfaces and models (GraphQL schema, execution coordinates)
- `sqrl-server-vertx-base/` - Full Vert.x implementation with database clients and Kafka integration
- `sqrl-server-vertx/` - Standalone deployment module with Docker support
- `sqrl-server-spring/` - Spring Boot implementation with R2DBC/JDBC database clients and Kafka integration

**sqrl-tools/** - Command-line tools and utilities:
- `sqrl-cli/` - Main CLI interface (entry point: `com.datasqrl.cli.DatasqrlCli`)
Expand All @@ -111,11 +110,11 @@ This is a multi-module Maven project with the following key components:
- **Java 17** with Maven build system
- **Apache Flink 1.19.2** for stream processing
- **Apache Calcite 1.27.0** for SQL parsing and optimization
- **Vert.x 5.0.0** for API server
- **Spring Boot 4.0** for API server (WebFlux reactive)
- **GraphQL Java 19.2** for API generation
- **Apache Kafka 3.4.0** for streaming
- **PostgreSQL 42.7.7** for storage
- **JUnit 5** with Testcontainers for testing
- **JUnit 6** with Testcontainers for testing

## Development Workflow

Expand All @@ -142,7 +141,7 @@ All dependency versions should be centralized as properties in the root pom.xml
```xml
<properties>
<jackson.version>2.19.1</jackson.version>
<vertx.version>5.0.1</vertx.version>
<spring-boot.version>4.0.2</spring-boot.version>
<kafka.version>3.4.0</kafka.version>
<flink.version>1.19.3</flink.version>
<httpcomponents.version>4.5.14</httpcomponents.version>
Expand Down Expand Up @@ -280,28 +279,27 @@ If tests fail due to Flink memory issues, uncomment the configuration line in `E

### Key Design Patterns
- **Visitor Pattern**: Extensively used for processing GraphQL model (`RootVisitor`, `QueryCoordVisitor`, `SchemaVisitor`)
- **Reactive Architecture**: Built on Vert.x event loop with CompletableFuture for async operations
- **Schema-First**: GraphQL schema loaded from `server-model.json` at runtime with pre-compiled execution paths
- **Reactive Architecture**: Built on Spring WebFlux with CompletableFuture for async operations
- **Schema-First**: GraphQL schema loaded from `vertx.json` at runtime with pre-compiled execution paths

### Runtime Model
The server operates on a compiled model where the DataSQRL compiler generates `server-model.json` containing all GraphQL execution metadata. The server loads this at startup and creates optimized execution paths - no runtime SQL generation occurs.
The server operates on a compiled model where the DataSQRL compiler generates `vertx.json` containing all GraphQL execution metadata. The server loads this at startup and creates optimized execution paths - no runtime SQL generation occurs.

### Database Abstraction
Multi-database support through `SqlClient` interface:
- PostgreSQL: Native Vert.x client with pipelining
- DuckDB: JDBC-based connection
Multi-database support through `SpringJdbcClient` interface:
- PostgreSQL: R2DBC async client for reactive database access
- DuckDB: JDBC-based connection with Virtual Threads
- Snowflake: JDBC-based connection with specialized configuration

### Server Configuration Files
- `server-model.json`: Runtime GraphQL model and execution coordinates
- `server-config.json`: Server configuration (ports, database connections)
- `vertx.json`: Runtime GraphQL model and execution coordinates
- `application.yaml`: Spring Boot configuration (ports, database connections)
- `snowflake-config.json`: Optional Snowflake-specific configuration
- `log4j2.properties`: Logging configuration

## Important Development Notes

### Module Dependencies
Always check existing dependencies in `pom.xml` files before adding new libraries. The project uses specific versions of Vert.x, GraphQL-Java, and database drivers.
Always check existing dependencies in `pom.xml` files before adding new libraries. The project uses specific versions of Spring Boot, GraphQL-Java, and database drivers.

### Database Operations
All database operations are asynchronous and non-blocking. Use the appropriate `SqlClient` implementation for the target database system.
Expand Down
133 changes: 67 additions & 66 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -66,10 +66,10 @@
</developers>

<modules>
<module>sqrl-cli</module>
<module>sqrl-discovery</module>
<module>sqrl-planner</module>
<module>sqrl-server</module>
<module>sqrl-cli</module>
<module>sqrl-testing</module>
</modules>

Expand Down Expand Up @@ -129,9 +129,8 @@
<graphql-java.version>25.0</graphql-java.version>
<graphql-micrometer.version>1.0.1</graphql-micrometer.version>
<guava.version>33.5.0-jre</guava.version>
<guice-multibindings.version>4.2.3</guice-multibindings.version>
<guice.version>7.0.0</guice.version>
<h2.version>2.4.240</h2.version>
<spring-framework.version>6.2.3</spring-framework.version>
<hikari.version>7.0.2</hikari.version>
<!-- Do not touch Hadoop version unless it has been updated on the Flink side.
Otherwise, it might break the Flink filesystem integration. -->
Expand All @@ -141,10 +140,11 @@
<jackson.version>2.21.0</jackson.version>
<jacoco.version>0.8.14</jacoco.version>
<jakarta.annotation.version>3.0.0</jakarta.annotation.version>
<jakarta.inject.version>2.0.1</jakarta.inject.version>
<javax.validation.version>2.0.1.Final</javax.validation.version>
<jjwt.version>0.13.0</jjwt.version>
<json-schema.version>2.0.0</json-schema.version>
<junit.jupiter.version>6.0.2</junit.jupiter.version>
<junit.version>6.0.2</junit.version>
<kafka.version>3.9.1</kafka.version>
<log4j.version>2.25.3</log4j.version>
<lombok.version>1.18.42</lombok.version>
Expand All @@ -165,7 +165,9 @@
<snakeyaml.version>2.5</snakeyaml.version>
<swagger-core.version>2.2.42</swagger-core.version>
<testcontainers.version>2.0.3</testcontainers.version>
<vertx.version>5.0.7</vertx.version>
<spring-boot.version>4.0.2</spring-boot.version>
<r2dbc-postgresql.version>1.0.7.RELEASE</r2dbc-postgresql.version>
<reactor-kafka.version>1.3.23</reactor-kafka.version>

<!-- Plugin versions -->
<dockerfile-maven-plugin.version>1.4.13</dockerfile-maven-plugin.version>
Expand Down Expand Up @@ -194,12 +196,7 @@
</dependency>
<dependency>
<groupId>com.datasqrl</groupId>
<artifactId>sqrl-server-vertx-base</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.datasqrl</groupId>
<artifactId>sqrl-server-vertx</artifactId>
<artifactId>sqrl-server-spring</artifactId>
<version>${project.version}</version>
</dependency>

Expand Down Expand Up @@ -326,27 +323,64 @@
<version>${graphql-java-extended-scalars.version}</version>
</dependency>

<!-- Jackson BOM - must be before Spring Boot to use our Jackson version -->
<dependency>
<groupId>com.fasterxml.jackson</groupId>
<artifactId>jackson-bom</artifactId>
<version>${jackson.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>

<!-- JUnit BOM - must be before Spring Boot to override its JUnit version -->
<dependency>
<groupId>org.junit</groupId>
<artifactId>junit-bom</artifactId>
<version>${junit.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>

<!-- Spring Boot BOM -->
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-dependencies</artifactId>
<version>${vertx.version}</version>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>${spring-boot.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>

<!-- required for vertx subscriptions -->
<!-- Jackson Scala module - must match jackson.version -->
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-scala_2.12</artifactId>
<version>${jackson.version}</version>
</dependency>

<!-- R2DBC PostgreSQL -->
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>r2dbc-postgresql</artifactId>
<version>${r2dbc-postgresql.version}</version>
</dependency>

<!-- Reactor Kafka -->
<dependency>
<groupId>io.projectreactor.kafka</groupId>
<artifactId>reactor-kafka</artifactId>
<version>${reactor-kafka.version}</version>
</dependency>

<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-core</artifactId>
<version>${projectreactor.version}</version>
</dependency>
<!-- vertx metrics -->
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
<version>${micrometer.version}</version>
</dependency>
<!-- required for vertx jdbc -->
<dependency>
<groupId>io.agroal</groupId>
<artifactId>agroal-pool</artifactId>
Expand Down Expand Up @@ -555,14 +589,6 @@
<scope>import</scope>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson</groupId>
<artifactId>jackson-bom</artifactId>
<version>${jackson.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>

<!-- JSON Schema -->
<dependency>
<groupId>com.networknt</groupId>
Expand Down Expand Up @@ -606,13 +632,6 @@
<artifactId>testcontainers-junit-jupiter</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
<!-- JUnit 4 is no longer necessary, so we exclude it -->
<exclusions>
<exclusion>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
Expand All @@ -632,13 +651,6 @@
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit</groupId>
<artifactId>junit-bom</artifactId>
<version>${junit.jupiter.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>

<!-- logging -->
<dependency>
Expand Down Expand Up @@ -674,6 +686,16 @@
<artifactId>flink-test-utils</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.jupiter</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
Expand Down Expand Up @@ -762,6 +784,11 @@
<artifactId>junit-jupiter-api</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.assertj</groupId>
Expand Down Expand Up @@ -792,11 +819,6 @@
<artifactId>maven-shade-plugin</artifactId>
<version>3.6.1</version>
</plugin>
<plugin>
<groupId>io.reactiverse</groupId>
<artifactId>vertx-maven-plugin</artifactId>
<version>2.0.2</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
Expand Down Expand Up @@ -1077,13 +1099,6 @@
</activation>
<build>
<plugins>
<plugin>
<groupId>io.reactiverse</groupId>
<artifactId>vertx-maven-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
Expand Down Expand Up @@ -1183,7 +1198,7 @@

<excludeArtifacts>
<!-- client jars are too large, skipping uploading for now -->
<excludeArtifact>sqrl-server-vertx</excludeArtifact>
<excludeArtifact>sqrl-server-spring</excludeArtifact>
<excludeArtifact>sqrl-cli</excludeArtifact>
<excludeArtifact>sqrl-test</excludeArtifact>
<excludeArtifact>sqrl-run</excludeArtifact>
Expand Down Expand Up @@ -1361,13 +1376,6 @@
</properties>
<build>
<plugins>
<plugin>
<groupId>io.reactiverse</groupId>
<artifactId>vertx-maven-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
Expand Down Expand Up @@ -1399,13 +1407,6 @@
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>io.reactiverse</groupId>
<artifactId>vertx-maven-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
Expand Down
Loading