A Clojure dataframe library for larger-than-memory datasets. Process millions of rows with lazy evaluation and parallel execution—all with familiar dataframe operations.
| Feature | Description |
|---|---|
| Larger than memory | Stream data from disk—no need to fit everything in RAM |
| Lazy & parallel | Operations are pipelined and executed across multiple threads |
| Relational operations | Filter, transform, group-by, aggregate, join, with arbitrary Clojure functions |
| File-to-file | Read CSV in, write CSV out, with built-in IO |
| Native types | Works with standard Clojure and Java types |
| Faster than Dask | Benchmarks show significant speedups on large datasets |
Add to project.clj:
[com.github.clojure-finance/clojask "2.1.2"]Or deps.edn:
com.github.clojure-finance/clojask {:mvn/version "2.1.2"}Basic example (using Employees-example.csv):
(require '[clojask.dataframe :as ck])
;; Load a CSV
(def df (ck/dataframe "Employees-example.csv"))
;; Preview the data (the second row shows each column's type)
(ck/print-df df)
;; | Employee | EmployeeName | Department | Salary | UpdateDate |
;; |------------------+------------------+------------------+------------------+------------------|
;; | java.lang.String | java.lang.String | java.lang.String | java.lang.String | java.lang.String |
;; | 1 | Alice | 11 | 300 | 2020/12/12 |
;; | 2 | Bob | 11 | 600 | 2020/12/01 |
;; ...
;; Set column types
(ck/set-type df "Salary" "double")
(ck/set-type df "UpdateDate" "date:yyyy/MM/dd")
;; Transform data: give Bob a raise, as a new column
;; (an operation over several columns must write to a new column)
(ck/operate df
(fn [name salary]
(if (= name "Bob") (+ salary 100) salary))
["EmployeeName" "Salary"]
"NewSalary")
;; Compute with 8 threads, output to file
(ck/compute df 8 "results.csv"
:select ["Employee" "EmployeeName" "Department" "NewSalary" "UpdateDate"])
;; results.csv:
;; Employee,EmployeeName,Department,NewSalary,UpdateDate
;; 1,Alice,11,300.0,2020/12/12
;; 2,Bob,11,700.0,2020/12/01
;; ...
Solid arrows show required sequence; dotted arrows show optional paths.
Available operations:
- Create:
dataframe - Types:
set-type,set-parser,set-formatter - Columns:
operate,rename-col,select-col,delete-col,reorder-col,get-col-names - Filter:
filter - Group:
group-by,aggregate - Combine:
inner-join,left-join,right-join,outer-join,rolling-join-forward,rolling-join-backward - Output:
compute(with options such as:select,:exclude,:melt,:header),print-df,preview
sort stands outside this pipeline: it sorts the dataframe's source file on disk and writes the result directly to an output file.
- OS: macOS or Linux
- JDK: 17 or newer (tested on 17, 21, and 25)
Upgrading from 2.0.x: clojask no longer uses Onyx, so the JVM flags it needed (--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED, --enable-native-access=ALL-UNNAMED) and the clojask.zookeeper.* / clojask.aeron.* / CLOJASK_* port and directory settings can be removed.
- API Reference — Full documentation for all functions
- Examples Repository — Real-world usage patterns
- Aggregation Functions — Built-in and custom aggregations
- Type System — Supported data types and parsing
Operations are collected lazily and executed together when you call compute, streaming data through your transformation pipeline without loading the full dataset into memory. Each compute runs in your JVM: one thread reads the input in batches, a pool of worker threads (the number you pass to compute) applies the operations and filters, and the calling thread writes the results in input order. Group-by and joins first sort the rows into groups by key, kept in temporary files under .clojask/ (or in memory with :in-memory true); group-by and outer joins write their results in group order.
Found a bug or have a question? Check the existing issues or open a new one.