Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@ Starting from a `demo-db.duckdb` with `nodes_user`, `nodes_city`, `edges_follows

```bash
uv run icebug-format \
--directed \
--source-db demo-db.duckdb \
--schema demo-db/schema.cypher
```
Expand Down Expand Up @@ -83,11 +82,11 @@ graph: IcebugMemGraph = IcebugMemGraph.from_arrow_tables(
to_node_arrow_table=cities, # pa.Table, first column is the primary key
)

# Directed or undirected homogeneous graph (same node type on both ends)
# Directed graph, or homogeneous graph with reverse edges added
graph: IcebugMemGraph = IcebugMemGraph.from_arrow_tables(
from_node_arrow_table=users, # pa.Table, first column is the primary key
rel_arrow_table=follows, # pa.Table with 'source' and 'target' columns
undirected=True, # undirected=True for undirected (to_node_arrow_table must be omitted)
add_reverse_edges=True, # to_node_arrow_table must be omitted
)

# Node tables are passed through unchanged
Expand All @@ -108,16 +107,16 @@ The `rel_arrow_table` source and target columns are resolved by name in priority

Any remaining columns are preserved as edge properties in `graph.indices`.

Set `undirected=True` to automatically add reverse edges (undirected graph). For undirected graphs, `to_node_arrow_table` must be omitted; the same node table is used for both sides of every edge.
Use `--add-reverse-edges` in the CLI, or `add_reverse_edges=True` in the Python API, to emit a symmetric adjacency by adding reverse edges. For reverse-edge expansion, `to_node_arrow_table` must be omitted; the same node table is used for both sides of every edge.

## Caveats

- icebug-format will always output a directed graph
- If you want an undirected graph to be converted, pass undirected=True to the CLI or Python API, and the reverse edges will be added automatically. But do note that undirected graphs are supported for rel tables with same node type on both ends only
- If an algorithm needs symmetric adjacency, pass `--add-reverse-edges` to the CLI or `add_reverse_edges=True` to the Python API. Reverse edges will be added automatically. Reverse-edge expansion is supported only for rel tables with the same node type on both ends.
- Reverse-edge expansion is all or nothing for a conversion. If your graph mixes edge types that should be symmetric, such as `friends`, with edge types that should stay directed, such as `follows`, run separate conversions or add reverse edges before calling icebug-format; `--add-reverse-edges` cannot be applied selectively per edge type.

---

## Further reading

[Blog post: Graph Archiving with Apache GraphAR](https://adsharma.github.io/graph-archiving/)

28 changes: 15 additions & 13 deletions doc/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ icebug-format \
[--edge-table EDGE_TABLE] \
[--schema SCHEMA_CYPHER] \
[--storage STORAGE_PATH] \
[--directed] \
[--add-reverse-edges] \
[--test] \
[--limit N] \
[--memory-limit LIMIT]
Expand Down Expand Up @@ -210,23 +210,25 @@ For each selected edge table `ET`:
Only edges whose endpoints both exist in the selected node mappings are emitted,
because the conversion uses inner-join semantics.

### Directed and Undirected Modes
### Direction and Reverse Edges

With `--directed`, each non-self-loop input edge produces one output edge:
By default, input edges are treated as directed. Each input edge produces one
output edge:

```text
source -> target
```

Without `--directed`, the graph is treated as undirected. Each non-self-loop
input edge produces two output edges:
With `--add-reverse-edges`, each non-self-loop input edge also emits a reverse
edge, producing symmetric adjacency for algorithms that need it. Each self-loop
appears once:

```text
source -> target
target -> source
```

Edge properties are copied onto both directions in undirected mode.
Edge properties are copied onto both directions when reverse edges are added.

### Test Limit

Expand All @@ -239,8 +241,8 @@ When multiple edge tables are selected, the per-table limit is:
floor(limit / number_of_edge_tables)
```

The limit is applied before undirected edge duplication. In undirected mode, a
per-table limit of `L` can therefore emit up to `2L` rows for that edge type.
The limit is applied before reverse-edge expansion. With `--add-reverse-edges`,
a per-table limit of `L` can therefore emit up to `2L` rows for that edge type.

The implementation does not define a deterministic ordering before applying the
limit, so callers should treat test-mode sampling as backend-dependent unless
Expand Down Expand Up @@ -366,15 +368,15 @@ Columns:
```text
n_nodes
n_edges
directed
add_reverse_edges
```

`n_nodes` is the sum of selected node table row counts.

`n_edges` is the sum of emitted rows across all generated indices tables. In
undirected mode this includes duplicated reverse edges.
`n_edges` is the sum of emitted rows across all generated indices tables. With
`--add-reverse-edges`, this includes generated reverse edges.

`directed` records whether `--directed` was supplied.
`add_reverse_edges` records whether `--add-reverse-edges` was supplied.

### Parquet Directory

Expand Down Expand Up @@ -491,7 +493,7 @@ A non-DuckDB backend must provide equivalent behavior for:
- assigning zero-based dense IDs with deterministic ordering
- joining edge tables to source and destination mapping tables
- filtering self-loops
- duplicating edges for undirected mode
- adding reverse edges for symmetric adjacency
- preserving edge property columns and values
- grouping emitted edges by `csr_source` to compute degrees
- creating cumulative CSR offsets for all source node IDs from `0` to `N - 1`,
Expand Down
57 changes: 33 additions & 24 deletions icebug_format/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,9 @@
ICEBUG_DISK_VERSION = "v1"


def _write_parquet_with_icebug_metadata(con, table_name: str, output_path: Path) -> None:
def _write_parquet_with_icebug_metadata(
con, table_name: str, output_path: Path
) -> None:
"""Export a DuckDB table to parquet with icebug_disk_version metadata."""
con.execute(
f"""
Expand Down Expand Up @@ -341,7 +343,7 @@ def get_display_name(table_name: str, prefix: str) -> str:
if table_name == prefix:
return prefix
if table_name.startswith(f"{prefix}_"):
return table_name[len(prefix) + 1:].lower()
return table_name[len(prefix) + 1 :].lower()
return table_name.lower()

# Export node tables: nodes_<display_name>.parquet
Expand All @@ -355,7 +357,9 @@ def get_display_name(table_name: str, prefix: str) -> str:
# Export edge tables: indices_<edge_name>.parquet, indptr_<edge_name>.parquet
for edge_table in edge_tables:
edge_name = (
edge_table[6:].lower() if edge_table.startswith("edges_") else edge_table.lower()
edge_table[6:].lower()
if edge_table.startswith("edges_")
else edge_table.lower()
)
indices_table = f"{csr_table_name}_indices_{edge_name}"
indices_file = parquet_dir / f"indices_{edge_name}.parquet"
Expand Down Expand Up @@ -396,7 +400,7 @@ def create_csr_graph_to_duckdb(
source_db_path: str,
output_db_path: str,
limit_rels: int | None = None,
undirected: bool = False,
add_reverse_edges: bool = False,
csr_table_name: str = "csr_graph",
node_table: str | None = None,
edge_table: str | None = None,
Expand All @@ -411,7 +415,7 @@ def create_csr_graph_to_duckdb(
source_db_path: Path to source DuckDB with edges table
output_db_path: Path to output DuckDB for CSR data
limit_rels: Limit number of relationships for testing
undirected: Whether graph is undirected
add_reverse_edges: Whether to add reverse edges for symmetric adjacency
csr_table_name: Name of table to store CSR data
node_table: Specific node table to use (default: auto-discover)
edge_table: Specific edge table to use (default: auto-discover)
Expand Down Expand Up @@ -501,14 +505,18 @@ def create_csr_graph_to_duckdb(

for et in edge_tables:
edge_name = et[6:].lower() if et.startswith("edges_") else et.lower()
src_node_type, dst_node_type = edge_relationships.get(edge_name, (None, None))
src_node_type, dst_node_type = edge_relationships.get(
edge_name, (None, None)
)

src_table = node_type_to_table.get(src_node_type)
dst_table = node_type_to_table.get(dst_node_type)

if src_table and dst_table:
num_src_nodes = node_counts.get(src_table, 0)
print(f"\n Processing {et}: {src_node_type} ({num_src_nodes} nodes) -> {dst_node_type}")
print(
f"\n Processing {et}: {src_node_type} ({num_src_nodes} nodes) -> {dst_node_type}"
)
else:
src_table = dst_table = node_tables[0] if node_tables else "nodes"
num_src_nodes = node_counts.get(src_table, 0)
Expand All @@ -519,13 +527,13 @@ def create_csr_graph_to_duckdb(
src_csr_table = f"{csr_table_name}_{src_table}"
dst_csr_table = f"{csr_table_name}_{dst_table}"

# For undirected graphs, from and to node tables must be the same.
if undirected:
# Reverse-edge expansion requires one node mapping for both endpoints.
if add_reverse_edges:
if src_table != dst_table:
raise ValueError(
f"Undirected graphs require the same node table on both sides of an "
f"Adding reverse edges requires the same node table on both sides of an "
f"edge, but edge table '{et}' connects '{src_table}' -> '{dst_table}'. "
f"Use --undirected for homogeneous edge tables."
f"Use --add-reverse-edges only for homogeneous edge tables."
)
dst_pk = src_pk
dst_csr_table = src_csr_table
Expand All @@ -551,31 +559,31 @@ def create_csr_graph_to_duckdb(
select_cols = "m1.csr_index AS csr_source, m2.csr_index AS csr_target"
if edge_cols:
select_cols += ", " + ", ".join([f"e.{c}" for c in edge_cols])
reverse_select_cols = "m2.csr_index AS csr_source, m1.csr_index AS csr_target"
reverse_select_cols = (
"m2.csr_index AS csr_source, m1.csr_index AS csr_target"
)
if edge_cols:
reverse_select_cols += ", " + ", ".join([f"e.{c}" for c in edge_cols])
reverse_cols = "csr_target AS csr_source, csr_source AS csr_target"
if edge_cols:
reverse_cols += ", " + ", ".join(edge_cols)

# Self-loops are not filtered from directed graphs.
# For undirected graphs, the reverse UNION excludes self-loops so
# each self-loop appears exactly once (forward only).
# Self-loops are not filtered. Reverse-edge expansion excludes
# self-loops so each self-loop appears exactly once (forward only).
join_clause = f"""
FROM orig.{et} e
JOIN src_map m1 ON e.source = m1.original_node_id
JOIN dst_map m2 ON e.target = m2.original_node_id"""

if limit_rels:
limit_per_table = limit_rels // len(edge_tables)
if not undirected:
if not add_reverse_edges:
rel_query = f"""
WITH {map_cte}
SELECT {select_cols} {join_clause}
LIMIT {limit_per_table}
"""
else:
# Reverse self-loops using CSR indices (already mapped)
rel_query = f"""
WITH {map_cte},
limited AS (
Expand All @@ -588,7 +596,7 @@ def create_csr_graph_to_duckdb(
WHERE csr_source != csr_target
"""
else:
if not undirected:
if not add_reverse_edges:
rel_query = f"""
WITH {map_cte}
SELECT {select_cols} {join_clause}
Expand Down Expand Up @@ -741,9 +749,10 @@ def main():
help="Number of edges to use in test mode (default: 50000)",
)
parser.add_argument(
"--undirected",
"--add-reverse-edges",
dest="add_reverse_edges",
action="store_true",
help="Treat graph as undirected (default: directed)",
help="Add reverse edges for symmetric adjacency (default: preserve input direction)",
)
parser.add_argument(
"--storage",
Expand Down Expand Up @@ -784,7 +793,7 @@ def main():
print(f"GraphAr directory: {args.graphar}")
print(f"CSR output database: {args.output_db}")
print(f"CSR table prefix: {args.csr_table}")
print(f"Undirected: {args.undirected}")
print(f"Add reverse edges: {args.add_reverse_edges}")
print(f"DuckDB memory limit: {args.memory_limit}")

try:
Expand All @@ -799,7 +808,7 @@ def main():
graphar_dir=args.graphar,
output_db_path=args.output_db,
csr_table_name=args.csr_table,
undirected=args.undirected,
add_reverse_edges=args.add_reverse_edges,
memory_limit=args.memory_limit,
)

Expand All @@ -823,7 +832,7 @@ def main():
print(f"Source database: {source_db_path}")
print(f"CSR output database: {args.output_db}")
print(f"CSR table prefix: {args.csr_table}")
print(f"Undirected: {args.undirected}")
print(f"Add reverse edges: {args.add_reverse_edges}")
print(f"DuckDB memory limit: {args.memory_limit}")

# Compute default storage path from output_db if not specified
Expand All @@ -843,7 +852,7 @@ def main():
source_db_path=source_db_path,
output_db_path=args.output_db,
limit_rels=test_limit,
undirected=args.undirected,
add_reverse_edges=args.add_reverse_edges,
csr_table_name=args.csr_table,
node_table=args.node_table,
edge_table=args.edge_table,
Expand Down
Loading
Loading