Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 47 additions & 33 deletions README.md

Large diffs are not rendered by default.

389 changes: 276 additions & 113 deletions docs/api_reference.md

Large diffs are not rendered by default.

124 changes: 103 additions & 21 deletions docs/graph_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,27 +12,41 @@ The graph module sits between neighbor search and visualization:
Data -> k-NN graph -> MST extraction -> Tree analysis / layout
```

TMAP keeps the public graph API small:
`tmap.graph` has three parts:

- `tree_from_knn_graph(knn)` computes a minimum spanning tree from a `KNNGraph`
- `Tree` exposes traversal, path, distance, and subtree helpers
- `Tree` exposes traversal, path, distance, hop-count, and neighborhood helpers
- analysis functions that combine the tree with per-point labels or values: `boundary_edges`, `confusion_matrix_from_tree`, `edge_delta`, `node_diversity`, `path_properties`, `subtree_purity`

The supported MST path is OGDF-based. There is no separate SciPy MST builder anymore.
The MST is computed by OGDF, the same library that draws the layout.

---

## Quick Start

If you fitted a `TMAP`, the tree is already there:

```python
from tmap.graph import tree_from_knn_graph
from tmap import TMAP

tree = tree_from_knn_graph(knn)
model = TMAP(metric="cosine").fit(X)
tree = model.tree_

print(tree.n_nodes)
print(len(tree.edges))
print(tree.root)
```

The estimator also has shortcuts for the most common queries: `model.path`, `model.distance`, `model.distances_from`, `model.hops` and `model.hops_from` do the same as the `Tree` methods below.

If you have a `KNNGraph` and no estimator, build the tree yourself:

```python
from tmap.graph import tree_from_knn_graph

tree = tree_from_knn_graph(knn)
```

Use this when:

- you already have a `KNNGraph`
Expand Down Expand Up @@ -60,25 +74,44 @@ This makes downstream layout and tree analysis practical.
### From LSHForest

```python
from tmap import LSHForest
from tmap import MinHash, LSHForest
from tmap.graph import tree_from_knn_graph

signatures = MinHash(num_perm=128, seed=42).batch_from_binary_array(fingerprints)

lsh = LSHForest(d=128, l=32)
lsh.batch_add(signatures)
lsh.index()

knn = lsh.get_knn_graph(k=20, kc=50)
tree = tree_from_knn_graph(knn)
```

### From USearch

`USearchIndex.query_knn` already returns a `KNNGraph`:

```python
from tmap.index import USearchIndex
from tmap.graph import tree_from_knn_graph

index = USearchIndex().build_from_vectors(X, metric="cosine")
tree = tree_from_knn_graph(index.query_knn(k=20))
```

### From another k-NN backend

For FAISS, Annoy, an alignment tool or anything else that gives you neighbor indices and distances, wrap the arrays in a `KNNGraph`:

```python
from tmap.graph import tree_from_knn_graph
from tmap.index.types import KNNGraph
from tmap.index import KNNGraph

knn = KNNGraph.from_arrays(indices, distances)
knn = KNNGraph.from_arrays(indices, distances) # both shape (n, k); use -1 for missing neighbors
tree = tree_from_knn_graph(knn)
```

That second path is the answer for USearch, Annoy, or any external neighbor search:
convert the arrays into `KNNGraph`, then extract the tree with `tree_from_knn_graph`.
To get a full map from those neighbors, pass the graph to the estimator instead: `TMAP().fit(knn_graph=knn)`.

---

Expand All @@ -91,7 +124,9 @@ At a high level:
3. Recover the returned MST edges as a `Tree`
4. Reattach edge weights from the original k-NN graph

For duplicated directed edges such as `i -> j` and `j -> i`, the smaller observed weight is kept.
For duplicated directed edges such as `i -> j` and `j -> i`, the smaller observed weight is kept. Self-matches, `-1` entries and non-finite distances are skipped.

The optional `config` argument takes a `LayoutConfig`. You rarely need it: by default the function runs OGDF with the cheapest settings, since only the tree topology is used.

---

Expand All @@ -101,12 +136,12 @@ The result is a `Tree` object with:

```python
tree.n_nodes
tree.edges
tree.weights
tree.edges # shape (n_edges, 2), int32
tree.weights # shape (n_edges,), float32
tree.root
```

`root` is chosen automatically from the highest-degree node in the extracted tree.
`root` is chosen automatically as the node with the most edges in the extracted tree.

---

Expand Down Expand Up @@ -134,6 +169,8 @@ for node, parent, depth in tree.dfs():
print(node, parent, depth)
```

Both start at `root` unless you pass `start=`.

### Children

```python
Expand All @@ -147,6 +184,8 @@ sizes = tree.subtree_sizes()
print(sizes[tree.root])
```

`sizes[i]` is the number of nodes below `i` (itself included) when the tree hangs from `root`.

---

## Paths and Distances
Expand All @@ -162,39 +201,82 @@ path = tree.path(10, 42)
### Tree distance

```python
distance = tree.distance(10, 42)
distance = tree.distance(10, 42) # sum of edge weights along the path
hops = tree.hops(10, 42) # number of edges along the path
```

### Distances from one source

```python
distances = tree.distances_from(10)
distances = tree.distances_from(10) # float32, one value per node
hops = tree.hops_from(10) # int32, one value per node
```

### Local subtree
### Nearby nodes

```python
nearby = tree.subtree(10, depth=2)
```

This returns every node within two edges of node 10, in any direction, closest first. With `depth=None` it returns the whole connected component.

These are useful for pseudotime, branch inspection, and neighborhood summaries.

---

## Analysis

The functions in `tmap.graph` take a tree plus one value or label per point:

```python
import numpy as np
from tmap.graph import (
boundary_edges,
confusion_matrix_from_tree,
edge_delta,
node_diversity,
path_properties,
subtree_purity,
)

labels = np.asarray(cell_types) # one label per point
values = np.asarray(molecular_weight) # one number per point

cross = boundary_edges(tree, labels) # edges joining two different labels
cmat, classes = confusion_matrix_from_tree(tree, labels)
jumps = edge_delta(tree, values) # |value change| across each edge
along = path_properties(tree, 10, 42, values) # values along the path from 10 to 42
spread = node_diversity(tree, method="mean") # mean weight of each node's edges
purity = subtree_purity(tree, labels, min_size=10) # NaN below min_size
```

A few ways to read them:

- `len(cross) / len(tree.edges)` is the share of edges that cross a label boundary. A low number means the labels form clean branches.
- `cmat[i, j]` counts the edges between points labeled `classes[i]` and `classes[j]`. Large off-diagonal counts show which groups touch each other on the tree.
- Nodes with high `node_diversity` have long edges, so they tend to sit at the bridges between clusters.
- `subtree_purity` is the share of each node's subtree that has its most common label, so it shows which branches belong to one group.

---

## Disconnected Graphs

If the k-NN graph has multiple components, the extracted tree will also have multiple components.
When `k` is too small, the k-NN graph can fall apart into several pieces, and a minimum spanning tree of it is a forest.

You can detect that with:
The estimator handles this by default. With `connect_components=True`, `TMAP.fit` finds the pieces, adds the cheapest edges that join them, and warns you. Afterward, `model.n_components_` tells you how many pieces there were and `model.n_bridges_` how many edges it added. Pass `connect_components=False` to keep the forest.

`tree_from_knn_graph` does not bridge anything. You can detect a forest with:

```python
n_components = tree.n_nodes - len(tree.edges)
```

On a forest, `path`, `distance` and `hops` raise `IndexError` for points in different pieces, `distances_from` gives `inf` and `hops_from` gives `-1` for unreachable points, and `bfs`, `dfs` and `subtree_sizes` only cover the piece that contains their start node.

Common reasons:

- `k` is too small
- `kc` is too small
- `kc` is too small (LSHForest only)
- the data genuinely has separated groups

Possible responses:
Expand Down Expand Up @@ -244,7 +326,7 @@ fingerprints = (np.random.rand(1000, 2048) < 0.1).astype(np.uint8)
mh = MinHash(num_perm=128, seed=42)
signatures = mh.batch_from_binary_array(fingerprints)

lsh = LSHForest(d=128, l=64)
lsh = LSHForest(d=128, l=32)
lsh.batch_add(signatures)
lsh.index()

Expand Down
29 changes: 22 additions & 7 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,29 +15,33 @@ Most users only need one pattern:
- [Visualization Guide](visualization_guide.md)
Learn when to use notebook widgets, HTML export, and `serve()`.
- [API Reference](api_reference.md)
Quick reference for `TMAP`, `TmapViz`, chemistry helpers, and layout functions.
Every public class and function: `TMAP`, `TmapViz`, the index and graph classes, layout functions, and the chemistry, protein and single-cell helpers.

## Choose Your API

### Estimator API

Use `TMAP(...).fit(X)` when you want the simplest path.
Use `TMAP(...).fit(X)` when you want the simplest path. After fitting, the same object answers tree queries (`path`, `distance`, `hops`, `distances_from`), places new data (`transform`, `add_points`), and saves itself (`save`, `TMAP.load`).

### Lower-level pipeline

Use `MinHash`, `LSHForest`, `layout_from_lsh_forest`, and related functions when you want direct control over hashing, indexing, or graph construction.
Use `MinHash`, `LSHForest`, `USearchIndex`, `layout_from_knn_graph` and related functions when you want direct control over hashing, indexing, or graph construction. [Notebook 12](../notebooks/12_lower_level_pipeline.ipynb) walks through it end to end.

## Quick Example

Run this from the repository root:

```python
import pandas as pd
from tmap import TMAP
from tmap.utils import fingerprints_from_smiles

df = pd.read_csv("../examples/cluster_65053.csv", nrows=3000)
df = pd.read_csv("examples/cluster_65053.csv", nrows=3000)
smiles = df["smiles"].tolist()

fps = fingerprints_from_smiles(smiles, fp_type="morgan", radius=2, n_bits=2048)
fps, valid = fingerprints_from_smiles(smiles, fp_type="morgan", radius=2, n_bits=2048, return_valid=True)
smiles = [smi for smi, ok in zip(smiles, valid) if ok]

model = TMAP(metric="jaccard", n_neighbors=20, seed=42).fit(fps)

viz = model.to_tmapviz()
Expand All @@ -46,28 +50,39 @@ viz.add_smiles(smiles)
viz.write_html("cluster_65053.html")
```

`return_valid=True` drops SMILES that RDKit can't parse, so the SMILES list stays aligned with the fingerprints. In a script, put the code under `if __name__ == "__main__":`, because `fingerprints_from_smiles` starts worker processes.

## Supported Input Paths

| Input | Metric | Good for |
|-------|--------|----------|
| Binary matrix | `jaccard` | Molecular fingerprints and other 0/1 features |
| Lists of feature indices, lists of strings, SciPy sparse matrix | `jaccard` | Sets of tokens, k-mers, very sparse features |
| Dense float matrix | `cosine` | Embeddings where direction matters |
| Dense float matrix | `euclidean` | Embeddings where magnitude matters |
| Distance matrix | `precomputed` | Distances you already computed elsewhere |
| `KNNGraph` | `fit(knn_graph=...)` | Neighbors from FAISS, Annoy, alignment tools and so on |

The first two rows use different search backends (USearch and MinHash + LSHForest). The [API reference](api_reference.md#input-routing) has the details.

## Choose The Right Output

| Goal | Method |
|------|--------|
| Quick notebook exploration | `viz.to_widget(...)` or `model.plot(...)` |
| Quick notebook exploration | `viz.show()` or `model.plot(...)` |
| Share one file | `viz.write_html(...)` or `model.to_html(...)` |
| Host a large map on a web server | `viz.write_static(...)` |
| Browse a large map locally | `viz.serve(...)` or `model.serve(...)` |
| Figure for a paper | `model.plot_static(...)` |

## Advanced Guides

- [MinHash Guide](minhash_guide.md)
- [LSHForest Guide](lshforest_guide.md)
- [USearch Backend](usearch_backend.md)
- [Graph Guide](graph_guide.md)
- [Layout Guide](layout_guide.md)

The files in `docs/` that are not linked here are planning notes or historical material and may not track the current public API line by line.
## Notebooks

The [`notebooks/`](../notebooks) folder has worked examples for cheminformatics, embeddings, single-cell data, proteins and the notebook widgets. The [README](../README.md#notebooks) lists them all.
Loading
Loading