Skip to content

Fix tutorial bugs (MQN TMAP, Windows hang, iterations) and data handling issues - #11

Merged
afloresep merged 10 commits into
masterfrom
fix/tutorial-bugs
Oct 29, 2026
Merged

afloresep merged 10 commits into
masterfrom
fix/tutorial-bugs

Conversation

@afloresep

@afloresep afloresep commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Fixes the bugs Dan Kovari hit while following the tutorial on Windows, plus a sweep of other failure points found along the same path.

Tutorial bugs

  • create_tmap with MQN used the jaccard metric on count vectors and failed with Binary matrix must contain only 0/1 values. Now uses euclidean for MQN, like representative_tmap. Also fixes chelombus-tmap --fingerprint mqn.
  • Fingerprint hang on Windows/macOS: under the spawn start method, a script without an if __name__ == "__main__": guard made multiprocessing.Pool respawn crashing workers forever. Switched to ProcessPoolExecutor, which fails fast, and re-raise with a message explaining the missing guard. Speed is on par with the old Pool. nprocesses=1 now computes in-process.
  • PQKMeans(iteration=) renamed to iterations= to match PQEncoder. The old keyword still works with a DeprecationWarning, and models saved with the old attribute are migrated on load.

Other fixes

  • DuckDB 2.0: cluster_io used duckdb.query(...).fetchone(), removed in DuckDB 2.0, so every cluster_io function that counts rows failed (and CI). Queries now use duckdb.execute(), which works on 1.x and 2.0.
  • TMAP builders crashed on any unparseable SMILES (layers had one entry per input, points only for valid ones). Both builders share one helper that filters with tmap2's valid flags. chelombus-tmap --cluster-file no longer also requires --smiles.
  • cluster_io:
    • sample_from_cluster(random_state=) was not reproducible (setseed ran in a separate query). Now orders by a seeded hash of each row's file name and row number: reproducible, and duplicate rows are sampled independently.
    • Configurable cluster/SMILES column names in sample_from_cluster, export_all_clusters and export_cluster.
    • Column lists are quoted and paths containing ' are escaped.
  • DataStreamer: reads .gz/.bz2 (including .sdf.gz), UTF-8 regardless of platform, glob patterns (as the pipeline help already claimed), and comma separated input via sep= (with csv quoting). Directories are read in sorted order, hidden files skipped. An existing path with glob characters (e.g. batch[1]/) is taken literally.
  • Import time: PQEncoder/PQKMeans load lazily, so import chelombus no longer pulls torch, numba and pqkmeans (0.23s instead of ~1.3s; matters for spawned fingerprint workers). A broken torch install falls back to CPU instead of breaking the import.
  • FingerprintFromSmiles warns when no SMILES in a batch parse (usually wrong column or separator).
  • Pipeline: 01_train_models.py dropped the smiles column from any chunk with an unparseable SMILES; now aligned with return_valid_idx. run_all.sh derives REPO from its location and treats an empty ID_COL as no id column. Fixed a README example broken by a comment after a line continuation.
  • Docs and notebooks: README Quick Start used a DataStreamer signature that does not exist. Added a Windows note (not supported, use WSL2) and the __main__ guard note for macOS/Windows scripts. Tutorial notebook no longer loads a model from a local path, uses iterations=, and points cluster stats at its own results.

Tests

  • New tests for each fix above; each was checked to fail on the old code.
  • Full suite passes on a fresh pip install ".[dev]" (DuckDB 2.0, as in CI) and on DuckDB 1.5 with CUDA.

- create_tmap: use euclidean metric for MQN (jaccard rejected count
  vectors with "Binary matrix must contain only 0/1 values"). Also
  fixes chelombus-tmap --fingerprint mqn.
- FingerprintFromSmiles: nprocesses=1 now computes in-process instead
  of starting a one-worker Pool.
- PQKMeans: rename `iteration` to `iterations` to match PQEncoder.
  `iteration=` still works with a DeprecationWarning, and models saved
  with the old attribute are migrated on load.
- Add tests for all three.

Reported by Dan Kovari while following the tutorial.
On Windows and macOS (spawn start method), a script calling
FingerprintFromSmiles without an `if __name__ == "__main__":` guard made
multiprocessing.Pool respawn crashing workers forever. Use
ProcessPoolExecutor, which raises BrokenProcessPool, and re-raise it as a
RuntimeError that explains the missing guard.

- Default nprocesses is now None (all cores, capped at 61 on Windows,
  where ProcessPoolExecutor rejects more).
- Shut the executor down without waiting so idle workers exiting does
  not add ~1s, keeping speed on par with the old Pool.
- Add a subprocess test that runs an unguarded script under spawn.
DuckDB 2.0 relations no longer have fetchone(), which broke every
cluster_io function that counts rows (and CI, which installs the latest
DuckDB). duckdb.execute() returns a cursor with fetchone/fetchall/df on
both 1.x and 2.0.
create_tmap and representative_tmap crashed on any unparseable SMILES
because tmap2 skips it when fingerprinting while the property layers,
SMILES and cluster ids still had one entry per input. Both now share one
helper that filters with tmap2's return_valid flags.

chelombus-tmap --cluster-file no longer also requires --smiles.
- sample_from_cluster(random_state=) now returns the same sample every
  time (setseed ran in a separate query and had no effect). Orders by a
  seeded row hash, independent of DuckDB threading.
- sample_from_cluster and export_all_clusters take cluster/smiles column
  names; export_cluster takes smiles_column for .smi output.
- Quote column lists and escape paths containing a quote.
- Remove references to a nonexistent chelombus[io] extra and a
  nonexistent 'chelombus visualize' command.
- Decompress .gz and .bz2 on the fly (including .sdf.gz).
- Read text as UTF-8 regardless of platform default.
- Accept glob patterns, as the pipeline --input help already promised.
- New sep argument for comma separated input (CSV was silently dropped).
- Read directories in sorted order and skip hidden files.
- Raise on an invalid input type instead of logging and continuing.
import chelombus no longer imports torch, numba and pqkmeans up front
(0.23s instead of ~1.3s). This matters most for fingerprint worker
processes on spawn platforms, which re-import the package.

A torch install that fails with something other than ImportError (e.g.
a missing DLL) now falls back to CPU instead of breaking the import.

FingerprintFromSmiles warns when no SMILES in a batch parse, which
usually means the wrong column or separator.
- 01_train_models wrote chunks without the smiles column whenever any
  SMILES failed, breaking DuckDB globs over mixed schemas. Align with
  return_valid_idx like 02_assign_clusters.
- run_all.sh derives REPO from its own location and treats an empty
  ID_COL as no id column.
- README: move an inline comment that broke a line continuation.
- README Quick Start used a DataStreamer signature that does not exist.
- Note that Windows is not supported (use WSL2) and that scripts need a
  __main__ guard on macOS/Windows.
- Triton ships with torch on Linux only.
- tutorial.ipynb: drop a cell loading a local model, use iterations=,
  point cluster stats at the tutorial's own results, note alignment.
- enamine_1B_clustering.ipynb: iterations=.
@afloresep afloresep changed the title Fix MQN create_tmap, Windows fingerprint hang, rename PQKMeans iterations Fix tutorial bugs (MQN TMAP, Windows hang, iterations) and data handling issues Oct 29, 2026
- DataStreamer: read sep-delimited input with csv.reader, so quoted
  fields ("CCO", "a,b") parse correctly.
- DataStreamer: an existing path is taken literally even if it contains
  glob characters (batch[1]/), glob expansion only when it does not exist.
- sample_from_cluster: seed the hash on each row's file name and row
  number, so duplicate rows are sampled independently. Still
  reproducible across DuckDB threading and if the folder is moved.
- FingerprintFromSmiles: the empty-result warning used `if smiles`,
  which raised for numpy arrays and pandas Series.
@afloresep
afloresep merged commit 10d0e8c into master Oct 29, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant