Repository navigation
Fix tutorial bugs (MQN TMAP, Windows hang, iterations) and data handling issues - #11
Merged
Merged
Conversation
- create_tmap: use euclidean metric for MQN (jaccard rejected count vectors with "Binary matrix must contain only 0/1 values"). Also fixes chelombus-tmap --fingerprint mqn. - FingerprintFromSmiles: nprocesses=1 now computes in-process instead of starting a one-worker Pool. - PQKMeans: rename `iteration` to `iterations` to match PQEncoder. `iteration=` still works with a DeprecationWarning, and models saved with the old attribute are migrated on load. - Add tests for all three. Reported by Dan Kovari while following the tutorial.
On Windows and macOS (spawn start method), a script calling FingerprintFromSmiles without an `if __name__ == "__main__":` guard made multiprocessing.Pool respawn crashing workers forever. Use ProcessPoolExecutor, which raises BrokenProcessPool, and re-raise it as a RuntimeError that explains the missing guard. - Default nprocesses is now None (all cores, capped at 61 on Windows, where ProcessPoolExecutor rejects more). - Shut the executor down without waiting so idle workers exiting does not add ~1s, keeping speed on par with the old Pool. - Add a subprocess test that runs an unguarded script under spawn.
DuckDB 2.0 relations no longer have fetchone(), which broke every cluster_io function that counts rows (and CI, which installs the latest DuckDB). duckdb.execute() returns a cursor with fetchone/fetchall/df on both 1.x and 2.0.
create_tmap and representative_tmap crashed on any unparseable SMILES because tmap2 skips it when fingerprinting while the property layers, SMILES and cluster ids still had one entry per input. Both now share one helper that filters with tmap2's return_valid flags. chelombus-tmap --cluster-file no longer also requires --smiles.
- sample_from_cluster(random_state=) now returns the same sample every time (setseed ran in a separate query and had no effect). Orders by a seeded row hash, independent of DuckDB threading. - sample_from_cluster and export_all_clusters take cluster/smiles column names; export_cluster takes smiles_column for .smi output. - Quote column lists and escape paths containing a quote. - Remove references to a nonexistent chelombus[io] extra and a nonexistent 'chelombus visualize' command.
- Decompress .gz and .bz2 on the fly (including .sdf.gz). - Read text as UTF-8 regardless of platform default. - Accept glob patterns, as the pipeline --input help already promised. - New sep argument for comma separated input (CSV was silently dropped). - Read directories in sorted order and skip hidden files. - Raise on an invalid input type instead of logging and continuing.
import chelombus no longer imports torch, numba and pqkmeans up front (0.23s instead of ~1.3s). This matters most for fingerprint worker processes on spawn platforms, which re-import the package. A torch install that fails with something other than ImportError (e.g. a missing DLL) now falls back to CPU instead of breaking the import. FingerprintFromSmiles warns when no SMILES in a batch parse, which usually means the wrong column or separator.
- 01_train_models wrote chunks without the smiles column whenever any SMILES failed, breaking DuckDB globs over mixed schemas. Align with return_valid_idx like 02_assign_clusters. - run_all.sh derives REPO from its own location and treats an empty ID_COL as no id column. - README: move an inline comment that broke a line continuation.
- README Quick Start used a DataStreamer signature that does not exist. - Note that Windows is not supported (use WSL2) and that scripts need a __main__ guard on macOS/Windows. - Triton ships with torch on Linux only. - tutorial.ipynb: drop a cell loading a local model, use iterations=, point cluster stats at the tutorial's own results, note alignment. - enamine_1B_clustering.ipynb: iterations=.
- DataStreamer: read sep-delimited input with csv.reader, so quoted
fields ("CCO", "a,b") parse correctly.
- DataStreamer: an existing path is taken literally even if it contains
glob characters (batch[1]/), glob expansion only when it does not exist.
- sample_from_cluster: seed the hash on each row's file name and row
number, so duplicate rows are sampled independently. Still
reproducible across DuckDB threading and if the folder is moved.
- FingerprintFromSmiles: the empty-result warning used `if smiles`,
which raised for numpy arrays and pandas Series.
afloresep
force-pushed
the
fix/tutorial-bugs
branch
from
October 29, 2026 12:06
b215146 to
bffb490
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the bugs Dan Kovari hit while following the tutorial on Windows, plus a sweep of other failure points found along the same path.
Tutorial bugs
create_tmapwith MQN used the jaccard metric on count vectors and failed withBinary matrix must contain only 0/1 values. Now uses euclidean for MQN, likerepresentative_tmap. Also fixeschelombus-tmap --fingerprint mqn.if __name__ == "__main__":guard mademultiprocessing.Poolrespawn crashing workers forever. Switched toProcessPoolExecutor, which fails fast, and re-raise with a message explaining the missing guard. Speed is on par with the oldPool.nprocesses=1now computes in-process.PQKMeans(iteration=)renamed toiterations=to matchPQEncoder. The old keyword still works with aDeprecationWarning, and models saved with the old attribute are migrated on load.Other fixes
cluster_iousedduckdb.query(...).fetchone(), removed in DuckDB 2.0, so every cluster_io function that counts rows failed (and CI). Queries now useduckdb.execute(), which works on 1.x and 2.0.chelombus-tmap --cluster-fileno longer also requires--smiles.cluster_io:sample_from_cluster(random_state=)was not reproducible (setseedran in a separate query). Now orders by a seeded hash of each row's file name and row number: reproducible, and duplicate rows are sampled independently.sample_from_cluster,export_all_clustersandexport_cluster.'are escaped.DataStreamer: reads.gz/.bz2(including.sdf.gz), UTF-8 regardless of platform, glob patterns (as the pipeline help already claimed), and comma separated input viasep=(withcsvquoting). Directories are read in sorted order, hidden files skipped. An existing path with glob characters (e.g.batch[1]/) is taken literally.PQEncoder/PQKMeansload lazily, soimport chelombusno longer pulls torch, numba and pqkmeans (0.23s instead of ~1.3s; matters for spawned fingerprint workers). A broken torch install falls back to CPU instead of breaking the import.FingerprintFromSmileswarns when no SMILES in a batch parse (usually wrong column or separator).01_train_models.pydropped thesmilescolumn from any chunk with an unparseable SMILES; now aligned withreturn_valid_idx.run_all.shderivesREPOfrom its location and treats an emptyID_COLas no id column. Fixed a README example broken by a comment after a line continuation.DataStreamersignature that does not exist. Added a Windows note (not supported, use WSL2) and the__main__guard note for macOS/Windows scripts. Tutorial notebook no longer loads a model from a local path, usesiterations=, and points cluster stats at its own results.Tests
pip install ".[dev]"(DuckDB 2.0, as in CI) and on DuckDB 1.5 with CUDA.