Industry Chain Matcher

A matrix-based system for automatically matching enterprise information (business scope, patents, etc.) to nodes in a predefined industry chain graph.

算法文档（中文） | 中文 README | Algorithm Visual Guide

---

Overview

Given an industry chain graph with m nodes and a corpus of n enterprise documents, this system efficiently maps each document to the most relevant chain node(s) using dense vector similarity and a set of configurable matching rules.

Key features:

• Batch matrix operations (no per-record loops) — scales to millions of documents
• Dual chain logic: industry-specific chains require an L0 (root) similarity gate; general-purpose chains do not
• Exclusion logic: "catch-all" nodes only match when no sibling-specific node matches
• LLM-assisted threshold calibration pipeline
• Optional GPU acceleration via CuPy

---

Algorithm at a Glance

Each document embedding d is compared against three sets of chain embeddings:

Sim_L0          = L0_Embed          @ D.T   →  (1,       n)
Sim_Spec        = Spec_Embeds       @ D.T   →  (m_spec,  n)
Sim_Other_Parent= OtherParent_Embeds@ D.T   →  (m_other, n)

Final match masks are computed as:

Chain type	Condition
Specific (Type A)	sim > threshold AND sim_L0 > T_L0
Specific (Type B)	sim > threshold (no L0 gate)
"Other" catch-all	parent sim > threshold AND type gate AND no sibling matched

See the inline code comments and README for algorithm details.

---

Quick Start

1. Install dependencies

pip install -r requirements.txt

2. Generate sample data

python sample_data/generate_sample_data.py

This creates a fictional 13-node industry chain with 200 mock source documents and matching threshold files — everything needed to run the full pipeline.

3. Run the matching pipeline

# Prepare configuration matrices (run once per chain definition)
python chain_matching_sop/prepare_chain_config.py

# Step 1: Compute raw similarity scores for all source batches
python chain_matching_sop/run_similarity.py

# Step 2: Apply thresholds and matching logic
python chain_matching_sop/apply_matching.py

# Step 3: Export final results as Parquet
python chain_matching_sop/export_results.py

4. Inspect results

import pandas as pd
df = pd.read_parquet("chain_matching_sop/results/final_matching_results.parquet")
print(df.columns.tolist())
# ['chain_id', 'chain_name', 'info_id', 'similarity', 'source_text']
print(df.head())

---

Directory Structure

industry-chain-matcher/
│
├── sample_data/                    # Sample data (safe to commit)
│   ├── generate_sample_data.py     # One-shot generator — run this first
│   ├── chain_embeddings.npz        # Chain node embeddings (generated)
│   ├── l0_embedding.npz            # L0 root embedding (generated)
│   ├── chain_type_classification.csv
│   ├── chain_definitions.json      # Text definitions for LLM calibration
│   ├── source_embeddings/          # Source document embeddings (generated)
│   └── source_texts/               # Source document texts (generated)
│
├── chain_matching_sop/             # Core matching algorithm
│   ├── config/
│   │   ├── data_config.py          # All file paths and format specs
│   │   └── sop_config.py           # Algorithm parameters
│   ├── data_preparation/           # prepare_chain_config.py modules
│   ├── similarity/                 # run_similarity.py modules
│   ├── matching/                   # apply_matching.py modules
│   ├── result_export/              # export_results.py modules
│   ├── utils/                      # I/O and matrix utilities
│   ├── prompt/                     # LLM prompt templates
│   ├── input_data/                 # Threshold configuration (see below)
│   │   ├── threshold_l0.json
│   │   ├── threshold_spec.csv
│   │   └── threshold_other.csv
│   ├── prepare_chain_config.py     # Step 0: build config matrices
│   ├── run_similarity.py           # Step 1: compute similarity scores
│   ├── sample_for_calibration.py   # Step 1.5a: sample for LLM review
│   ├── calibrate_thresholds_llm.py # Step 1.5b: LLM threshold calibration
│   ├── compute_compliance_stats.py # Step 1.5c: aggregate compliance rates
│   ├── export_thresholds.py        # Step 1.5d: generate threshold files
│   ├── apply_matching.py           # Step 2: apply matching logic
│   └── export_results.py           # Step 3: export results
│
├── embedding_code/                 # Upstream: chain definition & embedding pipeline
│   ├── llm_define_chains.py              # Step A1: LLM-generate text definitions
│   ├── parse_chain_definitions.py        # Step A2: Format and structure definitions
│   ├── embed_chain_definitions.py        # Step A3: Embed chain definitions
│   ├── preprocess_source_texts.py        # Step B1: Preprocess source text data
│   ├── embed_source_texts.py             # Step B2: Embed source text documents
│   ├── embed_utils.py                    # Shared embedding utilities
│   └── config.yaml                       # Configuration for embedding scripts
│
├── .env.example                    # Environment variable template
├── requirements.txt
├── README.md
└── README_zh.md                    # Chinese README

---

Using Your Own Data

To run on a real industry chain, you need to:

Hardware requirement: The embedding pipeline (embedding_code/) requires a CUDA-capable GPU. It uses flash_attention_2 + float16 and will fail on CPU-only machines. The core matching pipeline (chain_matching_sop/) has no GPU requirement.

Step A — Prepare chain embeddings (upstream pipeline)

1. Place your industry chain Excel file at sample_data/sample_industry_chain.xlsx

2. Configure embedding_code/config.yaml

3. Set your embedding model path:

   export EMBEDDING_MODEL_PATH=/path/to/your/embedding-model

4. Run in order:

   python embedding_code/llm_define_chains.py        # generate text definitions
   python embedding_code/parse_chain_definitions.py  # format definitions
   python embedding_code/embed_chain_definitions.py  # embed definitions

5. Update chain_matching_sop/config/data_config.py to point CHAIN_EMBEDDINGS_PATH, L0_EMBEDDINGS_PATH, and CHAIN_TYPE_CLASSIFICATION_PATH at your generated files.

Step B — Prepare source document embeddings

python embedding_code/preprocess_source_texts.py   # clean and split source texts

# Embed source texts (default column names)
python embedding_code/embed_source_texts.py --input_file source_texts_split/batch_0.parquet

# Embed a different text column
python embedding_code/embed_source_texts.py \
  --input_file patents/batch_0.parquet \
  --text_column custom_text_field \
  --id_column patent_id \
  --output_dir patent_embeddings

File naming convention: embed_source_texts.py derives the output filename from the input filename (e.g. source_part_0.parquet → source_part_0_embeddings.npz). The downstream data_config.py locates these files via SOURCE_EMBEDDINGS_PATTERN = "source_part_*.npz", so input files must be named with the source_part_ prefix. preprocess_source_texts.py uses this prefix by default. If you supply custom input files, follow the same naming convention or update SOURCE_EMBEDDINGS_PATTERN in data_config.py accordingly.

Update SOURCE_EMBEDDINGS_DIR in data_config.py to point to your embedding output directory.

Step C — Calibrate thresholds (optional but recommended)

After running Step 1, use the LLM pipeline to determine per-chain similarity thresholds:

python chain_matching_sop/sample_for_calibration.py          # sample by similarity bin
python chain_matching_sop/calibrate_thresholds_llm.py           # LLM judgment
python chain_matching_sop/compute_compliance_stats.py   # compute compliance rates
python chain_matching_sop/export_thresholds.py  # generate threshold files

Or manually edit chain_matching_sop/input_data/threshold_spec.csv, threshold_other.csv, and threshold_l0.json.

Data format reference

File	Required keys	Shape
chain_embeddings.npz	chain_names (str array), embeddings (float32)	(m, d)
l0_embedding.npz	chain_names (1-element array), embeddings (float32)	(1, d)
chain_type_classification.csv	chain_name, type (A or B)	—
source_part_*.npz	ids (int array), embeddings (float32)	(n_batch, d)
source_part_*.parquet	id (int), source_text (str)	—

---

Configuration

Copy .env.example to .env and fill in your values:

cp .env.example .env

Variable	Description
LLM_API_URL	OpenAI-compatible LLM endpoint (used for threshold calibration)
LLM_API_KEY	API authentication key
LLM_MODEL_NAME	Model name to use for LLM judgments
EMBEDDING_MODEL_PATH	Local path to the embedding model