ETL Pipeline for Multi-Source Bibliographic Data

[anibihakeem]

ETL Pipeline for Multi-Source Bibliographic Data

Sources: Scopus · Dimensions · PubMed · OpenAlex

---

1. The Problem

Every analytical function in functions/ and services/ assumes its input is already in the Web of Science (WoS) internal format: WoS column tags (AU, PY, SO...), WoS types, WoS conventions for multi-value fields. Scopus, Dimensions, PubMed, and OpenAlex each use different column names, file formats, delimiters, and representations of authors, affiliations, and references. Feed any of them to the dashboard unchanged and it either crashes or quietly returns wrong results.

This PR adds the translation layer: a source-agnostic ETL pipeline that converts any supported source into the exact schema the analytical functions expect. It's the Python counterpart to convert2df() in the R bibliometrix package.

---

2. What This PR Delivers

Four sources, five ways in, one pipeline:

Source	Retrieval	Format
Scopus	manual export (file)	CSV
Dimensions	manual export (file)	XLSX
PubMed	manual export (file)	MEDLINE TXT
PubMed	live API (E-utilities)	MEDLINE
OpenAlex	live API (REST)	nested JSON

All five paths run through the same transformation code. Adding another source later means writing one mapping dictionary, and nothing else.

New modules:

• www/services/standardizer.py: the transform layer (dispatch, mappings, type contracts, validation, entry point, export)
• www/services/api_retriever.py: live retrieval for OpenAlex and PubMed
• tests/test_pipeline.py: reproducible smoke tests
• sources/samples/: raw and standardized sample datasets

Plus fixes to seven existing functions that broke on standardized non-WoS data (§5).

---

3. How the Pipeline Works

A single call runs the whole thing:

from www.services.standardizer import run_pipeline
from www.services.api_retriever import fetch_openalex

df = run_pipeline(fetch_openalex("machine learning", 100), "openalex")
# validated, analysis-ready DataFrame in the WoS schema

Extract. Each file type gets the right loader: pd.read_csv, a Dimensions-aware load_dimensions_xlsx that skips the vendor disclaimer row, or the repo's existing parse_pubmed_data MEDLINE parser. API sources are fetched live (§4).

Transform. standardize(raw_df, source) runs four steps:

1. Dispatch. SOURCE_REGISTRY maps each source to its (mapping_dict, DB_label, delimiter). The registry is the only place routing happens, so there's no if source == … scattered through the transform logic.
2. Rename. A per-source mapping dictionary translates native columns into WoS tags (Scopus "Cited by" → TC, Dimensions "PubYear" → PY, PubMed "TA" → JI). I built these by reading real exports, not the docs.
3. Structural transforms. For the cases where renaming isn't enough: Dimensions packs pages into one Pagination field ("429-468"), which splits into BP/EP; PubMed's DP ("2026 Jul 3") becomes a 4-digit PY; PubMed's LID/AID clean down to the DI field.
4. Type contracts. Multi-value fields (AU, AF, C1, CR, DE, ID) become list[str]; PY and TC become int (nulls → 0); everything else becomes str (nulls → ""). Nothing survives as NaN or None, and that one guarantee heads off a whole category of downstream crashes.

SR derivation. The Short Reference comes from calling the repo's existing metaTagExtraction(df, "SR"). Reused, not rewritten.

Validate. validate() checks the output directly: complete schema, no nulls, list-columns that actually hold lists, int-columns that are actually integers. If something's off, it stops there instead of passing bad data downstream.

Load. export_standardized() writes dashboard-compatible XLSX/CSV. It joins list-columns with the codebase's own ; delimiter and writes empty cells as "", never NaN, because Excel reads NaN back as a float and that float crashes anything that iterates the cell.

---

4. Live API Retrieval

Beyond file imports, api_retriever.py adds live fetching from two separate APIs. The user gives a search string and picks a platform, and gets back records ready for the pipeline.

OpenAlex (fetch_openalex(query, max_records)):

• Cursor-based pagination, 100 per page to keep credit use down
• Exponential backoff on HTTP 429/5xx and on transport-level connection errors
• API key read from OPENALEX_API_KEY, never hardcoded or committed (OpenAlex made keys mandatory in Feb 2026)
• Flattens the nested JSON: authorships into author lists and per-author affiliations, biblio into volume/issue/pages, and the inverted-index abstract back into plain text

PubMed (fetch_pubmed(query, max_records)):

• NCBI E-utilities in two steps: esearch for PMIDs, efetch for the MEDLINE records
• The MEDLINE response goes through the repo's existing parse_pubmed_data(), so the file path and the API path share one parser and PubMed parsing lives in exactly one place

Both feed the same run_pipeline() as everything else, so no transformation logic is duplicated across the five paths.

---

5. Fixes to Existing Functions

Standardized data surfaced real bugs in the analytical layer. I fixed each one where it belonged: centrally in the pipeline when the fix helps every function, or in the specific function when the bug lived there.

1. Year arithmetic crashed. PY.max() - PY.min() on string years raised TypeError. Adding PY to the int contract fixed it once, centrally, instead of per function.
2. Country extraction misattributed or dropped countries. Two things. (a) Scopus's flat Affiliations field breaks the link between an author and their affiliation, so I remapped C1 to Authors with affiliations. (b) Scopus writes "Viet Nam" while countries.txt has "VIETNAM", so a COUNTRY_NORMALIZATION map canonicalizes affiliation strings before extraction.
3. References shattered into fragments. Scopus uses ; both inside a reference and between references, so I split on (?<=\));\s+ (a year boundary) instead of every semicolon.
4. WordCloud and Frequent Words crashed with invalid syntax. Both ran eval() on cell contents, which is unsafe and blows up on a plain ;-string. Replaced with tolerant deserialization: take a real list as-is, parse a list-repr with ast.literal_eval, otherwise split on ;.
5. Most Relevant Authors showed nothing, then crashed. A non-list AU was quietly swapped for [], then int(NaN) in the axis math crashed. Same tolerant deserialization, plus a NaN guard.
6. Country networks came back empty from file data (AU_CO/AU1_CO). The code walks C1 expecting a list, but a file-loaded string iterates character by character. Now it handles both.
7. Empty cells crashed SR and reference functions after an Excel round-trip. An empty cell comes back from Excel as NaN (a float), and for x in cell raises 'float' object is not iterable. The export layer now writes "", which removes the float path.

One thing ran through all of these: the codebase deserializes the same multi-value fields three different ways, split-on-;, eval(), and silent discard. These patches bring them together: lists in memory, ;-strings in files, both accepted everywhere. The remaining eval() calls should move to ast.literal_eval, since running file contents as code is a security hole.

---

6. Verification in the Actual Shiny Dashboard

I loaded standardized datasets from all four sources, Scopus (CSV), Dimensions (500 records), PubMed (MEDLINE), and OpenAlex (100, live), into the running dashboard and went through it panel by panel.

Works across all four sources (16 panels)

Main Information · Average Citations per Year · Three-Field Plot · Annual Scientific Production · Most Relevant Sources · Bradford's Law · Sources' Production over Time · Sources' Local Impact · Most Relevant Authors · Authors' Production over Time · Authors' Local Impact · Countries' Scientific Production · Countries' Production over Time · Most Frequent Words · Word Cloud · Collaboration Network.

That covers production, sources, authors, countries, words, and networks, so the schema holds up the same way across CSV, XLSX, MEDLINE, and JSON.

The image above shows "Countries Collaboration Network", generated from OpenAlex data (100 records, fetched live via the API). Collaboration arcs connect countries extracted from the standardized C1 field

The image above shows "Countries' Production over Time" generated from OpenAlex data. Cumulative publication counts by country (USA, France, Germany, Netherlands, China) plotted correctly across a 1988–2021 range.

Works where the source provides the data

Panel	Works on	Needs
Countries Collaboration Network	Scopus, Dimensions, OpenAlex	affiliations with country
Corresponding Author's Countries	Scopus, Dimensions, PubMed	RP field
Thematic Map	Scopus, PubMed	rich keywords
Most Global Cited Documents	Scopus, Dimensions	citation structure
Trend Topics	Dimensions	keyword time series
Citation Network	OpenAlex	referenced-work IDs
Most Local Cited References	Scopus only	parseable references
Lotka's Law	Scopus, Dimensions, PubMed	non-empty author distribution

Not verified

• Direct-citation analyses (Historiograph, Coupling, full Thematic Evolution, Local Cited Documents/Authors). These need references matchable to the collection's own records, which no source provides usably, and they're WoS-gated by design (§7).
• Lotka's Law on OpenAlex. Fails with 'float' object is not iterable, the same empty-cell issue as §5.7. Follow-up patch candidate.
• Most Cited Countries. Times out on the 500-record set (network performance, not a crash), and completes fine on smaller inputs.

python -m tests.test_pipeline reproduces the standardized output behind these panels.

---

7. Known Limitations

The direct-citation family doesn't run for non-WoS sources, for two separate reasons.

First, the port gates it on purpose: histNetwork checks the database and returns None with "Database not compatible with direct citation analysis" for non-WoS data, and the callers then crash on that unguarded None. Second, and more to the point, the data isn't there. These analyses need references that can be matched back to the collection's own SR keys, and no source provides that usably: Dimensions exports no references, OpenAlex gives opaque work-IDs instead of citation strings, and Scopus reference strings can't be decomposed reliably.

So this is a limit of the source data rather than the schema, and it's out of scope here. The natural next step is adding None-guards so these panels show a clear message instead of erroring out.

Minor: per-reference source extraction (CR_SO) on Scopus is unreliable because of the same comma-ambiguity, and the author-collaboration plot errors on degenerate graphs when a min-edge threshold empties a small sample, though the underlying matrix computes fine.

---

8. How to Run

pip install -r requirements.txt

# Live retrieval needs a free OpenAlex key: openalex.org/settings/api
export OPENALEX_API_KEY=your_key

# Pipeline smoke tests
python -m tests.test_pipeline

# Dashboard
shiny run app.py
# then load sources/samples/*_standardized.xlsx via "Load bibliometrix data"

# Live retrieval example
python -c "
from www.services.api_retriever import fetch_pubmed
from www.services.standardizer import run_pipeline
print(run_pipeline(fetch_pubmed('machine learning', 25), 'pubmed').head())
"