Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 22 additions & 10 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -128,50 +128,62 @@ jobs:
fi
echo "tamper correctly detected"

- name: DB integration - multi-fund round-trip in 4 languages
- name: DB integration - multi-fund import+export in 4 languages
run: |
set -e
python3 -m pip install --quiet lxml
tools/fetch-tools.sh # sqlite-jdbc -> .lib/
FX=FundsXML_Files/4.2.9/positions/Multi-Fund_Positions.xml
MX=FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml
DOC=FUNDSXML_MULTI_1
EQ="python3 Database_Integration/tools/xml_equiv.py"
XSD=".schema-cache/4.2.9/FundsXML.xsd"

# The multi-fund fixture must itself be schema-valid.
xmllint --noout --nonet --schema "$XSD" "$FX"

# Each language: separate import then export, then prove
# exported file == input file (and is schema-valid).

# Python
python3 Database_Integration/python/fundsxml_db.py roundtrip "$FX" py.xml
python3 Database_Integration/python/import_fundsxml.py py.db "$FX"
python3 Database_Integration/python/export_fundsxml.py py.db "$DOC" py.xml
xmllint --noout --nonet --schema "$XSD" py.xml
$EQ "$FX" py.xml

# Java (sqlite-jdbc)
CP=.lib/sqlite-jdbc-3.46.1.3.jar
javac -cp "$CP" -d /tmp/jdb Database_Integration/java/FundsXmlDb.java
javac -cp "$CP" -d /tmp/jdb \
Database_Integration/java/ImportFundsXml.java \
Database_Integration/java/ExportFundsXml.java
java --enable-native-access=ALL-UNNAMED -cp "$CP:/tmp/jdb" \
ImportFundsXml java.db "$FX"
java --enable-native-access=ALL-UNNAMED -cp "$CP:/tmp/jdb" \
FundsXmlDb roundtrip "$FX" java.xml
ExportFundsXml java.db "$DOC" java.xml
xmllint --noout --nonet --schema "$XSD" java.xml
$EQ "$FX" java.xml

# JavaScript (sql.js)
( cd Database_Integration/javascript && npm install --no-audit --no-fund )
node Database_Integration/javascript/fundsxml_db.mjs roundtrip "$FX" js.xml
node Database_Integration/javascript/import_fundsxml.mjs js.db "$FX"
node Database_Integration/javascript/export_fundsxml.mjs js.db "$DOC" js.xml
xmllint --noout --nonet --schema "$XSD" js.xml
$EQ "$FX" js.xml

# C# (.NET)
dotnet run --project Database_Integration/csharp -- roundtrip "$FX" cs.xml
# C# (.NET): separate import and export projects
dotnet run --project Database_Integration/csharp/import -- cs.db "$FX"
dotnet run --project Database_Integration/csharp/export -- cs.db "$DOC" cs.xml
xmllint --noout --nonet --schema "$XSD" cs.xml
$EQ "$FX" cs.xml

# Cross-language: every export identical to Python's.
$EQ py.xml java.xml && $EQ py.xml js.xml && $EQ py.xml cs.xml

# Single-fund sample still round-trips (lossy -> XSD-valid only).
python3 Database_Integration/python/fundsxml_db.py roundtrip "$MX" mx.xml
# Single-fund sample still imports+exports (lossy -> XSD-valid only).
python3 Database_Integration/python/import_fundsxml.py mx.db "$MX"
python3 Database_Integration/python/export_fundsxml.py mx.db FUNDSXML_FILE_1 mx.xml
xmllint --noout --nonet --schema "$XSD" mx.xml
echo "multi-fund round-trip equivalent in python/java/javascript/csharp"
echo "multi-fund import+export equivalent in python/java/javascript/csharp"

- name: Large-file - streaming aggregate / split / delta (constant memory)
run: |
Expand Down
53 changes: 32 additions & 21 deletions Database_Integration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,28 @@

![python](https://img.shields.io/badge/Python-verified-brightgreen) ![java](https://img.shields.io/badge/Java-verified-brightgreen) ![node](https://img.shields.io/badge/JavaScript-verified-brightgreen) ![csharp](https://img.shields.io/badge/C%23-verified-brightgreen) ![multi-fund](https://img.shields.io/badge/multi--node-yes-blue)

Standalone, copy-me examples that move data **both directions** between FundsXML
and a relational database, in the language you already use:
Standalone, copy-me examples in the language you already use. **Import and
export are separate programs** — one shows FundsXML → DB, the other DB →
FundsXML — so you can lift exactly the direction you need:

| Language | File | DB driver | Verified |
|----------|------|-----------|----------|
| Python | [`python/fundsxml_db.py`](python/fundsxml_db.py) | stdlib `sqlite3` | ✅ locally |
| Java | [`java/FundsXmlDb.java`](java/FundsXmlDb.java) | `sqlite-jdbc` (native `javax.xml`, no JAXB) | ✅ locally |
| JavaScript | [`javascript/fundsxml_db.mjs`](javascript/fundsxml_db.mjs) | `sql.js` (pure-WASM) | ✅ locally |
| C# / .NET | [`csharp/FundsXmlDb.cs`](csharp/FundsXmlDb.cs) | `Microsoft.Data.Sqlite` | ✅ locally + CI |
| Language | Import (FundsXML → DB) | Export (DB → FundsXML) | DB driver | Verified |
|----------|------------------------|------------------------|-----------|----------|
| Python | [`python/import_fundsxml.py`](python/import_fundsxml.py) | [`python/export_fundsxml.py`](python/export_fundsxml.py) | stdlib `sqlite3` | ✅ locally |
| Java | [`java/ImportFundsXml.java`](java/ImportFundsXml.java) | [`java/ExportFundsXml.java`](java/ExportFundsXml.java) | `sqlite-jdbc` (native `javax.xml`, no JAXB) | ✅ locally |
| JavaScript | [`javascript/import_fundsxml.mjs`](javascript/import_fundsxml.mjs) | [`javascript/export_fundsxml.mjs`](javascript/export_fundsxml.mjs) | `sql.js` (pure-WASM) | ✅ locally |
| C# / .NET | [`csharp/import/`](csharp/import/) | [`csharp/export/`](csharp/export/) | `Microsoft.Data.Sqlite` | ✅ locally + CI |

Every program is **self-contained**, heavily commented as a teaching artifact
(read one top-to-bottom and reimplement the pattern), and exposes the **same
CLI**:
Every program is **self-contained** (one file / one project, its own copy of
the small helper maps — nothing shared to import) and heavily commented as a
teaching artifact. The CLI is uniform across languages:

```
<prog> init <db> create the schema
<prog> import <db> <fundsxml.xml> FundsXML -> rows
<prog> export <db> <docId> <out> rows -> FundsXML
<prog> roundtrip <fundsxml.xml> <out> import then export through the DB
<import-prog> <db> <fundsxml.xml> FundsXML -> rows (creates the schema)
<export-prog> <db> <docId> <out.xml> rows -> FundsXML
```

The round-trip is then simply: run import, then run export, then compare.

## Relational model — multi-node

[`ddl/schema.sql`](ddl/schema.sql): `document → fund → portfolio → position`,
Expand Down Expand Up @@ -54,27 +55,37 @@ XSD-validity is asserted.

## Run

Each example is run as **import, then export** (the round-trip = both, then
compare). `DOC` is the document id the import prints.

```bash
tools/fetch-schema.sh 4.2.9 # XSD for validation
FX=FundsXML_Files/4.2.9/positions/Multi-Fund_Positions.xml
DOC=FUNDSXML_MULTI_1

# Python
python3 Database_Integration/python/fundsxml_db.py roundtrip "$FX" out.xml
python3 Database_Integration/python/import_fundsxml.py fx.db "$FX"
python3 Database_Integration/python/export_fundsxml.py fx.db "$DOC" out.xml

# Java (sqlite-jdbc fetched into .lib/ by tools/fetch-tools.sh)
tools/fetch-tools.sh
CP=.lib/sqlite-jdbc-3.46.1.3.jar
javac -cp "$CP" -d /tmp/db Database_Integration/java/FundsXmlDb.java
java --enable-native-access=ALL-UNNAMED -cp "$CP:/tmp/db" FundsXmlDb roundtrip "$FX" out.xml
javac -cp "$CP" -d /tmp/db \
Database_Integration/java/ImportFundsXml.java \
Database_Integration/java/ExportFundsXml.java
java --enable-native-access=ALL-UNNAMED -cp "$CP:/tmp/db" ImportFundsXml fx.db "$FX"
java --enable-native-access=ALL-UNNAMED -cp "$CP:/tmp/db" ExportFundsXml fx.db "$DOC" out.xml

# JavaScript
( cd Database_Integration/javascript && npm install )
node Database_Integration/javascript/fundsxml_db.mjs roundtrip "$FX" out.xml
node Database_Integration/javascript/import_fundsxml.mjs fx.db "$FX"
node Database_Integration/javascript/export_fundsxml.mjs fx.db "$DOC" out.xml

# C#
dotnet run --project Database_Integration/csharp -- roundtrip "$FX" out.xml
dotnet run --project Database_Integration/csharp/import -- fx.db "$FX"
dotnet run --project Database_Integration/csharp/export -- fx.db "$DOC" out.xml

# prove it: identical to the input, and schema-valid
# prove it: exported file == input file, and schema-valid
python3 Database_Integration/tools/xml_equiv.py "$FX" out.xml
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd out.xml
```
Expand Down
Loading
Loading