Add json parquet loader - #127
Conversation
|
@ahmdkaml is attempting to deploy a commit to the yashkewlani2020-gmailcom's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Hi! I really enjoyed working on this project and I feel like I'm starting to get familiar with the codebase. I've just opened a PR for #113. If everything looks good there, I'd like to work on #111 next. I also noticed the explainer issues. They look interesting, although I'm not sure if they're a current priority. If they are, I'd be happy to take one of those as well. |
There was a problem hiding this comment.
Pull request overview
This PR extends the dataset loader (read_table()) to accept additional input formats requested in #113, and expands the loader test suite accordingly.
Changes:
- Add
.jsonsupport toread_table()(records + split orientation fallback). - Add
.parquetsupport toread_table()(viapandas.read_parquet()with optional dependency handling). - Add tests for JSON/Parquet loading and profile parity across formats, plus CLI usage examples.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
faircode/loaders.py |
Adds JSON and Parquet branches to read_table(), including a missing-dependency error path for Parquet. |
tests/test_loaders.py |
Adds loader tests for JSON (records/split), Parquet, and CSV parity comparisons. |
faircode/cli.py |
Updates CLI docstring usage examples to include JSON and Parquet inputs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if suffix == ".json": | ||
| try: | ||
| return pd.read_json(path) | ||
| except ValueError: | ||
| return pd.read_json(path, orient="split") |
| raise RuntimeError( | ||
| "reading .parquet files requires the 'pyarrow' package " | ||
| "(install with: pip install faircode[parquet])" | ||
| ) from exc |
| if suffix == ".csv": | ||
| return pd.read_csv(path) | ||
|
|
||
| if suffix == ".json": |
|
Hey! Thanks for the interest in my project! This PR is fine i will be merging it and you can solve issue #111 next if u want! The explainers are js for others to understand what ai bias is and is a way for researchers to contribute! |
Summary
This PR extends
read_table()to support additional table formats:Add support for
.jsonsplitorientationAdd support for
.parquetusingpandas.read_parquet()Mirror the existing optional dependency handling for Excel by providing a clear error message when
pyarrowis not installed.Update the CLI usage examples to include JSON and Parquet.
Add tests covering:
Closes #113.