Skip to content

Add Amazon Redshift connector (#2) - #3

Open
jadstrike wants to merge 2 commits into
harshitboots:mainfrom
jadstrike:feature/redshift-connector
Open

Add Amazon Redshift connector (#2)#3
jadstrike wants to merge 2 commits into
harshitboots:mainfrom
jadstrike:feature/redshift-connector

Conversation

@jadstrike

Copy link
Copy Markdown
Contributor

What this adds

Amazon Redshift connector, as requested in the README's next-targets list (issue #2).

  • src/connectors/redshift.py — follows the standard connect / list_tables / fetch_table pattern used by the other connectors. Redshift is wire-compatible with PostgreSQL, so it connects via psycopg2zero new dependencies.
  • Amazon Redshift option in the Database Connectors section of the Streamlit UI (host, port — default 5439, database, username, password, table).

Tests

Added TestRedshiftConnector with three tests (list_tables, fetch_table, row limit). They monkeypatch psycopg2.connect with a stubbed connection, so the suite runs anywhere — no live cluster or AWS credentials required.

pytest tests/ -v

All existing tests still pass.

README

  • Redshift usage snippet under Database Connectors
  • Connector status table: Amazon Redshift → Stable (auth: user/pass)
  • Removed Redshift from the next-targets table and checked the roadmap box

Note

This branch is independent of #2's DuckDB work (both branch off main). If both are merged, the only overlap is the connector dropdown list in app.py — a trivial one-line conflict.

- New connector src/connectors/redshift.py following the standard
  connect/list_tables/fetch_table pattern. Redshift is wire-compatible
  with PostgreSQL, so it reuses psycopg2 — no new dependencies
- Amazon Redshift option in the Database Connectors UI (default port 5439)
- Tests covering list_tables, fetch_table and the row limit using a
  stubbed connection, so they run without a live cluster
- README: usage snippet, connector status table and roadmap updated
Comment thread tests/test_pipeline.py
from src.connectors.redshift import fetch_table
fetch_table("host", 5439, "dev", "awsuser", "password", "transactions", limit=500)
assert "LIMIT 500" in fake_connection.cursor().executed_query

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

def test_fetch_table_validates_table_name(self, fake_connection): from src.connectors.redshift import fetch_table with pytest.raises(ValueError, match="not found"): fetch_table("host", 5439, "dev", "awsuser", "password", "nonexistent_table")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this test in 087c44c — it passes along with the rest of the suite.

@harshitboots harshitboots left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can do this which will give more liberty to user while connecting to any tables

cursor.close()
conn.close()
return tables

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

` think we can use following code as its Doesn't work if user wants to query views or temporary tables (though they'd need to be in the whitelist query) as well now

actually it will resolve postgres and mysql issue also.

`def fetch_table(host: str, port: int, database: str, user: str, password: str, table: str, limit: int = 1000) -> pd.DataFrame:
"""Fetch a table from Redshift with SQL injection protection."""
conn = connect(host, port, database, user, password)
cursor = conn.cursor()

try:
    # Validate table exists to prevent SQL injection
    valid_tables = list_tables(host, port, database, user, password)
    if table not in valid_tables:
        raise ValueError(f"Table '{table}' not found in database. Available tables: {', '.join(valid_tables)}")
    
    cursor.execute(f"SELECT * FROM {table} LIMIT {limit}")
    columns = [desc[0] for desc in cursor.description]
    rows = cursor.fetchall()
    return pd.DataFrame(rows, columns=columns)
finally:
    cursor.close()
    conn.close()`

@jadstrike jadstrike Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 087c44c. fetch_table now validates the table name against list_tables() before querying and raises a ValueError listing the available tables if it's not found. Cursor/connection cleanup is now handled in a finally block. The same pattern would also apply to the Postgres and MySQL connectors — happy to address those in a follow-up PR.

harshitboots added a commit that referenced this pull request Jun 10, 2026
Feature: Added DuckDB Connector (Resolves Issue #3)
harshitboots pushed a commit that referenced this pull request Jun 13, 2026
- New connector src/connectors/duckdb_conn.py following the standard
  connect/list_tables/fetch_table pattern
- DuckDB option in the Database Connectors UI (file path + table)
- Tests covering list_tables, fetch_table and the row limit
- README: usage snippet, connector status table and roadmap updated
harshitboots added a commit that referenced this pull request Jun 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants