Hadoop to Databricks Table Validation

A Python library for validating data migration from Hadoop/HDFS to Databricks Unity Catalog. Validates schema, data quality, partitions, and actual row data to ensure complete and accurate migration.

What This Tool Does

This tool connects to both Hadoop (via Kerberos) and Databricks, then runs a series of validation checks to verify that your migrated data matches the source data. It handles common issues like type differences, partition mismatches, and data quality problems.

Validation Levels

QUICK Level

Validates basic structure and counts. Fast, suitable for initial checks.

What runs:

Schema validation
Row count validation

Use when: You want a quick sanity check before deeper validation.

STANDARD Level

Complete validation including data quality, sampling, and partition checks.

What runs:

Everything in QUICK level
Statistics validation (min/max/mean/stddev)
Null analysis
Data sampling (actual row matching)
Partition completeness validation (if partition=None)
Per-partition row count validation (if partition=None)

Use when: You need to verify the migration is complete and accurate.

Type Compatibility System (Phase 2)

The validator includes a smart type compatibility system that handles differences between Hive/Impala and Databricks Spark SQL types.

How It Works

When comparing columns between Hadoop and Databricks, types are classified as:

Exact Match - Same type on both sides (e.g., INT = INT)
Compatible - Different but safely comparable types (e.g., VARCHAR(100) → STRING)
Incompatible - Unsafe type difference that could cause data loss (e.g., DECIMAL(10,2) → DECIMAL(8,2))

Compatible Type Conversions

These type differences are considered safe and validation will pass (with informational warnings):

Hadoop Type	Databricks Type	Normalization Strategy
TINYINT	SMALLINT/INT/BIGINT	Both cast to BIGINT
SMALLINT	INT/BIGINT	Both cast to BIGINT
INT	BIGINT	Both cast to BIGINT
FLOAT	DOUBLE	Both cast to DOUBLE with rounding
VARCHAR(n)	STRING	Both trimmed, empty→null
CHAR(n)	STRING	Both trimmed, empty→null
DECIMAL(10,2)	DECIMAL(12,4)	Both cast to DOUBLE, rounded to 9 decimals
TIMESTAMP	TIMESTAMP	Both converted to UTC epoch seconds
BOOLEAN	INT/BIGINT	Both cast to INT (0/1)

Incompatible Type Differences

These type differences are considered unsafe and validation will fail:

Hadoop Type	Databricks Type	Risk
DECIMAL(10,2)	DECIMAL(8,2)	Precision too small - overflow risk
BIGINT	INT	Narrowing conversion - data loss risk
STRING	VARCHAR(50)	Unbounded→bounded - truncation risk
TIMESTAMP	BIGINT	Can't meaningfully compare
DECIMAL	STRING	Can't safely compare numeric vs text
Complex types	MAP, ARRAY, STRUCT	Not supported in Phase 2

Where Type Compatibility Is Applied

Schema Validation: Compatible types pass with warnings, incompatible types fail
Statistics Validation: Only compatible numeric columns are compared
Data Sampling: Only compatible columns are used for row matching

Detailed Validation Rules

1. Schema Validation

What it does: Compares column names and data types between Hadoop and Databricks, with smart type compatibility checking (Phase 2).

How it works:

Gets list of columns from both tables
Checks if column names match (case-sensitive)
Checks data types and classifies them as:
- Exact match: Same type (e.g., INT = INT)
- Compatible: Different but safe types (e.g., VARCHAR→STRING, DECIMAL(10,2)→DECIMAL(38,18))
- Incompatible: Unsafe type difference (e.g., DECIMAL(10,2)→DECIMAL(8,2), TIMESTAMP→BIGINT)

What it catches:

Missing columns in Databricks
Extra columns in Databricks
Incompatible type differences that could cause data loss
Safe type differences are shown as warnings but validation passes

Pass/Fail logic (Phase 2):

✅ PASSED: All columns present, all types are exact matches or compatible
❌ FAILED: Missing columns, extra columns, or incompatible types

Example output:

Schema Validation: PASSED
- Columns in Hadoop: 52
- Columns in Databricks: 52
- Columns matched: 52
  - Exact type matches: 48
  - Compatible types: 4
  - Incompatible types: 0

Compatible Types (4 columns):
| Column    | Hadoop Type     | Databricks Type | Compatibility    |
|-----------|-----------------|-----------------|------------------|
| name      | varchar(100)    | string          | char_to_string   |
| price     | decimal(10,2)   | decimal(38,18)  | decimal_widen    |
| quantity  | int             | bigint          | integer_widen    |
| timestamp | timestamp       | timestamp       | exact            |

Failure example:

Schema Validation: FAILED
- Columns in Hadoop: 52
- Columns in Databricks: 51
  - Exact type matches: 45
  - Compatible types: 4
  - Incompatible types: 2

Incompatible Types (2 columns):
| Column    | Hadoop Type     | Databricks Type | Reason                  |
|-----------|-----------------|-----------------|-------------------------|
| amount    | decimal(10,2)   | decimal(8,2)    | decimal_overflow_risk   |
| metadata  | timestamp       | bigint          | timestamp_vs_bigint     |

Missing in Databricks (1 column):
- status

2. Row Count Validation

What it does: Compares total number of rows between Hadoop and Databricks.

How it works:

Counts rows in Hadoop table/partition
Counts rows in Databricks table/partition
Calculates percentage difference
Allows configurable tolerance (default 0.1%)

What it catches:

Data loss during migration
Duplicate rows
Incomplete migration

Tolerance logic:

difference = abs(hadoop_count - databricks_count)
allowed_diff = hadoop_count * tolerance_threshold
passed = difference <= allowed_diff

Example output:

Row Count Validation: PASSED
- Hadoop: 1,234,567 rows
- Databricks: 1,234,567 rows
- Difference: 0 rows (0.000%)
- Tolerance: 0.1%

Failure example:

Row Count Validation: FAILED
- Hadoop: 1,234,567 rows
- Databricks: 1,200,000 rows
- Difference: 34,567 rows (2.8%)
- Tolerance: 0.1%
- Issue: 34,567 rows missing in Databricks

3. Statistics Validation (STANDARD level only)

What it does: Compares statistical properties (min, max, mean, stddev) of numeric columns with type compatibility (Phase 2).

How it works:

Identifies numeric columns (int, bigint, float, double, decimal)
Checks type compatibility for each numeric column
Skips columns with incompatible types (e.g., Hadoop=DECIMAL, Databricks=STRING)
Randomly samples X compatible columns (default 10, configurable)
Applies normalization if types differ but are compatible
Calculates min/max/mean/stddev for each sampled column
Compares values between Hadoop and Databricks with tolerance

Type compatibility (Phase 2):

Compatible columns are normalized before comparison (e.g., DECIMAL(10,2) and DECIMAL(38,18) both cast to DOUBLE)
Incompatible columns are skipped and reported
Examples:
- ✅ INT → BIGINT (compatible, both cast to double)
- ✅ DECIMAL(10,2) → DECIMAL(38,18) (compatible, normalized)
- ❌ DECIMAL → STRING (incompatible, skipped)

Random sampling: Each run samples different columns. Running validation 5 times with sample_size=10 will eventually check all 50 columns.

What it catches:

Data corruption (values changed during migration)
Aggregation problems
Calculation errors
Data type conversion issues

Tolerance logic:

For min/max: absolute difference check
For mean/stddev: relative difference check (0.1% default)

Example output:

Statistics Validation: PASSED
- Checking 10 randomly sampled numeric columns (out of 48 compatible, 2 skipped)

Skipped Columns (2 columns due to type incompatibility):
| Column    | Hadoop Type | Databricks Type | Reason                |
|-----------|-------------|-----------------|------------------------|
| status    | tinyint     | string          | tinyint_vs_string     |
| metadata  | decimal     | string          | decimal_vs_string     |

| Column              | Match | H_Min | H_Max | H_Mean  | D_Min | D_Max | D_Mean  |
|---------------------|-------|-------|-------|---------|-------|-------|---------|
| amount              | ✅    | 0.00  | 999.9 | 234.56  | 0.00  | 999.9 | 234.56  |
| quantity (int→big)  | ✅    | 1     | 1000  | 45.23   | 1     | 1000  | 45.23   |

Failure example:

Statistics Validation: FAILED
- Column 'amount':
  - Hadoop min: 0.00, Databricks min: -100.00 (MISMATCH)
  - Hadoop max: 9999.99, Databricks max: 9999.99 (OK)
  - Hadoop mean: 234.56, Databricks mean: 230.12 (MISMATCH)

4. Null Analysis (STANDARD level only)

What it does: Compares null counts for each column to detect data quality issues.

How it works:

Randomly samples X columns (default 10, configurable)
Counts null values in each column for both tables
Compares null counts

What it catches:

Null value handling issues during migration
Data quality degradation
ETL bugs that introduce or remove nulls

Example output:

Null Analysis: PASSED
- Checking 10 randomly sampled columns
- All null counts matched

Failure example:

Null Analysis: FAILED
- Column 'description':
  - Hadoop nulls: 1,234
  - Databricks nulls: 5,678
  - Difference: 4,444 rows became null in Databricks

5. Data Sampling (STANDARD level only)

What it does: Randomly samples actual rows from Hadoop and verifies they exist exactly in Databricks.

How it works:

Type Compatibility Check:
- Compares column types between Hadoop and Databricks
- Identifies compatible vs incompatible columns
- Only uses compatible columns for matching
Column Normalization:
- Applies normalization to handle type differences:
  - Timestamps → UTC epoch seconds
  - Decimals → rounded to 9 decimal places
  - Strings → trimmed, empty strings as null
  - Booleans → 0/1
  - Integers → cast to bigint
Random Sampling:
- Randomly samples N rows from Hadoop (default 100)
- True random sample (no ordering to avoid tie-breaking issues)
Row Matching:
- For each sampled row, searches for exact match in Databricks
- Uses normalized values for comparison
- Uses efficient null-safe JOIN operation across all compatible columns
- Match requires ALL compatible columns to have identical values (including NULLs)

Type Compatibility:

Compatible types (safe to compare):

tinyint → smallint → int → bigint (widening)
float → double
varchar(n)/char(n) → string
decimal(10,2) → decimal(10,4) (precision/scale safe)
timestamp → timestamp (normalized to epoch)
boolean ↔ int (as 0/1)
date ↔ date

Incompatible types (skipped):

string → varchar(n) (truncation risk)
decimal(10,4) → decimal(10,2) (rounding/overflow risk)
int → tinyint (narrowing)
array, map, struct (complex types)

Concrete Examples:

✅ Compatible (normalized and compared):

Hadoop: VARCHAR(100), Databricks: STRING → Both trimmed, empty→null, then compared
Hadoop: DECIMAL(10,2), Databricks: DECIMAL(12,4) → Both rounded to 9 decimals, then compared
Hadoop: INT, Databricks: BIGINT → Both cast to bigint, then compared
Hadoop: TIMESTAMP, Databricks: TIMESTAMP → Both converted to UTC epoch seconds, then compared

❌ Incompatible (skipped from sampling):

Hadoop: DECIMAL(10,2), Databricks: DECIMAL(8,2) → Overflow risk (target precision too small)
Hadoop: TIMESTAMP, Databricks: BIGINT → Can't normalize timestamp to match bigint
Hadoop: STRING, Databricks: VARCHAR(50) → Truncation risk (source unbounded)
Hadoop: MAP<STRING,STRING>, Databricks: MAP<STRING,INT> → Complex type, not supported in Phase 1

What it catches:

Missing rows in Databricks
Row-level data corruption
Subtle data transformation issues
Type conversion problems

Example output:

Data Sampling: PASSED
- Sample size: 100 rows
- Rows found: 100
- Match rate: 100.0%
- Columns checked: 48
- Columns skipped: 2 (incompatible types)

Incompatible Columns:
| Column    | Hadoop Type   | Databricks Type | Reason                    |
|-----------|---------------|-----------------|---------------------------|
| status    | tinyint       | string          | tinyint_vs_string         |
| metadata  | map<str,str>  | map<str,int>    | complex_type_phase1_skip  |

Failure example:

Data Sampling: FAILED
- Sample size: 100 rows
- Rows found: 87
- Rows not found: 13
- Match rate: 87.0%

Rows NOT FOUND in Databricks:
Row #5:
| Column     | Value              |
|------------|--------------------|
| order_id   | ORD-12345          |
| amount     | 234.56             |
| created_at | 2024-01-15 10:30:00|

6. Partition Completeness Validation (STANDARD level, partition=None only)

What it does: Verifies all partitions from Hadoop exist in Databricks.

When it runs: Only when validating the entire table (partition=None). Skipped when validating a single partition.

How it works:

Discover partitions from Hadoop HDFS:
- Walks HDFS directory structure
- Identifies partition directories (e.g., fab=X/year=2024/month=1/)
- Extracts partition key-value pairs
Get partitions from Databricks:
- Queries Databricks catalog for partition list
- Uses SHOW PARTITIONS command
Compare:
- Checks partition column schema matches
- Compares total partition counts
- Identifies missing partitions (in Hadoop but not Databricks)
- Identifies extra partitions (in Databricks but not Hadoop)

What it catches:

Incomplete migration (missing partitions)
Wrong partition column definitions
Extra partitions in target

Example output:

Partition Completeness: PASSED
- Partition columns: fab, year, month, day
- Hadoop partitions: 150
- Databricks partitions: 150
- Missing in Databricks: 0
- Extra in Databricks: 0

Failure example:

Partition Completeness: FAILED
- Partition columns: fab, year, month, day
- Hadoop partitions: 150
- Databricks partitions: 145
- Missing in Databricks: 5

Missing Partitions:
| fab      | year | month | day |
|----------|------|-------|-----|
| SMSNGS1  | 2024 | 1     | 15  |
| SMSNGS1  | 2024 | 1     | 16  |
| SMSNGS1  | 2024 | 1     | 17  |

7. Per-Partition Row Count Validation (STANDARD level, partition=None only)

What it does: Validates row counts for individual partitions (first 10 by default).

When it runs: Only when validating the entire table (partition=None). Skipped when validating a single partition.

How it works:

Gets list of partitions from Hadoop
Takes first 10 partitions
For each partition:
- Counts rows in Hadoop partition
- Counts rows in Databricks partition
- Compares with tolerance

What it catches:

Partial partition migration (partition exists but incomplete data)
Per-partition data quality issues
Partition-level data loss

Example output:

Per-Partition Row Counts: PASSED
- Partitions checked: 10
- All partitions matched within tolerance

| Partition             | Hadoop Rows | Databricks Rows | Match |
|-----------------------|-------------|-----------------|-------|
| fab=X/year=2024/m=1   | 12,345      | 12,345          | ✓     |
| fab=X/year=2024/m=2   | 13,456      | 13,456          | ✓     |

Failure example:

Per-Partition Row Counts: FAILED
- Partitions checked: 10
- Matched: 8
- Failed: 2

| Partition             | Hadoop Rows | Databricks Rows | Diff   |
|-----------------------|-------------|-----------------|--------|
| fab=X/year=2024/m=3   | 12,345      | 11,000          | -1,345 |
| fab=X/year=2024/m=4   | 13,456      | 12,000          | -1,456 |

Column Type Coverage

Different validations check different column types. Here's what gets checked for each data type:

Column Type	Null Analysis	Statistics	Empty Strings	Whitespace	Case Variations	Collation
Numeric (int, bigint, float, double, decimal, smallint, tinyint)	✅	✅	❌	❌	❌	❌
String (string, varchar, char)	✅	❌	✅	✅	✅	✅
Boolean	✅	❌	❌	❌	❌	❌
Date/Timestamp	✅	❌	❌	❌	❌	❌
Binary	✅	❌	❌	❌	❌	❌
Complex (array, map, struct)	✅	❌	❌	❌	❌	❌

Key Points:

Null Analysis checks ALL column types (nulls can exist in any data type)
Statistics (min/max/mean/stddev) only make sense for numeric columns
Empty Strings, Whitespace, Case Variations, Collation only apply to text data
Boolean, Date/Timestamp, Binary, Complex types only get null checks (no type-specific data quality validations)

This design ensures each validation runs only on relevant column types, avoiding errors and unnecessary checks.

Collation & Data Quality Checks (STANDARD level only)

For migrating English-only data, the validator includes several collation and string data quality checks:

Check Flow

STANDARD Validation Level
        │
        ▼
┌─────────────────────────────────────┐
│ CHECK 1: Collation Setting          │
│ Detect UTF8_BINARY_LCASE columns    │
│ ✅ PASS or ⚠️  WARN                  │
└────────────┬────────────────────────┘
             │
             │ (result passed to Check 2)
             ▼
┌─────────────────────────────────────┐
│ CHECK 2: Case Variations             │
│ Detect "TX" vs "tx" in data         │
│                                     │
│ IF partition column has variations: │
│   → ⚠️  NOTICE (existing structure)  │
│ ELSE IF case-insensitive collation: │
│   → ⚠️  WARN (duplicate risk)        │
│ ELSE:                               │
│   → ✅ PASS                          │
└────────────┬────────────────────────┘
             │
             ▼
┌─────────────────────────────────────┐
│ CHECK 3: Whitespace Issues           │
│ Detect spaces, tabs, special chars  │
│ ✅ PASS or ⚠️  WARN                  │
└────────────┬────────────────────────┘
             │
             ▼
┌─────────────────────────────────────┐
│ CHECK 4: Empty Strings vs NULL       │
│ Detect "" that should be NULL       │
│ ✅ PASS or ⚠️  WARN                  │
└─────────────────────────────────────┘

Check Descriptions

CHECK 1: Collation Setting

Examines Databricks column metadata
Detects UTF8_BINARY_LCASE (case-insensitive) columns
Default is UTF8_BINARY (case-sensitive, same as Hadoop)
⚠️ Warns if case-insensitive collation found

CHECK 2: Case Variations

Detects mixed case in data (e.g., "TX" vs "tx")
Depends on Check 1 results to determine severity
Identifies partition columns with variations
Different warnings based on context

CHECK 3: Whitespace Issues

Detects leading/trailing spaces
Detects multiple consecutive spaces
Detects tabs, non-breaking spaces (U+00A0)
Detects zero-width characters (U+200B, U+FEFF)
Independent check (affects all collation types)

CHECK 4: Empty Strings vs NULL

Identifies empty strings ("") that may behave differently than NULL
Hive blurs empty string and NULL, but Spark keeps them distinct
Affects COUNT, JOIN, and filter behavior
Independent check (affects all collation types)

Example Scenarios

✅ Best Case: Clean Data

Check 1: ✅ UTF8_BINARY (case-sensitive)
Check 2: ✅ No case variations
Check 3: ✅ No whitespace issues
Check 4: ✅ No empty strings

Result: Perfect migration, no data quality issues

⚠️ Case-Insensitive Collation Problem

Check 1: ⚠️  UTF8_BINARY_LCASE found
Check 2: ⚠️  "state" has "TX" vs "tx" variations
         → Risk: Values treated as duplicates
         → Fix: Change collation OR normalize case

Result: Need to fix collation or normalize data

⚠️ Partition Column Notice (Existing Structure)

Check 2: ⚠️  Partition column "fab" has "TX" vs "tx"
         → This structure ALREADY EXISTS in Hadoop (fab=TX/ and fab=tx/)
         → Migration PRESERVES this correctly (not a failure)
         → Users should be aware: WHERE fab='TX' only reads TX partition

Result: Data quality awareness, not migration problem

⚠️ Multiple Data Quality Issues

Check 1: ⚠️  Case-insensitive collation found
Check 2: ⚠️  Partition variations + data variations
Check 3: ⚠️  Leading/trailing spaces in 5 columns
Check 4: ⚠️  Empty strings in 3 columns

Result: Clean data in Hadoop before migration
        1. Normalize partition column case
        2. TRIM all string columns
        3. Convert empty strings to NULL
        4. Fix collation settings in Databricks

Key Points

Check 1 → Check 2 Dependency: Case variations only matter when collation is case-insensitive
Checks 3 & 4 are Independent: Run regardless of collation, affect all tables
All checks return warnings: Allow validation to continue while flagging issues
Partition column variations: Existing Hadoop structure, not a migration failure

Quality Score Calculation

The validator calculates an overall quality score based on which checks ran and passed.

Weights:

Schema: 25%
Row Count: 25%
Partitions: 20%
Statistics: 15%
Null Analysis: 8%
Per-Partition Row Counts: 5%
Data Sampling: 1%

Normalization: Score is normalized based on which checks actually ran. For example:

If you validate a single partition, partition checks are skipped
Max possible score is only from checks that ran (e.g., 0.74 instead of 1.0)
Normalized score = actual_score / max_possible_score
Displayed as percentage (e.g., 100% if all checks that ran passed)

Example:

Quality Score: 100.0% (4/4 checks passed)
- Schema: PASSED (0.25)
- Row Count: PASSED (0.25)
- Statistics: PASSED (0.15)
- Null Analysis: PASSED (0.08)
- Partitions: SKIPPED (validating single partition)
Total: 0.73 / 0.73 max possible = 100%

Configuration Parameters

validate_single_table() parameters

Parameter	Default	Description
`hadoop_path`	required	HDFS path to Hadoop table
`databricks_table`	required	Databricks table name (catalog.schema.table)
`partition`	None	Partition to validate. None = entire table
`validation_level`	"STANDARD"	"QUICK" or "STANDARD"
`tolerance_threshold`	0.0	Acceptable row count difference (absolute count, not percentage). If 0, requires exact match. If > 0 and difference ≤ threshold, triggers reconciliation to find differing rows and validates common subset.
`use_temp_table`	True	Write Hadoop data to temp Delta table (memory efficient)
`sample_fraction`	None	Sample fraction (0.01 = 1%) for testing
`max_columns_to_check`	10	Columns to randomly sample for stats/null checks. -1 = all
`sample_size`	100	Number of rows to sample for data sampling

Partition Parameter Behavior

partition=None (Validate Entire Table):

Validates ALL data across all partitions
Partition validation ENABLED:
- Partition completeness check runs
- Per-partition row count check runs
Use for: Final validation to ensure complete migration

partition={"fab": "X", "year": 2024} (Validate Single Partition):

Validates ONLY data in specified partition
Same partition filter applied to both Hadoop and Databricks
Partition validation SKIPPED (not applicable)
Use for: Quick testing on one partition before full validation

Memory Efficiency

Temp Delta Table Strategy (use_temp_table=True)

Problem: Large Hadoop datasets can cause out-of-memory errors when cached in Spark.

Solution:

Read Hadoop data once
Write to temporary Delta table in Databricks
Compare two Delta tables (both in Databricks)
Cleanup temp table when done

Benefits:

Works on small clusters (8GB+)
Efficient join/comparison operations
Automatic cleanup even if validation fails

Temp table naming:

{catalog}.{schema}.temp_validation_{timestamp}
Example: my_catalog.my_schema.temp_validation_1706123456

Random Sampling Strategy

Why Random Sampling?

Instead of always checking the same columns/rows, the validator randomly samples different data each run.

Benefits:

Better coverage over multiple runs
Catches edge cases in different columns
Validates different data subsets

Example:

Table has 50 columns
max_columns_to_check=10
Run 1: checks columns [5, 12, 23, 34, 45, ...]
Run 2: checks columns [3, 18, 29, 31, 48, ...]
Run 5 times → covers all 50 columns

Same for data sampling:

Each run samples different random rows
100 rows × 10 runs = 1,000 different rows validated

Output and Reporting

Console Output

Real-time progress during validation:

📂 Loading data from Hadoop HDFS...
   ✅ Loaded 1,234,567 rows from Hadoop

📂 Loading data from Databricks...
   ✅ Loaded 1,234,567 rows from Databricks

🔍 Running STANDARD validation...
   Checking 10 randomly sampled numeric columns (out of 52 total)
   Checking 10 randomly sampled columns for null patterns
   Randomly sampling 100 rows from Hadoop (out of 1,234,567 total)
   Checking column type compatibility...
   Compatible columns: 48, Incompatible: 2
   Checking 100 sampled rows in Databricks...

✅ Validation complete!
   Status: PASSED, Score: 100.0%

Validation Report

Detailed report with pandas DataFrames:

from validation_report import display_validation_report
display_validation_report(validation_result)

Shows:

Summary table (status, score, checks passed/failed)
Detailed results for each validation
Tables with specific failures
Recommendations for fixing issues

Typical Workflow

1. Test on Single Partition First

partition = {"fab": "SMSNGS1", "year": 2024, "month": 1, "day": 1}

result = validator.validate_single_table(
    hadoop_path="/user/hive/warehouse/db/table",
    databricks_table="catalog.schema.table",
    partition=partition,
    validation_level="STANDARD",
    max_columns_to_check=10,  # sample 10 columns
    sample_size=50  # sample 50 rows (fast)
)

2. Test with Sampling (1% of data)

result = validator.validate_single_table(
    hadoop_path="/user/hive/warehouse/db/table",
    databricks_table="catalog.schema.table",
    partition=None,  # entire table
    validation_level="STANDARD",
    sample_fraction=0.01,  # use 1% sample for speed
    max_columns_to_check=10
)

3. Full Validation (Production)

result = validator.validate_single_table(
    hadoop_path="/user/hive/warehouse/db/table",
    databricks_table="catalog.schema.table",
    partition=None,  # entire table
    validation_level="STANDARD",
    max_columns_to_check=-1,  # check ALL columns
    sample_size=500  # sample 500 rows for thorough validation
)

Common Issues and Solutions

Issue: "Incompatible columns skipped"

Cause: Type mismatch between Hadoop and Databricks

Solution:

Check the incompatible columns table in report
Fix type mismatches in Databricks schema
Re-run migration with correct types

Example:

Hadoop: amount DECIMAL(10,2)
Databricks: amount STRING
Fix: ALTER TABLE ... ALTER COLUMN amount TYPE DECIMAL(10,2)

Issue: "Rows not found in Databricks"

Cause:

Missing rows
Data corruption
Transformation issues

Solution:

Check "Rows NOT FOUND" section in report
Identify patterns in missing rows
Re-migrate affected data

Issue: "Partition missing in Databricks"

Cause:

Incomplete migration
Partition filter issue in migration script

Solution:

Check missing partitions list
Re-run migration for missing partitions
Verify partition discovery logic

Issue: "Statistics mismatch"

Cause:

Data type conversion issues
Calculation precision differences
Actual data corruption

Solution:

Check which statistics failed (min/max/mean/stddev)
Investigate specific columns
Verify data transformation logic

Performance Tips

Use temp Delta table (default) for large datasets
Start with QUICK level for initial checks
Use sample_fraction for very large tables (>100M rows)
Limit max_columns_to_check for wide tables (>100 columns)
Test single partition first before full table validation
Run validation multiple times with random sampling for better coverage

Requirements

PySpark with Databricks Runtime
Hadoop HDFS access with Kerberos authentication
Databricks Unity Catalog access
Network connectivity: Databricks ↔ Hadoop KDC, NameNodes, DataNodes

Files

hadoop_validator.py - Core validation logic
validation_report.py - Report display module
hadoop_connection.py - Hadoop/Kerberos connection utilities
quick_validation.ipynb - Main validation notebook
create_test_sample.ipynb - Create test samples from Hadoop data

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
.gitattributes		.gitattributes
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
README.md		README.md
hadoop_connection.py		hadoop_connection.py
hadoop_validator.py		hadoop_validator.py
quick_validation.ipynb		quick_validation.ipynb
validation_report.py		validation_report.py

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Hadoop to Databricks Table Validation

What This Tool Does

Validation Levels

QUICK Level

STANDARD Level

Type Compatibility System (Phase 2)

How It Works

Compatible Type Conversions

Incompatible Type Differences

Where Type Compatibility Is Applied

Detailed Validation Rules

1. Schema Validation

2. Row Count Validation

3. Statistics Validation (STANDARD level only)

4. Null Analysis (STANDARD level only)

5. Data Sampling (STANDARD level only)

6. Partition Completeness Validation (STANDARD level, partition=None only)

7. Per-Partition Row Count Validation (STANDARD level, partition=None only)

Column Type Coverage

Collation & Data Quality Checks (STANDARD level only)

Check Flow

Check Descriptions

Example Scenarios

Key Points

Quality Score Calculation

Configuration Parameters

validate_single_table() parameters

Partition Parameter Behavior

Memory Efficiency

Temp Delta Table Strategy (use_temp_table=True)

Random Sampling Strategy

Why Random Sampling?

Output and Reporting

Console Output

Validation Report

Typical Workflow

1. Test on Single Partition First

2. Test with Sampling (1% of data)

3. Full Validation (Production)

Common Issues and Solutions

Issue: "Incompatible columns skipped"

Issue: "Rows not found in Databricks"

Issue: "Partition missing in Databricks"

Issue: "Statistics mismatch"

Performance Tips

Requirements

Files

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages