Databricks/PySpark Expert — Test & Validate a Function Across 2 to 3 Delta Tables (Azure Databricks)
Publicada el 2026-07-21
Descripción de la oferta
Description: I need a Databricks/PySpark expert to help me test and validate whether a custom PySpark data-masking function is working correctly across several large Delta tables in Azure Databricks (Unity Catalog). Context: I've built a generic PySpark function that masks PII columns (replaces them with 'xxxxx') in Delta tables, based on a date-threshold business rule (e.g., mask records older than X years). I need to prove — with solid evidence — that this function is producing correct output on every table it's applied to, not just spot-checking a few rows. My current testing approach: Take a backup copy of the real (production) table Deliberately corrupt the PII columns on rows that should qualify for masking (set them to a dummy value like '0') Run the masking function against this corrupted copy Confirm every dummy value got correctly converted to 'xxxxx', and nothing else was affected Compare the result against the live production table using SQL EXCEPT (restricted to only the rows that should remain unmasked) to confirm nothing outside the masking scope was altered Current issue I'm stuck on: On one table (7.6M+ rows), the EXCEPT comparison returns ~1,400 mismatched rows out of millions, even though row counts match exactly and the masking itself is proven correct. I've traced part of it to a load_timestamp column that drifts between a live table and a point-in-time snapshot, but even after excluding that column, mismatches remain. I need help finding the remaining cause and building a clean, reliable comparison. What I need help with: Diagnose exactly which column(s) are causing the mismatch (likely another audit/metadata column with natural drift, unrelated to the actual masking logic) Build a reusable SQL/PySpark comparison script that: Does a column-by-column diff between two Delta tables (given a join key) to quickly spot which columns differ Produces a clean EXCEPT-based pass/fail check, correctly excluding known-noisy audit columns Run this same validated testing method across 5–6 more Delta tables (different schemas — I'll provide table names, primary keys, and PII column lists), to confirm the masking function works consistently, not just on one table Environment: Azure Databricks (Unity Catalog), Delta Lake, PySpark, Spark SQL. Tables range from ~1M to 40M+ rows. Deliverable: A working, documented notebook (SQL + PySpark) that reliably tests whether the masking function is working correctly on any given table — parameterized by table name, join key, PII columns, and columns to exclude from comparison — plus the completed test results for the 5–6 tables. Ideal skills: Databricks, Delta Lake, PySpark, Spark SQL, Unity Catalog, data QA/validation experience with large-scale tables.
Skills
Fuente original: freelancer