SQL View to PySpark Notebook
Publicada el 2026-07-30
Descripción de la oferta
Project Description I'm looking for an experienced Azure Databricks / PySpark developer to help convert an existing SQL Server view into a Databricks PySpark transformation. Project Overview I have: An existing SQL view with multiple CTEs. An existing Databricks Asset Bundle notebook that will be used as the template. A source Delta table already available in Unity Catalog. The task is to replace the existing business transformation with a new PySpark implementation while preserving the existing notebook framework. Source The source is a Delta table in Unity Catalog. The notebook should: Read data from the source table. Apply filters. Replicate the SQL CTE logic in PySpark. Perform required aggregations and calculations. Produce the final dataframe. Preserve the notebook's existing audit and validation framework. Write the output Delta table. Create a SQL View on top of the Delta table. SQL Logic The SQL contains: Multiple CTEs Window selection logic Aggregations CASE expressions GROUP BY Date filtering Calculated measures The objective is to produce the same output as the SQL view using PySpark DataFrame APIs. Existing Notebook The notebook already contains: Logging framework Metadata handling Schema validation Audit column generation Exception handling Delta write logic These should remain unchanged. Only the business transformation needs to be replaced. Deliverables Complete PySpark implementation. Clean, optimized, production-ready code. Logic matching the SQL view. Compatible with Databricks Runtime. Delta table creation. SQL View creation. Assistance with testing if required. Required Skills Azure Databricks PySpark Spark SQL Delta Lake SQL Server DataFrame API Window Functions Databricks Asset Bundles (preferred) Nice to Have Experience with Unity Catalog Production ETL development Azure Data Factory knowledge What I'll Provide Existing SQL View Existing Databricks notebook Business logic
Skills
Fuente original: freelancer