Senior Test Engineer – Databricks

About the Role

We are looking for a seasoned Senior Test Engineer specializing in Databricks to contribute in a data engineering and analytics setting. The primary responsibility is to architect and execute thorough testing strategies focused on Databricks, Apache Spark, Delta Lake, and modern data pipelines.

The candidate should have substantial experience in data testing, ETL/ELT verification, automation, SQL, and Python. Collaboration will be key as this role involves working closely with Data Engineers, Developers, Architects, and Business stakeholders.

Key Responsibilities

• Design and deploy comprehensive end-to-end testing strategies for data platforms and pipelines built on Databricks.
• Create and run automated tests targeting Databricks notebooks, workflows, job executions, and data transformations.
• Conduct in-depth testing on ETL/ELT pipelines encompassing data ingestion, transformation, and loading processes.
• Validate Delta Lake components including tables, schemas, partitions, record counts, data accuracy, and transformation logic.
• Test Apache Spark and PySpark jobs ensuring large-scale data processing integrity.
• Build automated frameworks for data quality assessment and reconciliation leveraging SQL and Python.
• Verify data accuracy across Bronze, Silver, and Gold (Medallion) architectural layers.
• Test various data quality aspects such as completeness, accuracy, consistency, uniqueness, referential integrity, and adherence to business rules.
• Ensure robustness against schema evolution, changing data types, null values, and duplicate records.
• Test both streaming and batch data pipeline architectures where applicable.
• Maintain and enhance unit, integration, system, and regression test suites.
• Use testing frameworks including PyTest, Python unittest, or equivalents for automation.
• Validate execution and failure recovery of Databricks workflows and pipelines.
• Integrate automated testing within CI/CD processes and set quality gates for deployment cycles.
• Support testing for Delta Live Tables/Lakeflow pipelines including data quality and dependencies.
• Perform data reconciliation between source and destination systems including validation for data migrations.
• Conduct performance and scalability tests of large-scale Spark and Databricks workloads.
• Identify bugs, conduct root cause analysis, and collaborate with engineering teams for issue resolution.
• Develop test plans, cases, automation scripts, defect logs, and documentation to uphold quality standards.
• Drive enhancements in DataOps, test automation workflows, and quality engineering methodologies.

Required Skills & Experience

• Minimum of 7 years in software or data testing, particularly within data engineering or platform testing domains.
• At least 3 years of practical experience in testing Databricks environments.
• In-depth knowledge of Databricks, Apache Spark, and Delta Lake technologies.
• Advanced SQL capabilities for crafting complex validation and reconciliation queries.
• Proficiency in Python and PySpark for automation and data verification tasks.
• Hands-on experience testing ETL/ELT data workflows.
• Strong understanding of data warehouse and Lakehouse architectural concepts.
• Expertise in data quality testing and reconciliation.
• Familiarity with API, integration, and full-cycle end-to-end testing.
• Experience using automated testing frameworks such as PyTest or unittest.
• Skill in integrating automated tests into CI/CD pipelines.
• Sound knowledge of Software Development Life Cycle (SDLC), Software Testing Life Cycle (STLC), defect management, and Agile methodologies.
• Experience working with Git and source code management systems.

Additional Information

Databricks supports unit testing for pipeline transformation logic, data quality validations, and managing dependent pipeline flows. Tools such as PyTest and Databricks Connect facilitate integration testing of Databricks jobs.