Data Engineer - Databricks Developer

About the Company

Our client, based in Auckland, is a prominent ANZ AI and Data Analytics consulting firm with three decades of experience partnering with leading healthcare, life sciences, financial services, and government organisations across Australia and New Zealand.

Role Overview

They are seeking a highly experienced Data Engineer to join their innovative team and contribute to the growth and success of their specialized Databricks Center of Excellence.

Key Responsibilities

• Collaborate extensively with data leadership to enhance sales, recruitment, consulting, client account management, and operational excellence within the practice.
• Take ownership in delivering and guiding Databricks-centric projects, acting as the in-house Databricks subject matter expert (SME).
• Support business development by advising clients on architecture, data engineering best practices, and scalable analytics using Databricks technologies.
• Foster innovation and maintain high performance standards through team collaboration focusing on output, ownership, and employee wellbeing.
• Lead project risk management, including planning, budgeting, deliverables, and alignment with executive stakeholders.
• Expand company market presence by developing proposals, statements of work, and growth strategies within existing customer accounts.
• Provide architectural and technical leadership on Databricks, modern cloud-native data platforms, and solutions to clients and internal teams.
• Mentor colleagues and promote cutting-edge technologies and tools aligned with client objectives and evolving data strategies.
• Adopt a business-first attitude ensuring technology implementations deliver impactful, competitive advantages for clients.

Technical Responsibilities

• Design and deploy scalable, robust data architectures utilizing Databricks including expertise with Delta Lake, Uniform (Iceberg), Delta Live Tables, and Unity Catalog.
• Lead engineering teams by fostering learning and innovation; implement best practices in data management and solution optimization.
• Engage directly with clients to tailor data engineering solutions that meet business requirements leveraging Databricks.
• Provide hands-on technical leadership in developing Data Lakehouse platform components tailored for client business needs.

Qualifications and Experience

• Minimum of 8 years in data engineering or architecture roles, with at least 4 years focused specifically on Databricks and its products such as Delta Lake, Delta Live Tables, and Unity Catalog.
• Extensive knowledge of Big Data platforms and cloud data warehouses.
• Strong familiarity with Databricks technical architecture on top public cloud providers (AWS, Azure, GCP) and experience establishing scalable Databricks environments from scratch.
• Proficient in developing industry-standard extensible data models for Silver and Gold layers, employing design methodologies like Star Schema and Data Vault.
• Advanced skills in Object-Oriented programming languages such as Java, Python, and PySpark, with expertise in NoSQL databases and optimizing complex data pipelines.
• Experience with container orchestration systems and AI/ML platforms.
• Competence in streaming data ingestion and modern data workflow design.
• History of leading large migration and modernization projects transitioning legacy data warehouses to the Databricks Lakehouse platform.
• Ability to estimate Databricks consumption for diverse Data & AI pipeline workloads across multiple environments.
• Experience designing and implementing large-scale Disaster Recovery strategies for Databricks platforms is highly desirable.
• Databricks Data Engineering Professional certification and proven success delivering multiple complex Databricks projects preferred.