Databricks Engineer

About Navitas Business Consulting

Founded in 2006, Navitas Business Consulting has become a respected leader in the digital transformation arena, serving clients across commercial, federal, state, and local sectors. The company focuses on delivering award-winning technology solutions that accelerate digital innovation and empower clients to leverage technology as a competitive edge.

Role Overview

We are looking for a skilled Databricks Engineer to design, develop, and maintain a robust Data & AI platform based on the Medallion Architecture consisting of raw (bronze), curated (silver), and mart (gold) data layers. The platform will automate complex data workflows and scalable ELT pipelines integrating data from enterprise sources such as PeopleSoft, D2L, and Salesforce. Your efforts will underpin high-quality, governed data availability supporting machine learning, AI, BI, and analytics at scale.

Key Responsibilities

• Create and optimize Databricks pipelines following Medallion Architecture principles to transform and curate data layers.
• Develop and operate ETL/ELT pipelines with Apache Spark and Delta Lake for scalable data transformation.
• Automate workflows using Databricks orchestration tools to manage dependencies and ensure smooth pipeline execution.
• Support schema evolution and data versioning to facilitate agile data development and continuous improvement.
• Integrate data from enterprise applications such as PeopleSoft, D2L, and Salesforce using suitable connectors and ingestion frameworks handling structured, semi-structured, and unstructured data.
• Standardize data ingestion with automated error handling, retries, and alerting mechanisms to ensure reliability.
• Implement data quality controls including validations, checks, and anomaly detection to maintain data integrity across all stages.
• Utilize monitoring and observability tools like Databricks metrics and Grafana for ETL performance tracking and failure alerts.
• Manage metadata and enforce governance with tools such as Unity Catalog for centralized control of data lineage and policies.
• Apply data security best practices including encryption, row-level security, and fine-grained access controls to comply with regulations such as GDPR and FERPA.
• Work alongside security teams to audit compliance controls and implement data masking, tokenization, and anonymization techniques.
• Provide ML/AI teams with high-quality datasets featuring reusable components for training and inference, supporting ML lifecycle workflows through MLflow.
• Architect cloud data storage solutions on Azure Data Lake Storage or Amazon S3, managing efficient data access and cost optimization.
• Create and maintain detailed documentation, including architecture diagrams, data dictionaries, and runbooks; conduct training and knowledge-sharing sessions for stakeholders.
• Submit weekly work schedules and progress reports, track milestones, and proactively communicate risks or dependencies.

Qualifications

• Proven hands-on experience in Databricks, Delta Lake, and Apache Spark for enterprise-scale data engineering.
• Expertise in developing, orchestrating, and monitoring ELT pipelines in cloud environments.
• Strong understanding and practical experience with Medallion Architecture layers and enterprise-grade data versioning and schema enforcement.
• Advanced skills in SQL, Python, and/or Scala for building data transformation and workflow logic.
• Experience integrating enterprise systems like PeopleSoft, Salesforce, and D2L into unified data platforms.
• Knowledge of data governance frameworks, metadata management, and lineage tracking tools.
• Preferred familiarity with Databricks Unity Catalog, MLflow MLOps tools, cloud platforms such as Azure or AWS, and data warehouse design methodologies including star and snowflake schemas.