Lead Data Engineer
Chennai, Tamil Nadu, India · Full Time
Be the first to apply
- Experience
- 7+ yrs
- Salary
- —
- Openings
- 1
- Posted
- 5 മണിക്കൂർ മുൻപ്
- Work mode
- In office
- Education
- B.Sc in Information Technology, B.Tech / B.E. in Computer Science Engineering
- Eligibility
- Candidates possessing a B.Sc in Information Technology or B.Tech/B.E. in Computer Science Engineering can apply.
- Resume
- Required to apply
Where you'll work
Sign in to tell us what does and doesn't work for you here — it sharpens every match we show you.
Job description
About the Role
GenSpark, powered by Pyramid Consulting, is seeking a Lead Data Engineer to spearhead the design, build, and deployment of scalable data solutions on Azure. This role involves crafting robust data pipelines and architectures that support vast data volumes with a focus on high performance and reliability.
Key Responsibilities
- Architect and implement scalable data engineering solutions using Azure services including Data Factory, Synapse, Databricks, and Azure SQL.
- Develop and optimize large-scale ETL/ELT workflows and data pipelines using PySpark, SparkSQL, and SQL.
- Design and maintain data lakes, data warehouses, and dedicated SQL pools on Azure to meet business analytics needs.
- Lead data ingestion and migration projects from legacy or on-premise systems to Azure cloud platforms.
- Ensure processing of petabyte-scale datasets is efficient, reliable, secure, and governed according to compliance standards.
- Collaborate cross-functionally with product, marketing, business intelligence, data science, and machine learning teams to translate requirements into analytical solutions.
- Support analytics initiatives leveraging Power BI, Tableau, and real-time/offline data solutions.
- Provide leadership through code reviews, design guidance, mentoring junior and senior engineers, and championing best engineering practices.
- Drive team recruitment efforts and foster growth within the Data Analytics and Engineering teams.
- Promote continuous improvement, automation, and operational excellence on the data platform.
Candidate Profile
- Over 7 years of relevant data engineering experience with deep proficiency in Azure cloud technologies.
- Hands-on expertise in Azure Data Factory, Synapse Analytics, Databricks, Azure SQL Database, and Data Lakes.
- Advanced skills in PySpark, SparkSQL, and SQL for building and tuning complex data transformations.
- Solid background in ETL/ELT design, data modeling, data warehousing, and constructing data lakes.
- Experience with Synapse Dedicated SQL Pools or Azure SQL Data Warehouse in managing large-scale data processing tasks.
- Capability to handle massive, event-driven datasets spanned across petabyte-scale volumes.
- Familiarity with SSIS, Azure Analysis Services, Power BI, and Tableau for comprehensive BI support.
- Strong focus on data security principles including role-based access control, governance policies, and quality assurance.
- Expertise in troubleshooting, performance tuning, and monitoring production data pipelines.
- Proven leadership with mentoring experience and effective collaboration across diverse technical teams including Agile/Scrum environments.
- Exceptional analytical and problem-solving skills with a knack for designing scalable solutions.
- Bonus: Exposure to contemporary data platform concepts such as Data Fabric architectures.
Eligibility Criteria
The position welcomes candidates holding a B.Sc. in Information Technology or B.Tech/B.E. degrees in Computer Science Engineering.
Level
Lead
Minimum education
Bachelor's Degree