Senior Data Engineer
Noida, Uttar Pradesh, India · Full Time
Be the first to apply
- Experience
- 4+ yrs
- Salary
- INR 2,500,000 – INR 3,500,000 / year
- Openings
- 1
- Posted
- 4 weeks ago
- Work mode
- In office
- Education
- Computer Science Engineering degree
- Eligibility
- Candidates who possess a minimum of 4 years of professional experience and hold a Computer Science Engineering degree are eligible to apply.
- Resume
- Required to apply
Where you'll work
Sign in to tell us what does and doesn't work for you here — it sharpens every match we show you.
Job description
Position Overview
We are recruiting for a forward-thinking technology company that focuses on creating sovereign, secure, and scalable digital platforms encompassing AI, Web3.0, cloud solutions, and health technology. The role demands a proactive Senior Data Engineer to develop and sustain robust data pipelines, implement lake house infrastructures, and manage the analytics environment using cutting-edge open-source tools to power data insights and GenAI analytics.
Key Responsibilities
- Construct and uphold data pipelines leveraging Airbyte to ingest data from diverse sources.
- Configure data connectors for various databases like PostgreSQL, MySQL, MongoDB, as well as APIs and SaaS platforms.
- Deploy Change Data Capture mechanisms for real-time data synchronization, including Kafka streaming integration.
- Devise strategies for both incremental and full data load synchronization, ensuring comprehensive data validation and quality assessments.
- Create and manage Airflow-directed ETL workflows with error handling, retries, and monitoring features.
- Schedule and orchestrate complex, multi-layered data pipeline processes.
- Collaborate with external databases, APIs, and systems integration.
- Design and optimize ClickHouse database schemas aimed at OLAP workloads and enhance SQL query performance.
- Implement materialized views and data aggregation methods, alongside partitioning and data retention policy management.
- Develop data marts tailored for analytic scenarios.
- Establish data lake house architecture on object storage like Cloudian, managing data partitioning by criteria such as date and region.
- Work with Parquet and ORC file formats, enforcing access control and governance protocols.
- Support development and maintenance of BI dashboards using Apache Superset and Power BI tools.
Candidate Qualifications
- Minimum of four years of professional experience in data engineering or related fields.
- Bachelor’s degree in Computer Science Engineering or an equivalent discipline.
Additional Information
- Position is based in Noida, Uttar Pradesh, requiring on-site presence.
- Immediate availability to join is preferred.
- Salary range between ₹2,500,000 and ₹3,500,000 per annum.
Level
Senior
Minimum education
Bachelor's Degree