Data Engineer

About the Role

We are seeking a proficient Data Engineer to architect, enhance, and maintain scalable data infrastructures that support analytics, AI, and machine learning applications. This role involves designing robust data pipelines, integrating diverse data sources, upholding data quality, and ensuring easy data access across the organization. It is well suited for professionals who enjoy tackling complex data challenges and collaborating with software engineers, data scientists, and product teams to convert raw data into actionable business insights.

Key Responsibilities

• Create, develop, and sustain scalable ETL/ELT pipelines handling both structured and unstructured data.
• Construct and oversee modern data warehouses and lakes.
• Implement dependable batch and real-time data processing systems.
• Aggregate data from APIs, databases, third-party platforms, and cloud services.
• Maintain high standards for data quality, integrity, security, and governance.
• Optimize data schemas and database performance for analytics and reporting needs.
• Monitor and troubleshoot to enhance the reliability and efficiency of data pipelines.
• Work jointly with Data Scientists, Machine Learning Engineers, Product Managers, and Software Developers to deliver data solutions.
• Apply CI/CD best practices and infrastructure automation in data workflows.
• Document data architecture, pipeline designs, and engineering protocols.
• Keep updated with the latest trends in cloud data engineering and big data technologies.

Required Qualifications

• Bachelor's degree in Computer Science, Software Engineering, Information Systems, or a related discipline.
• Minimum of four years’ experience in Data Engineering or backend/data platform development.
• Advanced skills in Python and SQL programming.
• Experience creating ETL/ELT pipelines using orchestration tools like Airflow, Prefect, or Dagster.
• Proficient knowledge of relational and NoSQL databases such as PostgreSQL, MySQL, MongoDB, or Cassandra.
• Hands-on experience with cloud platforms including AWS, Azure, or Google Cloud Platform.
• Familiarity with cloud data warehouses like Snowflake, BigQuery, Amazon Redshift, or Azure Synapse.
• Experience with distributed data frameworks such as Apache Spark.
• Understanding of streaming technologies such as Kafka or RabbitMQ.
• Experience using Docker, Kubernetes, and CI/CD pipelines.
• Strong grasp of data modeling, partitioning, indexing, and performance tuning.
• Experience with Git and collaborative development workflows.

Preferred Skills

• Background in building data platforms for AI or machine learning workloads.
• Familiarity with Delta Lake, Apache Iceberg, or Apache Hudi.
• Experience with dbt for analytics engineering tasks.
• Knowledge of Terraform or Infrastructure as Code practices.
• Understanding of data governance, metadata management, and data cataloging methodologies.
• Experience working within fast-growing technology or AI companies.

Technical Stack

• Programming Languages: Python, SQL
• Databases: PostgreSQL, MySQL, MongoDB
• Data Processing Frameworks: Apache Spark, Pandas
• Orchestration Tools: Apache Airflow, Prefect, Dagster
• Streaming Platforms: Kafka, RabbitMQ
• Cloud Providers: AWS, Azure, GCP
• Data Warehouses: Snowflake, BigQuery, Redshift, Synapse
• Containerization: Docker, Kubernetes
• Version Control: Git
• Infrastructure Management (preferred): Terraform

Desired Attributes

• Strong analytical aptitude and problem-solving capabilities.
• Deep understanding of scalable data architectures.
• Independent work ethic suited for a fast-paced environment.
• Effective communication and teamwork skills.
• Dedication to building reliable and high-performance data systems.
• Eagerness to stay updated with cutting-edge data technologies.

Additional Advantages

• Experience supporting Generative AI or large language model applications.
• Knowledge of vector databases such as Pinecone, Weaviate, or Milvus.
• Familiarity with data observability tools like Monte Carlo or Great Expectations.
• Experience in event-driven architectures and real-time analytics.
• Exposure to MLOps platforms and feature store technologies.

Why Join the Company?

• Opportunity to build next-generation AI and data-centered products.
• Collaborate with a highly skilled engineering team.
• Influence the architecture of modern data platforms.
• Work with extensive cloud infrastructure and advanced analytics.
• Competitive salary and substantial chances for career development.