Data Engineer / Python Developer / Data Analyst

About Aetherrise Solutions

Aetherrise Solutions Pvt. Ltd. is a progressive technology company dedicated to building innovative solutions that harness data, AI, and cloud computing for modern enterprises. Their expertise lies in converting raw data into meaningful insights through advanced machine learning models and scalable enterprise tools. They nurture a culture of innovation, ongoing education, and pragmatic problem-solving across Data Engineering, Data Analytics, AI, and Quality Assurance domains.

Role Overview and Responsibilities

• Design, develop, and maintain robust ETL/ELT data pipelines using technologies like Python, PySpark, SQL, AWS Glue, Azure Data Factory, and Apache Airflow.
• Build and manage cloud data platforms leveraging AWS and Microsoft Azure services to meet enterprise data engineering and analytics needs.
• Create comprehensive data ingestion workflows to source data from databases, APIs, files, streaming systems, and various structured and semi-structured formats.
• Apply PySpark, Apache Spark, Python, and SQL to develop efficient data transformation and processing solutions.
• Construct and optimize data lakes and warehouses using Amazon S3, Redshift, Snowflake, Azure Data Lake Storage, and Azure Synapse Analytics.
• Utilize a wide array of AWS services such as Lambda, EMR, Athena, Step Functions, SQS, SNS, and CloudWatch for data operations.
• Employ Azure tools including Databricks, Functions, Data Factory, ADLS Gen2, Synapse Analytics, and Azure Monitor for data workflows.
• Design and implement scalable data models, including star schema, fact, and dimension tables, for advanced analytics.
• Develop sophisticated SQL queries, stored procedures, and integrations to enhance reporting and analytics capabilities.
• Implement frameworks ensuring data quality, validation, reconciliation, and ongoing monitoring to maintain data integrity.
• Optimize Spark jobs, SQL queries, ETL processes, and cloud resource usage to boost performance and cost-effectiveness.
• Create Python-based automation scripts and utilities to streamline data processing and integration tasks.
• Configure and manage Apache Airflow for workflow orchestration, scheduling, and monitoring.
• Collaborate with cross-functional teams including business analysts, architects, developers, and stakeholders to fulfill data solution requirements.
• Support business analytics and BI efforts by delivering curated, reliable datasets.
• Implement version control and CI/CD practices via Git, GitHub, Azure DevOps, and cloud deployment pipelines.
• Diagnose and resolve issues related to pipeline failures, data inconsistencies, and production bottlenecks.
• Adhere to best practices in coding, testing, documentation, security, governance, and Agile/Scrum methodologies.

Preferred Candidate Profile

• Bachelor's or Master's degree in Computer Science, IT, Data Science, Engineering, or related disciplines.
• 3 to 8 years of experience in Data Engineering, Cloud Data Engineering, Data Analytics, or equivalent fields.
• Proficient programming skills in Python, SQL, and PySpark.
• Strong hands-on experience with AWS cloud data services (Glue, S3, Athena, Redshift, Lambda, EMR, Step Functions, CloudWatch).
• Practical knowledge of Microsoft Azure data services (Data Factory, Databricks, ADLS Gen2, Synapse Analytics, Azure Monitor).
• Solid understanding of ETL/ELT pipelines, data warehousing, data lakes, and big-data processing architectures.
• Experience in distributed data processing using Apache Spark and PySpark.
• Familiarity with Apache Airflow for workflow automation and orchestration.
• Knowledge of cloud-native data warehouses like Snowflake, Amazon Redshift, and Azure Synapse.
• Preferred familiarity with Databricks, Delta Lake, and Medallion Architecture concepts.
• Strong data modeling skills, including dimensional models and partitioning techniques.
• Experience with version control (Git), CI/CD, DevOps, Agile/Scrum workflows, and cloud security.
• Comfortable with data formats such as Parquet, JSON, CSV, and Avro.
• Excellent analytical, problem-solving, communication, and collaboration capabilities.
• Ability to work both independently and within fast-paced, collaborative teams.

Employee Benefits and Environment

• Flexible working arrangements including hybrid and remote opportunities.
• Comprehensive health insurance and wellness programs.
• Support for professional development, including certifications in AWS, Azure, Snowflake, Databricks, and related cloud technologies.
• Chance to engage in large-scale enterprise projects on AWS and Azure data platforms.
• Exposure to cutting-edge technologies in cloud computing, big data, analytics, and data engineering.
• Promising career advancement with continuous learning prospects.
• Employee-friendly policies promoting work-life balance including paid leaves and holidays.
• Recognition and rewards programs for employee contributions.
• Opportunities to work on challenging initiatives such as Data Engineering, Cloud Migration, Data Lake and Lakehouse architectures, and advanced Analytics projects.