AWS Data Engineer

AWS Data Engineer - Python AWS Glue PySpark & Attunity

We are seeking an experienced AWS Data Engineer with strong expertise in Python AWS Glue PySpark Attunity (Qlik Replicate) and AWS Cloud to design develop and maintain scalable enterprise data integration solutions. The ideal candidate will have hands-on experience building ETL pipelines developing cloud-native data lakes implementing real-time and batch data replication and automating infrastructure using AWS services.

The role requires collaboration with business analytics and application teams to deliver secure scalable and high-performing data engineering solutions while following DevOps and CI/CD best practices.

• 5-8 Years of experience in Data Engineering
• 3 Years of experience with AWS Data Services
• Experience with enterprise ETL/ELT and cloud migration projects

• Design develop and maintain scalable ETL/ELT pipelines using AWS Glue PySpark and Python.
• Build and optimize data transformation workflows for processing large-scale datasets.
• Develop reusable data ingestion and transformation frameworks.
• Implement batch and real-time data processing pipelines.

• Use Attunity (Qlik Replicate) for real-time and batch data replication.
• Integrate data from on-premises systems to AWS Cloud.
• Support heterogeneous source systems and cloud-based data platforms.

• Design and manage enterprise data lakes using Amazon S3.
• Ensure data quality security governance and lifecycle management.
• Implement serverless data processing using AWS Lambda.
• Integrate Glue Lambda S3 and other AWS services for end-to-end workflows.

• Configure and manage AWS IAM roles policies and permissions.
• Automate infrastructure provisioning using AWS CloudFormation.
• Support Infrastructure-as-Code (IaC) initiatives.
• Participate in CI/CD pipeline implementation and deployment automation.

• Monitor and optimize ETL jobs Glue workflows and cloud resources.
• Troubleshoot production issues and perform root cause analysis.
• Improve pipeline performance reliability and scalability.
• Maintain technical documentation and operational runbooks.

• Work with Business Analysts Data Architects Analytics teams and Application Developers.
• Gather business requirements and translate them into technical solutions.
• Participate in Agile ceremonies code reviews and design discussions.

• Amazon S3
• AWS Glue
• AWS Lambda
• AWS IAM
• AWS CloudFormation

• Python
• PySpark
• ETL / ELT
• Data Pipelines
• Data Integration
• Data Transformation

• Attunity (Qlik Replicate)
• Real-time Replication
• Batch Processing

• SQL
• Relational Databases
• Data Warehousing (preferred)

• Git
• CI/CD
• Infrastructure as Code (IaC)
• CloudFormation

• Performance Tuning
• Troubleshooting
• Root Cause Analysis
• Technical Documentation

• AWS Step Functions
• Amazon Redshift
• Amazon Athena
• AWS Lake Formation
• Apache Spark
• Airflow
• Terraform
• Docker
• Kubernetes
• Data Quality Frameworks

• Excellent communication skills
• Stakeholder management
• Strong analytical and problem-solving skills
• Leadership and ownership mindset
• Team collaboration
• Mentoring and coaching
• Planning and execution
• Risk management
• Customer-focused approach