Snr Data Engineer
Job Summary
- Leverage extensive experience (4 to 8 years overall ETL experience to assist in solution design and delivery along with build of new ETLs).
- We are seeking an experienced ETL Developer with strong expertise in Big Data (Spark Cloudera).
- Experience orchestrating workflows using AWS Step Functions (state machines) for reliable and scalable data pipelines.
- Ability to implement end-to-end serverless data architectures integrating Glue Lambda S3 and Redshift
- Build and maintain high volume ETL/ELT pipelines across Hadoop (HDFS Hive Spark Kafka) and AWS (Glue EMR Lambda Step Functions Redshift).
- Develop distributed data processing solutions using PySpark Spark SQL and scalable cloud serverless patterns.
- Implement reusable data ingestion frameworks for batch ability to design & implement Orchestration process and Leverage AI
- Optimize data workflows using partitioning bucketing compression file formats (Parquet/ORC).
- Understanding hybrid data lake architectures using S3 HDFS ensuring data governance and best practices are adheres
- Experience to deliver complex projects in an Agile environment
- Assist in Design and build the robust scalable and secure software solutions across the having no/least adoption
- Define clear technical specifications and make architecture decisions that align with business goals and long-term scalability.
- Implement best practices (including secure code guidelines) through the implementation of unit tests automation leverage and code reviews. Drive continuous improvement in code quality and maintainability.
- Troubleshooting issues and proactively solving problems as they arise ensuring the smooth operation of full stack applications
- Ability to understand the data flow diagram data modelling and Lineages
- Job orchestration using Airflow Control M Step Functions or event-driven triggers.
- Ensure data is protected and compliant with regulatory standards.
- Work closely with business stakeholders to enable high quality datasets.
- Work on best practice adoption and provide guidance to peers/juniors in team.
- Ability to respond on incidents and troubleshooting Spark performance issues job failures and cluster bottlenecks.
- Collaborate closely with team members QA and cross product teams to streamline release processes.
- Collaborate with business stakeholders to gather analyse and translate data into technical solutions
- Strong experience with the AWS data stack (S3 Glue EMR Lambda Kinesis Redshift Step Functions etc.).
- Strong hands-on expertise in Scala PySpark Spark optimization techniques HiveQL and distributed computing.
- Good understanding of Hadoop ecosystem (HDFS Hive Spark YARN Kafka).
- Good work experience in SQL in hive and impala
- Proficiency in at least one scripting/programming language: Python Shell scripting.
- Strong experience with CI/CD GitHub Git commands.
- Expertise in ETL and Data Warehousing and cloud concepts.
- Good understanding of data modelling (star/snowflake) partitioning strategies and schema evolution.
- Expertise in data profiling and decision making.
- Able to understand design and create data flow diagrams.
- Able to understand the architecture and design end-to-end data flow.
- Hands-on experience with Airflow or ControlM or other orchestrators.
- To monitor and support BAU and year end activities if needed.
- Exposure to security and compliance aspects in Cloud.
- Familiarity with serverless patterns and containerization (Docker ECS/EKS).
- Strong logical and analytical problem-solving and communication skills.
- Communicate effectively and concisely with multiple stakeholders and coordinate and collaborate with cross functional teams.
- AWS certifications (Data Engineer or Developer) are a plus.
We offer you a competitive total rewards package continuing education & training and tremendous potential with a growing worldwide organization.
DISCLAIMER:
Nothing in this job description restricts managements right to assign or reassign duties and responsibilities of this job to other entities; including but not limited to subsidiaries partners or purchasers of Alight business units.
Required Experience:
IC
About Company
From hire to retire, and beyond, your people depend on you for resources that will enhance their health, wealth, and wellbeing. Put the power in their hands with data and technology that provides clarity through timely, broader connected insights. Drive better outcomes for your people ... View more