Lead Data Engineer

Job Description

Role: Lead Data Engineer

Location: Onshore (Charlotte)

Who are we looking for

Were seeking an experienced pipeline-centric data engineer to put it to good use in building out ETL and Data Operations framework (Data Preparation / Normalization and Ontological processes).

Technical Skills:

• Lead the design development and maintenance of scalable data pipelines and ETL processes ensuring data integrity and accessibility for business intelligence and advanced analytics.
• Architect and manage robust data platforms on the AWS ecosystem leveraging services like Glue PySpark Apache Iceberg IAM S3 and Secrets Manager to build a secure and efficient data infrastructure.
• Provide technical guidance and mentorship to a team of data engineers fostering a culture of high performance and continuous learning.
• Utilize deep expertise in various RDBMS (e.g. MySQL Db2 PostgreSQL Snowflake) and different data formats (e.g. JSON Parquet) to drive strategic data initiatives.
• Champion the adoption of modern data technologies such as Apache Iceberg with AWS Glue to optimize data lake performance and analytics capabilities.
• Apply advanced proficiency in Generative AI to automate and streamline data analysis development and documentation processes.
• Business Acumen and Stakeholder Communication
• Translate complex business challenges into clear data-driven solutions effectively communicating technical concepts and project progress to both technical and non-technical stakeholders including senior leadership.
• Act as a key liaison between the data engineering team and business units providing data-backed insights and recommendations that directly influence business strategy.
• Manage concurrent projects in a dynamic research-oriented environment ensuring timely delivery and high-quality outcomes.
• Experience in insurance domain preferrable
• AWS Data Engineering certification good to have

Key Responsibilities

• Collaborate with business analysts and stakeholders to translate business needs into comprehensive source-to-target (S2T) data mappings.
• Lead the design and development of robust data pipelines using your deep understanding of existing ETL frameworks to build scalable and efficient solutions.
• Analyze and understand diverse source systems and data formats to create and validate synthetic datasets for testing and development.
• Develop automated data validation tools using Python to ensure data integrity across various ETL layers.
• Ensure Industry best practice is followed in all aspects of project.
• Develop robust development testing approach to minimize the quality integration and user testing issues.
• Leverage generative AI to enhance data analysis accelerate development Unit Testing and streamline documentation.
• Apply statistical analysis to large datasets uncovering key patterns identifying potential challenges and generating actionable insights.
• Partner with business leaders to understand their challenges and provide data-driven recommendations that improve processes and inform strategic decisions.

Qualification:

• Somebody who has at least 12 years of data engineering experience has played Lead Data Engineer role.
• Bachelors degree (or equivalent) in computer science information technology engineering or related discipline
• Education qualification: Any degree from a reputed college