Sr Data Engineer

Key Responsibilities

• Design and implement ELT/ETL solutions for batch and streaming ingestion integration refinement and publish patterns on the Lakehouse.
• Develop reusable data processing frameworks and configuration-driven pipelines using Python and PySpark (EMR Glue or comparable Spark runtimes).
• Build and maintain scalable orchestration workflows (e.g. Airflow) for production data delivery including retries historical loads and operational runbooks.
• Implement data quality checks validation frameworks and monitoring so data products meet defined contracts and SLAs.
• Apply DataOps practices: Git-based development CI/CD/CT for data pipelines automated testing and controlled promotion across environments.
• Contribute to data lifecycle practices (retention archival disaster recovery / resiliency considerations) in partnership with platform and governance teams.
• Support platform modernizationand cloud migration of legacy data flows into Lakehouse patterns (Iceberg on S3 governed catalog access).
• Collaborate with stakeholders to maptechnical designs to business processes non-functional requirements and consumption needs (Athena Redshift APIs exports streams).
• Establish and document standards naming/conventions and engineering practices; participate in Agile ceremonies and cross-team delivery.
• Provide technical leadership: mentor engineers conduct design and code reviews and continuously improve reliability performance and cost efficiency.

Minimum Qualifications:

• Bachelors degree in Computer Science Information Systems Engineering or a related technical field or equivalent work experience.
• 8 years of overall IT experience.
• 5 years of hands-on experience designing and developing enterprise-scale data engineering solutions.
• Strong experience developing scalable data pipelines and reusable frameworks using Python and PySpark.
• Experience implementing enterprise data ingestion integration and transformation solutions using AWS Glue dbt Apache Spark or comparable technologies.
• Strong understanding of data warehousing concepts dimensional modeling and modern data lake / Lakehouse architectures.
• Experience with cloud platforms such as AWS Microsoft Azure or Google Cloud Platform (AWS preferred).
• Strong SQL expertise with relational databases; familiarity with NoSQL databases is a plus.
• Experience with distributed data processing technologies such as Apache Spark Amazon EMR or Hadoop-based platforms.
• Experience using Git-based source control and Agile software development methodologies.
• Strong analytical problem-solving and communication skills with the ability to collaborate across technical and business teams.

Preferred Qualifications:

• Experience designing cloud-native data platforms using AWS services such as S3 Glue EMR Athena Redshift Lambda Lake Formation IAM and CloudWatch.
• Hands-on experience with Apache Iceberg (or similar open table formats) and governed Lakehouse patterns.
• Experience implementing CI/CD pipelines DataOps practices continuous testing (CT) and infrastructure automation (e.g. Terraform).
• Experience with workflow orchestration tools such as Apache Airflow MWAA AWS Step Functions or similar platforms.
• Experience developing RESTful APIs or other access layers and integrating with enterprise applications for data-product consumption.
• Experience building and supporting real-time or streaming platforms using Kafka Kinesis or Spark Structured Streaming.
• Strong understanding of data quality observability monitoring and automated validation frameworks.
• Knowledge of data governance metadata management lineage and enterprise data catalog solutions.
• Strong understanding of data security encryption access controls and healthcare regulatory compliance (HIPAA/PHI).
• Experience optimizing distributed workloads for scalability reliability and cloud cost efficiency.
• Experience mentoring engineers conducting design and code reviews and establishing engineering best practices.

Compliance and Regulatory Responsibilities: N/A

License/Certification: N/A

WE ARE AN EQUAL OPPORTUNITY EMPLOYER. HF Management Services LLC complies with all applicable laws and regulations. Applicants and employees are considered for positions and are evaluated without regard to race color creed religion sex national origin sexual orientation pregnancy age disability genetic information domestic violence victim status gender and/or gender identity or expression military status veteran status citizenship or immigration status height and weight familial status marital status or unemployment status as well as any other legally protected basis. HF Management Services LLC shall not discriminate against any disabled employee or applicant in regard to any position for which the employee or applicant is otherwise qualified.

If you have a disability under the Americans with Disability Act or a similar law and want a reasonable accommodation to assist with your job search or application for employment please contact us by sending an email to or your email please include a description of the accommodation you are requesting and a description of the position for which you are applying. Only reasonable accommodation requests related to applying for a position within HF Management Services LLC will be reviewed at the e-mail address and phone number supplied. Thank you for considering a career with HF Management Services LLC.

Know Your Rights

All hiring and recruitment at Healthfirst is transacted with a valid @ email address only or from a recruitment firm representing our Company. Any recruitment firm representing Healthfirst will readily provide you with the name and contact information of the recruiting professional representing the opportunity you are inquiring about. If you receive a communication from a sender whose domain is not @ or not one of our recruitment partners please be aware that those communications are not coming from or authorized by Healthfirst. Healthfirst will never ask you for money during the recruitment or onboarding process.

Hiring Range*:

• Greater New York City Area (NY NJ CT residents): $134600 - $194480

• All Other Locations (within approved locations): $119600 - $177905

As a candidate for this position your salary and related elements of compensation will be contingent upon your work experience education licenses and certifications and any other factors Healthfirst deems pertinent to the hiring decision.

In addition to your salary Healthfirst offers employees a full range of benefits such as medical dental and vision coverage incentive and recognition programs life insurance and 401k contributions (all benefits are subject to eligibility requirements). Healthfirst believes in providing a competitive compensation and benefits package wherever its employees work and live.

*The hiring range is defined as the lowest and highest salaries that Healthfirst in good faith would pay to a new hire or for a job promotion or transfer into this role.