Data Engineer

Job Description:

We are seeking a highly skilled resource to design and implement high-performance event-driven data pipelines ensuring low-latency data processing and a high-availability system for the large credit card processing system. The ideal candidate will work with the Ab Initio ecosystem (GDE EME Conduct>It) to build stateful services that ingest filter and transform data from sources like Kafka or message queues pushing updates to dashboards or downstream databases in near-real-time.

Key Responsibilities:

• Create complex Ab Initio continuous flow graphs including stateful joins sliding time windows and aggregations.
• Implement event-driven data pipelines using Kafka MQ and file streams.
• Ensure the resilience of continuous flows including checkpointing and recovery to guarantee exactly-once processing.
• Apply advanced Ab Initio components (e.g. Reformat Rollup Join Partition) to ensure low-latency performance.
• Proactively monitor live production streams to ensure 24/7 reliability and troubleshoot data issues
• Develop ETL pipelines for batch and real-time data ingestion and transformation.
• Implement and ensure data validation data security integrity and compliance across big data platforms.
• Monitor and troubleshoot performance issues in large-scale clusters.
• Collaborate with data scientists analysts and application teams to deliver high-quality data solutions.
• Automate workflows and improve operational efficiency using scripting and orchestration tools.

Required Skills & Qualifications:

• Deep understanding of credit card processing systems
• Deep knowledge of GDE (Graphical Development Environment) EME (Enterprise Meta>Environment) Conduct>It and Continuous Flows.
• Understanding of Kafka message queues and real-time stateful services.
• Proficiency in Unix/Linux shell scripting SQL and database technologies (e.g. Oracle Teradata).
• Experience in Java Scala Python or Kafka is a plus.
• Familiarity with Linux/Unix environments and shell scripting.
• Understanding of data security governance and compliance standards.

• Experience with cloud-based big data platforms
• Exposure to containerization (Docker Kubernetes) for big data workloads.
• Knowledge of CI/CD pipelines for data engineering projects.

Behavioral Skills:

• Good Communication skills
• 5 days Work from Office at Berkley Heights NJ
• Team Player
• Ability to work in a changing environment
• Strong problem-solving and analytical skills