Data Engineer Spark, Hadoop, Ozone CH, Flink

Job Title: Data Engineer - Spark Hadoop Ozone CH Flink

Location: Berkley Heights NJ (5 Days Onsite)

Duration: long term

Type: Contract W2 only

Experience: 7 - 10 Yrs

Job Description
We are seeking a highly skilled Big Data Engineer with strong experience in Apache Spark Hadoop ecosystem and Apache Ozone. The ideal candidate will design develop and optimize large-scale data processing systems ensuring high performance scalability and reliability for enterprise-level applications.
Key Responsibilities:

• Design and implement distributed data processing solutions using Apache Spark Hadoop Flink
• Develop and maintain Spark applications for data transformation aggregation and ETL processes using Scala Java or Python
• Utilize Apache Ozone for storing large-scale datasets ensuring efficient data access and management in a distributed environment
• Manage and optimize HDFS and Apache Ozone Kafka for scalable and fault-tolerant storage.
• Develop ETL pipelines for batch and real-time data ingestion and transformation.
• Implement and ensure data validation data security integrity and compliance across big data platforms.
• Monitor and troubleshoot performance issues in large-scale clusters.
• Collaborate with data scientists analysts and application teams to deliver high-quality data solutions.
• Automate workflows and improve operational efficiency using scripting and orchestration tools.

Required Skills & Qualifications:

• Strong expertise in Apache Spark (Core SQL Streaming).
• Hands-on experience with Hadoop ecosystem (HDFS YARN MapReduce).
• Proficiency in Apache Ozone for object storage and integration with Hadoop.
• Solid programming skills in Java Scala or Python.
• Experience with Hive HBase and Kafka is a plus.
• Knowledge of cluster management and resource optimization.
• Familiarity with Linux/Unix environments and shell scripting.
• Understanding of data security governance and compliance standards.
• Experience with cloud-based big data platforms
• Exposure to containerization (Docker Kubernetes) for big data workloads.
• Knowledge of CI/CD pipelines for data engineering projects.

Qualifications:

• Bachelors degree in computer science Software Engineering or a related field.
• Proficiency in business process modeling and documentation tools.
• Product implementation experience is preferred

About Next Gen Software Solutions:

Next Gen Software Solutions is a trusted provider of IT Staffing and consulting services dedicated to empowering businesses with cutting-edge technology solutions and exceptional talent. We specialize in delivering tailored IT consulting services innovative software solutions and connecting businesses with highly skilled IT professionals. Founded and led by a dedicated U.S. Army solider Next Gen Software Solutions is deeply rooted in the core values of integrity discipline commitment and experience-principles that guide every aspect of our operations.

Equal Employment Opportunity Statement:

Next Gen Software Solutions is an Equal Opportunity Employer. We are committed to fostering an inclusive and diverse workplace where all employees and applicants are treated respect and dignity. We do not discriminate based on race colour religion sex (including pregnancy sexual orientation or gender identity) national origin age genetic information veteran status or any other legally protected characteristic under applicable federal state or local laws.