Pyspark Python Developer

Role Purpose

The purpose of this role is to develop and test software modules based on client requirements and prepare project documentation while coordinating with cross-functional teams to ensure quality delivery.

We are seeking a highly skilled Senior PySpark Developer with strong expertise in Python programming PySpark SQL UNIX scripting and ELT/ETL data processing. The ideal candidate should have hands-on experience building scalable data pipelines processing large datasets and developing batch and streaming solutions in distributed environments. The candidate should be capable of independently designing developing optimizing and supporting data engineering solutions while effectively collaborating with stakeholders.

Must-Have Skills

Strong Python programming skills with expertise in coding data processing and performance optimization.

Extensive hands-on experience in PySpark development including application design job execution performance tuning and troubleshooting.

Hands-on experience in SQL coding

Experience in ELT/ETL pipeline development and data integration frameworks.

Proficiency in UNIX/Linux Shell Scripting including creating modifying and troubleshooting scripts.

Experience with batch and real-time/streaming data processing using modern data engineering technologies.

Ability to handle large-scale data processing and optimize data workflows for performance and scalability.

Good understanding of distributed computing concepts and big data architecture.

Experience in requirement gathering solution design technical documentation and implementation.

Strong analytical and problem-solving skills.

Excellent verbal and written communication skills with the ability to work effectively with technical and business stakeholders.

Stakeholder Collaboration-Collaborate with cross functional teams to understand requirements support development activities check system compatibility and ensure solutions meet expected standards.