PySpark Data Engineer – Python, ETL & Data Warehousing

Job Summary

Synechron is seeking a PySpark Data Engineer with 7 years of overall experience and at least 5 years of commercial experience in data-driven roles. The role will design develop test deploy and support scalable data pipelines data marts and data warehousing solutions using Python PySpark SQL and related data position will contribute to business objectives by delivering reliable data solutions improving data quality and accessibility supporting analytics and reporting and ensuring effective data processing across the full software development lifecycle.

Software Requirements

Required

• 7 years of overall professional experience in data engineering software development or related technology roles.

• 5 years of commercial experience in a data-driven role.

• Hands-on experience building data marts and ETL pipelines.

• Strong expertise in Python and PySpark for ETL scripting.

• Experience writing clean maintainable robust and testable Python code.

• Hands-on experience with Spark PySpark Hadoop MapReduce Hive and Pandas.

• Strong knowledge of SQL and Oracle query development.

• Experience working with SQL and NoSQL database management systems.

• Experience across the end-to-end software development lifecycle including:

  • Build and development.

  • User acceptance testing.

  • UAT defect resolution.

  • Production deployment.

  • Post-production support.

• Experience debugging PySpark code and investigating data processing issues.

• Strong understanding of data warehousing and data pipeline production practices.

• Ability to process structured semi-structured and unstructured data.

• Familiarity with Git CI/CD processes data testing and validation.

• Experience with data analysis data cleansing data linking imputation and feature engineering.

• Familiarity with workflow orchestration and scheduling tools.

• Experience collaborating with multiple technical and business teams.

Preferred

• Experience with Apache Airflow Oozie and Jenkins pipelines.

• Experience using Jupyter for data exploration prototyping and analysis.

• Knowledge of cloud-based data engineering platforms and services.

• Experience with data lake lakehouse distributed processing and streaming concepts.

• Familiarity with automated data quality monitoring and pipeline observability.

• Experience in banking financial services or other regulated data-intensive industries.

• Knowledge of data governance metadata management lineage security and privacy practices.

• Experience leading technical workstreams or coordinating delivery across multiple teams.

Overall Responsibilities

• Design develop test deploy and support scalable ETL pipelines and data marts using Python and PySpark.

• Build data processing solutions for structured semi-structured and unstructured data.

• Develop clean maintainable robust and reusable Python and PySpark code.

• Analyze business and technical requirements and translate them into data engineering solutions.

• Develop and optimize SQL and Oracle queries for data extraction transformation validation and analysis.

• Integrate data from multiple sources databases files and systems.

• Apply data cleansing data linking imputation transformation validation and feature engineering techniques.

• Support data warehouse development data modeling data integration and reporting requirements.

• Participate in build UAT UAT defect resolution production deployment and post-production support activities.

• Debug PySpark code investigate pipeline failures and resolve data quality and processing issues.

• Validate data outputs reconcile results and ensure that pipelines meet defined quality and business requirements.

• Collaborate with technical and non-technical stakeholders to clarify requirements resolve dependencies and deliver agreed outcomes.

• Participate in code reviews technical discussions testing deployment planning and production support activities.

• Identify opportunities to improve pipeline performance automation reliability maintainability and resource efficiency.

• Maintain technical documentation covering data flows pipeline logic data models dependencies test evidence and operational procedures.

• Consider security data privacy cost management and sustainability when designing and operating data solutions.

Technical Skills (By Category)

Programming Languages

Essential

• Python using a current and supported version.

• PySpark for distributed data processing and ETL development.

• Strong understanding of Python functions modules object-oriented programming exception handling testing and package management.

• Ability to write clean maintainable robust reusable and testable code.

• SQL for data extraction transformation validation analysis and query optimization.

• Understanding of data structures algorithms and software engineering principles.

Preferred

• Shell scripting for automation and operational support.

• Experience developing reusable Python packages and data-processing utilities.

• Knowledge of programming practices for distributed and production-scale data applications.

Databases/Data Management

Essential

• Strong knowledge of relational databases and Oracle query development.

• Experience with SQL and NoSQL database management systems.

• Understanding of data warehousing data marts data modeling and data integration.

• Knowledge of structured semi-structured and unstructured data processing.

• Experience with data cleansing data linking imputation reconciliation transformation and validation.

• Understanding of data quality data integrity data lifecycle and metadata requirements.

• Ability to analyze large datasets and identify data inconsistencies or processing issues.

Preferred

• Experience with dimensional modeling fact and dimension tables and analytical data warehouse design.

• Knowledge of data lake and lakehouse architectures.

• Familiarity with data lineage metadata management and data governance.

• Experience with feature engineering and preparing data for analytics or machine learning use cases.

• Knowledge of database performance tuning and query optimization.

Cloud Technologies

Essential

• Understanding of cloud-based data engineering concepts and distributed data processing.

• Awareness of cloud storage compute networking access management monitoring and deployment considerations.

• Ability to support data pipelines across development test UAT and production environments.

Preferred

• Experience developing and deploying PySpark data pipelines on cloud platforms.

• Familiarity with cloud-based data lakes data warehouses managed databases and workflow services.

• Knowledge of cloud monitoring infrastructure automation identity management and security controls.

• Understanding of cost-efficient and sustainable use of cloud data-processing resources.

Frameworks and Libraries

Essential

• Apache Spark and PySpark.

• Hadoop MapReduce and Hive.

• Pandas for data analysis and transformation.

• Python libraries for database connectivity file handling data validation and automation.

• Experience developing ETL and data-processing frameworks.

• Understanding of distributed processing partitioning transformations actions and performance considerations.

Preferred

• Apache Airflow or Oozie for workflow orchestration.

• Jupyter for data analysis exploration and prototyping.

• Libraries supporting data quality testing feature engineering and statistical analysis.

• Familiarity with streaming or near-real-time data-processing frameworks.

Development Tools and Methodologies

Essential

• Experience across the end-to-end SDLC including build UAT defect fixing deployment and post-production support.

• Git for source code versioning branching merging and code review.

• Familiarity with CI/CD processes and automated build or deployment workflows.

• Experience with data testing validation reconciliation and defect management.

• Knowledge of Agile or iterative software delivery practices.

• Ability to document data flows transformation logic data dependencies test results and operational procedures.

• Experience coordinating with multiple teams to resolve dependencies and deliver project outcomes.

Preferred

• Jenkins pipeline experience.

• Experience with automated data quality checks and test execution.

• Familiarity with pipeline monitoring logging alerting and incident management.

• Knowledge of infrastructure as code and automated environment deployment.

• Experience with performance monitoring and optimization of production data pipelines.

Security Protocols

Essential

• Understanding of secure data handling and data protection principles.

• Awareness of authentication authorization identity and access management encryption secrets management and secure connectivity.

• Ability to apply appropriate access controls to data pipelines databases files and processing environments.

• Understanding of data privacy data integrity auditability and secure transfer practices.

Preferred

• Experience implementing security controls across cloud and on-premises data environments.

• Knowledge of data masking tokenization role-based access control and audit logging.

• Familiarity with vulnerability management security testing and compliance-related data controls.

• Understanding of secure configuration and monitoring practices for data platforms.

Experience Requirements

• 7 years of overall experience in data engineering software development or related technology roles.

• 5 years of commercial experience in a data-driven role.

• Experience building data marts and ETL pipelines.

• Strong hands-on experience with Python and PySpark for ETL scripting.

• Experience with Spark Hadoop MapReduce Hive Pandas SQL and Oracle queries.

• Experience working with SQL and NoSQL database technologies.

• Experience across build UAT UAT defect resolution production deployment and post-production support.

• Experience debugging PySpark code and resolving data pipeline data quality and production issues.

• Strong understanding of data warehousing and production data pipeline practices.

• Experience handling structured semi-structured and unstructured data.

• Experience with CI/CD Git data testing validation workflow scheduling and pipeline support.

• Experience in banking financial services or other regulated data-intensive industries is preferred.

• Candidates may also qualify through equivalent practical experience relevant certifications professional training or demonstrated delivery of complex data engineering solutions.

Day-to-Day Activities

• Design develop test and maintain Python and PySpark ETL pipelines data marts data transformations and data warehouse components.

• Collaborate with technical and non-technical stakeholders participate in Agile meetings clarify requirements and resolve cross-team dependencies.

• Perform data analysis Oracle query development PySpark debugging data validation UAT defect fixing and production support.

• Review pipeline results monitor delivery progress document technical outcomes recommend improvements and make implementation decisions within approved standards.

Qualifications

• Degree in Computer Science Information Technology Engineering or an equivalent discipline; equivalent professional experience may be considered.

• Minimum of 7 years of overall experience including at least 5 years of commercial experience in data-driven roles.

• Certifications in data engineering cloud technologies Python Spark Agile or database technologies are preferred.

• Complete Synechron-required training related to information security data protection data governance workplace conduct and responsible technology use.

• Maintain continuous professional development in Python PySpark Spark data warehousing cloud data platforms SQL automation security and data engineering practices.

Professional Competencies

• Critical thinking data analysis technical investigation and structured problem-solving.

• Technical ownership teamwork dependency coordination and delivery accountability.

• Clear communication with technical and non-technical stakeholders.

• Adaptability continuous learning and effective response to changing data and delivery requirements.

• Innovation focused on reliable maintainable automated scalable and sustainable data solutions.

• Effective prioritization organization time management and delivery under multiple deadlines.

SYNECHRONS DIVERSITY & INCLUSION STATEMENT

Diversity & Inclusion are fundamental to our culture and Synechron is proud to be an equal opportunity workplace and is an affirmative action employer. Our Diversity Equity and Inclusion (DEI) initiative Same Difference is committed to fostering an inclusive culture promoting equality diversity and an environment that is respectful to all. We strongly believe that a diverse workforce helps build stronger successful businesses as a global company. We encourage applicants from across diverse backgrounds race ethnicities religion age marital status gender sexual orientations or disabilities to apply. We empower our global workforce by offering flexible workplace arrangements mentoring internal mobility learning and development programs and more.

All employment decisions at Synechron are based on business needs job requirements and individual qualifications without regard to the applicants gender gender identity sexual orientation race ethnicity disabled or veteran status or any other characteristic protected by law.

Candidate Application Notice