Python Spark AI Developer

We are looking for a skilled Python / Spark / AI Developer to join a growing technology team.

You will be responsible for developing scalable data pipelines replatforming legacy SQL workloads onto modern Spark and lakehouse technologies and supporting the development of APIs and AI-driven features.

The ideal candidate is a strong Python developer with hands-on production experience in Apache Spark/PySpark SQL Delta Lake and REST APIs.

Key Responsibilities

• Build Spark/PySpark data pipelines using Delta Lake.
• Replatform legacy T-SQL logic into Spark SQL.
• Develop modern type-safe and well-tested Python applications.
• Build REST APIs using FastAPI or equivalent frameworks.
• Implement authentication idempotency and job-status functionality.
• Validate migrated pipelines against legacy systems and provide parity evidence.
• Work from technical specifications design documents and ADRs.
• Develop automated tests using pytest.
• Contribute to CI/CD pipelines code reviews and technical documentation.

Minimum Requirements

• 4 years of professional Python development experience.
• 2 years of production experience with Apache Spark / PySpark.
• Strong knowledge of Spark SQL and DataFrame APIs.
• Production experience with Delta Lake or equivalent technologies such as Iceberg or Hudi.
• Strong SQL skills and experience working with legacy T-SQL.
• Experience with pytest and automated testing.
• Experience developing REST APIs preferably with FastAPI.
• Experience with Docker and containerised development environments.
• Strong Git and CI/CD experience.
• Experience with type-hinted Python and static analysis tools such as Pyright or Mypy.
• Experience with modern Python tooling such as uv or Poetry.
• Ability to work independently from written technical requirements and design documentation.
• Bachelors degree in Computer Science Engineering or a related field or equivalent experience.
• Proven track record of delivering production Python/Spark pipelines independently.

Advantageous Skills

• Experience with LLM / AI integration.
• Experience with Ollama vLLM LM Studio or hosted AI APIs.
• Knowledge of data governance privacy engineering and POPIA/GDPR principles.
• Experience with Hive Metastore Trino or Apache Ranger.
• Azure experience including ADLS Gen2 Synapse ADF Key Vault or Service Bus.
• Experience with structured logging OpenTelemetry or OpenLineage.
• Data migration and pipeline parity testing experience.
• Databricks Certified Developer for Apache Spark or equivalent certification.

How To Apply:

Contact Hire Resolve today for your next career-changing move

Our client is offering a highly competitive salary for this role based on experience.

Send your CV to: or connect with Mischa Bornman via LinkedIn.

Alternatively you can also contact me directly at Hire Resolve

We will contact you telephonically in 3 days should you be suitable for this vacancy. If you are not suitable we will put your CV on file and contact you regarding any future vacancies that arise.