Senior Databricks Engineer
Job Summary
- Design develop and maintain scalable data pipelines using Azure Databricks and Apache Spark.
- Build and optimize ETL/ELT processes for large-scale data integration and transformation.
- Develop and manage Delta Lake architectures to ensure data reliability governance and performance.
- Implement and maintain data ingestion frameworks from various structured and unstructured data sources.
- Collaborate with data architects analysts and business teams to translate requirements into technical solutions.
- Monitor troubleshoot and optimize Databricks workloads for performance and cost efficiency.
- Ensure adherence to data governance security and compliance standards.
- Automate deployment processes using CI/CD best practices.
- Mentor junior engineers and contribute to technical design discussions and architecture reviews.
- Participate in code reviews and drive engineering best practices across the team.
Required Skills:
Required Skills & Experience: Proven experience as a Solution Architect or similar role across multiple domains Strong expertise in Cloud platforms (AWS Azure or GCP) Experience in Data Architecture Data Engineering and Analytics platforms Knowledge of AI/ML solutions and frameworks Solid background in Software Engineering including microservices architecture and APIs Experience with Digital platforms customer experience solutions or front-end ecosystems Strong understanding of system integration security and scalability Excellent stakeholder management and communication skills
Required Education:
Bachelors or Masters degree in Computer Science Software Engineering Data Science or a related field.