Databricks Engineer Architect

Position:Databricks Engineer / Architect
Employment Type:Contractual
Work Arrangement:Remote
Experience:10 years overall with 4 years of hands-on Databricks experience
Location:Remote
About the Role
We are seeking an experiencedDatabricks Engineer / Architectto design develop and optimize modern data platforms and large-scale data engineering solutions using Databricks and cloud technologies.
The ideal candidate will have strong hands-on expertise inDatabricks Apache Spark Python/SQL Delta Lake data architecture ETL/ELT pipelines and cloud platforms. This role requires someone who can operate at both the engineering and architecture levelstranslating business requirements into scalable secure and high-performance data solutions.
Key Responsibilities
Design and architect scalable reliable and high-performance data platforms usingDatabricks.
Develop and optimize data pipelines usingPySpark Python SQL Spark and Delta Lake.
Design modernLakehouse architecturesand implement enterprise-grade data solutions.
Develop batch and streaming data pipelines and integrate data from multiple sources.
ImplementDelta Lakecapabilities including schema evolution partitioning optimization and data lifecycle management.
Work withDatabricks Workflows Jobs notebooks clusters and related platform capabilities.
Design and implement data ingestion transformation and orchestration frameworks.
Optimize Spark workloads Databricks clusters SQL queries and data pipelines for performance and cost.
Establish and implement data engineering best practices coding standards CI/CD and deployment processes.
Implement appropriatesecurity governance access controls and data qualitymechanisms.
Collaborate with data scientists analysts application teams cloud engineers and business stakeholders.
Provide technical leadership and mentorship to data engineering teams.
Evaluate existing data architectures and recommend improvements modernization strategies and technology adoption.
Troubleshoot complex production issues and provide root-cause analysis and long-term solutions.
Participate in architecture reviews technical design sessions and documentation of enterprise data solutions.
Required Technical Skills
Databricks & Data Engineering
Strong hands-on experience withDatabricks
Advanced knowledge ofApache Spark / PySpark
Strong proficiency inPython and SQL
Extensive experience withDelta Lake
Experience designing and implementing scalable ETL/ELT pipelines
Strong understanding of Lakehouse architecture and modern data platforms
Experience with batch and real-time/streaming data processing
Cloud Technologies
Strong experience with at least one major cloud platform:
Microsoft Azure- Azure Data Lake Storage Azure Data Factory Azure Synapse Azure Key Vault Azure DevOps
AWS- S3 Glue Lambda IAM Redshift CloudWatch
GCP- Cloud Storage BigQuery Dataflow Pub/Sub IAM
Preferred Qualifications
Databricks certifications such asDatabricks Certified Data EngineerorDatabricks Certified Data Engineer Professional
Experience withUnity Catalogand Databricks governance capabilities
Experience implementing enterpriseMedallion Architecture (Bronze/Silver/Gold)
Experience with structured streaming and event-driven architectures
Knowledge of data modeling and dimensional modeling
Experience with Kafka or other messaging/streaming platforms
Experience with data quality frameworks and observability
Familiarity with MLOps and integration with machine learning platforms
Strong communication documentation and stakeholder-management skills
Education
Bachelors or Masters degree in Computer Science Information Technology Engineering or a related field.
Equivalent professional experience may be considered.
What Were Looking For
The successful candidate should be:
A stronghands-on Databricks Engineerwho can also think at the architecture level.
Comfortable working independently in afully remote environment.
Capable of translating complex business requirements into scalable technical solutions.
Strong in problem-solving troubleshooting and performance optimization.
An effective communicator who can work with both technical and non-technical stakeholders.
Comfortable providing technical leadership and driving architecture decisions.

Required Skills:

PYTHONCLOUD STORAGECI/CDDOCUMENTATIONSQLBIGQUERYDATA MODELINGSPARKAPACHE SPARKSQL QUERIES