Databricks Data Specialist R
Job Location:
Monthly Salary:
Posted:
Application Deadline:
Vacancies:
Job Summary
We are seeking a highly skilled Databricks Engineer to design develop and optimize scalable data engineering and analytics solutions on the Databricks Lakehouse Platform. The ideal candidate will possess strong expertise in Databricks PySpark and SQL with hands-on experience building batch and real-time data pipelines implementing Lakehouse architectures and ensuring data governance and performance optimization.
- Design develop and maintain end-to-end data pipelines using Databricks and PySpark.
- Build and implement Lakehouse architectures utilizing Bronze Silver and Gold data layers.
- Develop and manage Delta Lake solutions with ACID transactions schema enforcement and data reliability features.
- Create monitor and optimize Delta Live Tables (DLT) pipelines.
- Implement scalable and efficient data ingestion processes using Auto Loader.
- Develop and manage real-time data processing solutions using Structured Streaming.
- Orchestrate schedule and monitor data workflows using Databricks Workflows.
- Design and implement Lakehouse data models to support reporting analytics and business intelligence requirements.
- Establish and enforce data governance security and access controls using Unity Catalog.
- Optimize Spark jobs SQL queries and overall platform performance to ensure efficiency and scalability.
- Collaborate with cross-functional teams including data analysts architects and business stakeholders to deliver high-quality data solutions.
- Databricks Platform
- Delta Lake
- Delta Live Tables (DLT)
- Unity Catalog
- Databricks Workflows
- PySpark and Apache Spark
- Structured Streaming
- Auto Loader
- SQL
- Lakehouse Data Modeling
- Strong understanding of data engineering best practices and scalable data architectures
- Azure Data Factory (ADF)
- Azure Synapse Analytics
- Microsoft Purview
- Microsoft Fabric
- AWS Glue
- AWS Lambda
- AWS Step Functions
- Apache Airflow
- DBT
- Fivetran
- Informatica
- Apache Kafka
- Power BI
- Collibra
- Alation
- BigQuery
- Bachelors or Masters degree in Computer Science Data Engineering Information Technology or a related discipline.
- Proven experience in designing and implementing cloud-based data engineering solutions and scalable data pipelines.
- Strong analytical troubleshooting and problem-solving capabilities.
- Experience working in agile and collaborative environments.
- Excellent communication and stakeholder management skills.
- Hands-on experience with modern Lakehouse architectures and enterprise-scale data platforms.
- Strong understanding of data governance security and compliance frameworks.
- Experience delivering both batch and real-time data processing solutions.
- Ability to work independently while collaborating effectively across global teams.
Databricks PySpark Apache Spark Delta Lake Delta Live Tables (DLT) Unity Catalog Structured Streaming Auto Loader SQL Lakehouse Architecture Azure AWS Airflow Kafka Power BI
- Databricks Engineering: Lead Data Engineer
We are seeking a highly skilled Databricks Engineer to design develop and optimize scalable data engineering and analytics solutions on the Databricks Lakehouse Platform. The ideal candidate will possess strong expertise in Databricks PySpark and SQL with hands-on experience building batch and real-time data pipelines implementing Lakehouse architectures and ensuring data governance and performance optimization.
- Design develop and maintain end-to-end data pipelines using Databricks and PySpark.
- Build and implement Lakehouse architectures utilizing Bronze Silver and Gold data layers.
- Develop and manage Delta Lake solutions with ACID transactions schema enforcement and data reliability features.
- Create monitor and optimize Delta Live Tables (DLT) pipelines.
- Implement scalable and efficient data ingestion processes using Auto Loader.
- Develop and manage real-time data processing solutions using Structured Streaming.
- Orchestrate schedule and monitor data workflows using Databricks Workflows.
- Design and implement Lakehouse data models to support reporting analytics and business intelligence requirements.
- Establish and enforce data governance security and access controls using Unity Catalog.
- Optimize Spark jobs SQL queries and overall platform performance to ensure efficiency and scalability.
- Collaborate with cross-functional teams including data analysts architects and business stakeholders to deliver high-quality data solutions.
- Databricks Platform
- Delta Lake
- Delta Live Tables (DLT)
- Unity Catalog
- Databricks Workflows
- PySpark and Apache Spark
- Structured Streaming
- Auto Loader
- SQL
- Lakehouse Data Modeling
- Strong understanding of data engineering best practices and scalable data architectures
- Azure Data Factory (ADF)
- Azure Synapse Analytics
- Microsoft Purview
- Microsoft Fabric
- AWS Glue
- AWS Lambda
- AWS Step Functions
- Apache Airflow
- DBT
- Fivetran
- Informatica
- Apache Kafka
- Power BI
- Collibra
- Alation
- BigQuery
- Bachelors or Masters degree in Computer Science Data Engineering Information Technology or a related discipline.
- Proven experience in designing and implementing cloud-based data engineering solutions and scalable data pipelines.
- Strong analytical troubleshooting and problem-solving capabilities.
- Experience working in agile and collaborative environments.
- Excellent communication and stakeholder management skills.
- Hands-on experience with modern Lakehouse architectures and enterprise-scale data platforms.
- Strong understanding of data governance security and compliance frameworks.
- Experience delivering both batch and real-time data processing solutions.
- Ability to work independently while collaborating effectively across global teams.
Databricks PySpark Apache Spark Delta Lake Delta Live Tables (DLT) Unity Catalog Structured Streaming Auto Loader SQL Lakehouse Architecture Azure AWS Airflow Kafka Power BI
Required Experience:
IC
About Company
Brillio is a global leader in Enterprise Digital Transformation Solutions, providing strategic consulting services and solutions using emerging technologies.
