DataBricks Lead

Position : Databrick Lead

Location : Remote

Term : C2C/W2 role

Duration : 12 months

Job Description:

We are looking for an experienced Databricks Lead to design develop and lead enterprise-scale data engineering and analytics solutions on the Databricks platform. The ideal candidate should have strong expertise in Databricks Spark cloud technologies ETL pipelines and modern data architecture. The candidate will work closely with cross-functional teams mentor engineers and drive end-to-end implementation of scalable data solutions.

Key Responsibilities

• Lead the design and development of scalable data solutions using Databricks.
• Build and optimize ETL/ELT pipelines using PySpark Spark SQL and Delta Lake.
• Work with large-scale structured and unstructured datasets.
• Design data ingestion frameworks from multiple data sources.
• Collaborate with business stakeholders architects analysts and engineering teams.
• Implement performance tuning and optimization techniques in Databricks.
• Develop and maintain data workflows and orchestration pipelines.
• Ensure data quality governance security and compliance standards.
• Lead code reviews technical discussions and best practices implementation.
• Mentor junior developers and provide technical leadership.
• Support cloud-based data platforms on AWS Azure or GCP.
• Participate in requirement gathering solution design and production support activities.

Required Skills

• Strong hands-on experience with Databricks
• Expertise in PySpark and Spark SQL
• Experience with Delta Lake
• Strong SQL and data warehousing concepts
• Experience in ETL/ELT development
• Knowledge of cloud platforms such as Azure AWS or GCP
• Experience with data modeling and big data processing
• Hands-on experience with orchestration tools like Airflow or ADF
• Knowledge of CI/CD pipelines and DevOps practices
• Strong analytical and problem-solving skills
• Excellent communication and leadership abilities

Preferred Skills

• Experience with Kafka or streaming technologies
• Knowledge of Unity Catalog and Databricks governance
• Experience with Snowflake Synapse or Redshift
• Familiarity with Terraform or Infrastructure as Code
• Agile/Scrum experience

Qualifications

• Bachelors degree in Computer Science Information Technology or related field
• 8 years of overall IT experience
• 4 years of hands-on experience with Databricks and Spark technologies
• Experience leading offshore/onshore teams is preferred

Nice to Have

• Databricks Certification
• Cloud Certifications (AWS/Azure/GCP)
• Experience in Healthcare Banking Retail or Telecom domains

Responsibilities in Leadership Role

• Lead technical discussions and architecture decisions
• Coordinate with clients and stakeholders
• Drive project delivery timelines and quality standards
• Provide mentoring and technical guidance to the team
• Identify process improvements and automation opportunities

Cloud BC Labs Inc is a digital transformation organization aimed at creating seamless solutions for clients to effectively manage their business operations. The company specializes in Business and Management Consulting AI/ML Data Analytics & Visualization Cloud Data Warehouse Migration Snowflake Implementation Informatica Implementation & Upgrade Staffing Services and Data Management Solutions