Senior Databricks Data Engineer

About Us
Capital Farm Credit is the largest rural lending cooperative in Texas serving 192 counties through nearly 70 credit offices. With over $12 billion in assets and more than 600 team members we provide essential financial services to farmers ranchers rural homeowners and agribusinesses. As part of the nationwide Farm Credit System we are dedicated to supporting rural communities and agriculture.

Why Join Us
We seek motivated individuals who share our core values: commitment trust value and family-like respect. As a customer-owned cooperative we align employee success with member success offering competitive pay growth opportunities and a supportive environment.

Our Benefits:

• Incentive Program: Company-wide goals-based rewards.
• Accrued Time Off: Earn 13 days of annual leave and 15 days of sick leave per year plus enjoy 10-12 paid holidays annually.
• Retirement: 401(k) with up to 9% employer contribution/match.
• Health Coverage: Affordable medical dental and vision plans.
• Parental Leave: 8 weeks of paid parental leave.
• Life & Disability Insurance: Employer-paid coverage.
• Education & Wellness: Tuition reimbursement and up to $400 for wellness expenses.

At Capital Farm Credit youll find more than a jobyoull find purpose.

EDUCATION AND EXPERIENCE

• Bachelors degree in computer science Information Systems Engineering Data Analytics or a related field or an equivalent combination of education and experience.
• Five (5) or more years of hands-on data engineering experience using Azure Databricks or comparable cloud data platforms including two (2) or more years serving as a senior engineer technical lead or mentor on data engineering teams or projects.
• Demonstrated experience in Databricks Spark PySpark Delta Lake Unity Catalog SQL Python notebooks jobs clusters and Lakehouse architecture is required.
• Experience with Azure Data Lake Storage (ADLS) Azure SQL Key Vault managed identities ETL/ELT processes dimensional modeling and cloud security fundamentals is required.
• Experience designing data solutions that support analytics machine learning artificial intelligence governed enterprise reporting and secure data consumption use cases is also required including knowledge of AI/ML data preparation practices such as feature engineering model-ready data design data quality lineage and secure access to sensitive data.
• Databricks Certified Data Engineer Associate certification or an equivalent current Databricks data engineering certification is required;
• Databricks Certified Data Engineer Professional certification is preferred.

JOB SUMMARY

• The Senior Databricks Data Engineer designs develops and supports scalable secure and governed cloud-based data solutions that enable enterprise analytics business intelligence regulatory reporting artificial intelligence initiatives and data products. This role serves as a senior technical resource for the enterprise data platform and provides day-to-day technical leadership to a team of data engineers including work planning and assignment design and code review mentoring and escalation support for complex production issues.
• In addition to building maintaining and optimizing production data pipelines this position leads the design and development of the organizations enterprise data platform using Azure and Databricks. The position applies and establishes modern Lakehouse engineering practices and technologies including Databricks Apache Spark PySpark Delta Lake Unity Catalog Azure Data Lake Storage SQL and Python with an emphasis on data governance performance automation security and production reliability. The Senior Databricks Data Engineer also drives data solution architecture by establishing and applying Lakehouse and Medallion architecture patterns developing governed and AI-ready data assets creating reusable engineering frameworks and supporting secure and consistent data consumption across the organization.

ESSENTIAL FUNCTIONS

• Provide day-to-day technical leadership and direction to a team of data engineers including assigning and prioritizing work estimating effort tracking delivery and removing technical obstacles in partnership with the manager.
• Mentor and develop engineers through design guidance pair programming knowledge sharing and structured onboarding of new team members contractors and partners.
• Lead technical design sessions and establish engineering standards patterns and best practices for ingestion transformation governance deployment observability and data consumption.
• Own technical quality across the team by conducting design and code reviews and enforcing testing documentation and release standards.
• Partner with data analytics application infrastructure security and AI teams to architect the overall enterprise data solution using Azure and Databricks.
• Design and maintain batch and near-real-time pipelines using Databricks ADF Spark/PySpark SQL REST APIs and ADLS.
• Build governed Bronze Silver and Gold data layers using Delta Lake Medallion Architecture schema enforcement incremental processing and data quality controls.
• Implement Unity Catalog RBAC lineage classification and secure data access/sharing across Databricks environments.
• Deliver reusable data products supporting Power BI enterprise and regulatory reporting analytics and AI/ML.
• Design AI-ready data pipelines and curated datasets that support machine learning generative AI feature engineering retrieval-augmented generation vector search and advanced analytics use cases.
• Support Databricks AI and ML capabilities including MLflow feature engineering patterns model-ready datasets model lifecycle support model monitoring and responsible AI governance controls.
• Manage orchestration scheduling monitoring alerting error handling testing and production support and serve as a senior escalation point for complex production incidents and root cause analysis.
• Optimize Spark SQL clusters storage and compute for performance reliability and cost efficiency.
• Implement solutions using Azure DevOps Git CI/CD automated testing and environment promotion.
• Collaborate with analytics application security infrastructure and business teams to deliver trusted enterprise data.
• Contribute to reference architectures reusable frameworks platform standards and implementation patterns for ingestion transformation governance deployment observability and data consumption.
• Support platform planning activities by providing roadmap input level-of-effort estimates resource considerations and assessments of technical risk.

REQUIRED SKILLS

• Ability to lead mentor and develop technical staff and to guide the work of other engineers without direct supervisory authority.
• Ability to lead technical design discussions evaluate alternatives build consensus and make sound architecture and implementation decisions.
• Ability to plan prioritize and coordinate work across multiple engineers concurrent projects and competing deadlines.
• Ability to communicate technical concepts clearly and collaborate with data analysts BI developers data scientists IT teams business stakeholders and other technical partners.
• Advanced proficiency with Databricks Delta Lake Unity Catalog Medallion architecture and Lakehouse design principles.
• Advanced skills developing troubleshooting and optimizing Spark and PySpark workloads.
• Strong programming and query-development skills using SQL and Python for data engineering transformation and automation.
• Hands-on knowledge of data pipelines in Databricks Azure Data Lake Storage (ADLS) Azure SQL Key Vault and managed identities.
• Ability to design build maintain and optimize scalable ETL/ELT pipelines and production data workflows.
• Knowledge of dimensional modeling data integration data layers and reusable enterprise data architecture patterns.
• Understanding of data governance lineage access controls data quality sensitive data protection and cloud security practices.
• Ability to identify and resolve performance issues involving Spark workloads SQL queries clusters and data pipelines.
• Working knowledge of Azure DevOps Git source control automated testing CI/CD pipelines release management and environment promotion.
• Knowledge of preparing governed high-quality model-ready datasets for machine learning and AI applications.
• Ability to diagnose data pipeline failures identify root causes implement solutions and maintain reliable production environments.
• Ability to analyze complex technical and data issues and develop scalable practical solutions.

DISCLAIMER

We are an Equal Employment/Affirmative Action employer. We do not discriminate in hiring on the basis of sex race color religious creed national origin physical or mental disability protected Veteran status or any other characteristic protected by federal state or local law. If you need a reasonable accommodation for any part of the employment process please contact us by email at and let us know the nature of your request and your contact information. Requests for accommodation will be considered on a case-by-case basis. Please note that only inquiries concerning a request for reasonable accommodation will be responded to from this e-mail address. For more information view the EEO - Know Your Rights and Pay Transparency Statement.

Applicants should personally complete and submit their application materials. Submissions generated through automated tools or third-party mass application services may not be reviewed.

Equal Opportunity Statement
Capital Farm Credit is committed to creating a diverse and inclusive workplace. The position title and requirements may be adjusted based on the candidates experience and qualifications. We welcome applicants of all backgrounds and do not discriminate based on race color gender religion national origin disability veteran status or any other protected status. A full job description is available upon request. Candidates selected for hire will be required to complete a background check including criminal history education verification and employment verification. A credit check will be required for roles that require NMLS registration.