Senior Databricks Platform Engineer (Remote US)

The Work

ICF is seeking a Senior Databricks Platform Engineer to support an independent US Government Agency where you will design build secure and operate governed Azure Databricks Lakehouse capabilities supporting a cloud-based analytics platform enterprise AI operations and emerging data initiatives.

You will develop reliable batch and streaming pipelines administer Databricks workspaces and access implement governance and automate delivery through infrastructure as code and CI/CD. Working with cloud architects AI engineers security analysts Business Analysts and mission owners you will help move data products from ingestion through validated analysis-ready datasets and production operations.

Job Location: Remote

What You Will Do

• Design configure and operate Azure Databricks workspaces clusters jobs workflows catalogs storage credentials external locations and service principals for enterprise initiatives.
• Build and optimize batch and real-time ingestion and transformation pipelines using Python PySpark SQL Delta Lake Delta Live Tables Auto Loader and Apache Spark.
• Implement medallion architecture dimensional models change data capture schema evolution data quality checks retry and rollback mechanisms and reproducible processing patterns.
• Configure Unity Catalog governance including role-based access lineage retention catalog and schema provisioning and secure integration with ADLS Gen2 and other approved storage.
• Develop frameworks and interfaces that enable AI and analytics teams to access governed training inference and reporting data across CDMA AlphaRex and future DMS initiatives.
• Automate Databricks deployments and environment configuration using Terraform Databricks Asset Bundles GitHub Actions Azure DevOps or comparable tools.
• Integrate Databricks with Azure Data Factory Kafka Airflow APIs Snowflake Synapse and other enterprise data services as required by the solution architecture.
• Monitor pipeline cluster and workspace health; diagnose performance and data issues; and use Azure Monitor Databricks logs Splunk or comparable tools to support root cause analysis and operational reliability.
• Support security ATO vulnerability remediation business continuity disaster recovery and operational readiness activities for Databricks environments and data products.
• Participate in Agile planning code reviews demos and retrospectives; document architecture runbooks data models governance decisions and deployment procedures in version-controlled repositories.

What You Will Bring With You

• BA/BS/MS in Computer Science Engineering Data Science or equivalent experience with more than 5 years of data engineering experience and at least 3 years specifically supporting Azure Databricks and related Azure data services.
• Expertise in Apache Spark Delta Lake Python SQL and Scala with hands-on experience designing and optimizing large-scale ETL/ELT pipelines.
• Experience with Azure Data Factory Azure Data Lake Storage Azure Synapse Analytics Databricks workflows data modeling and big data technologies.
• Experience configuring and managing Databricks workspaces clusters compute resources security controls access management and cost optimization.
• Experience implementing CI/CD or infrastructure as code for data platforms using Git and tools such as Terraform Databricks Asset Bundles GitHub Actions Azure DevOps or comparable technologies.
• Experience with Agile delivery data governance data security performance troubleshooting and clear communication with technical and business stakeholders.
• Ability to satisfy FDIC personnel eligibility and background-investigation requirements
• U.S. Citizenship required due to federal contract requirements.
• Candidate must reside in the U.S. be authorized to work in the U.S. and all work must be performed in the U.S.

What We Would Like You To Bring With You

• Experience with Azure Databricks ADLS Gen2 Unity Catalog Delta Live Tables Auto Loader Azure Data Factory and Azure Synapse.
• Databricks Certified Data Engineer Associate or Professional certification Databricks Generative AI Engineer certification or comparable demonstrated experience.
• Experience with streaming and real-time data using Kafka Confluent Structured Streaming or similar technologies.
• Experience with medallion architecture CDC dimensional modeling Apache Iceberg Snowflake Oracle SQL Server or other enterprise data platforms.
• Experience supporting Generative AI NLP RAG applications vector stores agents or Databricks AI/ML workloads.
• Experience with MLflow model deployment pipelines AWS SageMaker AWS Bedrock or cross-cloud data and AI architectures.
• Experience implementing BCDR high availability disaster recovery data retention and enterprise security controls for cloud data platforms.
• Experience supporting ATO processes NIST or federal compliance controls vulnerability remediation SonarQube Checkmarx or Zero Trust implementation.
• Experience with Docker Kubernetes REST APIs FastAPI or cloud-native application integration.
• Experience supporting federal data modernization programs and communicating technical decisions to agency stakeholders.

Professional Skills

• Explain data platform architecture trade-offs risks and operational impacts to technical and nontechnical stakeholders.
• Work effectively across data engineering AI cloud security application and operations teams.
• Use structured problem solving to diagnose pipeline performance access and deployment issues.
• Organize work across concurrent Agile priorities and maintain clear technical documentation and status communication.

Bot and Third-Party Applications

Please note that this application must be submitted directly by the applicant for consideration.Failure to do so may result in the application being excluded for consideration.Applicants needing an accommodation for disability or religious purposes in connection with the application process should contactfor assistance.

Working at ICF

Pay Range - There are multiple factors that are considered in determining final pay for a position including but not limited to relevant work experience skills certifications and competencies that align to the specified role geographic location education and certifications as well as contract provisions regarding labor categories that are specific to the position.

The pay range for this position based on full-time employment is: