Databricks Architect (Cosmos Vector DB Engineer (AI Serving Layer))

Databricks Architect (Cosmos / Vector DB Engineer (AI Serving Layer))

**Job Title: Databricks Data Architect** **Location:** New Jersey (Hybrid) **Responsibilities:** - Build and maintain ingestion frameworks (ADF / Databricks / Spark). - Implement Bronze to Silver transformations aligned to architecture. - Architect data quality checks schema validation and contract rules. - Optimize pipelines for performance and cost. - Own CI/CD pipelines (Azure DevOps). - Automate onboarding (repos pipelines policies templates). - Maintain Infrastructure as Code (IaC) and scripting (Terraform PowerShell / Python). - Troubleshoot pipeline failures and permission issues. - Design and implement curated data models in the Gold layer aligned to business use cases. - Build dimensional models (star/snowflake schemas) fact tables dimensions surrogate keys and handle Slowly Changing Dimensions (SCD). - Ensure consistency across domains (common dimensions conformed models) reusability and scalability of models and alignment with the enterprise semantic layer (Power BI / Fabric / downstream tools). - Implement governance patterns defined by architecture. - Enforce Unity Catalog standards naming conventions metadata policies and access controls (RBAC/ABAC). - Support delivery teams with onboarding debugging and design reviews. - Provide hands-on guidance and create reusable assets to reduce repeat questions. ** Skills:** - **Core Platform:** - Azure Data Platform (ADF ADLS Azure SQL). - Databricks (DLT Delta Lake Spark Workflows Unity Catalog). - Microsoft Fabric (preferred). - **Engineering:** - Advanced PySpark / Python. - Strong SQL. - Pipeline design (batch streaming). - **DevOps / Automation:** - Azure DevOps (pipelines repos policies). - PowerShell automation. - CI/CD and Infrastructure-as-Code concepts. - **Governance / Data Quality:** - Schema enforcement data contracts and data quality frameworks. - Understanding of metadata-driven design.