Data Architect

The Data Architect is responsible for designing building and evolving the enterprise data foundation supporting the Firms artificial intelligence analytics and business applications. This position translates complex enterprise data requirements into practical scalable architectures and delivers reliable governed data that technology teams and business stakeholders can use with confidence. This is a technical role with primary responsibility for enterprise data modeling lakehouse architecture data integration governance and data quality. The Data Architect will design and implement solutions within Azure Databricks develop data pipelines and models and establish standards and reusable patterns for the Firms growing data environment. The position will partner closely with AI Engineering DevOps Information Security Knowledge Management and other technology and business stakeholders to ensure data is accessible secure traceable and appropriately governed.

KEY RESPONSIBILITIES

• Lead the design implementation and ongoing evolution of the Firms Azure Databricks data architecture including lakehouse layers storage data models integration patterns and the roadmap from current-state systems to the target architecture.
• Partner with attorneys practice groups business teams and technology stakeholders to identify priority data requirements define data products and establish measurable standards for data quality freshness availability and usability.
• Design conceptual logical and physical data models for enterprise information including client matter people document financial and operational data establishing consistent definitions identifiers relationships and standards in partnership with data owners.
• Design build and maintain scalable data ingestion and transformation pipelines using Python SQL Apache Spark Delta Lake and related technologies selecting appropriate batch incremental change-data-capture or streaming approaches based on business requirements.
• Integrate data from enterprise databases APIs files document repositories and other systems through supported interfaces including the development of source-to-target mappings data contracts reconciliation processes and controls for schema changes and deletions.
• Design and administer data governance within Unity Catalog including catalogs schemas ownership structures access policies lineage classification retention and audit requirements.
• Partner with Information Security and data owners to design implement and validate access controls that appropriately reflect source-system permissions client and matter restrictions ethical walls and other confidentiality requirements.
• Establish and maintain data quality standards automated validation monitoring recovery procedures and service expectations. Troubleshoot data and pipeline failures and optimize reliability query performance compute utilization storage and overall platform costs.
• Develop curated datasets and governed data interfaces supporting analytics AI agents retrieval-augmented generation and other AI-enabled workflows while maintaining appropriate source traceability and access controls.
• Design and implement Lakebase PostgreSQL data stores supporting agentic applications including persistent agent state checkpoints and memory with appropriate user and matter isolation transactional access patterns retention and recovery.
• Establish reusable architectural standards technical documentation and engineering patterns and provide technical guidance design review code review and mentorship to AI Engineers and other technical team members.
• Partner with DevOps and other technology teams to support secure environments automated deployments development/test/production processes monitoring operational readiness and long-term platform supportability.
• Remain current on developments in data architecture Azure Databricks cloud data engineering AI data infrastructure governance and related technologies recommending enhancements where appropriate.

REQUIRED EDUCATION KNOWLEDGE & EXPERIENCE

• Bachelors degree in Computer Science Data Engineering Information Systems Engineering or a related technical discipline or equivalent combination of education and relevant professional experience.

• Significant experience designing implementing and operating enterprise data architectures with demonstrated ability to evaluate tradeoffs involving integration governance security performance scalability and cost.

• Advanced hands-on experience with Azure Databricks Apache Spark Delta Lake and Unity Catalog in production environments.

• Strong experience designing conceptual logical and physical data models including dimensional modeling entity relationships shared business definitions and historical data management.

• Advanced proficiency with SQL and Python including PySpark and experience developing maintainable production data pipelines using version control automated testing and code review practices.

• Demonstrated experience integrating data from multiple enterprise systems including resolving inconsistent identifiers and definitions and implementing reliable incremental processing reconciliation and data validation.

• Strong understanding of Azure lakehouse architecture including layered raw validated and curated data structures and the appropriate selection of ingestion transformation storage and serving patterns.

• Experience designing and administering Unity Catalog environments including catalog and schema structures privileges managed and external data lineage and integration with Azure storage and identity controls.

• Working knowledge of Azure Data Lake Storage Gen2 and Microsoft Entra ID including managed identities secrets management authentication authorization and secure connectivity.

• Experience designing production-grade data pipelines incorporating orchestration incremental processing or change data capture schema evolution retries safe reprocessing monitoring and automated data quality validation.

• Experience with PostgreSQL data modeling and transactional design including persistent application or agent state access controls retention and data lifecycle management.

• Demonstrated knowledge of Spark and SQL performance optimization compute sizing cost management environment management monitoring and disaster recovery or operational recovery practices.

• Strong understanding of enterprise data governance security confidentiality classification lineage retention auditability and role-based access controls. Demonstrated ability to translate complex technical concepts and architectural decisions for both technical and non-technical audiences.

• Strong collaboration communication analytical and problem-solving skills with the ability to work effectively across technology and business functions.

• Ability to provide technical leadership and mentorship while remaining actively involved in architecture engineering development and implementation.

PREFERRED SKILLS & KNOWLEDGE

• Masters degree in Computer Science Data Engineering Information Systems or a related discipline.

• Experience with master and reference data management entity resolution data stewardship and enterprise data governance across complex systems.

• Experience developing governed data foundations for artificial intelligence machine learning AI agents and retrieval-augmented generation (RAG) including document preparation metadata management vector search or knowledge graphs.

• Experience with Databricks SQL business intelligence integrations semantic models and governed datasets supporting enterprise reporting and analytics. Experience with infrastructure as code and automated deployment technologies such as Terraform Azure DevOps or GitHub Actions.

• Experience modernizing legacy data platforms or migrating enterprise data environments to cloud-based architectures.

• Experience working in legal services professional services financial services or another environment involving highly sensitive information complex confidentiality requirements and sophisticated access controls.

• Experience in technical consulting or another business-facing technology delivery role requiring direct engagement with business stakeholders and senior leaders.

PHYSICAL REQUIREMENTS

• Ability to sit and stand for extended periods.

• Ability to lift up to 15 pounds.

The expected salary range for this position is $200000 - $230000. Final compensation will be determined based on several factors including but not limited to relevant experience qualifications skill set and geographic location.

Pillsbury Winthrop Shaw Pittman LLP is an Equal Opportunity Employer.

If you require an accommodation in order to apply for a position please contact us at .