Principal Data Engineer

Location: Philadelphia PA

Onsite Flexibility: Hybrid

• Position Type: Right to Hire
• Contract Duration: 6 month conversion
• Pay Rate: $85.00$90.00 / Hour (USD)
• Schedule: 2 days on-site Tuesday / Wednesday
• Work Authorization: Applicants must be authorized to work for ANY employer in the U.S. We are unable to sponsor or take over sponsorship of an employment Visa at this time.

This role is focused on building and leading the data engineering foundation that powers real-time decisioning operational applications analytics ML/AI model development and data services across a medical client. The Principal Data Engineer will own the design delivery and maturity of production-grade data pipelines and data platforms with a primary emphasis on real-time streaming IoT telemetry Databricks Azure data services for APIs and microservices and reliable data products for downstream consumption.

We are looking for a Principal Data Engineer to serve as a hands-on technical and people leader for data engineering data platform architecture real-time streaming and production data services. This role will focus on designing building operating and improving data pipelines and data products while also bringing principal-level judgment to architecture stakeholder shaping delivery priorities team management and production readiness.

This is a hands-on engineering leadership role first. The ideal candidate should be comfortable spending significant time working directly with Databricks Spark SQL Python/PySpark Azure services streaming architectures data quality frameworks pipeline automation CI/CD and production troubleshooting. They should also be able to operate with the maturity of a principal-level leader: shaping unclear requirements making pragmatic technical decisions managing and mentoring engineers and driving work forward without waiting for perfect specifications.

This is a fast-moving startup-like environment. Requirements may be incomplete priorities may evolve and the right candidate will help create clarity while building quickly. We need someone who can move from ambiguous business need to reliable data capability with urgency discipline and ownership.

Stakeholder shaping is a critical part of this role. The Principal Data Engineer should be able to work directly with business product software engineering analytics ML/AI operations and leadership stakeholders to define what data needs to exist how it should be consumed what production guarantees are required and how success should be measured.

A background in commercial software SaaS digital products healthtech fintech IoT data platforms or other product-driven environments is strongly preferred. We want someone who understands that data pipelines and data services are not just technical artifacts. They are product capabilities that support real users real workflows operational decisions ML/AI systems APIs analytics and measurable business outcomes.

Hands-On Data Engineering and Platform Development

• Design build optimize and operate production-grade batch and streaming data pipelines on Azure and Databricks with a primary focus on real-time IoT and telemetry use cases within a Medallion architecture.
• Develop ETL/ELT workflows to ingest transform validate and serve large volumes of structured semi-structured unstructured and streaming data.
• Build and maintain reliable data products data services APIs and microservices that support operational applications analytics software engineering and ML/AI teams.
• Use Python PySpark Spark SQL SQL Delta Lake Databricks Workflows CI/CD and related tools to build maintainable testable and observable data systems.
• Troubleshoot complex production pipeline issues across Databricks Azure streaming systems APIs and source systems including root cause analysis corrective action and prevention planning.
• Move quickly from rough business need to prototype pilot and production-ready data capability while maintaining appropriate engineering discipline.

Real-Time Streaming IoT Telemetry and Operational Data Services

• Lead the design and delivery of real-time streaming ingestion and processing patterns for connected medical device telemetry event data and operational data feeds.
• Implement streaming solutions using Azure Event Hubs Azure Stream Analytics Databricks Delta Lake and related Azure integration patterns.
• Design cost-effective throughput partitioning delivery retention and replay strategies for high-volume event and telemetry workloads.
• Create consumption patterns that support APIs microservices operational applications near-real-time decisioning analytics and ML/AI use cases.
• Define reliability latency quality observability and supportability expectations for production streaming systems.

Databricks Lakehouse and Data Platform Architecture

• Set direction for Databricks-based data engineering patterns including Medallion architecture Delta Lake Spark optimization data modeling data quality and reusable pipeline design.
• Optimize production Databricks pipelines using PySpark Spark SQL Delta Lake partitioning strategies caching shuffle optimization cluster/job configuration and cost-aware design.
• Establish practical standards for pipeline structure code organization testing deployment monitoring documentation and ownership.
• Partner with data platform security infrastructure and engineering teams to ensure the data platform is scalable secure reliable and aligned with enterprise architecture.
• Make pragmatic architecture tradeoffs between speed durability cost governance performance and downstream business impact.

Stakeholder Shaping and Cross-Functional Partnership

• Work directly with business product analytics ML/AI operations software engineering and leadership stakeholders to clarify what data is needed why it matters how it will be used and what success looks like.
• Translate ambiguous business needs into concrete data requirements data product definitions architecture options delivery priorities and implementation plans.
• Ask practical questions early: who will use the data what decision or workflow does it support what latency and quality are required what happens if the data is wrong or late and how will we know the capability is creating value
• Help the organization avoid becoming a data ticket factory by shaping solutions not just executing requests.
• Communicate architecture decisions tradeoffs risks dependencies and delivery options clearly to technical and non-technical stakeholders.

Team Management and Principal-Level Technical Leadership

• Manage mentor and develop data engineers providing clear expectations technical guidance prioritization support feedback and accountability.
• Provide technical leadership through hands-on example strong engineering judgment clear recommendations and pragmatic decision-making.
• Lead design reviews code reviews production readiness reviews incident reviews and architecture discussions across data engineering initiatives.
• Establish and improve engineering standards for data quality testing CI/CD observability documentation runbooks cost management privacy and security.
• Proactively identify platform risks data gaps unclear ownership operational weaknesses and opportunities to improve reliability scalability and delivery speed.
• Influence without relying only on formal authority by building trust framing tradeoffs and helping cross-functional teams get to decisions.

ML/AI Analytics and GenAI Enablement

• Partner with ML engineers data scientists analytics engineers and analysts to deliver reliable data pipelines feature pipelines training datasets scoring inputs and feedback loops.
• Support the data foundation for predictive models risk scores operational decisioning GenAI workflows RAG document intelligence summarization and AI-enabled automation.
• Help define data contracts model-ready datasets feature definitions lineage and monitoring expectations for ML/AI and analytics use cases.
• Ensure downstream consumers understand the meaning freshness quality limitations and appropriate use of the data products they depend on.

Governance Data Quality Security and Production Operations

• Apply privacy-first security-aware and governance-aligned practices for regulated sensitive and operationally critical data.
• Design and implement data quality checks validation rules anomaly detection schema expectations alerting and operational monitoring.
• Ensure production pipelines and services are supportable observable documented recoverable and aligned with business continuity needs.
• Drive incident response and continuous improvement for data platform and pipeline issues including root cause analysis and preventative remediation.
• Balance innovation with reliability compliance privacy cost discipline and operational usefulness.

• Strong experience with Azure cloud services for data engineering streaming integration storage security and production operations.
• Experience designing and operating real-time streaming event-driven or near-real-time data pipelines in production or business-critical environments.
• Experience applying DevOps CI/CD testing version control code review documentation and automation practices to data engineering workloads.
• Experience building data services APIs microservices or reusable consumption patterns for downstream applications analytics ML/AI or operational workflows.
• Strong understanding of data quality observability monitoring lineage reliability cost optimization privacy and production support for data systems.
• Experience translating ambiguous business needs into technical designs architecture recommendations delivery plans and measurable outcomes.
• Ability to explain data architecture pipeline behavior tradeoffs assumptions risks and limitations to both technical and non-technical stakeholders.
• Strong ownership mindset and ability to drive work forward independently in a fast-moving evolving environment.

• Experience working as a principal staff lead manager or architect-level data engineering leader in a production environment.
• Experience managing direct reports setting team priorities developing engineers and improving team execution and accountability.
• Experience working in commercial software SaaS digital products healthtech fintech IoT consumer technology or other product-driven environments.
• Experience in startup scale-up innovation new product development or rapid-build environments where the candidate had to operate with ambiguity and drive work forward independently.
• Experience with Azure Event Hubs Azure Stream Analytics Azure Service Bus Azure Data Factory Azure Functions ADLS Azure Cosmos DB Event Grid or similar Azure services.
• Experience with medical device telemetry IoT data remote patient monitoring healthcare operations regulated data HIPAA-sensitive environments or safety-critical workflows.
• Experience building data platforms or data products that support APIs microservices operational applications ML/AI systems GenAI workflows RAG analytics and executive reporting.
• Experience with data contracts semantic layers feature stores model-ready datasets data lineage schema evolution CDC and operational feedback loops.
• Experience with performance tuning cost optimization FinOps practices data platform reliability and production incident management.
• Experience partnering with product managers software engineers ML engineers analysts business leaders and operations teams to turn data into usable business capabilities.
• Experience building MVPs validating assumptions iterating based on feedback and maturing prototypes into durable production systems.

• 10 years of professional experience in data engineering software engineering data platform engineering distributed systems analytics engineering or related technical fields.
• 7 years of hands-on experience designing building optimizing and operating production data pipelines data platforms or data services.
• 5 years of hands-on experience with modern cloud data platforms including Databricks Spark Delta Lake SQL Python/PySpark and production pipeline orchestration.
• 3 years of experience leading managing mentoring or providing technical direction to data engineers or related technical teams.

• 12 years of relevant professional experience in data engineering software engineering data platforms distributed systems analytics engineering commercial software or production data products.

• Medical Vision and Dental Insurance Plans
• 401k Retirement Fund

GTT is a minority-owned staffing firm and a subsidiary of Chenega Corporation a Native American-owned company in Alaska. We highly value diverse and inclusive workplaces and support Fortune 500 organizations across banking financial services technology life sciences biotech utilities and retail sectors throughout the U.S. and Canada.

Job Number: 26-09580 Industry: Data & Analytics

#LI-GTT #LI-Hybrid