System Engineer with Spark engine and SparkFlow framework

Role : System Engineer

Location : Charlotte NC (Hybrid)

Type: C2C

Role Summary

- Senior backend engineer building the core integrations between Unity and the enterprise Spark engine and SparkFlow framework

- Designs distributed high-throughput systems that power data processing and integration workflows at enterprise scale

- Operates with significant autonomy setting technical direction within a regulated large-scale engineering environment

Key Responsibilities

Platform & Integration Engineering

- Design build and operate integrations between Unity and the Spark engine and SparkFlow framework

- Develop robust APIs SDKs and connectors enabling reliable data movement and workflow orchestration

- Implement abstractions that simplify SparkFlow adoption for downstream platform consumers

Distributed Systems Design

- Architect scalable fault-tolerant systems handling large data volumes and concurrent workloads

- Apply best practices for performance tuning caching partitioning backpressure and resilience

- Drive design reviews RFCs and architectural decisions for core platform components

Reliability & Operations

- Build systems with first-class observability (metrics logs traces) and SLOs

- Establish operational excellence: runbooks alerting capacity planning and incident response

- Champion automated testing and progressive delivery practices

Required Qualifications

- 6 years of backend engineering experience with deep expertise in distributed systems

- Expert-level Python; strong API design (REST gRPC) and asynchronous programming

- Proven track record delivering large-scale production-grade systems

- Strong fundamentals in concurrency data structures and systems performance experience

Preferred Qualifications

- Hands-on experience with Apache Spark (PySpark Spark SQL Structured Streaming)

- Experience building or integrating with workflow/data orchestration frameworks (Airflow Autosys)

- Exposure to greenfield platform builds in regulated financial enterprises

- Experience with event-driven architectures (Kafka) and lakehouse formats (Iceberg Delta Hudi)

Technical Skills

- Languages: Python (expert) Scala or Java (plus) SQL

- Distributed Compute: Apache Spark

- Storage & Data: S3/ADLS/GCS Iceberg Delta Lake Parquet Postgres

- Streaming & Messaging: Kafka Pub/Sub

- APIs & Services: REST gRPC GraphQL OpenAPI

- Containers & Cloud: Docker Kubernetes Azure/GCP Terraform

- CI/CD: GitHub Actions

Soft Skills & Leadership Expectations

- Strong technical judgment and ability to operate