Senior Data Engineer Databricks

Rutherford NJ/ Jersey City NJ

• Build scalable production-grade ETL/ELT pipelines using Databricks (PySpark Spark SQL Delta Live Tables Workflows).
  
• Ingest structured semi-structured and streaming data into Bronze Silver and Gold layers.
  
• Develop optimized transformations data quality rules and reusable framework components.
  
• Implement best practices for job orchestration monitoring alerting and automation.
  
• Hands-on experience: Spark Delta Lake Workflows Unity Catalog.
  
• Strong SQL programming and performance tuning skills.
  
• Experience with cloud environments (AWS/Azure/GCP).
  
• Experience with modern data lakehouse concepts and distributed systems.
  
• Strong understanding of Lakeflow Connect LSDP/Lakehouse Medallion Architecture Data Validations Genie and Agent Bricks/RAG use cases.
  
• Should be able to explain these concepts using real project examples and architecture decisions.
  
• Knowledge of medallion architecture DLT and unity catalog within Databricks.
  

• Strong Python (PySpark) and SQL programming
  
• Databricks Spark Delta Lake Workflows Unity Catalog
  
• ETL/ELT pipeline development Medallion Architecture (Bronze/Silver/Gold)
  
• Delta Live Tables Auto-Loader Structured Streaming
  
• Data modeling dimensional (star/snowflake) normalization/denormalization
  
• CI/CD Git job orchestration
  
• Cloud experience AWS Azure or GCP
  
• 710 years in data engineering
  
• Knowledge of medallion architecture DLT and unity catalog within Databricks.
  

Nice-to-Have Skills

• Lakeflow Connect LSDP/Lakehouse Genie Agent Bricks/RAG use cases
  
• Data governance metadata management Unity Catalog advanced features
  
• Airflow dbt or similar orchestration tools
  
• Data security compliance and access models
  
• Cost optimization and performance tuning in cloud environments
  
• Corporate/enterprise data warehousing background