Senior Data Engineer
Job Summary
- Build a centralized data lake on GCP Data services by integrating diverse data sources throughout the enterprise.
- Develop maintain and optimize SPARK-powered batch and streaming data processing pipelines. Leverage GCP data services for complex data engineering tasks and ensure smooth integration with other platform components
- Design and implement data validation and quality checks to ensure datas accuracy completeness and consistency as it flows through the pipelines.
- Work with the Data Science and Machine Learning teams to engage in advanced analytics.
- Collaborate with cross-functional teams including data analysts business users operational and marketing teams to extract insights and value from data.
- Collaborate with the product team to design implement and maintain the data models for analytical use cases.
- Design develop and upkeep data dashboards for various teams using Looker Studio.
- Engage in technology explorations research and development POCs and conduct deep investigations and troubleshooting.
- Design and manage ETL/ELT processes ensuring data integrity availability and performance.
- Troubleshoot data issues and conduct root cause analysis when reporting data is in question.
- PySpark
- GCP - Big Query Dataproc Dataflow Dataplex Pub-Sub and Cloud Storage
- Advance SQL knowledge
- NoSQL (Preferably MongoDB)
- Programming languages - Scala/Python
- Great Expectation - similar DQ framework
- Familiarity with workflow management tools like Airflow Prefect or Luigi.
- Understanding of Data Governance DWH and Data Modelling
- Infrastructure as Code - Terraform
- Docker and Kubernetes
- Looker Studio
- AI and ML engineering knowledge
Required Experience:
Senior IC
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
About Yassir Yassir is the leading super App for on demand, ride-hailing, last-mile delivery, payment services and more, set to change the way daily services are provided. It currently operates in 45 cities across multiple countries. It has raised $150 million in Series B funding, fiv ... View more