Senior GCP Data Engineer – Python, PySpark & BigQuery

Experience: 6 Years
Job Location: As per project requirement

We are looking for an experienced GCP Data Engineer with strong expertise in Python PySpark SQL and Google Cloud Platform (GCP). The ideal candidate will be responsible for designing developing and optimizing scalable data pipelines and big data solutions for enterprise data processing analytics and data warehousing.

The candidate should have hands-on experience with GCP services such as BigQuery Cloud Composer (Airflow) Dataproc and Pub/Sub along with strong knowledge of Apache Spark Hadoop Hive ETL processes and data engineering best practices.

• Design develop and maintain scalable ETL/ELT data pipelines using Python PySpark and GCP services.

• Build and optimize large-scale data processing solutions using Apache Spark Hadoop Hive and Big Data technologies.

• Develop and manage data warehousing solutions using Google BigQuery.

• Implement data ingestion transformation and processing workflows using GCP services such as BigQuery Dataproc Pub/Sub and Cloud Composer (Airflow).

• Design and implement event-driven data processing pipelines using Kafka or Google Cloud Pub/Sub.

• Develop and optimize complex SQL queries data models and database structures for performance and scalability.

• Implement workflow orchestration and scheduling using Apache Airflow.

• Build and maintain CI/CD pipelines using Jenkins and Git-based source control tools.

• Perform data validation troubleshooting performance tuning and optimization of data pipelines.

• Follow coding standards object-oriented programming principles and software design patterns.

• Collaborate with product owners architects and cross-functional teams to understand requirements and deliver effective data solutions.

• Participate in Agile/Scrum ceremonies code reviews testing deployment and continuous improvement initiatives.

• Maintain technical documentation and ensure adherence to enterprise development standards.

Programming & Big Data

• Strong programming experience in Python.

• Hands-on experience with PySpark and Apache Spark.

• Good understanding of Hadoop and Hive.

• Strong knowledge of SQL and relational databases.

• Understanding of object-oriented programming and design patterns.

Google Cloud Platform (GCP)

• Hands-on experience with Google Cloud Platform.

• Google BigQuery for data warehousing and analytics.

• Google Cloud Dataproc for distributed data processing.

• Google Cloud Pub/Sub for event-driven data ingestion and processing.

• Cloud Composer / Apache Airflow for workflow orchestration.

Data Engineering & Integration

• Strong experience in ETL/ELT development and large-scale data processing.

• Data pipeline design development transformation and optimization.

• Experience with data warehousing and data analytics.

• Experience building event-processing pipelines using Kafka or Pub/Sub.

DevOps & Development Tools

• Experience with GitHub GitLab Bitbucket or SVN.

• Knowledge of Jenkins and CI/CD pipeline implementation.

• Familiarity with automated testing frameworks and source code management.

• Shell scripting experience.

• Google Cloud Professional Data Engineer certification.

• Experience with enterprise-scale Big Data and analytics solutions.

• Knowledge of distributed systems multi-tier architectures and algorithms.

• Experience with automated testing and deployment frameworks.

• Understanding of Agile/Scrum and SDLC methodologies.

• 6 years of hands-on software development experience in Big Data and Analytics.

• Strong practical experience with Python PySpark SQL and GCP.

• Experience designing and optimizing enterprise data pipelines.

• Strong analytical and problem-solving skills.

• Excellent communication and collaboration abilities.

• Ability to work independently and within cross-functional teams.

• Proactive approach to identifying technical improvements.

• Willingness to learn and adopt enterprise technologies and frameworks.