Senior Data Engineer

Overview:

As a Data Engineer you will be responsible for developing constructing testing and maintaining architectures such as databases and large-scale processing systems.

Responsibilities:

• Refine and enhance ETL processes using Apache Spark for efficient data handling.
  

• Utilize Python and PySpark to perform complex data processing tasks.
  

• Assess and understand business objectives to align data solutions effectively.
  

• Analyze data to identify trends and patterns providing actionable insights.
  

• Conduct in-depth data analysis and generate comprehensive reports on findings.
  

• Prepare data for prescriptive and predictive modeling to support business decision-making.
  

• Design and develop algorithms and prototypes for data analysis and processing.
  

• Integrate raw data from diverse sources into cohesive datasets.
  

• Implement and manage Redis for enhanced caching performance.
  

• Continuously explore and implement methods to improve data accuracy and reliability.
  

• Identify and acquire new data sources to enhance data repositories.
  

• Develop and maintain analytical tools and programs to facilitate data analysis.
  

• Work closely with data scientists architects and other stakeholders on various projects.
  

• Lead the migration efforts of existing data to new data warehouse infrastructures.
  

Qualifications:

• Degree in Computer Science Information Technology or a related field; a masters degree is a plus.
  

• Relevant certifications such as IBM Certified Data Engineer Google Professional Data Engineer are highly desirable.
  

• 3 years of Proven experience as a data engineer or in a similar role with a strong track record of successful data projects.
  

• In-depth knowledge of data models data mining and segmentation techniques.
  

• Advanced skills in Python and PySpark for data processing and analysis.
  

• Hands-on experience with SQL database design including T-SQL stored procedures and query optimization.
  

• Experience with Postgree and MongoDB
  

• Proficiency in implementing and managing Redis for caching and high-performance data retrieval.
  

• Proficiency with big data technologies such as Hadoop Hive Apache Ecosystem and Kafka.
  

• Experience with cloud platforms like Google Cloud for data storage and processing.
  

• Strong numerical and analytical skills with the ability to interpret complex data sets.
  

• Excellent communication skills with the ability to collaborate effectively with cross-functional teams.
  

• Strong problem-solving skills with a proactive approach to identifying and addressing data-related challenges.