Senior Engineer, Data Engineering

Job Description :

Lead Data Engineer One of the most innovative and high-prole teams in News Corp is looking for a lead data engineer to help accelerate its vision. The role will involve combining all of News Corps data across categories countries and organizations into a singular view of a customer enabling engagement measurement and targeting. The Dow Jones Lead Data Engineer will partner closely with product data & engineering teams to help make this data available and accessible to our businesses teams and partners in order to drive revenue and improve the customer experience.

Responsibilities

Design implement and maintain high performance big data infrastructure/systems & big data processing pipelines scaling to billions of structured and unstructured events daily

Design implement and maintain deep integration with up-stream systems

Implement data pipelines to automate the ingestion transformation and augmentation of data sources and provide best practices for pipeline operations.

Advanced SQL writing and experience in data mining (SQL ETL data warehouse etc.) and using databases in a business environment with complex datasets

Monitor and optimize the performance of the data platform to ensure reliability scalability and efiency.

Ensure data quality across systems (accuracy consistency completeness) by providing transparency and proactive monitoring.

Support products with the overall roadmap and ensure updates to senior leadership are 100% technically correct.

Data analysis understanding of business requirements and translation into logical pipelines & processes

Conduct timely and effective research in response to specific requests (e.g. data collection summarization analysis and synthesis of relevant data and information)

Evaluate and prototype new technologies in the area of data processing

Think quickly communicate clearly and work collaboratively with product data engineering QA and operations teams

Data Engineering & Data Modelling: Creating data architectures and logical/physical data models for databases and data warehouses.

Providing training on data engineering concepts and techniques to other members of the team

Writing documentation for data pipelines and data models

Mentor and guide a team of data engineers fostering a culture of continuous improvement innovation and knowledge sharing to enhance technical expertise and collaboration across the team.

Promote a high-performance team environment ensuring strong work ethic accountability and enthusiasm to drive successful project delivery.

Technologies we use

AWS (EMR EC2 Glue Athena Redshift Lambda S3 DynamoDB SNS)

Python Spark (PySpark)

Kubernetes Docker

Airow

Git for source code management

Jira

Qualifications

BS in Computer Science or other technical discipline

8 years of experience designing and developing big data processing systems using distributed computing

6 years experience in SQL

5 years experience in Python and Spark

6 years of experience working with cloud platforms such as AWS

4 years experience in Apache Airow

Expert knowledge in optimizing complex SQL queries and Hands on experience with NoSQL (DynamoDB)

An affinity for automation

Familiarity with networking and network application programming including HTTP/HTTPS JSON and REST APIs

Experience with at least one object oriented language (ex: Java)

Strong OO design data structure and algorithm design skills

Strong interest in emerging technologies: Spark EKS

Nice to Have

Experience in the digital advertising and marketing industry

Contribution to Open Source projects

Equal Opportunity Employer

All qualified applicants will receive consideration for employment without regard to race color religion sex age national origin protected veteran status disability status or any other protected characteristic.

Reasonable Accommodation

We are committed to providing reasonable accommodation for qualified individuals with disabilities in our job application and/or interview process. If you need assistance or accommodation in completing your application or participating in an interview due to a disability email us at . Please put Reasonable Accommodation in the subject line and provide a brief description of the type of assistance you need. This inbox will not be monitored for application status updates.

Please refer to the privacy notice at the bottom of this page for submitting any data access deletion or other data subject rights requests where permitted under your local laws and regulations.