AI Data Engineer AI Training data

You will contribute to the development of AI training datasets by building and maintaining Python-based data processing pipelines. You will work with large-scale structured and unstructured datasets transforming raw documents and text into high-quality validated data for AI model training. This role requires hands-on experience creating datasets from the ground up developing scalable data workflows and managing data processing in Azure environments. You will collaborate with data scientists and engineering partners to improve data quality processing efficiency and dataset reliability while ensuring data integrity traceability and security.

• Design develop maintain and optimize Python-based data pipelines for large-scale data ingestion processing and transformation.

• Build datasets from the ground up by collecting organizing cleaning validating and preparing raw structured and unstructured data.

• Develop workflows for OCR processing text extraction normalization filtering segmentation and deduplication.

• Execute and monitor large-scale batch processing jobs troubleshoot failures and implement automated retry and recovery mechanisms.

• Develop automated data quality checks to identify missing or corrupted files incomplete processing outputs duplicate records formatting issues and metadata inconsistencies.

• Prepare high-quality training-ready datasets through data filtering tokenization text segmentation and conversion into standardized formats.

• Manage bulk data uploads downloads and synchronization between local processing environments and Azure storage platforms.

• Maintain organized datasets with consistent naming conventions metadata version control and source traceability.

• Optimize data processing performance memory utilization storage efficiency and cloud resource consumption.

• Monitor dataset statistics processing coverage quality metrics and exceptions while generating operational reports.

• Collaborate with applied scientists and engineering teams to translate AI training requirements into scalable and repeatable data workflows.

• Maintain documented tested and version-controlled code while following established data security and governance requirements.

• 5 years of professional experience in data engineering data pipeline development or data science with substantial hands-on programming responsibilities.

• Strong proficiency in Python including developing reusable modules debugging testing logging and error handling.

• Demonstrated experience building and developing datasets from scratch including data ingestion cleaning transformation validation and preparation.

• Hands-on experience processing large-scale structured and unstructured datasets using memory-efficient parallel or distributed processing techniques.

• Practical experience with Azure Blob Storage or Azure Data Lake Storage including bulk data transfers storage organization and access troubleshooting.

• Experience with Python data-processing libraries and formats such as pandas NumPy JSON/JSONL CSV and Parquet.

• Working knowledge of SQL Git Linux environments and command-line tools.

• Strong troubleshooting analytical and problem-solving skills with a focus on data quality and integrity.

• Ability to communicate technical challenges and collaborate effectively with cross-functional engineering and data science teams.

• Experience developing OCR pipelines processing scanned documents multilingual text or document-layout data.

• Experience with Azure Databricks Azure Machine Learning Azure Data Factory or other workflow orchestration platforms.

• Familiarity with Hugging Face Datasets AI/ML tokenization near-duplicate detection and training/evaluation data separation.

• Experience with large-scale dataset curation and preparation for AI or machine learning model training.

• Knowledge of distributed data processing frameworks and technologies such as PySpark.

• Experience with dataset lineage reproducibility containerized environments and CI/CD practices.

• Experience optimizing data pipelines for performance reliability scalability and cost efficiency.