Senior Software Engineer- ETL Developer

Position Description:

Founded in 1976 CGI is among the largest independent IT and business consulting services firms in the world. With 94000 consultants and professionals across the globe CGI delivers an end-to-end portfolio of capabilities from strategic IT and business consulting to systems integration managed IT and business process services and intellectual property solutions. CGI works with clients through a local relationship model complemented by a global delivery network that helps clients digitally transform their organizations and accelerate results. CGI Fiscal 2024 reported revenue is CA$14.68 billion and CGI shares are listed on the TSX (GIB.A) and the NYSE (GIB). Learn more at .
Job Title: ETL Developer
Position: Software Engineer
Experience: 4- 7 Years
Category: Software Development/ Engineering
Shift: 1-10PM (Hybrid)
Main location: Bangalore / Chennai
Position ID: J0726-1184
Employment Type: Full Time
Education Qualification: Bachelors degree in Computer Science or related field or higher with minimum 3 years of relevant experience.
Position Description: We are looking for an Developer to join our team. The ideal candidate should be passionate about coding and developing scalable and high-performance applications. The successful candidate will be responsible for designing developing optimizing and supporting enterprise-grade ETL and ELT pipelines primarily using Azure Databricks PySpark SQL and related Microsoft Azure services. This is a hands-on engineering role focused on building scalable reliable and high-performing data pipelines across development testing and production environments. The engineer will work closely with data architects business analysts application teams quality assurance teams and production support teams to deliver secure and well-governed data solutions in a regulated financial-services environment.
#LI-MP14

Your future duties and responsibilities:

Design develop test deploy and support ETL and ELT pipelines using Azure Databricks.
Build scalable data transformation workflows using PySpark Spark SQL and Databricks notebooks.
Develop reusable Databricks frameworks utilities and common pipeline components.
Ingest and process data from relational databases flat files APIs cloud storage and enterprise applications.
Design and implement batch and near-real-time data pipelines.
Build and manage Databricks jobs workflows schedules dependencies retries and alerts.
Develop data cleansing validation enrichment reconciliation and transformation logic.
Implement data quality controls audit checks exception handling and restart capabilities.
Optimize Spark workloads SQL queries cluster configuration partitioning caching and file formats.
Work with Delta Lake to support reliable scalable and transactional data processing.
Design and maintain bronze silver and gold data layers within a lakehouse architecture.
Support source-to-target mapping data profiling data lineage and technical design activities.
Develop and maintain SQL queries stored procedures views and data reconciliation scripts.
Integrate Databricks pipelines with Azure Data Lake Storage and other Azure services.
Build and maintain automated deployment processes using Azure DevOps CI/CD pipelines.
Promote code notebooks jobs and configuration across development testing and production environments.
Monitor pipeline performance and troubleshoot failures data issues and production incidents.
Perform root-cause analysis and implement permanent fixes for recurring issues.
Create and maintain technical documentation data mappings deployment procedures and support runbooks.
Ensure solutions comply with CIBC security data governance risk management and regulatory requirements.

Required qualifications to be successful in this role:

Must-Have Skills:
Bachelors degree or diploma in Computer Science Software Engineering Data Engineering Information Technology or a related discipline.
Strong professional experience developing ETL or ELT solutions using Azure Databricks.
Strong hands-on experience with:
oDatabricks notebooks
oDatabricks Jobs and Workflows
oApache Spark
oPySpark
oSpark SQL
oDelta Lake
oCluster configuration and management
oPerformance tuning and optimization
Strong experience designing and developing complex ETL pipelines in Databricks.
Experience building reusable and parameterized data pipelines.
Strong SQL development skills including:
oComplex queries
oStored procedures
oViews and functions
oData reconciliation
oQuery optimization
oPerformance troubleshooting
Experience with Azure Data Lake Storage and cloud-based data ingestion.
Experience with structured semi-structured and unstructured data formats including CSV JSON Parquet and Delta.
Strong understanding of ETL and ELT patterns data integration principles and data processing frameworks.
Experience implementing data quality checks logging monitoring alerting and exception handling.
Experience with job orchestration scheduling dependency management and restartability.
Experience with source control and CI/CD deployment practices using Azure DevOps.
Experience supporting Databricks pipelines in production environments.
Strong analytical troubleshooting and problem-solving skills.
Strong written and verbal communication skills.
Good-to-Have Skills:
Experience with Databricks Unity Catalog including access controls governance and data lineage.
Experience with Databricks Asset Bundles or other Databricks deployment frameworks.
Experience with Databricks SQL Warehouses.
Knowledge of medallion and lakehouse architecture patterns.
Experience with Auto Loader and incremental data ingestion.
Experience with streaming pipelines using Spark Structured Streaming.
Experience with Delta Live Tables or Lakeflow Declarative Pipelines.
Experience with Python software engineering practices including modular development and unit testing.
Experience integrating Databricks with Azure Data Factory.
Experience with Azure API Management and REST API integrations.
Experience with PowerShell or other scripting languages.
Familiarity with data modelling techniques including dimensional and relational models.
Experience migrating legacy ETL workloads from Informatica SSIS or similar platforms to Databricks.
Previous experience working within banking capital markets commercial banking corporate banking or another regulated financial-services environment.
Knowledge of corporate banking lending credit risk client onboarding financial transactions or regulatory reporting data.
CGI is an equal opportunity addition CGI is committed to providing accommodation for people with disabilities in accordance with provincial legislation. Please let us know if you require reasonable accommodation due to a disability during any aspect of the recruitment process and we will work with you to address your needs.

Skills:

• Communication
• Database
• Banking
• Databricks
• SQLite

What you can expect from us:

Together as owners lets turn meaningful insights into action.

Life at CGI is rooted in ownership teamwork respect and belonging. Here youll reach your full potential because

You are invited to be an owner from day 1 as we work together to bring our Dream to life. Thats why we call ourselves CGI Partners rather than employees. We benefit from our collective success and actively shape our companys strategy and direction.

Your work creates value. Youll develop innovative solutions and build relationships with teammates and clients while accessing global capabilities to scale your ideas embrace new opportunities and benefit from expansive industry and technology expertise.

Youll shape your career by joining a company built to grow and last. Youll be supported by leaders who care about your health and well-being and provide you with opportunities to deepen your skills and broaden your horizons.

That same commitment to fairness extends to how we use technology. To support our recruitment team AI tools may be used to help assess applications though they never replace human judgement. All hiring decisions remain entirely in the hands of our recruitment professionals.

Come join our teamone of the largest IT and business consulting services firms in the world.