National Bank
2019 – 2020
Data engineering and cloud development for one of Canada's leading financial institutions.
Overview
Dual-role engagement combining data engineering with cloud development — building ETL pipelines, deploying a Delta Lake solution, and co-developing the bank's virtual assistant deployed to 20,000 employees.
Key Contributions
Data Engineering
Conducted data cleaning and processing on extensive datasets using PySpark in Databricks notebooks.
Designed and implemented a serverless solution to ingest the content of the bank's SharePoint pages and documents, utilizing Azure Functions, Azure Data Factory, and Databricks.
Deployed the Delta Lake solution for enhancing the capabilities of Data Lake Gen2.
Implemented serverless Azure Functions for data ingestion into Azure Data Lake from diverse data sources such as SharePoint, Microsoft Graph, Yammer, and QnAMaker.
Designed and developed ETL pipelines utilizing Data Factory and Databricks for efficient data transformation and transportation.
Operated in a CI/CD environment using Jenkins integrated with PowerShell Scripts.
Cloud Development
Contributed to the development and design of the bank's virtual assistant, leveraging the Microsoft Bot Framework, deployed to 20,000 employees in their Teams environment.
Developed and maintained .NET Core REST APIs.
Developed and maintained analytical pipelines with Databricks for the continuous improvement of the virtual assistant.
Designed and developed serverless applications (Azure Functions) to feed the QnAMaker cognitive service of the virtual assistant.
Employees using the virtual assistant
Data sources integrated
Solution deployed on Data Lake Gen2