Lead real-time data integration using Azure Databricks, PySpark, Kafka, ADF, and ADLS. Build and optimize medallion-style ETL pipelines processing over 10 million daily records. Improve transformation efficiency, data quality, validation, and downstream reliability by identifying upstream data issues, implementing root-cause fixes, and collaborating with stakeholders.
Real-time data pipelines using Azure Databricks, PySpark, Kafka, Azure Data Factory (ADF), and Azure Data Lake Storage (ADLS), enabling reliable ingestion and transformation of 10M+ records daily. • Developed and optimized SQL and PySpark transformations within medallion-style ETL workflows, improving processing efficiency, maintainability, and data quality for downstream analytics. • Identified requirement gaps and data quality issues stemming from upstream data sources and solutioned root-cause fixes by implementing validation rules, refining transformation logic, and collaborating with stakeholders to improve downstream data reliability.
Similar Jobs
Information Technology • Consulting
Leads data integration development using Azure, Hadoop, Spark, Scala, Java, SQL, and relational databases. Responsibilities include data modeling, analytics, solution design, requirements analysis, development, maintenance, and quality delivery in an agile, client-focused environment. The role requires strong problem-solving, interpersonal, communication, and technical skills, along with adaptability, motivation, and the ability to learn quickly.
Top Skills:
AgileSparkAzure CloudAzure Data Factory (Adf)Azure Data Lake Storage (Adls)Azure Databricks (Adb)Data AnalyticsData ModelingDelta LakeHadoopHbaseHdfsHiveIbm Db2J2EeJavaJdbcJSONMapreduceMicrosoft Sql ServerOracleScalaSpark SqlSpringSqoopWeb ServicesXMLYarn
Information Technology • Consulting
Leads hands-on data integration and engineering initiatives using Azure Databricks, Azure Data Factory, PySpark, Python, and SQL. Modernizes legacy on-premises systems into cloud-native architectures, implements Medallion Architecture and data warehousing, improves pipeline reliability and compute efficiency, develops CI/CD workflows, and mentors engineering teams delivering secure, actionable analytics for banking environments.
Top Skills:
AzureAzure Data FactoryAzure DatabricksCi/CdData WarehousingMedallion ArchitecturePysparkPythonSQL
Information Technology • Consulting
Leads hands-on data integration and engineering initiatives using Azure Databricks, Azure Data Factory, PySpark, Python, and SQL. Modernizes legacy on-premises systems into cloud-native architectures, implements Medallion Architecture and data warehousing, improves pipeline reliability and compute efficiency, establishes CI/CD workflows, supports banking-grade security, and mentors engineering teams delivering actionable analytics.
Top Skills:
Azure Data FactoryAzure DatabricksCi/CdData WarehousingMedallion ArchitecturePysparkPythonSQL
What you need to know about the Calgary Tech Scene
Employees can spend up to one-third of their life at work, so choosing the right company is crucial, not just for the job itself but for the company culture as well. While startups often offer dynamic culture and growth opportunities, large corporations provide benefits like career development and networking, especially appealing to recent graduates. Fortunately, Calgary stands out as a hub for both, recognized as one of Startup Genome's Top 100 Emerging Ecosystems, while also playing host to a number of multinational enterprises. In Calgary, job seekers can find a wide range of opportunities.
