Design, develop, and maintain data engineering solutions across the SDLC using Python (primary) and Scala (secondary). Build distributed data pipelines on Hadoop/Spark ecosystems, extend data processing platforms (Spark SQL, Hive, Starburst), manage CI/CD, and deploy on Kubernetes/OpenShift. Collaborate with stakeholders in Agile teams and ensure maintainable, reusable code.
Qualifications:
- Strong communication skills
- Experience of Agile development and scrums
- Banking and securities domain knowledge would be an added advantage
Skills Required:
- Strong experience working across the entire SDLC lifecycle
- Programming experience in one or more application or systems languages, Python - Primary, Scala - Secondary (basic knowledge)
- Strong experience working with Python concepts and libraries such as Jupyterhub, Airflow, Pandas, NumPy etc.
- Good experience with classes based OOP and design patterns
- Distributed Systems Design Experience - including understanding of distributed systems concepts and principles
- Knowledge and understanding of Kerberos and authentication
- Hadoop Ecosystem of Tools (Spark, Hive, Impala, etc).
- Experience extending and implementing core functionality and libraries in data processing platforms (Spark / Spark SQL, Hive, Starburst, etc)
- Strong experience working with the CI/CD pipeline and tools like Jenkins , Harness/Tekton, Udeploy/Ansible,Bitbucket, Jira
- Strong experience in cloud platforms like Kubernetes, OpenShift4
- Ability to deal with multiple stakeholders and follow through on open issues.
- A commitment to writing understandable, maintainable, and reusable software.
- Willingness to learn new languages and methodologies.
- Experience working with business partners and engineers to gather, understand, and bridge definitions and requirements.
- An innate desire to deliver and a strong sense of accountability for your work.
Education:
- Bachelor’s degree/University degree or equivalent experience
Similar Jobs
Agency • Information Technology
Design, develop, test, and deploy high-performance Apache Spark/Scala data processing applications and ETL pipelines on Cloudera (CDH). Optimize Spark and platform performance, build data pipelines using HDFS, Hive, Impala, HBase, and Kafka, ensure data integrity and security, collaborate with data scientists and analysts, and implement version control and CI/CD for deployments.
Top Skills:
SparkCi/CdCloudera (Cdh)FlumeGitGitlabHbaseHdfsHiveImpalaJenkinsKafkaNifiOoziePostgresScalaSpark SqlSqoop
Artificial Intelligence • Cloud • Computer Vision • Hardware • Internet of Things • Software
Build and own technically complex, end-to-end full-stack features and platform capabilities for AI-driven IoT products. Define technical strategy, ship backend/frontend/mobile work, scale data ingestion and APIs, partner with product/design/customer teams, mentor engineers, and drive cross-team projects from ambiguous problems to measurable outcomes.
Top Skills:
CursorGoIotJavaScriptLlmsPythonReactTelematicsTypescript
Big Data • Fintech • Mobile • Payments • Financial Services
Lead technical strategy and execution for the shared mobile app infrastructure, build and maintain React Native and native modules, improve observability, CI/CD, releases and developer tooling, define standards and runbooks, mentor engineers, coordinate cross-functional trade-offs, and ensure production availability and quality across mobile products.
Top Skills:
AppiumClaude CodeCrashlyticsExpoJavaScriptLaunchdarklyMaestroReact NativeRedux ToolkitSentryStatsigTypescript
What you need to know about the Calgary Tech Scene
Employees can spend up to one-third of their life at work, so choosing the right company is crucial, not just for the job itself but for the company culture as well. While startups often offer dynamic culture and growth opportunities, large corporations provide benefits like career development and networking, especially appealing to recent graduates. Fortunately, Calgary stands out as a hub for both, recognized as one of Startup Genome's Top 100 Emerging Ecosystems, while also playing host to a number of multinational enterprises. In Calgary, job seekers can find a wide range of opportunities.
