Design, develop, test, and deploy high-performance Apache Spark/Scala data processing applications and ETL pipelines on Cloudera (CDH). Optimize Spark and platform performance, build data pipelines using HDFS, Hive, Impala, HBase, and Kafka, ensure data integrity and security, collaborate with data scientists and analysts, and implement version control and CI/CD for deployments.
Key Responsibilities:
- Develop, test, and deploy data processing applications using Apache Spark and Scala.
- Optimize and tune Spark applications for better performance on large-scale data sets.
- Work with the Cloudera Hadoop ecosystem (e.g., HDFS, Hive, Impala, HBase, Kafka) to build data pipelines and storage solutions.
- Collaborate with data scientists, business analysts, and other developers to understand data requirements and deliver solutions.
- Design and implement high-performance data processing and analytics solutions.
- Ensure data integrity, accuracy, and security across all processing tasks.
- Troubleshoot and resolve performance issues in Spark, Cloudera, and related technologies.
- Implement version control and CI/CD pipelines for Spark applications.
Required Skills & Experience:
- Minimum 8 years of experience in application development.
- Strong hands on experience in Apache Spark, Scala, and Spark SQL for distributed data processing.
- Hands-on experience with Cloudera Hadoop (CDH) components such as HDFS, Hive, Impala, HBase, Kafka, and Sqoop.
- Familiarity with other Big Data technologies, including Apache Kafka, Flume, Oozie, and Nifi.
- Experience building and optimizing ETL pipelines using Spark and working with structured and unstructured data.
- Experience with SQL and NoSQL databases such as HBase, Hive, and PostgreSQL.
- Knowledge of data warehousing concepts, dimensional modeling, and data lakes.
- Ability to troubleshoot and optimize Spark and Cloudera platform performance.
- Familiarity with version control tools like Git and CI/CD tools (e.g., Jenkins, GitLab).
Similar Jobs
Artificial Intelligence • Big Data • Cloud • Information Technology • Software • Big Data Analytics • Automation
Hunt and close net-new enterprise accounts across Alberta, building executive relationships and value-based business cases. Drive pipeline via prospecting, partners, and events; lead complex sales cycles with Sales Engineers and Customer Success to deliver observability, security, cloud modernization, and AI-driven outcomes.
Top Skills:
AiopsAWSAzureCloud MigrationCybersecurityDevOpsDynatraceGCPObservabilityPlatform Engineering
.png)
Cloud • Insurance • Payments • Software • Business Intelligence • App development • Big Data Analytics
Design, deploy, and operate enterprise network infrastructure across data centers and cloud (GCP, AWS, Azure). Manage VPCs, VPNs, BGP, F5 BIG-IP, hybrid connectivity, and observability. Build IaC and automation with Terraform, GitLab CI/CD, and Ansible. Maintain monitoring, documentation, and participate in on-call support for critical incidents.
Top Skills:
AnsibleAWSAzureBgpCloud RouterCloudflareDnsF5 Big-IpFirewallsGCPGitlab Ci/CdHa VpnIp Address ManagementIpsec VpnLoad BalancingTerraformVpc
.png)
Artificial Intelligence • Blockchain • Fintech • Financial Services • Cryptocurrency • NFT • Web3
Lead high-impact reliability projects to improve resiliency, scalability, and deployment safety for thousands of services. Build secure configuration and secrets systems, improve canary-based release systems, partner with critical teams to reduce operational toil, drive observability and reliability best practices, participate in on-call rotations, and communicate architecture decisions to stakeholders.
Top Skills:
AWSAzureDatadogGCPGenerative AiGoKibanaRubyTerraform
What you need to know about the Calgary Tech Scene
Employees can spend up to one-third of their life at work, so choosing the right company is crucial, not just for the job itself but for the company culture as well. While startups often offer dynamic culture and growth opportunities, large corporations provide benefits like career development and networking, especially appealing to recent graduates. Fortunately, Calgary stands out as a hub for both, recognized as one of Startup Genome's Top 100 Emerging Ecosystems, while also playing host to a number of multinational enterprises. In Calgary, job seekers can find a wide range of opportunities.
