Azure Databricks Engineer

Azure Databricks Engineer Overview

Capgemini is seeking an experienced Azure Databricks Engineer to join their team, working across various locations within the UK. This role involves leading the technical implementation, troubleshooting, and optimization of the Azure Databricks platform in a production environment. The successful candidate will design and develop data pipelines using Databricks SQL and Delta Lake, manage cluster configurations, and support the deployment of data solutions in collaboration with data engineers, data scientists, and platform teams.

Key Responsibilities

• Lead the setup, configuration, and troubleshooting of Azure Databricks environments, ensuring high performance and reliability.
• Design and implement data pipelines, including full and incremental loads, change data capture, and data quality controls.
• Manage cluster operations, including workload monitoring, autoscaling, and resource management.
• Support the deployment and ongoing operation of Databricks environments in production, handling incident management and performance tuning.
• Implement security best practices using Azure Identity, Azure Security, and Key Vault to protect data and platform access.
• Integrate Databricks with various enterprise systems via REST APIs, databases, SaaS applications, and cloud storage solutions.
• Assist with application deployment within Databricks, including BI tool integration and scheduling tasks.
• Support migration activities and application deployment, including migration from tools like POSIT/RStudio.
• Collaborate with cross-functional teams to ensure seamless data workflows and platform stability.
• Create comprehensive technical documentation and facilitate knowledge transfer.
• Contribute to cost management efforts through resource tagging, monitoring, and budget controls.
• Participate in agile project delivery, promoting continuous improvement and team collaboration.
• Develop and uphold operational governance standards for platform support.

Candidate Requirements

• Hands-on experience with Azure Databricks, including platform engineering, workload optimization, and troubleshooting in a live environment.
• Strong programming skills in Python, PySpark, and SQL for data pipeline development.
• Experience with data ingestion, transformation, and CDC techniques, along with data quality and reconciliation processes.
• Proficiency in integrating Databricks with various systems via REST APIs, databases, SaaS, and cloud storage.
• Knowledge of Azure services such as Azure Identity, Networking, Security, and Key Vault.
• Experience supporting production Databricks environments, including incident management and performance tuning.
• Understanding of cloud cost management practices, including resource tagging and budget controls.
• Experience with application deployment, BI tooling, and scheduling within Databricks.
• Familiarity with migration support activities and application deployment processes.
• Excellent collaboration skills working with Data Engineers, Data Scientists, and Platform Teams.
• Ability to produce detailed technical documentation and share knowledge effectively.
• Experience working within agile teams and delivering iterative projects.
• Desirable: Experience with AI/ML integration, RAG, Vector Databases, and regulated industry environments.

Benefits and Culture

• Access to extensive training programs, including thousands of courses and external certifications from top providers.
• Flexible, hybrid working arrangements that combine office, client site, and home working.
• A supportive environment that encourages innovation, exploration, and career growth.
• Opportunity to work on impactful projects that leverage the latest technology to drive business value.
• Be part of a global organization committed to digital and sustainable transformation, with a heritage spanning over 55 years.

Degree Requirement: Not Specified

Visa Sponsorship May be