Azure Databricks Technical Lead

Azure Databricks Technical Lead Overview

Capgemini is seeking an experienced Azure Databricks Technical Lead to join their team, with the role based anywhere within the UK. This position involves leading the deployment, management, and optimization of Azure Databricks environments, ensuring high performance and reliability for client projects. The role requires hands-on expertise in platform engineering, data pipeline development, system integration, and operational support within Azure cloud services.

Key Responsibilities

• Take charge of implementing and troubleshooting Azure Databricks platforms, ensuring configurations are optimized for workload performance.
• Manage serverless Databricks environments, focusing on workload tuning, autoscaling, and resource management.
• Design and develop data pipelines using SQL and Delta Lake, including ingestion, transformation, and incremental data loads.
• Implement change data capture (CDC) techniques, maintain data quality, and handle schema changes and error reconciliation.
• Integrate Databricks with various external systems through REST APIs, connecting to databases, SaaS applications, and cloud storage solutions.
• Ensure secure authentication and credential management within the Azure ecosystem.
• Support Azure platform services such as Identity and Access Management, Networking, Security, and Monitoring to maintain operational excellence.
• Monitor production environments, troubleshoot issues, and optimize performance to ensure high availability.
• Implement operational governance, including resource tagging, cost monitoring, and budget controls to optimize expenses.
• Assist in application deployment, migration activities, and support BI tooling integration and scheduling.
• Collaborate with Data Engineers, Data Scientists, and Platform teams to facilitate seamless data workflows and knowledge sharing.
• Contribute to technical documentation and participate in Agile development cycles.
• Optional: Support AI/ML projects, including Large Language Model (LLM) integration, RAG, and Vector Database exposure.
• Optional: Provide support for regulated industry compliance requirements.

Qualifications and Skills

• Extensive hands-on experience with Azure Databricks, including platform engineering, configuration, and troubleshooting.
• Proficiency in managing serverless Databricks environments, with skills in workload optimization and autoscaling.
• Strong development skills in Python, PySpark, and SQL for data pipeline creation.
• Experience designing ingestion, transformation, and CDC processes, with a focus on data quality and error handling.
• Knowledge of integrating Databricks with various systems via REST APIs and experience with database and SaaS integrations.
• Understanding of Azure platform services such as Identity and Access Management, Networking, Security, and Monitoring.
• Experience supporting production environments, including incident management and performance tuning.
• Ability to implement operational governance, resource tagging, and cost controls effectively.
• Excellent collaboration skills working with cross-functional teams and stakeholders.
• Experience working within Agile frameworks.
• Desirable: Migration experience with POSIT/RStudio, AI/ML and LLM integration, RAG, and Vector Databases.
• Optional: Experience working in regulated industries.

Benefits and Culture

Employees will have access to a wide range of training and development programs, including external certifications from leading providers. The company promotes a hybrid working model, offering flexibility to balance work and personal life. Employees are encouraged to pursue continuous learning and innovation, contributing to impactful projects that help clients transform their businesses. Capgemini values diversity, inclusion, and sustainability, fostering a positive and dynamic work environment.

Capgemini is a global leader in business and technology transformation, with over 340,000 employees worldwide. The company leverages its expertise in AI, cloud, and data to deliver end-to-end solutions that address complex business challenges, helping organizations accelerate their digital and sustainable transitions.

Degree Requirement: Not Specified

Visa Sponsorship May be