Azure Databricks Technical Lead

Capgemini

Azure Databricks Technical Lead Overview

Company Name Capgemini
Job Role Azure Databricks Technical Lead
Qualifications Not Specified
Category IT Jobs
Job Type Full Time
Location London

Capgemini is seeking an experienced Azure Databricks Technical Lead to join their team, working remotely across the UK. This role involves leading the development and management of data solutions on the Azure Databricks platform, ensuring high performance, security, and efficiency. The successful candidate will be responsible for designing, implementing, and troubleshooting complex data pipelines, managing platform configurations, and integrating Databricks with various enterprise systems.

Key Responsibilities

  • Lead the architecture, development, and deployment of solutions utilizing Azure Databricks, aligning with client needs and industry standards.
  • Provide expert troubleshooting and performance tuning within Databricks environments to ensure optimal operation.
  • Configure serverless Databricks environments, managing workload distribution, autoscaling, and resource policies.
  • Design and develop data pipelines using SQL, Delta Lake, and PySpark, supporting both full and incremental data loads, including CDC processes.
  • Implement data quality measures, schema management, and reconciliation processes to maintain data integrity.
  • Integrate Databricks with external systems through REST APIs, connecting to databases, SaaS platforms, cloud storage, and enterprise applications securely.
  • Manage Azure platform services such as Identity and Access Management, Networking, Security (Key Vault), and Monitoring to support Databricks operations.
  • Assist in application deployment, migration, and scheduling, including BI tool integration.
  • Collaborate with Data Engineers, Data Scientists, and Platform teams to ensure seamless workflows and knowledge sharing.
  • Create comprehensive technical documentation and conduct knowledge transfer sessions.
  • Support production environments through monitoring, incident response, and performance optimization.
  • Implement operational governance, including resource tagging, cost monitoring, and budget control, to optimize cloud expenditure.
  • Support discovery, migration, and zone management activities related to Databricks applications.
  • Participate in agile project delivery, contributing to planning, stand-ups, and continuous improvement.
  • Support AI/ML initiatives, including LLM integration, RAG, and vector database exposure, especially in regulated industries.

Candidate Requirements

  • Hands-on experience with Azure Databricks, including platform engineering, workload management, and troubleshooting in production environments.
  • Strong programming skills in Python, PySpark, and SQL for data pipeline development.
  • Deep understanding of data engineering concepts such as ingestion, transformation, CDC, and data quality management.
  • Experience with REST API integration and connecting Databricks to various external systems securely.
  • Familiarity with Azure cloud services like Identity and Access Management, Networking, Security, and Monitoring tools.
  • Proven ability to support and operate production Databricks environments, including incident management and performance tuning.
  • Knowledge of operational governance, resource tagging, and cost management within Azure.
  • Ability to support application deployment, migration, and BI tool integration within Databricks.
  • Excellent collaboration skills, working effectively with Data Engineers, Data Scientists, and Platform teams.
  • Strong technical documentation and knowledge transfer capabilities.
  • Experience working within agile teams and delivering iterative solutions.
  • Desirable skills include experience with RStudio/POSIT migration, AI/ML and LLM integration, RAG, vector databases, and regulated industry standards.

Additional Information

This role is offered on a permanent basis with flexible, hybrid working arrangements across the UK, including locations such as London and Manchester. Capgemini emphasizes its commitment to diversity and inclusion, being a Disability Confident Employer, and offers extensive training opportunities, including certifications from leading providers. The company encourages a positive work-life balance and supports flexible working requests.

Capgemini is a global leader in digital and sustainable transformation, with over 340,000 employees worldwide, delivering innovative solutions across strategy, design, engineering, and technology. The organization values continuous learning, collaboration, and making a meaningful impact through technology.


Degree Requirement: Not Specified

Visa Sponsorship May be

To apply for this job please visit careers.capgemini.com.