Staff Software Engineer, Observability & Profiling

Anthropic

Staff Software Engineer, Observability & Profiling Overview

Company Name Anthropic
Job Role Staff Software Engineer, Observability & Profiling
Qualifications Not Specified
Category IT Jobs
Job Type Full Time
Location London

Anthropic is seeking experienced Software Engineers to join their Observability team within the Infrastructure department. The team is responsible for the monitoring and telemetry infrastructure that supports all engineers and researchers at Anthropic. This includes managing metrics, logging pipelines, distributed tracing, profiling, error analytics, alerting, and the dashboards and query interfaces that make operational data actionable. As part of this team, you will directly influence the reliability and operational excellence of Anthropic’s research and product systems.

About the Role

In this position, you will design and build scalable telemetry ingestion and storage systems for metrics, logs, traces, and error data across multiple clusters. You will develop observability solutions that provide deep, low-overhead insights into system behavior, enabling engineers to quickly detect and diagnose issues. Your responsibilities include maintaining and improving core observability platforms, leading migrations, and architectural enhancements to improve system reliability, reduce costs, and support growth.

You will also develop instrumentation libraries, SDKs, and auto-instrumentation tools based on eBPF technology to generate high-quality telemetry data with minimal code changes. Your work will help reduce mean time to detection and resolution by creating cross-signal correlation systems, unified query interfaces, and AI-assisted diagnostic tools that span from kernel events to application traces. Additionally, you will turn continuous profiling and resource utilization telemetry into actionable insights to optimize CPU, memory, and accelerator performance across the fleet.

Key Responsibilities

  • Design and implement scalable telemetry pipelines for metrics, logs, traces, and errors across Anthropicâ??s multi-cluster infrastructure.
  • Create observability solutions that provide deep insights into system behavior with minimal overhead.
  • Own and evolve core observability platforms, leading migrations and architectural improvements to enhance reliability, reduce costs, and support scaling efforts.
  • Develop instrumentation libraries, SDKs, and eBPF-based auto-instrumentation tools to emit high-quality telemetry data, with or without code modifications.
  • Reduce incident detection and resolution times by building systems for cross-signal correlation, unified querying, and AI-powered diagnostics.
  • Drive operational efficiency by transforming telemetry data into actionable optimization insights for CPU, memory, and accelerator fleets.
  • Collaborate with Research, Inference, Product, and Infrastructure teams to tailor observability solutions to their specific needs.

Qualifications

  • Practical experience in building and managing large-scale observability or monitoring systems.
  • Deep understanding of the entire signal pipelineâ??from instrumentation and ingestion to querying and analysis.
  • Knowledge of high-throughput telemetry systems and the tradeoffs involved in data collection, storage, and querying at scale.
  • Comfort troubleshooting below the application layer, including kernel, network, or hardware diagnostics.
  • Excellent communication skills and a collaborative mindset to work effectively with internal teams.
  • Enthusiasm for building foundational infrastructure and navigating complex technical challenges independently and within teams.
  • Minimum of a bachelorâ??s degree or equivalent experience in a relevant field, demonstrated through coursework, training, or professional background.
  • At least 10 years of industry experience, including operating large-scale observability systems.
  • Experience with eBPF-based observability tools in production environments, such as tracing, profiling, or network visibility.
  • Proficiency in continuous profiling at fleet scale, managing overhead budgets, and symbolization techniques.
  • Kernel and syscall-level debugging skills and performance engineering expertise.
  • Experience profiling or instrumenting workloads on accelerators.
  • Knowledge of metrics systems handling high-cardinality data and large telemetry storage solutions.
  • Familiarity with OpenTelemetry instrumentation, collector pipelines, and sampling strategies like tail-based sampling.
  • Interest in applying AI and large language models to operational workflows, including automated root cause analysis, anomaly detection, and intelligent alerting.

Benefits

  • Annual salary between £325,000 and £390,000 GBP.
  • Hybrid working model requiring staff to be in the office at least 25% of the time, with some roles possibly needing more in-office presence.
  • Visa sponsorship support, with efforts made to secure visas for successful candidates, assisted by legal experts.
  • Opportunity to work on impactful AI research projects within a collaborative environment.
  • Modern office facilities in San Francisco, with options for remote work and relocation assistance.
  • Generous vacation and parental leave policies.
  • Optional equity donation matching program.
  • A culture emphasizing diversity, inclusion, and ethical AI development.

Anthropic is committed to building a diverse team and encourages applicants from underrepresented groups to apply. The company emphasizes the importance of ethical AI and social responsibility in its work.


Degree Requirement: Not Specified

Visa Sponsorship Promising

To apply for this job please visit job-boards.greenhouse.io.