Software Engineer Query Engines

Palantir

Software Engineer Query Engines Overview

Company Name Palantir
Job Role Software Engineer Query Engines
Qualifications Not Specified
Category IT Jobs
Job Type Full Time
Location London

Palantir Technologies is seeking a talented Software Engineer specializing in Query Engines to join their team in London, United Kingdom. This role is focused on developing and enhancing the core data processing infrastructure that supports Palantir Foundry, a platform used by organizations to make critical data-driven decisions. The position involves working on open-source data technologies such as Apache Spark, Apache DataFusion, and Apache Iceberg, with a focus on query planning, execution, and system optimization.

As a Software Engineer in this role, you will be responsible for designing and implementing sophisticated query planning and optimization features that convert complex computations into efficient execution plans. You will extend existing data processing engines by adding new capabilities, improving query operators, and optimizing aspects like parallelism, memory management, and data movement. Additionally, you will work on developing Foundryâ??s Iceberg catalog and engine integrations, managing table metadata, transactional operations, and ensuring efficient read/write processes.

Your work will also involve creating shared transformation semantics and execution interfaces that support various workloads across Palantirâ??s platform. You will focus on enhancing incremental processing pipelines to enable reuse of previous results, thereby improving efficiency and correctness when handling new data. Evaluating and integrating advances from the open-source data systems community will be a key part of your role, ensuring that new technologies are validated for performance and reliability in real-world applications.

Investigating and resolving issues related to correctness and performance across different layers of the data stack is essential, along with developing tests and benchmarks to prevent regressions. Collaboration with product teams and customers to translate operational needs into engine capabilities that align with Palantirâ??s security and data management standards is also expected.

Technologies We Use

  • Programming languages including Java, Scala, Rust, and Python
  • Open-source data processing frameworks such as Apache Spark, DataFusion, Comet, and Velox
  • Table management and catalog interoperability via Apache Iceberg
  • In-memory data processing with Apache Arrow and columnar storage with Apache Parquet
  • Industry-standard build tools like Gradle, Cargo, and GitHub

Core Responsibilities

  • Designing and implementing query planning and optimization features for efficient data computation
  • Extending data processing engines with new capabilities and improving query operators and data handling
  • Developing and maintaining Iceberg catalog and engine integrations, including transactional support
  • Building shared semantics and interfaces for diverse workloads across Palantirâ??s platform
  • Enhancing incremental processing pipelines for better reuse and efficiency
  • Evaluating open-source data system innovations for performance and reliability
  • Investigating and resolving correctness and performance issues, creating tests and benchmarks
  • Working closely with product teams and clients to develop engine features aligned with operational needs and security standards

What We Value

  • A strong ownership mentality and a commitment to system correctness, as these systems support critical decisions and operations
  • Curiosity about data system internals, including query optimization, execution operators, table formats, and distributed processing
  • Excellent debugging skills and motivation to trace problems across multiple layers and languages
  • A practical approach to performance improvements, grounded in profiling and real workload testing
  • An interest in applying deep systems engineering to solve real-world problems with empathy for users
  • Experience with systems like Spark, DataFusion, Iceberg, or similar, and a willingness to learn across the stack
  • Ability to collaborate effectively across teams and contribute to open-source projects (valued but not mandatory)

What We Require

  • At least four years of professional experience in building and operating production-grade data systems
  • A background in Computer Science, Mathematics, Software Engineering, Physics, or related fields, or equivalent practical experience
  • Strong coding skills in Java, Rust, Scala, or C++, demonstrated through previous work
  • Experience with database engines, distributed data processing, storage systems, or similar infrastructure, especially in query planning, execution, or optimization
  • Solid understanding of algorithms, data structures, and concurrency, with the ability to troubleshoot complex correctness and performance issues
  • Excellent communication skills, both written and verbal, with a proven ability to work well across teams and incorporate feedback
  • A high standard for quality and attention to detail in software development

Palantir values excellence regardless of background and is dedicated to making the application process accessible to all. They offer accommodations for applicants with disabilities and emphasize that candidates will never be asked to pay or share financial information during the interview process. The company encourages work from their offices to foster collaboration and innovation, with many teams offering hybrid work options. Some roles may allow remote work on an exceptional basis, but applicants must work from the same city or country as their employment location. Palantir is committed to diversity, inclusion, and supporting the growth of all employees.


Degree Requirement: Not Specified

Visa Sponsorship Promising

To apply for this job please visit jobs.lever.co.