Staff Software Engineer - Distributed Tracing Backend (Tempo)

Company and Role Overview

Grafana Labs develops Grafana Cloud, an open observability cloud designed for scale and flexibility. Trusted by industry leaders such as Microsoft, NVIDIA, and Bloomberg, Grafana Cloud helps organizations maintain reliability, resolve incidents swiftly, and optimize telemetry data. With a fully remote team across over 40 countries, Grafana Labs fosters a transparent, autonomous, and inclusive environment grounded in open-source values.

This opportunity targets candidates located in Spain, Sweden, UK, Ireland, or Germany.

Role Description

The Staff Software Engineer will focus on Grafana's Tempo, an open-source distributed tracing backend powering Grafana Cloud Traces and Enterprise Traces. Tempo enables trace searching, metrics generation from spans, and correlation of tracing data with logs and profiles.

In 2026, the team will transition from foundational Tempo architecture enhancements to advancing product features and operational excellence, expanding Tempo into a platform powering Grafana's next-gen observability offerings.

Key Responsibilities

• Ensure Grafana Cloud Traces offers seamless customer experiences by resolving usability issues and eliminating hidden failure modes.
• Scale operations to support a triple-digit number of Tempo cells through autoscaling, parameterized rollouts, and toil reduction.
• Advance Tempo into a platform with enhanced APIs, trace aggregation, TraceQL metrics math, and interfaces supporting AI and downstream products.
• Optimize performance for faster query latencies and support high ingestion rates and extended query ranges.
• Adapt Tempo for high-cardinality, bursty workloads and AI-driven workflows like assistant-guided triage.
• Lead and oversee multi-quarter initiatives from conception through deployment, including architecture ownership and design reviews.
• Design API interfaces structured for both human and automated consumption to support next-gen integrations.
• Promote operational excellence by managing SLOs, reducing toil, automating workflows, and leading incident response and on-call duties.
• Collaborate closely with Product Managers and related teams for integrated product delivery.
• Mentor engineers through code reviews, pair programming, and knowledge sharing.
• Engage with the open-source community by reviewing contributions and guiding project direction.

Ideal Candidate Profile

• Proven leadership in delivering complex, long-term software initiatives enhancing team capability.
• Extensive hands-on experience with distributed data systems in production, including ingestion pipelines, storage engines, and query components.
• Strong software engineering skills with an emphasis on writing maintainable, efficient code and balanced optimization.
• Experience in Go or equivalent systems programming languages like Rust, C, or C++.
• Operational experience managing production services, on-call rotations, and prioritizing SLOs as key features.
• Customer-centric problem-solving approach emphasizing iterative delivery and pragmatic feedback loops.
• Exceptional communication skills suited for fully remote, asynchronous teamwork.

Additional Qualifications (Preferred)

• Expertise with tracing technologies, OpenTelemetry, or large-scale observability systems.
• Experience designing query languages or programmatic APIs (e.g., SQL/TraceQL).
• Knowledge of columnar storage formats like Parquet or analytical on-disk formats.
• Experience operating scalable multi-tenant SaaS infrastructures on Kubernetes.
• Development for AI/LLM consumption including structured APIs and metadata endpoints.
• Active open-source contributor or maintainer and comfort engaging in open communities.
• Prior use of Grafana, Prometheus, Loki, or Tempo in production or similar setups.
• Experience working in a fully remote, globally distributed environment.

Working Environment and Benefits

• Fully remote, global team culture promoting collaboration, transparency, and creativity.
• Competitive salary range for Ireland residents: EUR 117,600 - EUR 141,120, plus equity and bonuses.
• Support for modern AI-assisted development workflows with company-funded tooling budgets.
• Career growth with structured pathways and approachable leadership.
• 30 days annual leave including 3 mandatory company-wide shutdown days to encourage rest and disconnection.
• Commitment to equal opportunity employment and diversity in the workplace.

Additional Information

Grafana Labs strongly supports inclusive hiring practices. Recruitment processes may include AI-assisted resume screening combined with manual assessments. Applicants’ personal data usage adheres to the company's privacy policies.