Staff Software Engineer - Machine Learning for Search

About the Role

The Applied AI team at Databricks is pioneering AI and machine learning driven products that transform how users interact with their data. Our customers generate vast assets ranging from tables and notebooks to dashboards and machine learning models, some numbering in the hundreds of millions. Facilitating effective asset discovery is critical to their productivity.

As part of advancing our Search product, we are looking for an experienced Staff Software Engineer to lead improvements on search relevance. The focus for 2025 will be on optimizing search ranking algorithms, enhancing query comprehension, expanding coverage of assets, and building robust evaluation systems to support scalable, seamless search experiences.

Key Responsibilities

• Lead the creation and deployment of machine learning-based search relevance models integrated within Databricks’ product ecosystem.
• Develop automated ML and NLP pipelines spanning data preprocessing, query understanding and rewriting, retrieval and ranking, as well as comprehensive model evaluation to accelerate experimentation.
• Collaborate closely with product managers and cross-disciplinary teams to drive technology-led initiatives that inform innovative business strategies and product development in the search and discovery domain.
• Architect and implement a rigorous evaluation framework measuring search ranking advancements through both offline and online methodologies.

Candidate Profile

• Holds a Bachelor's degree in Computer Science or related field; advanced degrees (M.S. or PhD) preferred.
• Brings over 10 years of experience building scalable search relevance systems, either in production environments or impactful research settings.
• Demonstrated skills applying large language models (LLMs) to improve search relevance outcomes.
• Proficiency in one or more areas: query understanding, natural language processing (NLP), text mining, recommendation systems, personalization, discovery, or conversational AI.
• Strong foundation in computer science fundamentals.
• Contributions to popular open-source projects are a plus.

Why Join Databricks?

We are at the forefront of creating cutting-edge AI solutions that redefine data interaction. Here, you will have the opportunity to shape AI-driven product futures, engage with advanced models, and be part of a world-class team of AI and ML specialists.

About Databricks

Databricks is recognized as the Data and AI company, trusted by over 20,000 organizations worldwide including major global brands and 70% of Fortune 500 companies. With its headquarters in San Francisco and more than 30 offices worldwide, Databricks delivers a unified platform featuring technologies such as Genie, Lakebase, Agent Bricks, Lakeflow, Lakehouse, and Unity Catalog.

Diversity & Inclusion Commitment

Databricks fosters a culture of inclusion and equal opportunity, valuing diverse perspectives and employing fair hiring processes that do not discriminate based on age, ethnicity, gender identity, disability, or other protected traits.

Compliance Note

For roles requiring access to export-controlled technology or source code, authorization and licensing are at the employer’s discretion. Applicants may be declined based solely on government licensing considerations.