Search and Ranking Q&A

Q: What if there are a very large number of retrieval results, like millions? For our head queries like “books,” we have many millions of results. How do you fix this?

Yes, we handle this! Our method:

• Help you use your existing search retrieval(s) systems to maximize recall with potentially low precision
• Dynamically allocate every response without caching so that all relevant items get some delivery proportional to how much value they create
• Add retrieval sampling and pacing to force good faith, eventual retrieval and attempted delivery of every eligible item
• Use real-time, within-session personalization and sponsored search and novelty exploration to further customize and optimize head query performance in a way that may resemble a “relevant recommendations feed”

Commercial search for marketplaces is two-sided. There isn’t a definitive “correct” or “best” single result. Instead, head and torso queries function more like “directed browse” surfaces, and all acceptable results need some potential delivery. This is more similar to performance advertising than document search.

Promoted.ai specializes in this type of “matching” search. We solve this by leveraging your existing search retrieval systems to maximize probabilistic recall and using our 2nd stage and blender systems to maximize precision, measured as relevance and business value. We do not recommend a complex single search retrieval system that maximizes too many things. Instead, search retrieval should be designed to be efficient and to maximize probabilistic recall: if repeatedly called X times, all relevant items will appear with Y probability depending on the corpus size and the retrieval set size. We recommend using a (potentially abstracted, depending on the search retrieval system) combination of different retrieval queries with a union: a combination of keyword and vector retrieval and different rankings including “top” “new” and “sampled.” The goal is a fault-tolerant, highly efficient, easy to explain system that maximizes recall. Nearly all modern search systems can be configured to achieve these goals given a powerful 2nd stage ranking and allocation system to correct imprecisions like Promoted.

This is in contrast to many simpler search systems, which use a “quality score” to rank search results by “goodness” and then use the same “top K” query. This strategy will cause only the first few hundred relevant results to be repeatedly delivered, resulting in an “iceberg” effect of 90% or more of the relevant inventory never getting delivered. This is especially problematic for “cold start” and new items. Promoted solves this.

Q: What do users experience? What do they see?

Users see your search and recommendations as they already see them, but more relevant! They buy more, and therefore, sellers sell more. Promoted is purely a backend service, and the user experience is not changed at all. We believe that users would like their existing search systems work correctly, not for new search experiences.

If you would like, Promoted supports “presentation explanation” labels like “personalized for you” or “because you’ve bought this before” or “new to you” types of synthetic labels to explain recommendations. We pass these as Response.Insertion.Properties for you to pass through to your UI for display.

Q: How many items do you rank per request?

We don’t have a hard limit. In practice, a reasonable limit is about 1,000 items. More items can increase system infrastructure costs and latency beyond the business metric efficiency gains. Larger retrieval sets can be handled using probabilistic sampling from retrieval.

Q: Do different surfaces and applications have different objectives?

Yes, you can set different optimization objectives for different surfaces and even users. All of our customers eventually evolve to this state. For example, one customer has a different objective for new versus established users in combination with different objectives for search and discovery surfaces.

Q: How do you optimize for relevance (our goal) when our business goal is GMV and revenue?

We don’t use a single “quality score” to rank. We estimate “human semantic relevance” and “business value” as separate dimensions. We maximize business value (typically a combination of sales and user-purchase-propensity) subject to sufficient relevance. This is a more complex allocation algorithm beyond a simple “sort.” We also support additional objectives like new seller activation, diversity, and seller behavior incentives like downrankings for cancellations or poor responses.

Q: How fast is Promoted.ai? What are your latencies?

Promoted is very fast! We support multi-region deployments with services deployed in your data centers. We implement our systems in C++ for maximum performance. If you need even more performance, we have “on-premise” licensing options available. Customers see a range between 20ms to 150ms p95 latencies.

Q: You mentioned several different types of searches at our company. Do you support all of these different search and discovery systems?

Yes, Promoted is a general search and discovery allocation and ranking system. We support many kinds of surfaces and products. For you, we can support all your search surfaces so that you can unify optimization and metrics.

Q: Do you use Learn to Rank algorithm?

Technically, no, we do not use the “Learn to Rank” algorithm. However, if your team uses this algorithm, we can use it as a feature in our models. We use “multi-task engagement” models and “relevance” models that are more typically used in Meta and Pinterest newsfeed / homefeed ranking.

Q: What is the largest catalog size that you support?
We do not have a catalog limit size. All our customers have complex inventory including hierarchies, ephemeral, non-fungible, and highly dynamic properties and complex value functions.

Our systems generally perform better at larger sizes because:

• There is more to optimize, so more room for impact
• There is more data to learn from, so models make better predictions

Q: Does Promoted provide search and discovery retrieval systems?

Promoted does not currently provide a retrieval system as a service as part of our standard offering. Instead, we help you configure and optimize your existing retrieval systems to be more efficient in combination with Promoted’s ranking and allocation systems.

Q: How do you integrate with our systems?

Generally, we integrate just as if you had a dedicated and mature internal search and discovery engineering team. We do all our own data processing, and we can rank any search system in an agnostic way.
https://docs.promoted.ai/docs/how-to-integrate-with-promoted

Q: How do you think about measurement compared to ML?

We LOVE measurement. We run our own metrics processing infrastructure, and we are obsessed with consistent and correct cross-platform and cross-product measurement. Correct measurement is critical to getting the best AI results.