Introspection Report

Understanding Promoted's powerful analysis and debugging tool.

Purpose

Promoted provides tools for detailed analysis of the metrics behind individual search queries and the corresponding ranking of search, discovery, and native ad results on each request. This powerful tool is called Introspection, which is shown through a comprehensive report. The Introspection report unifies search queries and ranked insertions in a Google spreadsheet interface. The spreadsheet consists of multiple tabs, each providing insight into different parts of Promoted's delivery. The main tab provides insights into item ranking, including the associated scores, rules, and feature values. Additional tabs contain key data such as user engagement sequences and the full raw delivery log, aiding in debugging.

Consolidating the Introspection report on Google Sheets allows for easy sharing, sorting, and commenting.

Use Cases

Click here to learn how Outschool, a Promoted customer, leverages the Introspection report for debugging search functionality.

Introspection Report Sections

There are five tabs in the Google Sheet corresponding to different sections of introspection, ranging from user engagement data to listing details:

SectionPurpose
delivery_detailThe item rankings and the corresponding scores and metrics that determine why each item ranks as it does for that particular search query.
delivery_metadataSelect user and model data from the delivery request in a readable format, in addition to the full raw delivery log linked at the top.
blender_configThe code for specific scores and metrics that make up the allocation rules.
delivery_cohortsThe experiments (e.g., for A/B testing), if any, are running for that user, and which cohort the user is in.
seq_user_eventsA shortened list of current user sequences, which is essentially a list of all previous user engagement events like navigates and purchases. This is used in modeling to better predict clicks and conversions.
seq_user_events_allA complete list of user sequences.

delivery_detail

The first tab within the Introspection report shows an overview of the search report, containing the ranked results and the scores that determined the rankings for a particular search query.
The search query is contained in cell B2, and the listings are shown in column B in order of Promoted’s rankings. Items that were evaluated by Promoted but not ranked are shown beneath the ranked listings. Looking down the row containing each item, you will see all of the item's associated scores and metrics. These scores are what determine the ranking.

Key Scores and Metrics on delivery_detail

Content

ColumnValueDescription
A (hidden)Content IDThe ID of the content item, for debugging.
BContent NameThe name or title of the item.
CPromoted Response PositionThe order of the placement of listings by Promoted.

Retrieval

ColumnValueDescription
DRequest Retrieval ScoreThe retrieval score generated by the platform's own search retrieval system.
EPlatform Retrieval RankWhere the platform would have ranked it without Promoted. This enables the identification of any anomalies or irregularities in the ranking.

Highlighted Features

ColumnValueDescription
FFeature XDepending on the client, a particular feature may be highlighted in the retrieval section. This feature is shown as an example.

Predictions

This includes the prediction results from the model.

ColumnValueDescription
GPredict ClickA prediction of the likelihood that a user will click on that particular listing.
HPredict Post-Click PurchaseA prediction of the likelihood that the user will purchase after clicking on the listing.

Blender

Blender is Promoted's system that allocates items. This section can be expanded to show the rest of the scores that factor into the Blender Utility Score calculation.

ColumnValueDescription
I to R (hidden)(various Blender scores)Intermediate Blender values used in the overall Blender utility score calculation. These may be used for further insight and debugging.
SBlender Utility ScoreA “score” that is calculated by unique metrics, found in blender_config, which determines the ranking. The rules that makeup blender are shown in the blender_config tab. This is customizable and different for every marketplace.
TBlender AllocationWhat Blender rule, which is found in the blender_config sheet, is being applied to that specific position.

Selected Features and Counters

ColumnValueDescription
URelevance 1 RatingA score generated by Promoted's relevance model, that signifies how relevant the item is to the search query.
VBoosted_profile_config.is_boostedA boolean value that shows whether a particular item is eligible to be boosted.
WBoosted_profile_config.bid_valueHow much an advertiser or listing owner is willing to pay for a given advertisement.
X and beyond(various)The remaining columns represent various features and counters. It is a comprehensive but abbreviated list; see the feature importance report for a full accounting.

Hidden Values and Scores

Some rows and columns are hidden by default for a more concise view. For more detail, expand the sections on the Google Sheet. Rows 4–6 contain additional details about the type of feature (user, content, request) and the relative importance of each feature to the model.

Boosted Listings

Promoted implements native ads through “boosted” listings. Boosting is implemented via a separate blender rule that combines model predictions, the bid value, and other metrics to rank eligible listings higher in the search results than they ordinarily would have been. This implementation can range from simple to complex, depending on the needs of your marketplace. This feature ensures that native ads are prioritized and strategically positioned within the search results for maximum impact, enhancing their visibility without disrupting the user experience.

There are three boosted items in this search result: B9, B11, and B14.

The Boosted listings are highlighted in this screenshot.

The Boosted listings are highlighted in this screenshot.

delivery_metadata

The delivery_metadata tab is the delivery log laid out in a readable format. This tab contains information about the user and the request. This tab’s purpose is for easy readability and debugging. The raw delivery log is linked at the top of the tab for further insight.

blender_config

This page contains the configuration code for the Blender, a critical component in content delivery. The Blender algorithm integrates quality scores, applies positioning algorithms, and prioritizes content based on defined attributes to ensure result diversity. Here, you can review the specific rules each item undergoes for ranking. These rules collectively define the Blender's operation. Initial functions determine eligibility criteria, such as whether an item qualifies for a 'boosted' position. Subsequent functions include score calculations and sorting functions that directly influence search result rankings.

Read more about Blender here Configuring Blender

delivery_cohorts

This tab identifies which experiments (e.g., for A/B testing), if any, are found to be running for that user.

seq_user_events

This tab details the current sequences, which are past user events that may have been observed by a user and are used in modeling.

As shown in this seq_raw_logs this user has made impressions that were logged. This user has clicked or navigated onto items and has taken some items to "CHECKOUT". This information is used in modeling, which produces more tailored and personalized search results for a specific user.