Introspection Report
Understanding Promoted's powerful analysis and debugging tool.
Purpose
Promoted provides tools for detailed analysis of the metrics behind individual search queries and the corresponding ranking of search, discovery, and native ad results on each request. This powerful tool is called Introspection, which is shown through a comprehensive report. The Introspection report unifies search queries and ranked insertions in a Google spreadsheet interface. The spreadsheet consists of multiple tabs, each providing insight into different parts of Promoted's delivery. The main tab provides insights into item ranking, including the associated scores, rules, and feature values. Additional tabs contain key data such as user engagement sequences and the full raw delivery log, aiding in debugging.
Consolidating the Introspection report on Google Sheets allows for easy sharing, sorting, and commenting.
- Example fictional Introspection report (reviewed in this document)
- A high-level presentation of introspection on our Solutions webpage
Use Cases
Click here to learn how Outschool, a Promoted customer, leverages the Introspection report for debugging search functionality.
Introspection Report Sections
There are five tabs in the Google Sheet corresponding to different sections of introspection, ranging from user engagement data to listing details:
| Section | Purpose |
|---|---|
| delivery_detail | The item rankings and the corresponding scores and metrics that determine why each item ranks as it does for that particular search query. |
| delivery_metadata | Select user and model data from the delivery request in a readable format, in addition to the full raw delivery log linked at the top. |
| blender_config | The code for specific scores and metrics that make up the allocation rules. |
| delivery_cohorts | The experiments (e.g., for A/B testing), if any, are running for that user, and which cohort the user is in. |
| seq_user_events | A shortened list of current user sequences, which is essentially a list of all previous user engagement events like navigates and purchases. This is used in modeling to better predict clicks and conversions. |
| seq_user_events_all | A complete list of user sequences. |
delivery_detail
The first tab within the Introspection report shows an overview of the search report, containing the ranked results and the scores that determined the rankings for a particular search query.
The search query is contained in cell B2, and the listings are shown in column B in order of Promoted’s rankings. Items that were evaluated by Promoted but not ranked are shown beneath the ranked listings. Looking down the row containing each item, you will see all of the item's associated scores and metrics. These scores are what determine the ranking.
Key Scores and Metrics on delivery_detail
Content
| Column | Value | Description |
|---|---|---|
| A (hidden) | Content ID | The ID of the content item, for debugging. |
| B | Content Name | The name or title of the item. |
| C | Promoted Response Position | The order of the placement of listings by Promoted. |
Retrieval
| Column | Value | Description |
|---|---|---|
| D | Request Retrieval Score | The retrieval score generated by the platform's own search retrieval system. |
| E | Platform Retrieval Rank | Where the platform would have ranked it without Promoted. This enables the identification of any anomalies or irregularities in the ranking. |
Highlighted Features
| Column | Value | Description |
|---|---|---|
| F | Feature X | Depending on the client, a particular feature may be highlighted in the retrieval section. This feature is shown as an example. |
Predictions
This includes the prediction results from the model.
| Column | Value | Description |
|---|---|---|
| G | Predict Click | A prediction of the likelihood that a user will click on that particular listing. |
| H | Predict Post-Click Purchase | A prediction of the likelihood that the user will purchase after clicking on the listing. |
Blender
Blender is Promoted's system that allocates items. This section can be expanded to show the rest of the scores that factor into the Blender Utility Score calculation.
| Column | Value | Description |
|---|---|---|
| I to R (hidden) | (various Blender scores) | Intermediate Blender values used in the overall Blender utility score calculation. These may be used for further insight and debugging. |
| S | Blender Utility Score | A “score” that is calculated by unique metrics, found in blender_config, which determines the ranking. The rules that makeup blender are shown in the blender_config tab. This is customizable and different for every marketplace. |
| T | Blender Allocation | What Blender rule, which is found in the blender_config sheet, is being applied to that specific position. |
Selected Features and Counters
| Column | Value | Description |
|---|---|---|
| U | Relevance 1 Rating | A score generated by Promoted's relevance model, that signifies how relevant the item is to the search query. |
| V | Boosted_profile_config.is_boosted | A boolean value that shows whether a particular item is eligible to be boosted. |
| W | Boosted_profile_config.bid_value | How much an advertiser or listing owner is willing to pay for a given advertisement. |
| X and beyond | (various) | The remaining columns represent various features and counters. It is a comprehensive but abbreviated list; see the feature importance report for a full accounting. |
Hidden Values and Scores
Some rows and columns are hidden by default for a more concise view. For more detail, expand the sections on the Google Sheet. Rows 4–6 contain additional details about the type of feature (user, content, request) and the relative importance of each feature to the model.
Boosted Listings
Promoted implements native ads through “boosted” listings. Boosting is implemented via a separate blender rule that combines model predictions, the bid value, and other metrics to rank eligible listings higher in the search results than they ordinarily would have been. This implementation can range from simple to complex, depending on the needs of your marketplace. This feature ensures that native ads are prioritized and strategically positioned within the search results for maximum impact, enhancing their visibility without disrupting the user experience.
There are three boosted items in this search result: B9, B11, and B14.
The Boosted listings are highlighted in this screenshot.
delivery_metadata
The delivery_metadata tab is the delivery log laid out in a readable format. This tab contains information about the user and the request. This tab’s purpose is for easy readability and debugging. The raw delivery log is linked at the top of the tab for further insight.
blender_config
This page contains the configuration code for the Blender, a critical component in content delivery. The Blender algorithm integrates quality scores, applies positioning algorithms, and prioritizes content based on defined attributes to ensure result diversity. Here, you can review the specific rules each item undergoes for ranking. These rules collectively define the Blender's operation. Initial functions determine eligibility criteria, such as whether an item qualifies for a 'boosted' position. Subsequent functions include score calculations and sorting functions that directly influence search result rankings.
Read more about Blender here Configuring Blender
delivery_cohorts
This tab identifies which experiments (e.g., for A/B testing), if any, are found to be running for that user.
seq_user_events
This tab details the current sequences, which are past user events that may have been observed by a user and are used in modeling.
As shown in this seq_raw_logs this user has made impressions that were logged. This user has clicked or navigated onto items and has taken some items to "CHECKOUT". This information is used in modeling, which produces more tailored and personalized search results for a specific user.
Updated about 2 months ago