Introspection Report

Purpose

Promoted provides tools for detailed analysis of the metrics behind individual search queries and the corresponding ranking of results. This powerful tool is called Introspection, which is shown through a comprehensive report. The Introspection report unifies search queries and ranked insertions in a unified Google spreadsheet interface. The main tab provides insights into item ranking, including the associated scores, rules, and feature values. Additional tabs contain key data such as user engagement sequences and the full raw delivery log, aiding in debugging.

Consolidating the Introspection report on Google Sheets allows for easy sharing, sorting, and commenting. Click here to access a fictional Introspection Report.

Use Cases

Click here to learn how Outschool, a Promoted customer, leveraged the Introspection report for debugging search functionality.

Introspection Sections

There are five tabs in the Google Sheet corresponding to different sections of introspection, ranging from user engagement data to listing details:

  1. delivery_detail: The main tab of the Introspection report showcases the item rankings and the corresponding scores and metrics that determine why each item ranks as it does for that particular search query.
  2. delivery_metadata: This tab houses select user and model data from the delivery request in a readable format, in addition to the full raw delivery log linked at the top.
  3. blender_config: This page houses the code for specific scores and metrics that make up the allocation rules.
  4. delivery_cohorts: This tab identifies which experiments (e.g., for A/B testing), if any, are running for that user, and which cohort the user is in.
  5. seq_raw_logs: This tab details the current user sequences, which is essentially a list of all previous user engagement events like navigates and purchases. This is used in modeling to better predict clicks and conversions.

delivery_detail Tab

The first tab within the Introspection report shows an overview of the search report, containing the ranked results and the scores that determined the rankings for a particular search query.
The search query is contained in cell B2, and the listings are shown in column B in order of Promoted’s rankings. Items that were evaluated by Promoted but not ranked are shown beneath the ranked listings. Looking down the row containing each item, you will see all of the item's associated scores and metrics. These scores are what determine the ranking.

Key Scores and Metrics on the delivery_details Tab

Retrieval

Column D-E

Promoted Response Position and Request Retrieval Score: The placement of listings by Promoted is stated in column C, while column D states where the platform would otherwise have ranked it. This enables the identification of any anomalies or irregularities in the ranking.

Predictions

Column F-G

Predict Click: A prediction of the likelihood that a user will click on that particular listing.

Predict Post-Click Purchase: A prediction of the likelihood that the user will purchase after clicking on the listing.

Hidden Scores

Column H-P

Some rows and columns are hidden by default. Column A contains the contents of the corresponding item, and rows 4–6 contain additional details about the type of feature (user, content, request) and the relative importance of each feature to the model.

Some of the columns describing the intermediate Blender values are also hidden. More details on Blender are below.

Blender

Column Q-R

Blender Utility Score: A “score” that is calculated by unique metrics, found in blender_config, which determines the ranking. The rules that makeup blender are shown in the blender_config tab. This is customizable and different for every marketplace.

Blender Allocation: What Blender rule, which is found in the blender_config sheet, is being applied to that specific position.
This section can be expanded to show the rest of the scores are calculated within the Blender Utility Score.

Selected Features and Counters

Column S-T

Boosted_profile_config.is_boosted: This is a boolean value that shows whether a particular item is eligible to be boosted.

Boosted_profile_config.bid_value: This is how much an advertiser or listing owner is willing to pay for a given advertisement.

Other Important Scores under Selected Features and Counters

Impressions 30 days: The number of impressions on that item for the past 30 days.

Clicks 30 days: The number of clicks on that item for the past 30 days.

Checkout 30 days: The number of checkouts for that item in the last 30 days.

Purchase 30 days: The number of purchases of that item in the last 30 days.

Column U-on

Other scores and metrics that are assigned to the items that are considered in the ranking.

Boosted Feature

Promoted implements native ads through “boosted” listings. Boosting is implemented via a separate blender rule that combines model predictions, the bid value, and other metrics to optimally rank particular listings higher in the search results. This implementation can range from simple to complex, depending on the needs of your marketplace. This feature ensures that native ads are prioritized and strategically positioned within the search results for maximum impact, enhancing their visibility without disrupting the user experience.

As demonstrated in the Introspection report above, there are three boosted items in this search result: B9, B11, and B14.

The listing "Graphic Designer || Figma Designer," "Senior Graphic Designer | UI Designer", and "Expert Graphic Designer | Web Designer | Main Page" are boosted listings that have been elevated in the search ranking, increasing visibility and click-through rates. The Introspection report helps identify where and why these listings appear.

blender_metadata Tab


The delivery_metadata tab is the delivery log laid out in a readable format. This tab contains information about the user and the request. This tab’s purpose is for easy readability and debugging.
The raw delivery log is linked at the top of the tab for further insight.

blender_config Tab

This page contains the configuration code for the Blender, a critical component in content delivery. The Blender algorithm integrates quality scores, applies positioning algorithms, and prioritizes content based on defined attributes to ensure result diversity. Here, you can review the specific rules each item undergoes for ranking. These rules collectively define the Blender's operation. Initial functions determine eligibility criteria, such as whether an item qualifies for a 'boosted' position. Subsequent functions include score calculations and sorting functions that directly influence search result rankings.

Read more about Blender here Configuring Blender

delivery_cohorts Tab

This tab identifies which experiments (e.g., for A/B testing), if any, are found to be running for that user.

seq_raw_logs Tab

This tab details the current sequences, which are past user events that may have been observed by a user and are used in modeling.

As shown in this seq_raw_logs this user has made impressions that were logged. This user has clicked or "NAVIGATE"(ed) onto items and has taken some items to "CHECKOUT". This information is used in modeling, which produces more tailored and personalized search results for a specific user.