Daily ML Reports

Promoted.ai continuously generates multiple models for optimizing your search, feed, promotions, and ads. Every day, Promoted generates several reports to let you know that the system is still working and to show how optimization systems are continuously improving.

The four most common use cases for these reports are:

• To verify that optimization has started and is continuing to run
• To monitor that there is no sudden change in metrics or unusual behavior like a model failing to update for a long period of time, indicating a potential system failure
• To show which features are most important for driving optimization performance and if these features have suddenly changed recently. A sudden change could indicate a system failure.
• If newly added features are beginning to contribute to driving optimization performance and if this impact is increasing over time. This verifies that new feature addition was successful.

Shared Slack Channel for Model Training Notifications

Promoted creates a shared slack channel for daily model updates and feature reports. You can directly respond to these messages to raise them to Promoted's or your own team's attention. There are two types of messages in this channel:

• Daily Model Statistics: numeric model quality statistics computed from offline evaluation about the latest model
• Daily Feature Reports: links to spreadsheets describing what features were seen and used in which models, how important these features were, and if these feature importances are changing over time.

"Daily model statistics" looks like Figure 1. It shows:

[SUMMARY] Release Decisions

Promoted has trained new click and post-click conversion models. If two estimates of model performance for the new model exceed the current production model, then the production model is "Updated." This happens automatically when both the AUC and NCE tests succeed. It is common that several days pass before a model is updated due to ordinary variance, selection bias, and random chance. However, if a model does not update after a week or more, or if the new model dramatically outperforms or underperforms the current production model (+/-1%), then an additional investigation may be warranted, reach out to Promoted for guidance.

[PERFORMANCE] Per-Day Model Stats

Promoted uses a variety of machine learning metrics to estimate model health. Promoted uses these metrics to look for major differences between models or evaluation days that could indicate a system regression. Generally, non-data scientists do not need to worry about this section.

Daily Feature Report Links

When new models are trained, new feature reports are generated. These reports are always generated with the latest daily model, even if the latest daily model was not published to production, see Feature Importance Report. These reports show the latest feature importances and how they have changed over time. The three reports are:

• Descriptive Feature Importance: A simplified, unified report that best answers "what features drive model performance overall" in an intelligible way. Excludes item engagement features to help show the importance of content features that are otherwise made redundant by item engagement features.
• Debug Feature Importance: A more technical report that shows feature importance for all features for all models. More useful for engineering and debugging.
• Feature Coverage Report: Statistics of features used in the models.