A/B Testing

Overview

Promoted’s framework allows for the assignment of experiment arm labels to users, facilitating any type of experiment assignment. This is crucial for enabling Promoted to recognize and interact with these assignments effectively, enhancing the relevance and accuracy of its predictions and rankings.

Collaboration and Experiment Management

While Promoted complements existing experiment tools, it actively collaborates on experiment evaluation and can conduct internal A/B testing, relevant to both Promoted's system improvements and potential customer insights. Customers can either run their own experiments and share the assignments with Promoted or request Promoted to manage the experiment assignments.

Importance of Experiment Labels

It's essential for Promoted to receive experiment group labels. This is especially true for Promoted-related experiments, which assess the safety and impact of integration and help determine whether users experience Promoted's rankings or the default setup. Early experiments may compare different objectives, like click-through versus conversion rates, aiding in refining the business objectives integration.

Promoted-internal experiments

Promoted's system can autonomously assign users to different groups for experiments primarily affecting its own predictions and rankings. Promoted is constantly running experiments internally which may or may not be relevant to expose to end users, such as new infrastructure.

UI/UX Experimentation and Model Training

Promoted's model needs comprehensive data to predict user behavior accurately. For instance, in UI experiments, knowing the context like display format (FULL versus COMPACT) helps the model learn that reviews impact click probability only in FULL displays. Similarly, experiments altering product titles (e.g., 'wool socks' versus 'cozy socks') should provide these variations during the delivery API call. This ensures the model can accurately learn from these changes, avoiding the need to relearn from scratch after the experiment ends.

Outcome Analysis

Customers can collaborate with Promoted to query the data warehouse for analysis to evaluate experiment outcomes. Outcomes are analyzed in data notebooks, and Promoted can provide periodic exports of assignments and the metrics needed for comprehensive analysis. This service is tailored to the unique goal metrics of each customer, acknowledging that many customers prefer their own experiment tools for consistency and comprehensive analysis, including metrics beyond Promoted's scope.

Promoted also generates custom reports on feature importance, aiding in understanding the impact of UI/UX changes on user behavior and optimizing future experiments.

Conclusion

In conclusion, Promoted serves as a versatile partner in the A/B testing process, offering robust integration capabilities, detailed analytical support, and a commitment to improving experiment outcomes and user experience insights.