A/B Testing
Overview
Promoted’s framework allows for the assignment of experiment arm labels to users, facilitating any type of experiment assignment. This is crucial for enabling Promoted to recognize and interact with these assignments effectively, enhancing the relevance and accuracy of its predictions and rankings.
Collaboration and Experiment Management
While Promoted complements existing experiment tools, it actively collaborates on experiment evaluation and can conduct internal A/B testing, relevant to both Promoted's system improvements and potential customer insights. Customers can either run their own experiments and share the assignments with Promoted or request Promoted to manage the experiment assignments.
Importance of Experiment Labels
It's essential for Promoted to receive experiment group labels. This is especially true for Promoted-related experiments, which assess the safety and impact of integration and help determine whether users experience Promoted's rankings or the default setup. Early experiments may compare different objectives, like click-through versus conversion rates, aiding in refining the business objectives integration.
Promoted-internal experiments
Promoted's system can autonomously assign users to different groups for experiments primarily affecting its own predictions and rankings. Promoted is constantly running experiments internally which may or may not be relevant to expose to end users, such as new infrastructure.
UI/UX Experimentation and Model Training
Promoted's model needs comprehensive data to predict user behavior accurately. For instance, in UI experiments, knowing the context like display format (FULL versus COMPACT) helps the model learn that reviews impact click probability only in FULL displays. Similarly, experiments altering product titles (e.g., 'wool socks' versus 'cozy socks') should provide these variations during the delivery API call. This ensures the model can accurately learn from these changes, avoiding the need to relearn from scratch after the experiment ends.
Outcome Analysis
Customers can collaborate with Promoted to query the data warehouse for analysis to evaluate experiment outcomes. Outcomes are analyzed in data notebooks, and Promoted can provide periodic exports of assignments and the metrics needed for comprehensive analysis. This service is tailored to the unique goal metrics of each customer, acknowledging that many customers prefer their own experiment tools for consistency and comprehensive analysis, including metrics beyond Promoted's scope.
Promoted also generates custom reports on feature importance, aiding in understanding the impact of UI/UX changes on user behavior and optimizing future experiments.
Conclusion
In conclusion, Promoted serves as a versatile partner in the A/B testing process, offering robust integration capabilities, detailed analytical support, and a commitment to improving experiment outcomes and user experience insights.
Updated 23 days ago