Services | A/B/n Testing Framework

In order to extend your business process execution framework (BPXF) from reporting what happened to helping you decide what should happen and why, you need data-driven experiments that can be set up, executed, measured, and evaluated using an A/B/n testing framework.

A/B/n testing is an objective, iterative, small-step, data-driven method of comparing two or more versions of something (e.g. a message, a page, a business process, a business rule, etc) to determine which is more effective at bringing about a desired outcome.  This kind of testing is valuable because it can help to remove guesswork and instinct from the decision-making process.

Tests can be simple (one variable with a control) or complex (multiple variables or “multi-variate).  The “A” in A/B/n testing refers to the “control” or original testing variable, and “B” refers to the “variation” or new version of the testing variable.  A simple example might be an existing “Click for More Information” call to action (CTA)  control and a new “Get Your Free eBook Now!” variation.  The “n” implies that there can be more than one new version in a given experiment.

This kind of testing can help gather both qualitative and quantitative user insights and can provide data that can help explain customer behavior, engagement, pain points, and satisfaction.

Successfully specifying, designing, implementing, and integrating an A/B/n testing framework should always include all stakeholders who will use and benefit from it, real-world use cases that illustrate how they intend to use it, product selection based on these requirements, connection to strategic planning goals and metrics, and business process and technical integration.

Beware the urge to oversimplify in order to move more quickly.  Creating a system like this is an opportunity to plan and consider carefully in order to deliver capabilities that will enable and serve the needs of the organization for years or to waste precious time and treasure delivering a system with serious limitations that doesn’t work well, requires a lot of manual interaction, and produces questionable value.

Pro Tips

• Initially some stakeholders may only be considering user interface (UI) testing — unaware of or not yet thinking about “back end” tests involving business process, rule, or data variations or that depend on integrations with flexible, granular application programming interfaces (APIs) or third-party systems like enterprise content management, to manage and deliver original and new, test content.
• Be on the lookout for testing capabilities that require engineering support to set up, run, interpret, or remove tests, which can seriously impact your organization’s business agility and can unintentionally limit who can experiment.  Great ideas often come from unexpected sources.
• And watch out for testing platforms that litter your application code with temporary instrumentation that must be added, tested, and then removed using your development, testing, and deployment process.  Not only will this slow testing to a crawl by tying it to your engineering processes and limited resources, but each cycle will then result in the possibility of introducing new defects while adding, removing, and deploying the test code.

Organizations that survive and remain competitive make decisions based on data.  Enlist the experience needed to fully envision and successfully implement a testing system that supports agile, democratized, data-driven decision-making!