Answered by:
Bayesian A/B testing  compilation error
Question

I am trying to implement a simple Bayesian A/B testing. Suppose I have two websites: A and B. Site A gets 500 visits and 10 of them make a purchase. Similarly for site B, 700 visit it and only 15 purchase. I am interested in probability that the rate of purchase for site A is bigger than that of site B.
For this task, I came up with the following code:
// data int numSiteASuccesses = 10; int numSiteBSuccesses = 15; int siteATotal = 500; int siteBTotal = 700; // latent model variables Variable<double> siteARate = Variable.Beta(1, 10); Variable<double> siteBRate = Variable.Beta(1, 10); Variable<bool> siteAIsBetter = siteARate > siteBRate; // observed variables Variable<int> siteASucces = Variable.Binomial(siteATotal, siteARate); Variable<int> siteBSucces = Variable.Binomial(siteBTotal, siteBRate); siteASucces.ObservedValue = numSiteASuccesses; siteBSucces.ObservedValue = numSiteBSuccesses; // inference InferenceEngine engine = new InferenceEngine(); //Beta siteARatePosterior = engine.Infer<Beta>(siteARate); //Beta siteBRatePosterior = engine.Infer<Beta>(siteBRate); Bernoulli siteAIsBetterPosterior = engine.Infer<Bernoulli>(siteAIsBetter); // printing Console.WriteLine("P(Site A is better than B) = " + siteAIsBetterPosterior);
When trying to run it, a get a model compilation error and a very lengthy error message. What could be the issue with my code?
Thursday, June 8, 2017 6:36 AM
Answers

Infer.NET does not provide comparison between two variables with Beta distributions. Your best option is to infer the two rates and estimate the probability that A>B by drawing samples from the inferred distributions.
 Marked as answer by usptact Monday, June 12, 2017 3:16 PM
Thursday, June 8, 2017 10:09 AMOwner
All replies

Infer.NET does not provide comparison between two variables with Beta distributions. Your best option is to infer the two rates and estimate the probability that A>B by drawing samples from the inferred distributions.
 Marked as answer by usptact Monday, June 12, 2017 3:16 PM
Thursday, June 8, 2017 10:09 AMOwner 
Thank you, Tom! This is how I changed the code:
Beta siteARatePosterior = engine.Infer<Beta>(siteARate); Beta siteBRatePosterior = engine.Infer<Beta>(siteBRate); int numSamples = 1000000; double diff = 0; double preferA = 0, preferB = 0; for (int i = 0; i < numSamples; i++) { diff = siteARatePosterior.Sample()  siteBRatePosterior.Sample(); if (diff > 0) preferA++; else preferB++; } double probSiteAIsBetter = (double)(preferA / numSamples); double probSiteBIsBetter = 1  probSiteAIsBetter; Console.WriteLine("\nP(Site A is better than B) = " + probSiteAIsBetter); Console.WriteLine("P(Site B is better than A) = " + probSiteBIsBetter);
Are there any underwater stones if employing this approach? Are there more efficient ways finding the probability of interest?
Thursday, June 8, 2017 9:15 PM 
That's exactly the method I had in mind. If you want something more efficient then you could try using Beta.GetProbLessThan to integrate out one of the variables.Thursday, June 8, 2017 10:56 PMOwner