locked
Bayesian A/B testing - compilation error RRS feed

  • Question

  • I am trying to implement a simple Bayesian A/B testing. Suppose I have two websites: A and B. Site A gets 500 visits and 10 of them make a purchase. Similarly for site B, 700 visit it and only 15 purchase. I am interested in probability that the rate of purchase for site A is bigger than that of site B.

    For this task, I came up with the following code:

    // data
    int numSiteASuccesses = 10;
    int numSiteBSuccesses = 15;
    
    int siteATotal = 500;
    int siteBTotal = 700;
    
    // latent model variables
    Variable<double> siteARate = Variable.Beta(1, 10);
    Variable<double> siteBRate = Variable.Beta(1, 10);
    
    Variable<bool> siteAIsBetter = siteARate > siteBRate;
    
    // observed variables
    Variable<int> siteASucces = Variable.Binomial(siteATotal, siteARate);
    Variable<int> siteBSucces = Variable.Binomial(siteBTotal, siteBRate);
    
    siteASucces.ObservedValue = numSiteASuccesses;
    siteBSucces.ObservedValue = numSiteBSuccesses;
    
    // inference
    InferenceEngine engine = new InferenceEngine();
    
    //Beta siteARatePosterior = engine.Infer<Beta>(siteARate);
    //Beta siteBRatePosterior = engine.Infer<Beta>(siteBRate);
    Bernoulli siteAIsBetterPosterior = engine.Infer<Bernoulli>(siteAIsBetter);
    
    // printing
    
    Console.WriteLine("P(Site A is better than B) =  " + siteAIsBetterPosterior);
    When trying to run it, a get a model compilation error and a very lengthy error message. What could be the issue with my code?

    Thursday, June 8, 2017 6:36 AM

Answers

  • Infer.NET does not provide comparison between two variables with Beta distributions.  Your best option is to infer the two rates and estimate the probability that A>B by drawing samples from the inferred distributions.
    • Marked as answer by usptact Monday, June 12, 2017 3:16 PM
    Thursday, June 8, 2017 10:09 AM
    Owner

All replies

  • Infer.NET does not provide comparison between two variables with Beta distributions.  Your best option is to infer the two rates and estimate the probability that A>B by drawing samples from the inferred distributions.
    • Marked as answer by usptact Monday, June 12, 2017 3:16 PM
    Thursday, June 8, 2017 10:09 AM
    Owner
  • Thank you, Tom! This is how I changed the code:

    Beta siteARatePosterior = engine.Infer<Beta>(siteARate);
    Beta siteBRatePosterior = engine.Infer<Beta>(siteBRate);
    
    int numSamples = 1000000;
    double diff = 0;
    double preferA = 0, preferB = 0;
    for (int i = 0; i < numSamples; i++)
    {
        diff = siteARatePosterior.Sample() - siteBRatePosterior.Sample();
        if (diff > 0)
            preferA++;
        else
            preferB++;
    }
    
    double probSiteAIsBetter = (double)(preferA / numSamples);
    double probSiteBIsBetter = 1 - probSiteAIsBetter;
    
    Console.WriteLine("\nP(Site A is better than B) =  " + probSiteAIsBetter);
    Console.WriteLine("P(Site B is better than A) =  " + probSiteBIsBetter);

    Are there any underwater stones if employing this approach? Are there more efficient ways finding the probability of interest?

    Thursday, June 8, 2017 9:15 PM
  • That's exactly the method I had in mind.  If you want something more efficient then you could try using Beta.GetProbLessThan to integrate out one of the variables.
    Thursday, June 8, 2017 10:56 PM
    Owner