locked
How to learn a Dirichlet (Migrated from community.research.microsoft.com) RRS feed

  • Question

  • bearstroller posted on 04-16-2009 9:11 AM

    If my model contains a variable Dirichlet prior, how do I specify its variable type in Infer.NET? Do I have to use the DirichletEstimator to customize it or there is more direct approaches?

    Friday, June 3, 2011 4:52 PM

Answers

All replies

  • minka replied on 04-16-2009 10:34 AM

    Learning a Dirichlet distribution from data is not yet supported via the modelling API.  But it is on our list of things to do.  This would allow, for example, fitting a latent Dirichlet allocation model (which currently cannot be done in Infer.NET).

    Friday, June 3, 2011 4:53 PM
  • bearstroller replied on 04-16-2009 11:29 AM

    Thank you, hope you work that out soon :)

    btw, I wonder if I can just add a new distribution type to solve this problem. Since \alpha in LDA is learned by maximum likelihood estimation, it corresponds to ignore the local conditional of the node itself from the node updating equation in VMP (the first term of r.h.s of equation (8) in VMP JMLR 2005 paper), everything else of VMP can be left unchanged, I guess maybe I can add a distribution type with some modification. Is that possible?

    Friday, June 3, 2011 4:53 PM
  • minka replied on 04-18-2009 5:15 AM

    While it is possible to add a distribution type in Infer.NET, I think what you mean here is to add a new factor, connecting the alpha parameters to the word probabilities.  The VMP update for alpha in the factor would perform maximum-likelihood estimation.  This is indeed along the lines of what we have planned.  If you want to have a go at it yourself, take a look at our example of how to add a new factor:  http://research.microsoft.com/en-us/um/cambridge/projects/infernet/docs/How%20to%20add%20a%20new%20factor%20and%20message%20operators.aspx

    Friday, June 3, 2011 4:53 PM
  • bearstroller replied on 04-18-2009 7:54 AM

    Great!...Really thank you...

    Friday, June 3, 2011 4:53 PM
  • bearstroller replied on 04-19-2009 8:17 AM

    Can you help me to check whether my understanding is correct? I suppose that the factor is to represent what is \alpha, instead of a gateway between \alpha and \theta, so the parameter is only \theta. But what should I do in the Alpha function? What's more, is my AlphaOp correct?

     

     

    namespace MicrosoftResearch.Infer.Factors
    {
        public static class LatentDirichlet
        {
            [ParameterNames("Theta")]
            public static Variable<Vector> Alpha(Variable<Vector>[] theta)
            {
                Variable<Vector> alpha = Variable.New<Vector>();
                // what to do here? how to let the engine know that alpha is connected to theta?
                // how to get the distribution from theta?
                return alpha;
            }
        }
        [FactorMethod(typeof(LatentDirichlet), "Alpha")]
        public static class AlphaOp
        {
            public static IList<Dirichlet> ThetaAverageLogarithm(IList<Dirichlet> theta)
            {
                int K = theta[0].Dimension;
                int N = theta.Count;
                Vector ss = new Vector(K);
                Vector sumk_T = new Vector(N);
                for(int i=0;i<N;i++)
                {
                    sumk_T = 0.0;
                    for (int k = 0; k < K; k++)
                        sumk_T += theta[k];
                }
                for(int k=0;k<K;k++)
                {
                    ss[k] = 0.0;
                    for(int i=0;i<N;i++)
                    {
                        ss[k] += MMath.Digamma(theta[k]);
                        ss[k] -= MMath.Digamma(sumk_T);
                    }
                }

                // Compute the alpha parameters.
                Vector alpha = LatentDirichlet.EstimateDirichlet(ss, N);
               
                // Send the alpha parameters out as messages.
                IList<Dirichlet> msg_out = new IList<Dirichlet>(N);
                for(int i=0;i<N;i++)
                    msg_out = new Dirichlet(alpha);
                return msg_out;
            }
        }
       
        public static class MathHelper
        {
            public static Vector EstimateDirichlet(Vector ss, int D, Vector alpha_init)
            {
                int iter_max_step = 1000;
                double iter_epsilon = 1e-5;
                int iter = 0;
                double iter_step = Double.MaxValue;

                Vector alpha = alpha_init.Clone();
                int K = ss.Count;
                Vector psi_alpha = new Vector(K), g = new Vector(K), h = new Vector(K);
                Vector alpha_new = new Vector(K);
                double sum_alpha, z, c, c1, c2, diff;

                while( iter_step > iter_epsilon && iter < iter_max_step )
                {
                    sum_alpha = 0.0;
                    c1 = 0.0;
                    c2 = 0.0;
                    for (int i=0;i<K;i++)
                    {
                        sum_alpha += alpha;
                        psi_alpha = MMath.Digamma(alpha);
                    }
                    for(int i=0;i<K;i++)
                    {
                        psi_alpha = MMath.Digamma(sum_alpha) - psi_alpha;
                        g = psi_alpha * D + ss;
                        h = -D * MMath.Trigamma(alpha);
                        c1 += g / h;
                        c2 += 1.0 / h;
                    }
                    z = D * MMath.Trigamma(sum_alpha);
                    c = c1 / (1.0 / z + c2);
                    iter_step = 0.0;
                    for(int i=0;i<K;i++)
                    {
                        alpha_new = alpha - (g - c) / h;
                        diff = alpha_new - alpha;
                        iter_step = iter_step + diff * diff;
                    }
                    iter_step = Math.Sqrt(iter_step);
                    for (int i = 0; i < K; i++)
                    {
                        alpha = alpha_new;
                    }
                    iter = iter + 1;
                }
                return alpha;
            }
            public static Vector EstimateDirichlet(Vector ss, int D)
            {
                int K = ss.Count;
                Vector alpha_init = new Vector(K);
                for(int i=0;i<K;i++)
                {
                    alpha_init = Math.Exp(ss) + 2;
                }
                return EstimateDirichlet(ss, D, alpha_init);
            }
        }
    }

    Friday, June 3, 2011 4:53 PM
  • minka replied on 05-16-2009 11:22 PM

    The factor should represent p(theta | alpha).  Therefore it should take alpha as argument and return theta.  Currently you’ve got it the other way around.  See for example the BernoulliFromBeta factor (Variable.Bernoulli with Beta argument).

     When implementing the factor function, the arguments are not variables.  They are just ordinary values.  It should take a Vector argument and return a Vector of probabilities, by sampling from the Dirichlet distribution with given parameters.

    Friday, June 3, 2011 4:53 PM
  • bearstroller replied on 05-17-2009 9:28 AM

    thank you...

    Friday, June 3, 2011 4:53 PM