locked
Discrete Bayes network parameters problem RRS feed

  • Question

  • Hi

    I am using the Infer.net package for representing a bayes network with discrete distribution. I am learning the parameters of the bayes network from a training sample of 6000 tuples(records).

    The bayes network represents a cars database. The node Body style has 2 parents - Model and Year. Body style can take any of 28 different values. 

     

    In the training sample that I use to learn the parameters for the Bayes net; Every car with Model = Tucson has Body = SUV.
    However, during inference P(Body =SUV | Model = Tucson) is only 0.06. All the other body types have probability of 0.03, even though there are no tuples with Model = Tucson with other body types in the training sample. This ia because of the uniform prior. However it is surprising to me that there is not much of a  difference between 0.06 and 0.03.

    There are also many tuples with Model = Tucson and Body = SUV. (198 tuples out of 6000 tuples in the training set.)

    Is there any way which that you know that can offset this problem.

     

    Thanks

    Rohit

     

    Monday, July 4, 2011 1:36 AM

All replies

  • Hi

    I tried to debug the problem. I found that whenever I have a node with two children, the dirichlet distribution is not updating properly.

    In the above case, Node Body has 2 parents - Model and Year. The Dirichlet distribution corresponding to Model = Tucson and Year = 2005 says Body = SUV is 2 despite the count being 198 in the training data.

    I do not have this problem when I have a node with only 1 or no parents.

    I tried running the code provided at this thread.  I am having the same problems with that code too.

    Monday, July 4, 2011 6:09 AM
  • In the first post, I rounded the values 0.03 and 0.06..

    the actual probabilities are P(Body = SUV | Model = Tucson) = 0.06889 and for all other Body values it is 0.03449.

     

    Rohit

     

    Monday, July 4, 2011 7:33 AM
  • Hi Rohit

    To help us respond quicker, can you post a stripped down version of the code you are running along with enough data to illustrate the problem?

    Many thanks

    John

    Monday, July 4, 2011 9:17 AM
    Owner
  • Powertrain---> Engine---> Type
                         ^
                                 |
                                 |

                      Model          

    (Engine is determined by both Powertrain and Model)

    -----

    using System;

    using System.Collections.Generic;

    using System.Linq;

    using System.Text;

    using MicrosoftResearch.Infer.Models;

    using MicrosoftResearch.Infer.Maths;

    using MicrosoftResearch.Infer.Distributions;

    using MicrosoftResearch.Infer;

    using MicrosoftResearch.Infer.Collections;





    namespace BayesNet
    {

        public class CarBayesNet
        {

            // Model variables

            public Variable<int> NumCases = Variable.New<int>();

            public VariableArray<int> CarModel;

            public VariableArray<int> CarType;

            public VariableArray<int> PowerTrain;

            public VariableArray<int> EngineType;

            // Probability tables and conditional probability tables

            public Variable<Vector> PT_M;

            public Variable<Vector> PT_P;

            public VariableArray2D<Vector> CPT_E;

            public VariableArray<Vector> CPT_T;

            // Prior distributions for the probability tables - make these variable so we can

            // change at run-time without model recompilation

            public Variable<Dirichlet> PT_M_Prior = null;

            public Variable<Dirichlet> PT_P_Prior = null;

            public VariableArray2D<Dirichlet> CPT_E_Prior = null;

            public VariableArray<Dirichlet> CPT_T_Prior = null;

            // Inferred posterior distributions over the probability tables

            public Dirichlet PT_M_Posterior = null;

            public Dirichlet PT_P_Posterior = null;

            public Dirichlet[,] CPT_E_Posterior = null;

            public Dirichlet[] CPT_T_Posterior = null;

            // Inference engine

            public InferenceEngine InfEngine = new InferenceEngine() { ShowProgress = false };



            public void CreateModel(int numModels, int numTypes, int numPowerTrains, int numEngines)
            {

                NumCases = Variable.New<int>(); // Set this at run time

                // The ranges and value ranges

                Range n = new Range(NumCases);

                Range m = new Range(numModels);

                Range t = new Range(numTypes);

                Range p = new Range(numPowerTrains);

                Range e = new Range(numEngines);



                // Define the priors as variables - these will be set (observed) at run time

                PT_M_Prior = Variable.New<Dirichlet>();

                PT_P_Prior = Variable.New<Dirichlet>();

                CPT_E_Prior = Variable.Array<Dirichlet>(m, p);

                CPT_T_Prior = Variable.Array<Dirichlet>(e);

                // Probability table and conditional probability table variables (parameters)

                PT_M = Variable<Vector>.Random(PT_M_Prior);

                PT_P = Variable<Vector>.Random(PT_P_Prior);

                CPT_E = Variable.Array<Vector>(m, p);

                CPT_E[m, p] = Variable<Vector>.Random(CPT_E_Prior[m, p]);

                CPT_T = Variable.Array<Vector>(e);

                CPT_T[e] = Variable<Vector>.Random(CPT_T_Prior[e]);

                // Set the value ranges for the probability tables

                PT_M.SetValueRange(m);

                PT_P.SetValueRange(p);

                CPT_E.SetValueRange(e);

                CPT_T.SetValueRange(t);

                // Describe the structure

                CarModel = Variable.Array<int>(n);

                CarModel[n] = Variable.Discrete(PT_M).ForEach(n);

                PowerTrain = Variable.Array<int>(n);

                PowerTrain[n] = Variable.Discrete(PT_P).ForEach(n);

                EngineType = AddChildFromTwoParents(CarModel, PowerTrain, CPT_E);

                CarType = AddChildFromOneParent(EngineType, CPT_T);

            }

            // Infer the probability tables and conditional probability tables

            public void InferParameters(int[] modelData, int[] typeData, int[] ptData, int[] engineData)
            {

                // Set the observations:

                NumCases.ObservedValue = modelData.Length;

                CarModel.ObservedValue = modelData;

                CarType.ObservedValue = typeData;

                PowerTrain.ObservedValue = ptData;

                EngineType.ObservedValue = engineData;



                // Set the uniform priors for the probability tables

                int numModels = CarModel.GetValueRange().SizeAsInt;

                int numPowerTrains = PowerTrain.GetValueRange().SizeAsInt;

                int numEngines = EngineType.GetValueRange().SizeAsInt;

                int numTypes = CarType.GetValueRange().SizeAsInt;



                Dirichlet[,] cpt_E_Prior = new Dirichlet[numModels, numPowerTrains];

                Dirichlet[] cpt_T_Prior = new Dirichlet[numEngines];

                for (int i = 0; i < numModels; i++)

                    for (int j = 0; j < numPowerTrains; j++)

                        cpt_E_Prior[i, j] = Dirichlet.Uniform(numEngines);

                for (int i = 0; i < numEngines; i++)

                    cpt_T_Prior[i] = Dirichlet.Uniform(numTypes);

                PT_M_Prior.ObservedValue = Dirichlet.Uniform(numModels);

                PT_P_Prior.ObservedValue = Dirichlet.Uniform(numPowerTrains);

                CPT_E_Prior.ObservedValue = cpt_E_Prior;

                CPT_T_Prior.ObservedValue = cpt_T_Prior;



                // Run the inference

                InfEngine.InferAll(PT_M, PT_P, CPT_E, CPT_T);

                PT_M_Posterior = InfEngine.Infer<Dirichlet>(PT_M);

                PT_P_Posterior = InfEngine.Infer<Dirichlet>(PT_P);

                CPT_E_Posterior = InfEngine.Infer<Dirichlet[,]>(CPT_E);

                CPT_T_Posterior = InfEngine.Infer<Dirichlet[]>(CPT_T);

            }

            // Query the distribution over car type conditioned on each of a set of model types

            public Discrete[] QueryCarTypesFromModelTypes(int[] models)
            {

                NumCases.ObservedValue = models.Length;

                CarModel.ObservedValue = models;

                CarType.ClearObservedValue();

                PowerTrain.ClearObservedValue();

                EngineType.ClearObservedValue();



                PT_M_Prior.ObservedValue = PT_M_Posterior;

                PT_P_Prior.ObservedValue = PT_P_Posterior;

                CPT_E_Prior.ObservedValue = CPT_E_Posterior;

                CPT_T_Prior.ObservedValue = CPT_T_Posterior;



                // Run the inference

                return InfEngine.Infer<Discrete[]>(CarType);

            }

            // Model code for adding a child from a single parent

            public static VariableArray<int> AddChildFromOneParent(

              VariableArray<int> parent, VariableArray<Vector> cpt)
            {

                var d = parent.Range;

                // data range

                var child = Variable.Array<int>(d);

                using (Variable.ForEach(d))

                using (Variable.Switch(parent[d]))

                    child[d] = Variable.Discrete(cpt[parent[d]]);

                return child;

            }

            // Model code for adding a child from two parents

            public static VariableArray<int> AddChildFromTwoParents(

              VariableArray<int> parent1, VariableArray<int> parent2, VariableArray2D<Vector> cpt)
            {

                var d = parent1.Range;

                // data range

                var child = Variable.Array<int>(d);

                using (Variable.ForEach(d))

                using (Variable.Switch(parent1[d]))

                using (Variable.Switch(parent2[d]))

                    child[d] = Variable.Discrete(cpt[parent1[d], parent2[d]]);

                return child;

            }

        }



        class Program
        {

            static void Main(string[] args)
            {

                int[] modelData = new int[] { 0, 0, 0, 0,1, 2, 1, 3, 3, 1, 1, 2, 3, 3, 0 };

                int[] typeData = new int[] { 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0 };

                int[] ptData = new int[] { 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2 };

                int[] engineData = new int[] { 0, 0, 0, 0, 1, 2, 1, 3, 3, 1, 1, 2, 3, 3, 1 };

                int numModels = 4;

                int numTypes = 2;

                int numPowerTrains = 3;

                int numEngines = 4;



                // Learn the parameters

                CarBayesNet bayesNet = new CarBayesNet();

                bayesNet.CreateModel(numModels, numTypes, numPowerTrains, numEngines);

                bayesNet.InferParameters(modelData, typeData, ptData, engineData);



                // Print out the PT distributions

                Console.WriteLine("\nCar model PT");

                Console.WriteLine(bayesNet.PT_M_Posterior);

                Console.WriteLine("\nPower Train PT");

                Console.WriteLine(bayesNet.PT_P_Posterior);

                Console.WriteLine("\nEngine CPT");

                Console.WriteLine(Distribution<Vector>.Array(bayesNet.CPT_E_Posterior));

                Console.WriteLine("\nType CPT:");

                Console.WriteLine(Distribution<Vector>.Array(bayesNet.CPT_T_Posterior));



                // Query the model

                int[] modelTypes = System.Linq.Enumerable.Range(0, numModels).ToArray();

                var carTypesFromModelTypes = bayesNet.QueryCarTypesFromModelTypes(modelTypes);

                Console.WriteLine("\nDistributions for car types from model types");



                for (int i = 0; i < numModels; i++)

                    Console.WriteLine("Probability of type, given model {0}: {1}", i, carTypesFromModelTypes[i]);

                Console.ReadLine();
            }

        }

    }


    The output for this program is :


    Car model PT
    Dirichlet(6 5 3 5)

    Power Train PT
    Dirichlet(5 11 2)

    Engine CPT
    [0,0] Dirichlet(2 1 1 1)
    [0,1] Dirichlet(1 1 1 1)
    [0,2] Dirichlet(1 2 1 1)
    [1,0] Dirichlet(1 1 1 1)
    [1,1] Dirichlet(1 2 1 1)
    [1,2] Dirichlet(1 1 1 1)
    [2,0] Dirichlet(1 1 1 1)
    [2,1] Dirichlet(1 1 2 1)
    [2,2] Dirichlet(1 1 1 1)
    [3,0] Dirichlet(1 1 1 1)
    [3,1] Dirichlet(1 1 1 2)
    [3,2] Dirichlet(1 1 1 1)

    Type CPT:
    [0] Dirichlet(5 1)
    [1] Dirichlet(2 5)
    [2] Dirichlet(3 1)
    [3] Dirichlet(1 5)

    Distributions for car types from model types
    Probability of type, given model 0: Discrete(0.522 0.478)
    Probability of type, given model 1: Discrete(0.4816 0.5184)
    Probability of type, given model 2: Discrete(0.5384 0.4616)
    Probability of type, given model 3: Discrete(0.4671 0.5329)

    ----

    Note that Engine CPT(Engine has 2 parents) for [0, 0] is

    [0,0] Dirichlet(2 1 1 1)

    But, shouldn't it be Dirichlet(5 1 1 1) according to the data.

    ----

    Rohit

    Monday, July 4, 2011 9:45 AM
  • Hi Rohit - this is a bug related to 2D arrays. This will be fixed in the next release. For now, work with jagged arrays which works fine (and is almost always the preferred option anyway. See http://social.microsoft.com/Forums/en-US/infer.net/thread/8e60b7d2-741b-4ee2-99f2-51596d578ad5 for a related discussion of this relating to CPTs (look at the last couple of posts).

    John

    Monday, July 4, 2011 1:28 PM
    Owner
  • Hi John- I am having troubles converting the 2D arrays into jagged arrays.

    Here is my attempt to convert the code that I have provided in the post above, from 2D to jagged array. I would appreciate if you could help me find the reason for the exception.

    In the method AddChildFromTwoParents, I am getting ArgumentException was unhandled:

    VariableArray<VariableArray<Vector>,Vector[][]> vVector[][]0 cannot be indexed by vint[]0[index0].
    Parameter name: index

    ---

    using System;

    using System.Collections.Generic;

    using System.Linq;

    using System.Text;

    using MicrosoftResearch.Infer.Models;

    using MicrosoftResearch.Infer.Maths;

    using MicrosoftResearch.Infer.Distributions;

    using MicrosoftResearch.Infer;

    using MicrosoftResearch.Infer.Collections;





    namespace BayesNet
    {

        // Aliases for distributions over 1-D and 2-D probability tables

        using DirichletArray = DistributionRefArray<Dirichlet, Vector>;

        //using DirichletArray2D = DistributionRefArray2D<Dirichlet, Vector>;

        using DirichletArrayArray = DistributionRefArray<DistributionRefArray<Dirichlet, Vector>, Vector[]>;

        public class CarBayesNet
        {

            // Model variables

            public Variable<int> NumCases = Variable.New<int>();

            public VariableArray<int> CarModel;

            public VariableArray<int> CarType;

            public VariableArray<int> PowerTrain;

            public VariableArray<int> EngineType;

            // Probability tables and conditional probability tables

            public Variable<Vector> PT_M;

            public Variable<Vector> PT_P;

            public VariableArray<VariableArray<Vector>, Vector[][]> CPT_E;

            public VariableArray<Vector> CPT_T;

            // Prior distributions for the probability tables - make these variable so we can

            // change at run-time without model recompilation

            public Variable<Dirichlet> PT_M_Prior = null;

            public Variable<Dirichlet> PT_P_Prior = null;

            public Variable<DirichletArrayArray> CPT_E_Prior = null;

            public Variable<DirichletArray> CPT_T_Prior = null;

            // Inferred posterior distributions over the probability tables

            public Dirichlet PT_M_Posterior = null;

            public Dirichlet PT_P_Posterior = null;

            public DirichletArrayArray CPT_E_Posterior = null;

            public DirichletArray CPT_T_Posterior = null;

            // Inference engine

            public InferenceEngine InfEngine = new InferenceEngine() { ShowProgress = false };



            public void CreateModel(int numModels, int numTypes, int numPowerTrains, int numEngines)
            {

                NumCases = Variable.New<int>(); // Set this at run time

                // The ranges and value ranges

                Range n = new Range(NumCases);

                Range m = new Range(numModels);

                Range t = new Range(numTypes);

                Range p = new Range(numPowerTrains);

                Range e = new Range(numEngines);



                // Define the priors as variables - these will be set (observed) at run time

                PT_M_Prior = Variable.New<Dirichlet>();

                PT_P_Prior = Variable.New<Dirichlet>();

                CPT_E_Prior = Variable.New<DirichletArrayArray>();

                CPT_T_Prior = Variable.New<DirichletArray>();

                // Probability table and conditional probability table variables (parameters)

                PT_M = Variable<Vector>.Random(PT_M_Prior);

                PT_P = Variable<Vector>.Random(PT_P_Prior);

                CPT_E = Variable.Array(Variable.Array<Vector>(m), p);
                CPT_E.SetTo(Variable<Vector[][]>.Random(CPT_E_Prior));

                CPT_T = Variable.Array<Vector>(e);

                CPT_T.SetTo(Variable<Vector[]>.Random(CPT_T_Prior));

                // Set the value ranges for the probability tables

                PT_M.SetValueRange(m);

                PT_P.SetValueRange(p);

                CPT_E.SetValueRange(e);

                CPT_T.SetValueRange(t);

                // Describe the structure

                CarModel = Variable.Array<int>(n);

                CarModel[n] = Variable.Discrete(PT_M).ForEach(n);

                PowerTrain = Variable.Array<int>(n);

                PowerTrain[n] = Variable.Discrete(PT_P).ForEach(n);

                EngineType = AddChildFromTwoParents(CarModel, PowerTrain, CPT_E);

                CarType = AddChildFromOneParent(EngineType, CPT_T);

            }

            // Infer the probability tables and conditional probability tables

            public void InferParameters(int[] modelData, int[] typeData, int[] ptData, int[] engineData)
            {

                // Set the observations:

                NumCases.ObservedValue = modelData.Length;

                CarModel.ObservedValue = modelData;

                CarType.ObservedValue = typeData;

                PowerTrain.ObservedValue = ptData;

                EngineType.ObservedValue = engineData;



                // Set the uniform priors for the probability tables

                int numModels = CarModel.GetValueRange().SizeAsInt;

                int numPowerTrains = PowerTrain.GetValueRange().SizeAsInt;

                int numEngines = EngineType.GetValueRange().SizeAsInt;

                int numTypes = CarType.GetValueRange().SizeAsInt;



                //Dirichlet[,] cpt_E_Prior = new Dirichlet[numModels, numPowerTrains];
                Dirichlet[][] cpt_E_Prior = new Dirichlet[numPowerTrains][];

                Dirichlet[] cpt_T_Prior = new Dirichlet[numEngines];

                for (int i = 0; i < numPowerTrains; i++)
                {
                    cpt_E_Prior[i] = new Dirichlet[numModels];
                    for (int j = 0; j < numModels; j++)

                        cpt_E_Prior[i][j] = Dirichlet.Uniform(numEngines);
                }
                for (int i = 0; i < numEngines; i++)

                    cpt_T_Prior[i] = Dirichlet.Uniform(numTypes);

                PT_M_Prior.ObservedValue = Dirichlet.Uniform(numModels);

                PT_P_Prior.ObservedValue = Dirichlet.Uniform(numPowerTrains);

                CPT_E_Prior.ObservedValue = (DirichletArrayArray)Distribution<Vector>.Array(cpt_E_Prior);

                CPT_T_Prior.ObservedValue = (DirichletArray)Distribution<Vector>.Array(cpt_T_Prior);



                // Run the inference

                InfEngine.InferAll(PT_M, PT_P, CPT_E, CPT_T);

                PT_M_Posterior = InfEngine.Infer<Dirichlet>(PT_M);

                PT_P_Posterior = InfEngine.Infer<Dirichlet>(PT_P);

                CPT_E_Posterior = InfEngine.Infer<DirichletArrayArray>(CPT_E);

                CPT_T_Posterior = InfEngine.Infer<DirichletArray>(CPT_T);

            }

            // Query the distribution over car type conditioned on each of a set of model types

            public Discrete[] QueryCarTypesFromModelTypes(int[] models)
            {

                NumCases.ObservedValue = models.Length;

                CarModel.ObservedValue = models;

                CarType.ClearObservedValue();

                PowerTrain.ClearObservedValue();

                EngineType.ClearObservedValue();



                PT_M_Prior.ObservedValue = PT_M_Posterior;

                PT_P_Prior.ObservedValue = PT_P_Posterior;

                CPT_E_Prior.ObservedValue = CPT_E_Posterior;

                CPT_T_Prior.ObservedValue = CPT_T_Posterior;



                // Run the inference

                return InfEngine.Infer<Discrete[]>(CarType);

            }

            // Model code for adding a child from a single parent

            public static VariableArray<int> AddChildFromOneParent(

              VariableArray<int> parent, VariableArray<Vector> cpt)
            {

                var d = parent.Range;

                // data range

                var child = Variable.Array<int>(d);

                using (Variable.ForEach(d))

                using (Variable.Switch(parent[d]))

                    child[d] = Variable.Discrete(cpt[parent[d]]);

                return child;

            }

            // Model code for adding a child from two parents
            public static VariableArray<int> AddChildFromTwoParents(
                VariableArray<int> parent1, VariableArray<int> parent2,
                VariableArray<VariableArray<Vector>, Vector[][]> cpt)
            {
                var d = parent1.Range;
                // data range
                var child = Variable.Array<int>(d);
                using (Variable.ForEach(d))
                using (Variable.Switch(parent1[d]))
                using (Variable.Switch(parent2[d]))
                    child[d] = Variable.Discrete(cpt[parent1[d]][parent2[d]]);
                return child;
            }

        }



        class Program
        {

            static void Main(string[] args)
            {

                int[] modelData = new int[] { 0, 1, 2, 1, 3, 3, 1, 1, 2, 3, 3, 0 };

                int[] typeData = new int[] { 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0 };

                int[] ptData = new int[] { 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2 };

                int[] engineData = new int[] { 0, 1, 2, 1, 3, 3, 1, 1, 2, 3, 3, 1 };

                int numModels = 4;

                int numTypes = 2;

                int numPowerTrains = 3;

                int numEngines = 4;



                // Learn the parameters

                CarBayesNet bayesNet = new CarBayesNet();

                bayesNet.CreateModel(numModels, numTypes, numPowerTrains, numEngines);

                bayesNet.InferParameters(modelData, typeData, ptData, engineData);



                // Print out the PT distributions

                Console.WriteLine("\nCar model PT");

                Console.WriteLine(bayesNet.PT_M_Posterior);

                Console.WriteLine("\nPower Train PT");

                Console.WriteLine(bayesNet.PT_P_Posterior);

                Console.WriteLine("\nEngine CPT");

                Console.WriteLine(bayesNet.CPT_E_Posterior);

                Console.WriteLine("\nType CPT:");

                Console.WriteLine(bayesNet.CPT_T_Posterior);



                // Query the model

                int[] modelTypes = System.Linq.Enumerable.Range(0, numModels).ToArray();

                var carTypesFromModelTypes = bayesNet.QueryCarTypesFromModelTypes(modelTypes);

                Console.WriteLine("\nDistributions for car types from model types");



                for (int i = 0; i < numModels; i++)

                    Console.WriteLine("Probability of type, given model {0}: {1}", i, carTypesFromModelTypes[i]);

            }

        }

    }

     

     

     

     

    Monday, July 4, 2011 10:16 PM
  • You've got the ranges flipped around in the declaration of CPT_E:

    CPT_E = Variable.Array(Variable.Array<Vector>(m), p);

    According to the code later, you want CPT_E to be indexed by PowerTrain first, then CarModel.  So the declaration should be:

    CPT_E = Variable.Array(Variable.Array<Vector>(p), m);

    By the way, if you name your variables, you will get more helpful error messages.

    Tuesday, July 5, 2011 12:28 PM
  • You have got the indexing the wrong way round in the definition of the jagged array. In the model, you should have

    CPT_E = Variable.Array(Variable.Array<Vector>(p),m);
    
    

    Then, when you are setting the priors:

    Dirichlet[][] cpt_E_Prior = new Dirichlet[numModels][];
    for (int i = 0; i < numModels; i++)
    {
     cpt_E_Prior[i] = new Dirichlet[numPowerTrains];
     for (int j = 0; j < numPowerTrains; j++)
      cpt_E_Prior[i][j] = Dirichlet.Uniform(numEngines);
    }
    
    

    John

    Tuesday, July 5, 2011 1:07 PM
    Owner
  • Thanks a lot John and Tom. It works now!

     

    Rohit

    Wednesday, July 6, 2011 10:10 AM