locked
Modelling errors in a task that generates T / F questions (Migrated from community.research.microsoft.com) RRS feed

  • Question

  • alexjames posted on 06-17-2010 10:35 AM

    Hello, I wonder if anyone can help.

    I have a task that asks the user a set questions within a single trial. I have N categories. Each category has a label, and each category has a list of questions. Each question has a true or false answer only. Is it possible, from the information generated by the answers to these questions across multiple trials, to predict the probability of an error being made within a different task?

    For example:

    Task A, Category A, Category B, Category , Category A, Category B, Category C

    <trials executed and data generated for questions>

    Task B, Category B, Category B, Category , Category C, Category C, Category A

    Can I use the data generated in task A to inform what the user may do in task B?

    best

    alex

    Friday, June 3, 2011 5:49 PM

Answers

  • alexjames replied on 06-25-2010 3:44 AM

    Hello, that is working well. 

    I uncovered a post of John's that covers the retrieval of a distribution across a vector of probabilities which I plan to use for summarising the performance of Task [A, B, C ... n] sub-tasks. This should let me abstract task predictions at progressively higher levels (unless I have misunderstood John' s post):

    http://community.research.microsoft.com/forums/t/4204.aspx

    best

    alex

    Friday, June 3, 2011 5:49 PM

All replies

  • DavidKnowles replied on 06-17-2010 11:04 AM

    Hi Alex- 

    It is certainly possible but how to approach your model depends on what modelling assumptions you want to make. If I understand your problem correctly, then each task consists of a set of questions, and each question has a category. One approach might be to assume that each individual has a probability of answering correctly for each category of question. You could learn these probabilities on the multiple trials you have, and use them to predict performance on the different task. 

    If that seems like a sensible approach to you I could sketch out some code to achieve this. 

    Best

    David. 

    Friday, June 3, 2011 5:49 PM
  • jwinn replied on 06-17-2010 3:02 PM

    Certainly.  You could assume, for example, that the user has a certain level of knowledge in each category and represent this as a probability that the user will get a question right in that category.  By looking at how the user performed on previous questions, you can infer posterior distributions over these  probabilities and then use those to predict what the use would do in future questions (e.g. Task B).  For example if your categories are Maths, Science and Geography, you would associate a random variable representing the probability of getting a question right in each of these categories and infer it from Task A - you would then get three Beta distributions out - one for each category. (a Beta distribution is a distribution over a probability between 0 and 1 and allows you to represent the uncertainty in that probability).

    To understand more about Beta distributions and how to build this kind of model with Infer.NET, watch the video on this page:
    http://microsoftpdc.com/Sessions/VTL03

    Best
    John W.

    Friday, June 3, 2011 5:49 PM
  • alexjames replied on 06-18-2010 5:23 AM

    Hello David/ John

    I have started to experiment with your suggestions, and I wonder if you could point me in the right direction with the below.

    If I have Task A, which has category 'math' data attached, I should be able to get  Beta posterior for this category, however attaching the data (TFFFTFFF...) could use a hand.

    David, if you could provide a code outline would be very helpful.

    Would it be possible to combine the e.g. 3 beta distributions I will ultimately get for Task A into one 'master' indicator of the task?

    Best

    Alex

            //START

                // Data about tasks: true = correct, false = incorrect
                bool[] TaskA_Math = { false, true, true, true, true, true, true, true, false, false };

                //B, C etc

                // Range for A
                int numA = TaskA_Math.Length;
                Range taskARange = new Range(numA);

                //B, C etc

                //  T F  data samples : how do I add the boolean samples?
                VariableArray<bool> data = Variable.Array<bool>(taskARange);
                using (Variable.ForEach(taskARange))
                {
                    //data [taskARange] = <???>  T / F data stream
                }

                // Inference engine (EP)
                InferenceEngine engine = new InferenceEngine();

                // Infer probability of Task A Math so can predict Task B_Math etc.
                Console.WriteLine("isCorrect = {0}", engine.Infer<Beta>(data ));
               
                //Next trial

                //END

     

    Friday, June 3, 2011 5:49 PM
  • alexjames replied on 06-18-2010 6:56 AM

    Hello David

    That would be great if you could lend a hand, I have made a start as below. I am interested in making an inference from the N categories to one 'master' category also, to allow further organisation of the task.

    best

    alex

    Friday, June 3, 2011 5:49 PM
  • DavidKnowles replied on 06-20-2010 11:26 AM

    The code below could be a starting point. We assume there is some hidden probability of getting the correct answer "taskA_performance", which we give a uniform prior over the interval [0,1]. We then infer the posterior over taskA_performance given the observed data.

                // Data about tasks: true = correct, false = incorrect
                bool[] TaskA_Math = { false, true, true, true, true, true, true, true, false, false };

                // Range for A
                int numA = TaskA_Math.Length;
                Range taskARange = new Range(numA);

                VariableArray<bool> data = Variable.Array<bool>(taskARange);

                // The prior
                var taskA_performance = Variable.Beta(1, 1);

                // The likelihood model
                data[taskARange] = Variable.Bernoulli(taskA_performance).ForEach(taskARange);

                // Attach data
                data.ObservedValue = TaskA_Math;

                // Inference engine (EP)
                InferenceEngine engine = new InferenceEngine();

                // Infer probability of Task A Math so can predict Task B_Math etc.
                Console.WriteLine("isCorrect = {0}", engine.Infer<Beta>(taskA_performance).GetMean());

     

    Friday, June 3, 2011 5:49 PM
  • alexjames replied on 06-21-2010 5:47 AM

    Thanks David, that's just what I wanted to start me off.

    I also found a few tutorials that have been helpful, the clinical infer.net and a MLSS practical:

    http://research.microsoft.com/en-us/um/cambridge/projects/infernet/docs/clinical%20trial%20tutorial.aspx

    http://mlg.eng.cam.ac.uk/mlss09/infernet.zip

    best

    alex

    Friday, June 3, 2011 5:49 PM
  • alexjames replied on 06-25-2010 3:44 AM

    Hello, that is working well. 

    I uncovered a post of John's that covers the retrieval of a distribution across a vector of probabilities which I plan to use for summarising the performance of Task [A, B, C ... n] sub-tasks. This should let me abstract task predictions at progressively higher levels (unless I have misunderstood John' s post):

    http://community.research.microsoft.com/forums/t/4204.aspx

    best

    alex

    Friday, June 3, 2011 5:49 PM