Answered by:
Modelling errors in a task that generates T / F questions (Migrated from community.research.microsoft.com)
Question

alexjames posted on 06172010 10:35 AM
Hello, I wonder if anyone can help.
I have a task that asks the user a set questions within a single trial. I have N categories. Each category has a label, and each category has a list of questions. Each question has a true or false answer only. Is it possible, from the information generated by the answers to these questions across multiple trials, to predict the probability of an error being made within a different task?
For example:
Task A, Category A, Category B, Category , Category A, Category B, Category C
<trials executed and data generated for questions>
Task B, Category B, Category B, Category , Category C, Category C, Category A
Can I use the data generated in task A to inform what the user may do in task B?
best
alex
Friday, June 3, 2011 5:49 PM
Answers

alexjames replied on 06252010 3:44 AM
Hello, that is working well.
I uncovered a post of John's that covers the retrieval of a distribution across a vector of probabilities which I plan to use for summarising the performance of Task [A, B, C ... n] subtasks. This should let me abstract task predictions at progressively higher levels (unless I have misunderstood John' s post):
http://community.research.microsoft.com/forums/t/4204.aspx
best
alex
 Marked as answer by Microsoft Research Friday, June 3, 2011 5:49 PM
Friday, June 3, 2011 5:49 PM
All replies

DavidKnowles replied on 06172010 11:04 AM
Hi Alex
It is certainly possible but how to approach your model depends on what modelling assumptions you want to make. If I understand your problem correctly, then each task consists of a set of questions, and each question has a category. One approach might be to assume that each individual has a probability of answering correctly for each category of question. You could learn these probabilities on the multiple trials you have, and use them to predict performance on the different task.
If that seems like a sensible approach to you I could sketch out some code to achieve this.
Best
David.
Friday, June 3, 2011 5:49 PM 
jwinn replied on 06172010 3:02 PM
Certainly. You could assume, for example, that the user has a certain level of knowledge in each category and represent this as a probability that the user will get a question right in that category. By looking at how the user performed on previous questions, you can infer posterior distributions over these probabilities and then use those to predict what the use would do in future questions (e.g. Task B). For example if your categories are Maths, Science and Geography, you would associate a random variable representing the probability of getting a question right in each of these categories and infer it from Task A  you would then get three Beta distributions out  one for each category. (a Beta distribution is a distribution over a probability between 0 and 1 and allows you to represent the uncertainty in that probability).
To understand more about Beta distributions and how to build this kind of model with Infer.NET, watch the video on this page:
http://microsoftpdc.com/Sessions/VTL03Best
John W.Friday, June 3, 2011 5:49 PM 
alexjames replied on 06182010 5:23 AM
Hello David/ John
I have started to experiment with your suggestions, and I wonder if you could point me in the right direction with the below.
If I have Task A, which has category 'math' data attached, I should be able to get Beta posterior for this category, however attaching the data (TFFFTFFF...) could use a hand.
David, if you could provide a code outline would be very helpful.
Would it be possible to combine the e.g. 3 beta distributions I will ultimately get for Task A into one 'master' indicator of the task?
Best
Alex
//START
// Data about tasks: true = correct, false = incorrect
bool[] TaskA_Math = { false, true, true, true, true, true, true, true, false, false };
//B, C etc
// Range for A
int numA = TaskA_Math.Length;
Range taskARange = new Range(numA);
//B, C etc
// T F data samples : how do I add the boolean samples?
VariableArray<bool> data = Variable.Array<bool>(taskARange);
using (Variable.ForEach(taskARange))
{
//data [taskARange] = <???> T / F data stream
}
// Inference engine (EP)
InferenceEngine engine = new InferenceEngine();
// Infer probability of Task A Math so can predict Task B_Math etc.
Console.WriteLine("isCorrect = {0}", engine.Infer<Beta>(data ));
//Next trial
//ENDFriday, June 3, 2011 5:49 PM 
alexjames replied on 06182010 6:56 AM
Hello David
That would be great if you could lend a hand, I have made a start as below. I am interested in making an inference from the N categories to one 'master' category also, to allow further organisation of the task.
best
alex
Friday, June 3, 2011 5:49 PM 
DavidKnowles replied on 06202010 11:26 AM
The code below could be a starting point. We assume there is some hidden probability of getting the correct answer "taskA_performance", which we give a uniform prior over the interval [0,1]. We then infer the posterior over taskA_performance given the observed data.
// Data about tasks: true = correct, false = incorrect
bool[] TaskA_Math = { false, true, true, true, true, true, true, true, false, false };// Range for A
int numA = TaskA_Math.Length;
Range taskARange = new Range(numA);VariableArray<bool> data = Variable.Array<bool>(taskARange);
// The prior
var taskA_performance = Variable.Beta(1, 1);// The likelihood model
data[taskARange] = Variable.Bernoulli(taskA_performance).ForEach(taskARange);// Attach data
data.ObservedValue = TaskA_Math;// Inference engine (EP)
InferenceEngine engine = new InferenceEngine();// Infer probability of Task A Math so can predict Task B_Math etc.
Console.WriteLine("isCorrect = {0}", engine.Infer<Beta>(taskA_performance).GetMean());Friday, June 3, 2011 5:49 PM 
alexjames replied on 06212010 5:47 AM
Thanks David, that's just what I wanted to start me off.
I also found a few tutorials that have been helpful, the clinical infer.net and a MLSS practical:
http://research.microsoft.com/enus/um/cambridge/projects/infernet/docs/clinical%20trial%20tutorial.aspx
http://mlg.eng.cam.ac.uk/mlss09/infernet.zip
best
alex
Friday, June 3, 2011 5:49 PM 
alexjames replied on 06252010 3:44 AM
Hello, that is working well.
I uncovered a post of John's that covers the retrieval of a distribution across a vector of probabilities which I plan to use for summarising the performance of Task [A, B, C ... n] subtasks. This should let me abstract task predictions at progressively higher levels (unless I have misunderstood John' s post):
http://community.research.microsoft.com/forums/t/4204.aspx
best
alex
 Marked as answer by Microsoft Research Friday, June 3, 2011 5:49 PM
Friday, June 3, 2011 5:49 PM