Hi everyone, I am quite interested in the infer.net technology and have been trying for many days to figure out how I can apply it to problems in my domain. I am not as interested in classification problems; my interest is more in finding a robust regression tool. My question is that I am not sure if any of the basic systems (Bayes Point Machines; Gaussian Processes; Mixture of Gaussians; etc.) presented here are relevant to the problems I am looking at. I am trying to predict time varying quantities (for example, the output "y" variable might be the % change of a quantity from the current value to the value 10 days in the future). For my input "X" data, I have data that could either be real valued or categorical. For example:
Date Input1 Input2 Input3 Input4 Output 8/17/2011 2.911 3.983 1.800 9.386 0.0818 8/16/2011 2.911 3.983 1.800 9.386 -0.0135 8/15/2011 2.911 3.983 1.800 9.386 -0.0445 8/12/2011 2.911 3.995 1.800 9.386 -0.0872 8/11/2011 2.916 3.999 1.809 9.401 -0.0784 8/10/2011 2.928 4.008 1.809 9.408 -0.0359 8/9/2011 2.928 4.008 1.809 9.428 -0.0279 8/8/2011 2.941 4.008 1.809 9.428 -0.1077 8/5/2011 2.941 4.008 1.810 9.428 -0.0369 8/4/2011 2.951 4.008 1.863 9.428 -0.0800 8/3/2011 2.951 3.991 1.853 9.487 -0.0214 8/2/2011 2.951 3.991 1.853 9.502 -0.0574 8/1/2011 2.948 3.991 1.853 9.486 -0.0503 7/29/2011 2.949 3.968 1.853 9.417 -0.1350 7/28/2011 2.949 3.926 1.853 9.404 -0.1497 7/27/2011 2.915 3.859 1.853 9.382 -0.1991 7/26/2011 2.917 3.859 1.853 9.382 -0.1575 7/25/2011 2.917 3.859 1.853 9.382 -0.2318 7/22/2011 2.917 3.859 1.853 9.382 -0.1834 7/21/2011 2.917 3.859 1.853 9.382 -0.1660 7/20/2011 2.917 3.875 1.853 9.382 -0.1200 7/19/2011 2.917 3.875 1.853 9.382 -0.1152 7/18/2011 2.917 3.875 1.853 9.382 -0.0450 7/15/2011 2.923 3.875 1.853 9.382 -0.0222 7/14/2011 2.923 3.863 1.853 9.382 -0.0179 7/13/2011 2.923 3.863 1.853 9.382 -0.0172 7/12/2011 2.923 3.863 1.853 9.382 0.0199 7/11/2011 2.923 3.863 1.853 9.382 0.0419 7/8/2011 2.935 3.875 1.853 9.382 0.0205 7/7/2011 2.935 3.875 1.853 9.382 0.0014 7/6/2011 2.935 3.859 1.853 9.380 -0.0215
The basic idea is that I don't know the relationship between the inputs and the outputs. In the example above, there are only 4 inputs, but in my real problem there are many more. But there is certainly a lot of structure here, and I have a feeling that this is something that infer.net should be able to help with. The other question I have is that, in the example above, there are 4 inputs, each representing descriptive data about a different entity; i.e., each entity is represented by a single column of data. By entity I simply mean a quantity that in my application domain is related to the output I'm trying to model. For example, the data could be weather data, say, normalized average temperature readings of nearby cities, and I am trying to predict the temperature in a nearby, unknown city using this data. If I instead wanted to give, say, 3 columns of data for a given entity, how could I do this? In the example of weather data, suppose that in addition to having the temperature data for each city, I also have humidity and wind speed. Is there some way to explain to infer.net that the temperature, humidity, and wind speed for a given city are all part of a "package" for that city? The reason I ask is because I wouldn't want to be making inferences based on meaningless comparisons (say, comparing the wind speed of City A to the humidity of City B in order to predict the temperature in City C).
From searching the microsoft research site, I see that there is a program called "Tark" that pertains to time varying quantities. Is this more relevant to my problem? Can Tark work together with infer.net? Any advice or clarification would be much appreciated.
PS: I have also been investigating work on hierarchical temporal memory (HTM) such as the systems developed by Numenta (http://www.numenta.com/). It occurs to me that there is significant overlap between HTMs and the functionality provided in infer.net. Is it possible (or even desirable) to try to implement HTMs using infer.net?
The Gaussian Process classifier and Bayes Point Machine can be modified to predict a continuous output. For example, in the Gaussian Process classifier where it builds the observation model:
// The observation model VariableArray<bool> y = Variable.Observed(outputs, j).Named("y"); Variable<double> score = Variable.FunctionEvaluate(f, x[j]); y[j] = (Variable.GaussianFromMeanAndVariance(score, 0.1) > 0);
you can change y to be double and remove the >0 part:
// The observation model VariableArray<double> y = Variable.Observed(outputs, j).Named("y"); Variable<double> score = Variable.FunctionEvaluate(f, x[j]); y[j] = Variable.GaussianFromMeanAndVariance(score, 0.1);
I don't understand your question about grouping data for each city. Why should you not use wind speed of city A to predict temp in city C? And if you are not using it, why include the wind speed of city A in the data?
Tark is a special tool for finding ordering patterns in a sequence of discrete events. The data you've provided is not of this form. One way that you could combine Tark with Infer.NET is to use Infer.NET to cluster the time points (the mixture model example does this). The cluster number of each time point is a discrete value describing the time point. You could apply Tark to the sequence of cluster numbers to discover temporal patterns among them.
- Edited by Tom MinkaMicrosoft employee Monday, September 05, 2011 8:01 AM
Hi, thanks for your response Tom. I will play with the Gaussian Process classifier as you suggest.
In terms of the data grouping question, I think I didn't properly express what I am concerned about. It's not so much that I don't want to make spurious comparisons, it is more that I want to express the concept that a group of observations form a "chunk" that describe one single entity. Suppose instead of cities and temperature that we were looking at stocks. We might have several data points for each stock; the "standardized" daily % return in the stock's price, the most recent % change in the book value per share, the industry category (i.e., "Chemicals" or "Industrial"), etc. We would want to keep all of those observations together for that single stock. If we then wanted to predict things about that stock, say by looking at other stocks that might be in the same industry or something, it seems that you would be losing huge amounts of useful information if your classifier treated each column of data (say, the % return for the price of stock A) as a separate entity without having a concept that this information forms a bundle for each stock. Does that make sense?
Your suggestion about Tark seems extremely promising and I will certainly try to get that to work. I would humbly suggest that such a Tark/infer.net crossover application would make a great candidate for a future worked example/tutorial, since I would imagine that many potential applications are concerned with time series data, such as econometric and sociology applications, and leveraging two cutting edge tools (both from Microsoft) strikes me as being pretty exciting and compelling.
Thanks again for your help!
Standard regression models such as Gaussian Processes don't have the concept of grouping that you describe. However, it is possible to define custom regression models that have grouping in Infer.NET. You just need to work out how the grouping should influence the regression.