none
Using the Forum to Measure Team Expertise

    General discussion

  • Hello,

    As a project manager of the company I currently working, I was thinking about creating a productivity indicator based on the number of "medals" of the msdn/expression forums a person in my team have. The ideia would be motivate them helping and participating in the forums to ask questions but also to answer questions increasing the number of points.

    Instead of start certifing people out there, this would be a initial movement on the very difficult task of measuring the ability, expertise and comunicability of the people. If someone in my team have several points, it reflects on experience and knolodge obtained. This would be much cheaper than certification and completely related to the real and actual issues that are up.

    Please note that in my understanding the company must have more than on indicator, since "numbers don't lie" but they can fool you.

    Have you ever had any experience on this?

    Regards,

    Igor.
    Project Manager at INOVATIVA Tecnologia www.inovativatec.com
    Thursday, September 24, 2009 5:05 PM

All replies

  • Oh, the numbers here DO lie :)

    The current implementation of the points system rewards people more for posts that are helpful to the community.  That's great, except it requires that people reading the messages to take some action (to vote the post as helpful, or mark/propose the post as an answer).

    In the forums I've been following, I see that very few people take one of these actions on a helpful post.  So maybe it depends on the particular forums you're asking people to participate in.  If you have a few specific ones in mind, you should check and see if people frequently use the "vote as helpful" and "propose as answer" features.

    If you do find a way to do this, please make sure to come back to this thread in a few weeks/months to let other folks know how you did it :)

    Thanks,
    Mark
    (Note: I'm not on the forums team)
    Want to know if your hardware will work on Windows 7? Check out these links: **Windows 7 Upgrade Advisor ** **Microsoft Logo'd Hardware List**
    Friday, September 25, 2009 4:20 AM
  • Hello Mark,

    I see what say... Usually it happens to me somthing similar when I ask a question to a forum and somebody replies. Than I take much time to test the proposed sollution, and I completely forget to go there and "mark as an answer". But, after some time I receive an e-mail form the moderator that "proposes the reply as an answer". Than I usually take an action, like continue the discussion, mark as an anser and do nothing (here is the problem) if I still did not have time enough to investigate the issue.

    I was thinking about using this forum to somehow motivate people in my team to use the forum and look for posts instead of wasting ours trying to figure out something that was already discussed here. In the other hand, answering questions and getting poinst for that would potentially show the technical evolution. But as you highlighted, it it would not be a trusteable/fair indicator.

    Thank you very much for your opinion and let's see if somebody else has another contribution to this issue.

    Igor.


    Project Manager at INOVATIVA Tecnologia www.inovativatec.com
    Friday, September 25, 2009 11:52 AM
  • Interesting, I didn't know that there were messages sent to people suggesting they mark something as an answer.  It's possible I just missed them before.  The alerts system needs a little fine tuning :)

    I think what you're trying to do is a fantastic idea, so I hope you're successful.


    Want to know if your hardware will work on Windows 7? Check out these links: **Windows 7 Upgrade Advisor ** **Microsoft Logo'd Hardware List**
    Saturday, September 26, 2009 8:14 AM
  • Hello Mark,

    Sometimes (when I receive a reply for a question I made, but forget to mark it as an answer), I receive a message like this:

    A reply from XXXXXXXXXX - MSFT has been marked as a proposed answer in the forums thread you have subscribed to. If it answers your question, mark it as the answer!

    I suppose there are someone in MSFT (or maybe the moderator) that notes that the discussin died in the post and click on "propose as an answer" or somethink like this. I know there is an option for that.

    Regards,

    Igor.


    Project Manager at INOVATIVA Tecnologia www.inovativatec.com
    Monday, September 28, 2009 1:59 PM
  • That is an interesting proposal Igor.

    As mentioned, the 'medals' don't tell a convincing story. Two points are awarded just for responding and ten points are awarded for being credited with helping with the 'answer'. One can amass a lot of points, and thereby, medals, by regularly replying without being really helpful. Sometimes folks try to get a lot of points without really helping -they just reply, and reply, and keep replying. And points are not always awarded properly. You help someone and they give someone else credit. Or they never return to the thread and no one gets credit. But on average, there is a 'workable' indicator.'

    One of the ways that I 'guage' participation is by examining the users activity. I look at the ratio of Posts to Answers. For a Moderator, I expect to see a ratio of 1 Answer per 5 Posts (or better). This is an imperfect 'quality' indicator.

    Looking at your activity, I see that you have 3 Answers out of 100 Posts, or 3%, and a low 'Points' total. I would guess that you are in a 'learning' mode -asking a lot of questions and getting help. At some point, when you feel confident with your skills, you may choose to start helping others and posting replies. Then your points will increase, and more importantly, the number of Answers attributed to you will also begin to increase.

    Looking at my own profile, you will see that my answer ratio is about 43%, and the high point count validates that most of my activity is from 'helping' others.

    So don't be misled by the medals or points, look at the qualitative measures too.

    And I wish more managers would consider actively encouraging their staff to participate in the Forums. It is my contention that you really learn your craft when you try to help others. You have to understand it, you have to be able to communicate it, and you have to arrive at a solution. And that is what we seek with our development staff.

    I hope that this helps, and that you do encourage your staff to help in the Forums.


    You may be only one person in the world, but you may also be the world to one person.
    Saturday, October 10, 2009 4:02 AM
  • Hello Arnie,

    I think this ratio you created is great for measuring quality (where everybody who is planning to manage competencies should look for). I see some problem here when someone posts a question, receives a good answer but actually do not "mark it as an answer" in the forum. This wold not contribute to the ratio of the guy that answered the question. I see some moderators "porpose the reply as an answer", but I don't think this is always the case. How do you manage that? Do you think this is very common?

    Igor
    Project Manager at INOVATIVA Tecnologia www.inovativatec.com
    Saturday, October 10, 2009 10:02 AM
  • I don't 'manage' it, I inore it. That is why I indicated that it was an 'imperfect' metric.

    When you are first starting out, it 'hurts' to not get the attribution you deserve. However, if one keeps up the effort, and remains consistent and on message, the contributions (and quality) will still be obvious.

    As I wrote before, some folks do not properly attribute an answer, or attribute it to the incorrect person, or not at all. Some Moderators attempt to help, we try to close the threads when possible. but sometimes it is not clear if the original poster recieved the answer that helped him/her move forward.

    And on occasion, I will see a response marked as 'Answer' when it is not technically correct. But it did help the OP solve his/her issue.

    So ignore the descripancies and focus on the imperfect metric. It is better than the other options.


    You may be only one person in the world, but you may also be the world to one person.
    Saturday, October 10, 2009 12:54 PM
  • This really needs to be addressed. I've seen people with signature text explicitly encouraging other forum members to mark their posts as "helpful" where so, yet those same people never seem to mark other people's posts as "helpful"!!! Now is that hypocritical, or what?

    I've also seen cases where these same folks deliberately run their own "Answers" as a reply to the real "Answer" on a thread, where the reply "Answer" contains only trivial modifications or amendments to the real "Answer". Such posts appear designed to persuade the moderators to award "Answer" points to both the real "Answerer" and to the person who contributed the trivial modification.

    The members who rarely vote on other people's posts (and most especially those who consistently contribute trivial run-behind-the-real-answer answers that attempt to ride on the real answer's slip-stream, as well as never marking other people's posts as "helpful"), should be marked down significantly on their own points. Until members start giving points to other people for significant and genuine contributions, they don't deserve any. (My suggestion here is that each member's points should be determined not just by their answers, but also, by how they mark other people's answers.)


    Matthew Slyman M.A. (Camb.)
    Wednesday, June 22, 2011 10:26 AM
  • It depends on personal standard. A lot of posts I see here are either duplicates or variants of existing posts that are easiler to solve than other posts. I don't think the replies in the posts are more helpful to replies in other posts - they are mostly duplicates of existing answers and do not add to the forum's knowledge pool.

    Of course, the thread starter would have a quite different opinion on what is helpful. And everyone has one vote.



    The following is signature, not part of post
    Please mark the post answered your question as the answer, and mark other helpful posts as helpful, so they will appear differently to other users who are visiting your thread for the same problem.
    Visual C++ MVP
    Wednesday, July 27, 2011 3:11 AM
    Moderator