The late great management guru, Peter Drucker, helped to innovate theidea that there is a whole economic class known as the Knowledge Workeronce said: "Nobody has really looked at productivity in white collarwork in a scientific way. But whenever we do look at it, it isgrotesquely unproductive." In other words, trying to define how tomeasure the productivity or performance of Knowledge workers (commonlycalled white collar workers), is an exercise in futility.
Davenport's bookpoints out that it is hard to create a common correlative measurementaround "knowledge workers" as a whole class. In fact, he describes thatthe typical way of dealing with knowledge workers is the HSPALTA approach: Hire smart people and leave them alone. Unfortunately, this doesn't really examine how to improve the system or help people improve themselves.
Maybe we will eventually discover some future magical formula thatmeasures this performance and how to improve it. In the past, inAgricultural societies, we had found ways to improve agriculturaloutput. Farmers from the dawn of time will tell you that "farming is anart", but the truth is that farming is also a science. Art issubjective, hard to measure, quantify and teach. Science is morestructured and actually can be taught (although not necessarily easily).During the Industrial Age, we achieved similar goals for manufacturingoutput. Now that we are in the Information Age, we are stumped, becauserather than a physical unit output, it is more of a mental qualitativeoutput, and that seems to us a very subjective element.
The good news is, as Davenport points out, there is at least one way to measurethe quality of knowledge. It's been done for centuries: the Peer Reviewprocess. It's most common in academia, whereby a group of your peersexamines your output and gives an analysis of what they think of it.It's how Masters and PhDs are still given out for the most part,worldwide.
I think that this is a good thing for us because that Peer Review process is atechnique that can be applied to unstructued knowledge on our site. Inits simplest form it is a Ratings systemwhereby anyone reading a piece of information on the site can vote 1through 5 on what they think of the article. It's entirely subjectivebut if you get a large number of ratings, it tends to average out whatpeople think of the information. This can apply to structured as wellas unstructured knowledge. This is the first level of a Ratings model.
That's a very basic notion. In fact, to be more useful, you may want tocollect all those ratings per a person's knowledge output and store itand those knowledge output items as part of their identity. Thus, youcan see what a person has contributed and produced and what peoplegenerally think of their output (their level of quality). This is amore evolved Rating system, generally referred to as a Reputation model.
Then, in turn, you could use a person's current rating as a weightingfactor to any rating they apply to others; i.e., normalize the value ofthe function of "my current rating" multipled by the rating value theyascribe. Thus when an industry luminary says you have a good idea, itweighs more towards the rating of that information, than when a novicerates it. Thus you have a weighted average of your Reputation based onwho actually rates your articles. This is a second evolution of Ratingsinto a weighted or a Ranked Reputation model.
How do you yourself become such an "industry luminary"? Essentially, alot of high-ranked people giving you good ratings implies that a lot ofknowledgable or influential people think that your output has a highlevel of quality. Thus, you would appear higher on the rankingshopefully amongst those lofty people who are the luminaries.
Community and social computing
with Tags: reputation X
rawn 100000R0P5 Tags:  reputation knowledge_workers unstructured_knowledge trust_models 2,338 Visits
We had a call today posing the question of how to work with Influencersin your community. I compared community influencers similar to how theindustry works with Analysts. Many organizations have Analyst Relationsteams whose sole job is to interact and communicate with analysts thatcover their products or services. Analysts come from many origins, butare usually recognized as experts in their topic, and hold a primaryjob function to cover the topic, or consult and interact with manycompanies, the press or other customers about the topic.
It's not a far stretch to say that leading bloggers, forum members, andothers who interact in a community are starting to gain (or evenbecome) the same status. The primary difference for most is the"amateur" status: rather than "being an expert" as their official job(the professional analyst), they are experts because of an existingjob, function or coverage they have. However, my point is that weshouldn't disregard them because of their amateur status. In fact,quite the opposite: this represents an opportunity to work with otherwho can help spread the word.
For the converted, this is not news, but in general this is still a newnotion. In fact, many organizations haven't even considered how toaddress this new population of amateur-analysts. They are looking for anew generation of PR/AR people who do get it. Businessweek's July 24th issue raises this issue.
On the other hand, some have (see the Nike examplein that article). Or on the other hand, there's the current debate onblogging-for-pay, where some bloggers are looking to get paid for theiractivities when in relation to product mentions, etc. I find thisdebate not very surprising when you consider this is already exists inthe field of professional-analysts. What is happening is that they aretrying to make a transition from amateur to professional status.
That is a bigger leap than a lot of people may think.
It's more than a matter of showing that you have X number of people whoread your blog, hence you should be paid $Y. In fact, what it points tois the somewhat mysterious/mystical reputationfactor. Anyone who tells you they have an equation for fame is talkinghorse's eggs (it's one my mom's favorite Bengali sayings: ghorar-dim).
This circles back to our Influencers and trying to figure out who theyare, and how to support them. With the social networking software oftoday, however, it is possible to learn some general behaviors like howmany people read something you've written, how often they may come backto it, what they think of your posting, etc. (Obviously, this onlyenters the picture when the influencer in questionwants to actually participate in this. Otherwise, you'd be running intosome heavy privacy invasions.) You need a number of different softwareelements to find that out, including a good metrics collection andanalysis system, a ratings system, per-user histories, peer networks,etc. This is more than a few separate pieces of social software we aretalking about, which means a lot of investment in developing the righttools. It's already going on in the industry as apparent by the growingnumber of social software sites and products that many new startups aretrying to capitalize on. So, there is and will be many different waysfor companies to start leveraging social systems.
But what is the value to the company? How do you even value somethinglike that, especially when they are so widespread across many differentsocial networking sites, tools, all of which provide different types offeatures, metrics, etc.
Many web sites in the industry have generally agreed upon the UniqueVisitor metric as a common measurement for people coming to sites, buthow do you measure reputation.Just because you have a lot of people reading the post, does notnecessarily mean you have any kind of influence over them. In fact, thepopulation may actively dislike an influencers views. Amusingly enough,this reminds of how Howard Stern became a big hit: not just his fansbut also those anti-fans listened to his show. Because of how thetraditional Wanamaker advertising model works (see my earlier post),it is considered a success. Is this still true in the evolvingpin-point targetted marketing model available on the Internet? (I'msure Mr Stern's producers certainly hope so: can he deliver on his$500M salary).
Do we need an industry-wide reputation metric? Is that even possible?Would this have any impact on what people use now: the traditional CPMadvertising/marketing metric model?
Well to get off the pontification of where it may go, I'll step into what we are thinking in terms of a reputation model next...
It's interesting to see the evolution of reward mechanisms in MMORPGs. In multiplayer situations, it is often difficult to tell who completes a task when there are a number of steps involved. For example, in a raid on WoW against a single monster, who gets recognition for the kill when it takes 40-80 people working as a team to kill that beast? It's defined now in WoW but this same problem existed all the way back to the days of text MUDs.
In the first generation, essentially there was no thought involved in this. Whoever made the killing blow got the points for the kill. Very primitive and the cause of many an argument. The next few improvements--it came in several different ways--was to add a list on the create being attacked where it kept track of who the attackers were and distribute the points based on how many points of damage each person made on the creature, so the distribution of experience points for the kill was more even. WoW provides a newer version of this whereby each person engaged in the raid gets dollars per kill, which once the creature was dead, they could spend to buy the reward items.
However, this still raises a problem when there are items that several people want to share from the common pool of rewards. In the people-administrative way, the guild--there's usually a guild involved--makes a decision on who gets what. This is where it breaks down again: what happens when there is an item so unique that the top contendors or leaders will argue for it. As my friend Eric described, that is the death knell of many a guild.
If you think this is a situation limited to MMORPGs and games, you're mistaken. This same situation exists when you have a community that is working for a shared goal, and there is some reward that needs to be distributed across the membership. In team scenarios in corporate environment, that is not very different than negotiations for annual bonuses. The differentiating factor is that in MMORPGs as in other communities, the group of people are not part of a single formal organization with defined managers and advocates. Instead it is up to each person to argue their own case. In other words, it is even more difficult in communities.
I don't have any answer here, but it is important to recognize that the situations are similar here, and compare what methods of distribution or approaches people take.