🔒 Locked

Statistical Analysis on the new "rating" function?

Started by IADaveMark Jun 28, 2004 at 2:48 PM 47 replies 7.3k views

Original Post

Jun 28, 2004 at 2:48 PM

To the mods: I would love to see a graph of a query from the DB showing the distribution of member rating entries. Specifically, I would be interested in answering questions such as: What is the shape of the curve? In theory, if done correctly, the shape would be the typical "normal distribution" or "bell curve". i.e. Most people would be 5's while only a few would be 1's or 10's. I figure that is not the case - but acutally the reverse is true... most votes are skewed towards the outsides of the range: many 8-10s and 1-3s. What are the mean and median of the sample? Is the mean or median is up around 8 or 9, or lower? That would show that a 9 or 10 rating really isn't "very helpful" but only slightly better than normal. In a way it's like the gymnastics or skating scores where you can fall on your ass and still have a "9.625" or some such rediculous score. How close are the mean and median to each other (presence of outliers?) The shape of the curve really comes into play here as well... are most of the votes 8-10 with some of the votes being 1's that drag down the average? This should be something that is easily extracted and dumped into a program such as Excel. I hope that this would be something you guys are willing to do. Thanks!

Bossk

Jun 28, 2004 at 3:27 PM

What I'm curious about, is when you have not yet rated someone, whether your rating counts for nothing, or it counts as 5 (right in the middle of the scale) ...

Like do 5 ratings actually help/hurt people? Or do they not alter ratings at all?

If they don't actually affect your score, then you could say that every person on the site who has not yet rated you has actually rated you a 5, bringing the median much much closer to the middle of the 1-10 scale.

IADaveMark

Jun 28, 2004 at 3:39 PM

I can almost guarantee that it is on an "entry only" basis... that if you haven't rated someone, there is NO effect whatsoever. If that were the case, they would have had to seed the data with a "5" for n*(n-1) entries.

twix

Jun 28, 2004 at 3:42 PM

Quote:
Original post by Bossk
Like do 5 ratings actually help/hurt people? Or do they not alter ratings at all?

It appears that they don't affect anything.

mittens

Jun 28, 2004 at 3:59 PM

I'm curious on two points:

Why it matters?
Who cares?
How it concerns you.

Yeah, I can count. Really.

Promit

Jun 28, 2004 at 4:46 PM

Quote:
Original post by twix
Quote:
Original post by Bossk
Like do 5 ratings actually help/hurt people? Or do they not alter ratings at all?

It appears that they don't affect anything.

Agreed, LOL [grin]

coldacid

Jun 28, 2004 at 4:59 PM

Quote:
Original post by mittens
I'm curious on two points:

Three, mean thee?

Quote:
Why it matters?

It's an interesting study in social dynamics here at GDNet.

Quote:
Who cares?

The site's amateur statisticians and social scientists.

Quote:
How it concerns you.

It helps us understand how the system is working (I mean this more in a "on-target" way, not tinkering with it). And helps you guys refine it so that it works better.

Quote:
Yeah, I can count. Really.

Are you sure about that? :D

Max_Payne

Jun 28, 2004 at 5:34 PM

Not exactly on the same subject, but related: I think it would be nice that aside the total rating of a user, it shows the rating you gave that user (because often you may not remember rating the person, or your opinion may change with time).

IADaveMark

Jun 28, 2004 at 5:47 PM

Quote:
Original post by mittens
I'm curious on two points:

Why it matters?
Who cares?
How it concerns you.

I'm curious on three points:

Why does it matter to you why it matters to me?
Why you figure out that if it didn't matter to at least ME, I wouldn't have asked in the first place?
Why my asking concerns YOU?

What a completely moronic response to post.

Michael Tanczos

Jun 28, 2004 at 9:01 PM

For ratings, here is some data. Votes are recorded as a number between 0 and 1:

So far,

5,795 ratings have taken place
The average vote is 0.724 (or a 7)
The median vote is a 0.9 (or a 9)
The standard deviation is 0.38

It would appear that people are more likely to vote for people they like than dislike, but are more prone to vote in extremes either way.

Here is the total rating data thus far.

0	9990.1	310.2	1730.3	500.4	1400.5	810.6	880.7	2380.8	5870.9	5271	2884

---
Michael Tanczos

Samith

Jun 28, 2004 at 9:10 PM

I usually mostly vote for people I like. I think I've voted down one or two people and voted up quite a few. The only ratings I've given are 9s, 10s, and 0s. Probably because I want to make the most impact possible.

IADaveMark

Jun 28, 2004 at 9:19 PM

OK, guys... this is a completely useless rating system as it is laid out currently. Only true purists are going to really use the gradients. Most people will peg it to the edges. You might as well go to a completely binary system of respected or not respected.

C-Junkie

Jun 28, 2004 at 9:23 PM

Nah.

I think those few thousand people who didn't use 10 or 0 might disagree.

ArchWizard

Jun 28, 2004 at 9:24 PM

Quote:
Original post by C-Junkie
Nah.

I think those few thousand people who didn't use 10 or 0 might disagree.

Actually, that table represented uses of the other ratings, not the number of people who use the other ratings. I probably account for at least 15 of those non-extreme ratings.

Michael Tanczos

Jun 28, 2004 at 9:31 PM

Quote:
Original post by InnocuousFox
OK, guys... this is a completely useless rating system as it is laid out currently. Only true purists are going to really use the gradients. Most people will peg it to the edges. You might as well go to a completely binary system of respected or not respected.

I'm willing to wait on that. A great deal of ratings that have been done by staff and mods already have been binary in nature to seed the system. But the way it looks.. the fine gradient is unnecessary. Instead have a smaller gradient of maybe 4 choices with no selectable middle.

---
Michael Tanczos

IADaveMark

Jun 28, 2004 at 9:35 PM

67% of the ratings were either 0 or 10.
86% of the ratings were {0,8,9,10}
97% of the ratings were {0,7,8,9,10}

I am working on a more usable, less abusable system that simply allows you to add someone to your respected user list. The rating is generated by not only the number of people who respect you but by THEIR respect rating as well. It is run in a batch mode so that as THEIR respect rating increases, yours does as well. (It helps to be respected by people who are respectable!)

Samith

Jun 28, 2004 at 9:41 PM

What if someone with a low respectability respects you, will it drag you down? Because that would suck. Otherwise, the idea sounds alright.

Michael Tanczos

Jun 28, 2004 at 9:42 PM

Quote:
Original post by InnocuousFox
The rating is generated by not only the number of people who respect you but by THEIR respect rating as well. It is run in a batch mode so that as THEIR respect rating increases, yours does as well. (It helps to be respected by people who are respectable!)

This really is a point of contention.. since ratings are placed at a point in time where the person being rated may still be inexperienced. The people they rate highly at a particular moment in time might not be the same people that they would be inclined to rate highly at a future moment in time.

---
Michael Tanczos

BitBlt

Jun 28, 2004 at 9:53 PM

This is getting heavy. Pretty soon we'll be dealing with cosmology and the space-time continuum.

Cypher Ae

Jun 28, 2004 at 9:56 PM

Quote:
Original post by coldacid
The site's amateur statisticians and social scientists.

Social scientist - now there's a condtradiction in terms.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.