Changing the Rating Equation - Suggestions

Jabapac

Gender: Male
Age: 36

Defago wrote:

The Rated and Ranked are almost identical at this moment. If we do reduce m to 5, then we'll have two charts with different priorities, which would make two distinct charts (one more volatile and the other one is the one we know), whereas now we have 2 charts with one same priority, making them identical.

How identical?
Hybrid Theory by Linkin Park is rated only 63/100 and has the rank of 392. which is relatively high!.. and there are many more examples of that tendency.

Defago wrote:

That's all good, that's why we have a Ranked chart.

Not really, many users here think the ranked charts are more accurate, but maybe many also think the current rated charts are more accurate. as I see it, the ranked charts are not absolute accurate to get enough to. because the ranking system ignores the dislikes, which the ratings allow them by low rating.

Defago
Your Most Favorite User

Gender: Male
Age: 31
Location: Lima

When I claim the charts are similar it's not out of my personal opinion. The charts are objectively similar.

73% of the Top 100 Ranked albums are present on the Top 100 Rated albums.
73% of the Top 100 Rated albums are present on the Top 100 Ranked albums.
There's a 73% similarity between the both charts, objectively.

This means there's only 27 albums which are not common to both charts. Out of the 27 albums on the Top 100 Rated but not on the Top 100 Ranked, 17 are present on the Top 200 Ranked. Only 10 albums on the Top 100 Rated are NOT present on the Top 200 Ranked. These include Good Kid, M.A.A.D City, Trouble Will Find Me, Songs Of Leonard Cohen, In A Silent Way, Lonerism, Red, Spirit Of Eden, Five Leaves Left, Laughing Stock and Cosmo's Factory.

The value of correlation (a number between 0 and 1 which measures how similar the order between the charts is) is +0.69 between both Top 10, +0.63 between both Top 25, +0.6 between both Top 50 and +0.38 between both Top 100. This last value is lower because of Trouble Will Find Me by The National, which has been rated highly but is not ranked highly (the implementation of my suggestion will increase the number of Trouble Will Find Mes on the rated chart and reduce the number of classic, known-by-everyone albums). To give you a clearer view, a correlation value of +0.4 or more indicates a strong positive relationship. This is to be interpreted as that there is a strong similarity in the ordering of both charts.

So, yeah. This isn't me thinking they're kinda similar. This is me affirming they're 73% identical and that their orderings are related with a correlation factor of 0.6 at the median.

Last edited by Defago on 05/21/2013 01:35; edited 1 time in total

undefined

Defago wrote:

When I claim the charts are similar it's not out of my personal opinion. The charts are objectively similar.

73% of the Top 100 Ranked albums are present on the Top 100 Rated albums.
73% of the Top 100 Rated albums are present on the Top 100 Ranked albums.
There's a 73% similarity between the both charts, objectively.

This means there's only 27 albums which are not common to both charts. Out of the 27 albums on the Top 100 Rated but not on the Top 100 Ranked, 17 are present on the Top 200 Ranked. Only 10 albums on the Top 100 Rated are NOT present on the Top 200 Ranked. These include Good Kid, M.A.A.D City, Trouble Will Find Me, Songs Of Leonard Cohen, In A Silent Way, Lonerism, Red, Spirit Of Eden, Five Leaves Left, Laughing Stock and Cosmo's Factory.

The value of correlation (a number between 0 and 1 which measures how similar the order between the charts is) is +0.69 between both Top 10, +0.63 between both Top 25, +0.6 between both Top 50 and +0.38 between both Top 100. This last value is lower because of Trouble Will Find Me by The National, which has been rated highly but is not ranked highly (the implementation of my suggestion will increase the number of Trouble Will Find Mes on the rated chart and reduce the number of classic, known-by-everyone albums). To give you a clearer view, a correlation value of +0.4 or more indicates a strong positive relationship. This is to be interpreted as that there is a strong similarity in the ordering of both charts.

So, yeah. This isn't me thinking they're kinda similar. This is me affirming they're 73% identical and that their orderings are related with a correlation factor of 0.38 at the median.

This is why you pay attention in math class kids. It comes in handy

Windowblinds

Happymeal wrote:

I didn't even notice this suggestion. Interesting I guess. Wouldn't affect me personally either way so whichever, whatever.

Thany you for this amazingly informative contribution.

Happymeal

Windowblinds wrote:

Thany you for this amazingly informative contribution.

It's not much of a problem, I'm always here to add in my own unique and fun commentary.

albummaster
Janitor

Gender: Male
Location: Spain
Site Admin

Defago wrote:

Currently, the equation to determine an item's rating uses a Bayesian equation to avoid items with a low amount of ratings to crowd the top of the charts. Without this measure, an album with one single rating of 100 would be the most rated album on the whole site, which would make the Best Rated Albums chart useless.

The actual equation is as follows (taken from BEA's How Does It Work page)

Average Rating = (n ÷ (n + m)) × av + (m ÷ (n + m)) × AV

Which is more easily read as

Average Rating = (n × av + m × AV) / (n + m)

where
n is the amount of ratings received
m is a fixed value, currently 20
av is the average of all ratings received
AV is the average of all ratings on the site, which I'm estimating to be around 75

As mentioned before, the idea behind doing things this way instead of a regular, vanilla average is to avoid items jumping to the top with a single rating, because our top rated charts would be filled with unknown stuff which has gotten one single rating.

I'd like, however, to suggest reducing the "m" constant. It used to be fixed at 10, and then it was raised to 20 recently. Having such a high "m" value is negative, in my opinion. The core effect of this is that items with fewer ratings, regardless of the actual ratings, get a lower average. An item with ten ratings will have a maximum possible average of 83.

On one side, I understand the need to use the Bayesian average to avoid inaccuracies. However, I believe a reduced number such as 5 would do the trick, while also providing incentives for people to listen to new albums.

Sorry for not responding sooner, I'm having problems getting online at the moment. As pointed out, the minimum number of ratings to qualify for a top-rated chart was increased to 20 not so long ago.

Items with more ratings tend to be the most popular/most listened, which I agree is 'hiding' some lesser-known albums from appearing in these lists, but the key objective must be to have a stable chart that stands up to scrutiny. I think reducing this figure to five would lead to a very unstable chart, but I can definitely see a case for reverting back down to 10.

The reason that the minimum number of ratings was increased was because it was felt that ten ratings wasn't enough to provide a statistically valid figure. On the other hand, the site doesn't seem to have enough volume at the moment to justify using a higher number as there are too many items with not enough ratings and it takes too long for an item to get that many votes. It's difficult to keep everybody happy because, whatever happens, there will be people either way that won't get what they want, but I'd personally be happy to reduce this figure again until the site grows a bit more.

Jabapac

Gender: Male
Age: 36

albummaster wrote:

but I'd personally be happy to reduce this figure again until the site grows a bit more.

I agree, as m= 1% of the total no. of ratings of the top 10 ranked albums as you suggested before in other topic.

Defago
Your Most Favorite User

Gender: Male
Age: 31
Location: Lima

albummaster wrote:

Sorry for not responding sooner, I'm having problems getting online at the moment. As pointed out, the minimum number of ratings to qualify for a top-rated chart was increased to 20 not so long ago.

Items with more ratings tend to be the most popular/most listened, which I agree is 'hiding' some lesser-known albums from appearing in these lists, but the key objective must be to have a stable chart that stands up to scrutiny. I think reducing this figure to five would lead to a very unstable chart, but I can definitely see a case for reverting back down to 10.

The reason that the minimum number of ratings was increased was because it was felt that ten ratings wasn't enough to provide a statistically valid figure. On the other hand, the site doesn't seem to have enough volume at the moment to justify using a higher number as there are too many items with not enough ratings and it takes too long for an item to get that many votes. It's difficult to keep everybody happy because, whatever happens, there will be people either way that won't get what they want, but I'd personally be happy to reduce this figure again until the site grows a bit more.

OK, thanks a lot.

As a side note, and for a future time when there are not so many site changes on the list, would it be possible to have a "volatile Top Rated" chart as an additional chart? Kind of like a top rated but removing M completely and the minimum amount of ratings (only on that chart), so as to have a really chaotic chart people could go to to find good random album recommendations.

drakonium
coucou

Location: More than one

Defago wrote:

As a side note, and for a future time when there are not so many site changes on the list, would it be possible to have a "volatile Top Rated" chart as an additional chart? Kind of like a top rated but removing M completely and the minimum amount of ratings (only on that chart), so as to have a really chaotic chart people could go to to find good random album recommendations.

This would be FANTASTIC.

albummaster
Janitor

Gender: Male
Location: Spain
Site Admin

Defago wrote:

OK, thanks a lot.

As a side note, and for a future time when there are not so many site changes on the list, would it be possible to have a "volatile Top Rated" chart as an additional chart? Kind of like a top rated but removing M completely and the minimum amount of ratings (only on that chart), so as to have a really chaotic chart people could go to to find good random album recommendations.

Yes, I'll look at adding something like that.