Best Ever Albums

Re: Changing the Rating Equation

albummaster — Wed, 22 May 2013 03:38:36 GMT

Author: albummaster

Posted: 05/22/2013 07:38

Defago wrote:

OK, thanks a lot.

As a side note, and for a future time when there are not so many site changes on the list, would it be possible to have a "volatile Top Rated" chart as an additional chart? Kind of like a top rated but removing M completely and the minimum amount of ratings (only on that chart), so as to have a really chaotic chart people could go to to find good random album recommendations.

Yes, I'll look at adding something like that.

Re: Changing the Rating Equation

drakonium — Tue, 21 May 2013 14:14:24 GMT

Author: drakonium

Posted: 05/21/2013 18:14

Defago wrote:

As a side note, and for a future time when there are not so many site changes on the list, would it be possible to have a "volatile Top Rated" chart as an additional chart? Kind of like a top rated but removing M completely and the minimum amount of ratings (only on that chart), so as to have a really chaotic chart people could go to to find good random album recommendations.

This would be FANTASTIC.

Re: Changing the Rating Equation

Defago — Tue, 21 May 2013 14:10:59 GMT

Author: Defago

Posted: 05/21/2013 18:10

albummaster wrote:

Sorry for not responding sooner, I'm having problems getting online at the moment. As pointed out, the minimum number of ratings to qualify for a top-rated chart was increased to 20 not so long ago.

Items with more ratings tend to be the most popular/most listened, which I agree is 'hiding' some lesser-known albums from appearing in these lists, but the key objective must be to have a stable chart that stands up to scrutiny. I think reducing this figure to five would lead to a very unstable chart, but I can definitely see a case for reverting back down to 10.

The reason that the minimum number of ratings was increased was because it was felt that ten ratings wasn't enough to provide a statistically valid figure. On the other hand, the site doesn't seem to have enough volume at the moment to justify using a higher number as there are too many items with not enough ratings and it takes too long for an item to get that many votes. It's difficult to keep everybody happy because, whatever happens, there will be people either way that won't get what they want, but I'd personally be happy to reduce this figure again until the site grows a bit more.

Re: Changing the Rating Equation

Jabapac — Tue, 21 May 2013 10:34:07 GMT

Author: Jabapac

Posted: 05/21/2013 14:34

albummaster wrote:

but I'd personally be happy to reduce this figure again until the site grows a bit more.

I agree, as m= 1% of the total no. of ratings of the top 10 ranked albums as you suggested before in other topic.

Re: Changing the Rating Equation

albummaster — Tue, 21 May 2013 06:43:56 GMT

Author: albummaster

Posted: 05/21/2013 10:43

Defago wrote:

Currently, the equation to determine an item's rating uses a Bayesian equation to avoid items with a low amount of ratings to crowd the top of the charts. Without this measure, an album with one single rating of 100 would be the most rated album on the whole site, which would make the Best Rated Albums chart useless.

The actual equation is as follows (taken from BEA's How Does It Work page)

Average Rating = (n ÷ (n + m)) × av + (m ÷ (n + m)) × AV

Which is more easily read as

Average Rating = (n × av + m × AV) / (n + m)

where
n is the amount of ratings received
m is a fixed value, currently 20
av is the average of all ratings received
AV is the average of all ratings on the site, which I'm estimating to be around 75

As mentioned before, the idea behind doing things this way instead of a regular, vanilla average is to avoid items jumping to the top with a single rating, because our top rated charts would be filled with unknown stuff which has gotten one single rating.

I'd like, however, to suggest reducing the "m" constant. It used to be fixed at 10, and then it was raised to 20 recently. Having such a high "m" value is negative, in my opinion. The core effect of this is that items with fewer ratings, regardless of the actual ratings, get a lower average. An item with ten ratings will have a maximum possible average of 83.

On one side, I understand the need to use the Bayesian average to avoid inaccuracies. However, I believe a reduced number such as 5 would do the trick, while also providing incentives for people to listen to new albums.

Re: Changing the Rating Equation

Anonymous — Mon, 20 May 2013 23:26:37 GMT

Author: Anonymous

Posted: 05/21/2013 03:26

Windowblinds wrote:

Thany you for this amazingly informative contribution.

It's not much of a problem, I'm always here to add in my own unique and fun commentary.

Re: Changing the Rating Equation

Windowblinds — Mon, 20 May 2013 23:21:33 GMT

Author: Windowblinds

Posted: 05/21/2013 03:21

Happymeal wrote:

I didn't even notice this suggestion. Interesting I guess. Wouldn't affect me personally either way so whichever, whatever.

Thany you for this amazingly informative contribution.

Re: Changing the Rating Equation

Anonymous — Mon, 20 May 2013 21:00:35 GMT

Author: Anonymous

Posted: 05/21/2013 01:00

Defago wrote:

When I claim the charts are similar it's not out of my personal opinion. The charts are objectively similar.

73% of the Top 100 Ranked albums are present on the Top 100 Rated albums.
73% of the Top 100 Rated albums are present on the Top 100 Ranked albums.
There's a 73% similarity between the both charts, objectively.

This means there's only 27 albums which are not common to both charts. Out of the 27 albums on the Top 100 Rated but not on the Top 100 Ranked, 17 are present on the Top 200 Ranked. Only 10 albums on the Top 100 Rated are NOT present on the Top 200 Ranked. These include Good Kid, M.A.A.D City, Trouble Will Find Me, Songs Of Leonard Cohen, In A Silent Way, Lonerism, Red, Spirit Of Eden, Five Leaves Left, Laughing Stock and Cosmo's Factory.

The value of correlation (a number between 0 and 1 which measures how similar the order between the charts is) is +0.69 between both Top 10, +0.63 between both Top 25, +0.6 between both Top 50 and +0.38 between both Top 100. This last value is lower because of Trouble Will Find Me by The National, which has been rated highly but is not ranked highly (the implementation of my suggestion will increase the number of Trouble Will Find Mes on the rated chart and reduce the number of classic, known-by-everyone albums). To give you a clearer view, a correlation value of +0.4 or more indicates a strong positive relationship. This is to be interpreted as that there is a strong similarity in the ordering of both charts.

So, yeah. This isn't me thinking they're kinda similar. This is me affirming they're 73% identical and that their orderings are related with a correlation factor of 0.38 at the median.

This is why you pay attention in math class kids. It comes in handy

Re: Changing the Rating Equation

Defago — Mon, 20 May 2013 20:42:28 GMT

Author: Defago

Posted: 05/21/2013 00:42

When I claim the charts are similar it's not out of my personal opinion. The charts are objectively similar.

73% of the Top 100 Ranked albums are present on the Top 100 Rated albums.
73% of the Top 100 Rated albums are present on the Top 100 Ranked albums.
There's a 73% similarity between the both charts, objectively.

This means there's only 27 albums which are not common to both charts. Out of the 27 albums on the Top 100 Rated but not on the Top 100 Ranked, 17 are present on the Top 200 Ranked. Only 10 albums on the Top 100 Rated are NOT present on the Top 200 Ranked. These include Good Kid, M.A.A.D City, Trouble Will Find Me, Songs Of Leonard Cohen, In A Silent Way, Lonerism, Red, Spirit Of Eden, Five Leaves Left, Laughing Stock and Cosmo's Factory.

The value of correlation (a number between 0 and 1 which measures how similar the order between the charts is) is +0.69 between both Top 10, +0.63 between both Top 25, +0.6 between both Top 50 and +0.38 between both Top 100. This last value is lower because of Trouble Will Find Me by The National, which has been rated highly but is not ranked highly (the implementation of my suggestion will increase the number of Trouble Will Find Mes on the rated chart and reduce the number of classic, known-by-everyone albums). To give you a clearer view, a correlation value of +0.4 or more indicates a strong positive relationship. This is to be interpreted as that there is a strong similarity in the ordering of both charts.

So, yeah. This isn't me thinking they're kinda similar. This is me affirming they're 73% identical and that their orderings are related with a correlation factor of 0.6 at the median.

Re: Changing the Rating Equation

Jabapac — Mon, 20 May 2013 18:40:57 GMT

Author: Jabapac

Posted: 05/20/2013 22:40

Defago wrote:

The Rated and Ranked are almost identical at this moment. If we do reduce m to 5, then we'll have two charts with different priorities, which would make two distinct charts (one more volatile and the other one is the one we know), whereas now we have 2 charts with one same priority, making them identical.

How identical?
Hybrid Theory by Linkin Park is rated only 63/100 and has the rank of 392. which is relatively high!.. and there are many more examples of that tendency.

Defago wrote:

That's all good, that's why we have a Ranked chart.

Not really, many users here think the ranked charts are more accurate, but maybe many also think the current rated charts are more accurate. as I see it, the ranked charts are not absolute accurate to get enough to. because the ranking system ignores the dislikes, which the ratings allow them by low rating.

Re: Changing the Rating Equation

GeevyDallas — Mon, 20 May 2013 08:20:34 GMT

Author: GeevyDallas

Posted: 05/20/2013 12:20

This needs to happen

Re: Changing the Rating Equation

HigherThanTheSun — Mon, 20 May 2013 07:38:44 GMT

Author: HigherThanTheSun

Posted: 05/20/2013 11:38

Yeah I like this idea. It relies on us checking the rated chart regularly and rating the top albums in case they've just got a small amount of really good ratings but that's cool I reckon, people would do it.

If it doesn't work can always be changed back anyway.

Re: Changing the Rating Equation

Defago — Mon, 20 May 2013 00:58:24 GMT

Author: Defago

Posted: 05/20/2013 04:58

JMan wrote:

I'm fine with the rating equation the way it is.

I beg you to expand your opinion.

Even if the changes were not to your liking, it wouldn't affect you chart-wise, because the Top-Rated charts already include a default filter which shows only items with 20 or more ratings. If you want things to remain the same on the chart, just don't change that filter. If this change is implemented and you'd like to view the "new" top rated chart, all you'd have to do is lower the filter to any value between 0 and 5.

Re: Changing the Rating Equation

Anonymous — Mon, 20 May 2013 00:54:00 GMT

Author: Anonymous

Posted: 05/20/2013 04:54

I'm fine with the rating equation the way it is.

Re: Changing the Rating Equation

Anonymous — Sun, 19 May 2013 23:16:15 GMT

Author: Anonymous

Posted: 05/20/2013 03:16

I agree. As it is, I have no incentive to look over the top-rated albums list because pretty much everything there is pretty highly ranked, anyway.

Re: Changing the Rating Equation

Anonymous — Sun, 19 May 2013 22:44:35 GMT

Author: Anonymous

Posted: 05/20/2013 02:44

I didn't even notice this suggestion. Interesting I guess. Wouldn't affect me personally either way so whichever, whatever.

Re: Changing the Rating Equation

Defago — Sun, 19 May 2013 22:32:53 GMT

Author: Defago

Posted: 05/20/2013 02:32

Jabapac wrote:

I could agree to get the m back to 10. but reduce the effect of amount of ratings too much is wrong I think. it's not about popularity over equality, but equality depends on popularity partly. they are not totally separated, because the more popular the more criticized.

That's all good, that's why we have a Ranked chart.

The Rated and Ranked are almost identical at this moment. If we do reduce m to 5, then we'll have two charts with different priorities, which would make two distinct charts (one more volatile and the other one is the one we know), whereas now we have 2 charts with one same priority, making them identical.

Re: Changing the Rating Equation

MrFrogger — Sun, 19 May 2013 22:10:35 GMT

Author: MrFrogger

Posted: 05/20/2013 02:10

This is a good idea

Re: Changing the Rating Equation

Jabapac — Sun, 19 May 2013 21:49:24 GMT

Author: Jabapac

Posted: 05/20/2013 01:49

I could agree to get the m back to 10. but reduce the effect of amount of ratings too much is wrong I think. it's not about popularity over equality, but equality depends on popularity partly. they are not totally separated, because the more popular the more criticized.

Changing the Rating Equation

Defago — Fri, 17 May 2013 16:53:00 GMT

Author: Defago

Posted: 05/17/2013 20:53

Currently, the equation to determine an item's rating uses a Bayesian equation to avoid items with a low amount of ratings to crowd the top of the charts. Without this measure, an album with one single rating of 100 would be the most rated album on the whole site, which would make the Best Rated Albums chart useless.

The actual equation is as follows (taken from BEA's How Does It Work page)

Average Rating = (n ÷ (n + m)) × av + (m ÷ (n + m)) × AV

Which is more easily read as

Average Rating = (n × av + m × AV) / (n + m)

where
n is the amount of ratings received
m is a fixed value, currently 20
av is the average of all ratings received
AV is the average of all ratings on the site, which I'm estimating to be around 75

As mentioned before, the idea behind doing things this way instead of a regular, vanilla average is to avoid items jumping to the top with a single rating, because our top rated charts would be filled with unknown stuff which has gotten one single rating.

I'd like, however, to suggest reducing the "m" constant. It used to be fixed at 10, and then it was raised to 20 recently. Having such a high "m" value is negative, in my opinion. The core effect of this is that items with fewer ratings, regardless of the actual ratings, get a lower average. An item with ten ratings will have a maximum possible average of 83.

On one side, I understand the need to use the Bayesian average to avoid inaccuracies. However, I believe a reduced number such as 5 would do the trick, while also providing incentives for people to listen to new albums.

Example on both situations

m=5

An album gets five 100 ratings by members of the site, getting an average of 87.5 and climbing to the top of its year's Rated Chart. As it's on a higher spot and easier to see, people will listen to the album. Then they will either agree with the previous ratings, in which case the album is deserving of it's position, or they will disagree, in which case a couple of average ratings would drop it back to an intermediate position on that year's rated charts.

m=20

An album gets five 100 ratings. Its average is 80, which is barely enough for it to be noticed. Few people check it out, and a potentially good album is missed by the regulars.

I understand this would make the charts more volatile, but I don't see a problem with that. We already have a Ranked chart which measures popularity, it's unnecessary to have the Rated chart measure popularity as well. In my opinion, the Ranked chart should stay as it is, combining amount of charts with position, which takes into account both popularity and how much people like it; the Rated chart should try to leave popularity as much out as possible.

Also, a value of m of 5 wouldn't enable albums to rise to the top with a single rating - it's VERY hard to find an album with five 100's. If an album does get five 100's and rises to the top, then it's worth checking out at least. The people who check it and listen to it will decide whether it stays high or sinks low, in case the five ratings were not in accordance to what most raters think about the item in question.

tl;dr: Reduce the amount of invisible AV ratings. 20 is too high. Since we already have a chart measuring popularity, we should have the other chart exclude the "popularity" factor and concentrate on the "quality" factor.

All of this was done assuming an AV of 75, which while it might be wrong, the argument stands for any possible value it might take.