Is it me, or is the Internet all up in arms about the whole “don’t look at the scores when reading a review” situation? It seems to be the big topic of debate recently.
 |
First, IGN had a podcast where they talked about the games that they felt they had overrated, going in-depth with the idea of the scores themselves and what it means in a review. Then, Destructoid ran a series of articles that detailed why reviews are all out of whack, solely based on the scores. And because those same articles got onto digg, the whole community has something to say, even the big sites. You can see this in the Godfather Wii review on IGN. Scroll down to Matt Casamissina’s comments and you can see the evidence.
I won’t lie though. I have been known to skim through the review and go straight to the score. I usually only do this when reading about a game I don’t plan on buying though. If there’s a game I am interested in, I’ll probably read the entire thing. So it’s not like I’m a complete douchebag.
But it is a little disheartening to know that most people probably don’t actually read the text part of the review, as that is the most important part. The reviewer puts far more time into writing the review than assigning a number grade.
But what about you guys/gals? Are you guilty of the “quick glance”, or are you actually respectful to your fellow journalists?
I typically read the reviews for games I’m at all interested in (unless I’m already planning on buying it and am paranoid about spoilers), but scores are a great way to find out about games you might otherwise not have bothered looking at. As a recent example, reading the title of "Puzzle Quest" and glancing quickly at a couple of screenshots as I’m scrolling past, the game doesn’t at all arouse my interest- after all, I’m at Gamespot to read the Armored Core 4 review. But then I notice that, hey, it got an 8.5. So I read the review and find out that this is pretty much a game designed specifically for me to fall in love with. Without the score to catch my eye, I would have completely overlooked it. This has happened more than a few times and it highlights that scores really can be usefull.
The key thing regarding scores for me is they give me a quick, standard way of normalization and comparison across reviews and reviewers. (which sites like metacritic eventually came around and automated). Let’s say I find a review of a game I might like, but it’s from a publication I’ve never before read. I can very quickly get a feel for how they treat games overall, what genres they tend to score higher, and how this review compares to their average reaction to a game. This allows me to select other reviews of the same game, picking from the most positive ones to see what I might like, the most negative ones to see what the drawbacks of the game are, and somewhere around the mean, to see where the balance between the two lies. Admittedly, the numbers are a drastic oversimplification, but when used properly, they give me the context I need to correctly interpret all of the detailed
reviewer comments regarding their subjective play experiences, and help me to find the reviews which will most closely predict my own experience.
Stefan’s point is about how I feel – I get a feel for how games are generally rated on a site and then I compare them to the score of whatever review I’m looking at. If it’s a genre I’m not particularly interested in, I’m content to look at the score, but then I’ve seen plenty of reviews where the score given to a game was below 3/5 (or 7/10) but the review was actually pretty positive because the review score took graphics or other non-gameplay elements into account when they’re not as important to the game.
I dislike numbers but understand people (including myself) are lazy and don’t always want to read a whole review. The solution videolamer
has come up with is to ignore the problem, remain on our pedestal by not giving number scores, and ensure the site is not accessible.
I almost blogged about the Destructoid articles. They are so wrong. Well, they are right if you assume number reviews are going no where, but
it’s still just a band aid. Read more of my preaching here – http://videolamer.com/index.php/734
Yeah, I commented on those articles, saying there probably shouldn’t be any numbers to begin with. I don’t believe in them, solely because they can change so easily. As someone stated, an average can mean a rather unimportant aspect of the game was really bad, but the game was still really fun to play. So what’s the point, and why put so much stock in something like that?
And yes, I know that makes me sound like a hypocrite, where I look at review scores sometimes but don’t believe in them. I get to be a hypocrite every once in awhile.
I just don’t like that Desctructiod article because the writer doesn’t know basic properties of mathematics
That same guy gave a 4 to Twilight Princess. A FOUR! It’s not that great, but it’s not that bad either. And it seems he used that grade for the basis of his article (which I’m ashamed I only skimmed though), where he tried to prove why he graded it like that, while every other reviewer/reader follows completely different rules.
I thought he was all for using the full range of numbers, meaning a 5 is average in his book. What games is he playing that make TP look below average and can I play them too?
That’s actually a really interesting idea…I wonder what would happen if a gaming site enforced a normal distribution in their review scores. This would give a definite meaning to an "above average" score, and a "below average score". It would mean that a game which was above average was actually considered to be better than the average game they reviewed, rather than simply being higher than the midway point on an arbitrary universal scale of how good video games are. The really interesting part is also that the old scores would change over time as new video games were reviewed….which is consistant with the idea of presenting a score as relative to other existing games, instead of the current pretense that a score is on a fixed, universal scale, but also takes into account the graphics as compared to other games of today. Assuming consistant scoring criteria were applied (which can never be perfectly done, but you can approach it), then a score of 4/10 would guarantee that there were a lot of better games.
Guys, he said that 5 is the average of a 1 to 10 scale. That’s making me cry.
Yeah, I don’t agree with that scoring system at all. It might improve certain aspects of the review system, but it just seems obscure to me. If Zelda is a 4, then whats the difference between a 3 and 4 game? It seems that if you push the grades down like that to have more room at the top for really good games (which Zelda is, btw), then you clump the ones on the bottom more closely. And it then goes back to not even having scores in the first place.
Okay, I might was well explain what I meant. The average of a set of numbers is obvious to compute, and if you add up 1 to 10, you’ll find an average of 5.5. Close, but not actually 5. What the Destructoid author is describing is the median, which 5 certainly is. But even with that out of the way, he’s still not using them right. 5 is the median when dealing with the set up numbers 1 through 10. Is there any worth in computing these values for the numbers you are using, or is it better to calculate based on a set of scores you’ve given? In the case of the latter, your average and median are going to be very different. As you can see, numbers are dangerous in the reviewing world because right now they’re only used as symbols; they could easily be replaced with letter grades or stars. When you start doing serious mathematical analysis with them, it gets tricky, becuse no one is really using them that way. That is why I believe in a four or five star system. If you take away the math, these systems allow a writer to easily break a game down into obvious categories of crap or all time classic, while the middle scores are left for things that are average (not in the mathematical sense) or great but not classic.
For the record I understood why you were complaining about 5 not being the average of 1 and 10. I like the 4 star scale, good idea. One is generally crappy, zero is shit. Two means you should look into it if the game is particularly up your alley, and three and four should be self explanatory.
Beyond averaging {1,2,3,4,5,6,7,8,9,10}, the average of {1,10} is still 5.5. This is also sort of obvious, but it means that whether you think of the scale as a set of all possible scores or as two numbers defining the outer boundaries using 5 as the average is still wrong. (Unless your scale is 0-10)