2/10 this article blows chunks

“Quantitative rating systems”? What?

Whatever you’d want to call them. Numeric rating systems. You know, your ten out of ten scores, your five star scores, and your A-F grading scores as well. Well, technically, I’m specfiically referring to rating systems using the ratio scale type, and them being applied to mediums which they are not applicable to.

Scale types?

In measurement, there’s a number of different scales of values that different operations can be performed on. Today, we’ll be discussing the ratio, interval, and ordinal scale types.

We begin at the “ratio” scale type. All manner of operations can be used on it. Essentially, it’s just referring to integers, and the many operations you can perform on them. For example, distance is measured in the absolute scale type; you can have zero distance, you can have positive distance, negative distance; you can multiply distance, divide distance, add distance, substract distance, and you can compare the values distances of to see which quantity is lesser or greater.

Then, the next rung down is the interval scale type. It’s akin to the “ratio” scale type, but without a concept of zero, or nothing. A good example is temperature, say in celsius (well, “absolute zero” in terms of temperature refers to something else, but that’s another discussion). What is meant by that is that 0°C does not represent a lack of temperature; the particles are still moving, but it’s merely on this scale that zero still represents a quantity rather than nothing. It still has all the same operations, but with that additional caveat.

The following scale type is the ordinal scale type. This one has all the properties of the interval scale type, but with all operations that add or remove quantities gone. All you have left are comparisons; this scale type exists purely in the qualitative realm. For example, let’s say you have a list of all your favorite foods, and you wish to compare them by taste. Taste is not something you can measure by quantity; it is purely by quality. You can say you prefer a canteloupe to a watermelon, but you hate durians even more than watermelons. You cannot numerically add an interval between tastes, you only have this quality to compare the values between.

So what?

I bring this concept up in regards to the use of the ratio scale type to non-quantitative things. A ratio scale type is appropiate for something that can be determined quantitatively. For example, a math test. You have a set amount of questions, and you can either get them right or wrong. You can measurethe amount of question answered correctly through this scale, and measure different values using this ratio scale type to paint an image of which amounts are larger or smaller, or more correct and less correct.

This rating system has also been brought to the world of media. Films, books, video games, music, are all judged and graded as if they were a test, by reviewers and critics…but this misses the point entirely. These are media, not math tests, and their worth cannot be measured quantitatively.

Pokemon Omega Ruby and Alpha Sapphire earned a 7.8 for having “too much water”. But I ask, what does that score accurately mean? If I took a game that likewise had the same score, does that mean that they’re exactly, precisely the same quality? What about an 8.8 video game, like The Legend of Zelda: Twilight Princess, supposedly? Is Twlight Princess one “point” better than that other game? What does that even mean? How bad must a game be to score 0/10? There are numerous games with a 10/10 rating system. Are they totally perfect, without flaws? What if a 10/10 game is more fun than another 10/10 game? What does any of this mean?!

I now ask this question: why are we using a quantitative rating system to rate something that is purely qualitative?

Applying a ratio scale type score to something that is wholly qualitative is ultimately utterly incomprehensible. You cannot meaningfully rate anything with this. You can’t meaningfully rate anything with these scores.

I can understand the need for something quick and dirty. You want to gauge the quality of something, so you throw out a number, with some vague agreed societal notion that this number reflects a certain quality. But as time passes, as standards change, tightening or loosening, this system only becomes more and more nonsensical overtime.

Whenever I see something like a video game being graded or ranked with a number, I’m just left to think…what does this even mean? It hardly paints a meaningful depiction of that media’s quality, leaving me guessing on what that actually means.

The Right Scale Type to Use

I’ve complained enough about why a ratio scale type is like putting a square peg in a round hole; it’s simply not sufficient for accurately judging something.

Something that should be rated qualitatively should use a scale type that only has comparisons as its only metrics.

And the ordinal scale type is the one to use for these purposes!

So, throw away your five stars, your ten out of tens, your A-F grading scores, and put them firmly in the bin. It’s time to use an ordinal scale system!

“But!” you might be asking, “how am I to use this new ordinal scale type system to determine which American fast food franchise is the most delicious and the most devastating to my body?!”

Well…

Tier Lists

I am so grateful that tier lists have become more popular in recent ages. It’s as if this problem was silently acknowledged by normies and through an odd turn of events, people have actually been embracing a better solution for quite some time now.

Tier lists are an example of this ordinal scale type. The only operation that can be performed on tiers is comparison. A tier, or an item within said tier, is either higher, lower, or equal to its contemporaries. The only way to measure their values are through ordered comparisons.

And due to this, you can actually paint a proper picture of quality now! You are now unbounded by minimums or maximums, and can rank something as high as low as you please to something else. I can now say that Ace Attorney Investigations 2 is not a “10/10” game, but rather, it is better in comparison to Ace Attorney Trials & Tribulations, and Trials & Tribulations is better than Phoenix Wright: Ace Attorney.

Through this, you can now determine something’s quality by determing merely how it is in comparison to something else. And now there’s so much more clarity because of this! No longer do you need to call a bit of media five stars. You can just say it is better or worse than the media around it.

And you can still use this in the shorthand! You can say a game is an “S tier”, whereas another is a “C tier”, and due to this common rating system, people can perfectly understand what you mean. And you don’t need to limit yourself to those, either; you could say “SS tier” or “Z tier” and people would understand your message just as well. You can have as many arbitrary tiers as you need to rank something’s quality, removing that issue of having multiple “10/10” media.

Another Thing: The Separation of an Experience

I do have one more thing to mention, though.

Let’s say some reviewer plays a game, and chooses to rank its different attributes with different numbers. He’ll rank the game’s sound, its gameplay, its graphics, everything, all with different quantities.

…but what does this even mean?

A game is experienced in the hollistic. You do not listen to its soundtrack, and then indulge in its mechanics, or then be presented with a slideshow of its many textures, models, and shaders. You go through all of this at once. And thus, the experience should be graded as a WHOLE, not in bite-sized pieces.

The Sad Reality

I’ve been a part in some contests or two where rating was a two-hit catastrophe. First, the item being judged is not a math test, it is a qualitative thing whose value cannot have a number attributed to it. And then, second, the judges decided to completely forgo the basic idea of rating the thing as is, and not in nonsensical separations.

Not only that, they don’t even make things simple - they assign some arbitrary amount of points, with arbitrary maximums and minimums, distributing them in god knows which way - and everyone constantly complains about this system not making any sense, and demanding their own changes which do not actually address the fundamental problem of using a ratio scoring scale versus an ordinal one, nor acknowledging the reality of what they’re reviewing.

As proposed earlier, a tier list is a perfect measure for these things. You could even go a step further and order the things within their own tiers for an even clearer picture of quality.

Yes, it may take time to go through all items and compare each of them rather than assigning some arbitrary score. But it’ll be much less headache, I can guarantee you that, and you’ll have made a clearer picture of what each thing’s actual quality is.

Shameless Plagiarism

Much of this article is just reiterating what this video discusses, without all of that YouTuber cruft. I saw this video many years ago and it really just solidified and clarified an issue I’d wanted to verbalize for a very long time.

LambHoot suggests a slightly different ordinal system, assigning an arbitrary “game point” amount for each title he reviewed. The system is nice, but again, I think it still falls into that trap of having arbitrary numbers. It’s more digestible to say a game is “S tier” rather than “this game is worth 500 game points”. You can always make new tiers, or bump things around within their tiers, but let’s say you have a game worth 49 points and a game worth 50 points, and a new game comes around that quality, you’d have to make new numbers and new decimal places to wedge that game within. It’s too much nonsense! A tier list is a better solution, in my eyes.

Takeaway

Stop using “quantitative rating systems” (i.e., ratio scale type scoring systems) for non-quantitative things. Instead of five stars or ten out of ten, use a tier list (or some ordinal scale) to paint a better picture of each item’s quality.

Don’t judge things by their parts, judge them by the hollistic sum of their parts.