Answering Your Questions on the Most Acclaimed Games Project

Last week, I shared a friend’s attempt to calculate the top 100 most acclaimed video games of all time. Here’s that article and accompanying list. At first people treated it as little more than a curiosity; they perused it for their own personal favourites, celebrated inclusions, lamented omissions, but ultimately moved on.

However, later on, a few wanted to know more: how it was calculated, what lists were included, what games were eligible, why Metroid: Other M wasn’t in the top 10 (OK, maybe not that one). So in order to respond to these questions—and also just to satisfy our own desire to speak in more depth about the project—James and I sat down for an informal interview.

Kris: May as well jump right into the juicy questions: How have you actually calculated all of this?

James: Yeah that seems to be by far the most common question I get. As soon as it dawns on people that the types of data that I’m using don’t exactly combine easily, they want to know how I’ve solved that. And the first thing that I want to say is that I take no credit for this method. Part of it came from pieces of information that I gleaned from Acclaimed Music, another big part came from a friend of mine who is an expert in statistics (thanks Ian!), and all I really did was fill in the gaps.

How it works though… I won’t reveal everything because I’m still sorting out the details, but essentially it’s a two step process: the first step involves sorting through all the lists and recording every instance of one game being ahead of another. This is how I make all time lists and end of year lists comparable to each other: in each you can only get a win against games that were eligible for the list. The lists can be weighted during this step if there’s a justification for it but weighting isn’t necessary; they’re weighted organically by what games were eligible for them.

The next step involves turning all this into a definitive ranking. I won’t share exactly how that’s done because I got a lot of help with this part and so the method really isn’t mine to share. Also not entirely sure that I’ll be sticking with it due to how computationally intensive it is. It’s very clever though, and essentially what it spits out is a number representing not only the number of “wins” vs the number of “losses”, but also the strength of the games those wins and losses were against.

Kris: Yeah, and I think that’s something that wasn’t really communicated that well in the initial article. A lot of people just treated it as a curiosity, not realising just how rigorous the process was.

James: Agreed. A few of the people who I showed it to assumed that I’d just assigned some kind of arbitrary weighting to each of the lists—and then another arbitrary rating to each position on the list—and then just tallied up the points or something. But if it were that arbitrary then I don’t think I would’ve been motivated to do it. I really wanted this to be better than Metacritic, and for that I needed a method that was a little more scientific.

Kris: Did you want to go more into the problems with Metacritic?

James: Most of it was covered in the article, wasn’t it? I’m not sure now. But basically the main problems are that they don’t take lists into account, and they assume that scores on different websites are comparable to each other. That’s it in a nutshell. I know I’m pretty harsh on Metacritic when we’re speaking privately, but honestly as long as you understand its limitations it’s probably still a somewhat useful website. It’s just annoying when people treat their top games list as the definitive best games of all time, because that’s obviously not the case once you start looking at lists as well.

Kris: There was something you said before about lists being inherently better than scores. Want to explain that?

James: Oh yeah, right. Well, the problem is that you always need some kind of reference point for scores. For example if a website gives a game 80%, it’s not immediately clear what that’s actually supposed to mean. 80% sounds like a pretty good score, but is it? Ultimately it depends on how lenient the specific website is, and that’s going to vary.

With lists there’s never any confusion because the number itself tells you everything you need to know. If I say something is the 130th best game of all time, then it’s immediately obvious what that means: 129 games are “better” than it, the rest are “worse”. And yeah, I guess that’s another problem with Metacritic: it uses scores.

Kris: Makes sense. I’m generally quite vocal about my disdain for numbered scores in reviews, so I only ever do it if the publication I’m writing for requires it. Which is, well, most of them. Even with a score guide for a website, too many people will just focus on the number without any of the context. There’s plenty who judge not just the game but the review and the reviewer based solely on the number, sometimes without even reading the full thing. Lists and rankings are much more useful when it comes to quick information, as you say.

James: Just to be clear though, lists definitely have their problems as well, with the main one being that you have to know what was eligible for them. And that’s why I’m only accepting lists from major publications at the moment. I need to be able to assume that all the main contenders at least had a chance of making the list, and for that to be true, someone on staff needs to have actually played them. I can safely assume that for a website like IGN, but for publications that don’t focus as much on video games, it becomes a lot less guaranteed.

That’s probably the biggest challenge with this method that I’m using, actually: determining eligibility. The way I’m handling it now is decent, but it could definitely be better. In terms of how to present data though? Yeah, lists are definitely the way to go. I’ll only ever be publishing lists for this project.

Kris: Something else that people were curious about is if they could actually see what lists a game was mentioned on, and use that to get some insight into why the game is ranked where it is.

James: Unfortunately it’s not practical at the moment. There are more then 300 lists now and there’s just no way to display all that information conveniently. One day I hope to have this up on its own website, which would allow me to have a separate page for each game where people could view all the lists that it was mentioned on, but that’s still a long way off. The other thing to consider is that a game’s place on a list doesn’t actually tell you much until you know what kind of list it was.

For example, there’s a big difference between getting first on an unranked end of year list, and first on a ranked all time list, with the latter obviously being a lot better. So yeah, basically, putting all of that information up would just make the list an unreadable mess. That said, if people are curious about a specific game then I’d be pretty happy to just look it up for them. It only takes me a few clicks, so it’s no trouble.

Kris: And I can pass on any requests to you that I get from people I know. So feel free to sling comments at either of us.

James: Exactly. Honestly though, it’s really not worth obsessing over the rankings at this stage. There’s still a ton of data entry that I need to do, so the rankings will be bouncing all over the place. The list has already changed pretty significantly, and it’s only been a week or so since we published it.

Kris: Actually that’s something we could talk about… Should we reveal some of the changes in the rankings since the list was published?

James: I think I’d rather just publish another version of the list, and I’m not ready to do that yet. Plan is to get through all the end of year lists back to the year 2000, and then publish a big top 250 or something like that. Not sure how long that will take, but that’s the plan. Something that I could do right now though is a top 100 just for the last decade. Because the lists for 2010-2019 are basically all done. Any interest in that?

Kris: There might be, I’m not sure. Come to think of it, no one commented on the lack of recent games in the top 100.

James: One of my friends did. He asked if Disco Elysium was eligible since it was released so close to the end of the decade. And it was, but unfortunately it placed just outside the top 100. 110th or something, I think. The actual cut off is the end of 2019, by the way. None of the 2020 end of year lists have been entered yet.

Kris: I think that about wraps up all the main questions. Did you want to add any general comments about the list? Why you think it’s useful, whether you’re happy with it and so on?

James: It’s hard to answer these questions because I’m honestly still undecided about a lot of it. And at the end of the day it doesn’t even matter what I think because people will make up their own minds. All I ask is that they don’t assume they know what I think about it, because that’s very unlikely. Before making it I joked that there were two possible outcomes: it would either be the ultimate resource for people looking for what games to play next, or it would be definitive proof that games media sucks and the critics have no idea what they’re talking about. And looking at it now, I think it’s actually a bit of both.

Most of the rankings are probably deserved, but some… The list is going to change though, so is it even worth trying to speak definitively about it now? It probably isn’t. The main thing to keep in mind is that it will only ever be a reflection of games media as a whole. So if the community doesn’t find the list useful or does think it’s a “good” list, then that points to a broader problem with the industry. It’s a list of the most acclaimed games, not necessarily a list of the best games.

Kris: And of course you’re talking about the Western games media here, since those are the only eligible lists at the moment.

James: Yeah, that’s a problem that I’ll need to fix at some stage. The list is heavily biased towards English lists, and especially North American lists, just because that was the most convenient starting point for me. English is the only language I speak, so I’ll need help when it comes time to add non-English lists. And I’ll also have to make some adjustments to account for staggered release dates, especially for Japanese games. It will be a lot of work, but I would like to do it eventually. For now though, yes, the list is heavily biased.

Kris: That’s all I had, so last chance for general comments?

James: Nothing, I just hope people liked it and they’re looking forward to the next update.

Kris: Which will be a top 250, right?

James: Either that or top 100 of the last decade.

Kris: Either way there’ll hopefully be some interest in it. That’s all the questions though, so thanks for the interview.

And thank you for reading this far. As ever, more feedback, commentary, and sharing of the project is extremely welcome (even if you’re just curious where Morrowind landed, like I was). There is still more to come, but hopefully this should give further insight. Until then!