The problem with Steam recent reviews score
Right when the Summer sale started on Steam, this is what you could see on the page of March of the Living:
Just 3 days later, here’s what it looks like:
I’d like to be able to explain to you precisely what happened here but the fact is we don’t know exactly how this recent reviews score is handled. My best guess is that some on these “recent” reviews are now more than a month old but hey look! We still have an overall “very positive” score so what was the point of that “very negative” recent score to start with?…
While I’m glad the “very negative” score is gone, this whole recent reviews score has some problems with it.
What value does a recent reviews score have?
Games in Early Access are constantly evolving and it’s quite possible a game who entered Early Access a year ago is quite different today. It might become better or maybe some controversial decisions were made and people are not enjoying it as much anymore. Having a sense of “where” the game really is today might be useful here as it should in theory give a better idea of the real state of the game at the moment.
For example maybe the game entered Early Access with a ton of bugs but several months later new players just don’t see these problems as they have been fixed. In this case, the recent reviews should show the game in a more positive and more accurate way.
Sure, people can change their review and switch a negative score into a positive one but I doubt many people actually take the time to go back to each games to check if their initial review still match the current state of the game (some do but I’d be surprised if it was anything more than a small minority).
For games not in Early Access though I’m not sure I see how this can be helpful. Since the release of March of the Living for example we only added more options (difficulty setting, fast forward, etc.) that didn’t change how you play the game if you were satisfied with it and we fixed bugs. We didn’t changed core systems and didn’t implement a controversial rule. What does a completely opposite recent reviews score tells then?…
Emphasis on a tiny minority of reviews
It took only 13 reviews to get MotL a “very negative” score over 363 reviews that gave it a “very positive” one. How are the last 13 reviews a better representation of what the game is? While MotL is a success, it’s not a massive hit so new reviews are coming in slowly (even very popular games only ever get a tiny fraction of reviews compared to sales they make) so if it happens the last few persons didn’t like the game then you’re stuck with a completely different score than the global one you have even if the game is exactly the same as before.
Devs have pretty much figured that a game need to have a minimum number of reviews to start getting a recent score. Is 13 really a decent minimum though? It seems to put the bar really low for something that is displayed even before the price (some people with smaller screen resolution don’t even see the price unless they scroll so the recent review score is the first thing they see, specially if it appears in red above the unnoticeable good overall score in blue right under).
Forget about customer service when they can just ask for a refund
Since there weren’t a lot of recent negative reviews I answered them all to see if I could do something to improve the game or help players in any way.
In one case the negative review was written because “the game crashes” in a way that was never reported before. No problem, I asked the player for the logs and if it’s really related to the game and not the PC of the player then I should be able to help him. Well sorry but a refund was already asked and I couldn’t get my hands on any logs but of course the cryptic “game crashes” negative review remains.
Now it’s okay because the recent score is gone and the overall review is still very positive but there’s no way to validate the cause was the game, the player got his money back and out of 13 reviews this review was used anyway to compile the recent score. I can stay up all night trying to offer good customer service but really what’s the point here if you can just dump a bad review and ask for refund.
Then there’s of course this kind of review:
Okay so you play 20 hours and post this kind of review, I see where you are coming from. Thankfully, only 1 person out of 15 found this review helpful but this review still counts in the 13 reviews that put a very visible “very negative” score at the top of the page. Shouldn’t unhelpful reviews be ignored from the score?
Looking at that person’s history of reviews you can see that it doesn’t mean anything (click to zoom):
Now DayZ won’t suffer from this specific review as it has over 2,000 recent reviews but when you only have 13… It’s easy for just a few people to screw with the recent reviews score. If the helpfulness of reviews was considered in the score then maybe it would be enough to “fix” this but it’s apparently not the case.
Even with a review like that the overall score remains “very positive” so really, what’s the point of putting the focus on that one for a different score?
How to improve such system
First, for a game that is not in Early Access, I’m having a really hard time seeing the value in this. Sure, we could still update the game anyway and upset players but it wasn’t the case here so using a time notion to calculate a different score doesn’t seem useful at all.
Maybe there could be a threshold of usefulness for reviews to be considered in the score. I could post a negative reviews that just says “iojiosfauead” and right now it appears it’d still be counted to calculate the score. This would easily solve the problem of someone posting “worst game ever” on every single games he reviewed.
How are “funny” reviews considered in this? I never got this feature. If a review is funny I’ll know by reading it. I don’t need a survey to know this. So are funny reviews compiled in the score?
Should there be a minimum time a player must play to post a review? It’s a tricky question as the question following this is “how much time?” but then Steam figured a time limit to request a refund so surely it could be done. If you’re having technical difficulties launching the game you can just ask for a refund so no harm was done. If you just don’t enjoy the game and play for 10 minutes and leave a negative review based on this experience… well, okay, technically you “played” but… eh. Of course people could just leave a game running a few hours without running just to be able to post a review but I think this would be fairly rare. Only a few dedicated persons would bother with such scheme.
Why be so concerned about that
Well, first I care about the game I created. I’m proud of it so when “very negative” is the first thing people see when looking at it and this judgment comes from only 13 reviews I can’t help but be concerned, specially when just under that it says “very positive”.
This is also my salary. This is how I pay for food each day so it’s hard for me to ignore. Maybe it’s just a coincidence but the sales of the game went up at the same time the recent very negative reviews score disappeared. If only ten persons changed their mind about buying the game because they saw a “very negative” score then it has a big impact on me. When you sell thousands of copies during a seasonal sale on Steam then a few losses don’t make a big difference but for a game like MotL it’s a big deal. Every single sale counts.
So I can’t help but pay attention to this stuff. If 500 reviews would give the game a very negative score then I’d have to accept the game doesn’t please people. When only 13 reviews can have that impact while over 300 reviews gave the game a very positive score… well, as hard as I might try I can’t ignore this.
I can deal just fine with one of my game rated badly. It is what it is. This game just doesn’t please people. I do get concerned however when a store puts a game well received in a very contradicting light based on only 13 reviews.