I have heard the idea of using peer review (other developers reviewing your game) floated in a number of places as a way of deciding what games should/shouldn't be let onto the OUYA store.
On the surface, it seems like a pretty good idea. However, I just came across this article which serves as a warning for how it could also go horribly wrong, as was the case for XBLIG:
Clearly some of those issues are specific to Microsoft (such as the legal stuff), but I'm interested to hear what other people think on the idea
On the surface, it seems like a pretty good idea. However, I just came across this article which serves as a warning for how it could also go horribly wrong, as was the case for XBLIG:
Thomas Steinke said:The peer review system just plain sucked. Most people may not know how games actually made it to the system. Microsoft, to avoid obvious legal problems, devised the following system: You would submit a game or update to a game on the system and it would go in a review list. At this point other developers on the system (yes, your direct competition) would review your game (yes, before it is released to the public or anyone has seen it). At this time the reviewer would either submit a pass or fail vote on the game. If the game received 7 passes it would appear on the system, if it received 2 fails for any reason, it would be booted out of the system. Whether your game passed or failed, a 7 day wait period was imposed on the title preventing you from resubmitting or updating the game in this period. If you resubmitted the game without fixing the issue you risked being permanently banned from the entire system. After 30 days if your title did not clear the system (for example people weren’t interested in it enough to vote on it at all), you failed automatically and needed to restart the process. If you needed to release a Multilanguage title, in addition you need 2 reviews for each language you want to release in (this made releasing or even updating localized titles nearly impossible).
Aside from the obvious problems that arise from letting your competition see early releases of your titles and ultimately deciding if your game should make it to the system, Microsoft intentionally left the criteria for review vague with rules like “Your game cannot be frustrating or confusing”. With rules like that you could have fun imagining the creative ways that one could judge a title unfit to release. Each fail costing the developer another week plus the time it takes to accumulate the votes again, potentially holding titles in limbo for months.
It would seem that this would quickly degrade into the chaos of developers maliciously failing each other to prevent them from competing with them, however in practice, as if in some sort of odd social experiment, the opposite actually occurred. Everyone had a mutual goal of getting their title though this tedious process so it became common practice to do “kickback reviewing”, where one developer would “review”(pass) your game with the expectation that you would do the same. It became so prevalent that it actually degraded the quality of the titles on the system as a whole. You see it wasn’t really in your interest ever to fail a game; it just meant that that developer would come back and invent some reason to fail your title. For that reason, fails even legitimate ones became fairly rare. As a reviewer myself (an honest one I felt) there were many times when I elected just not to vote on a game instead of failing it.
Surely Microsoft would step in to enforce order. Due to legal reasons, no one at Microsoft was allowed to view any content on the system until it was officially released to the public. Not even the discussion that occurred in the game’s review forum.
Their solution to this was the moderators or MVPs. These MVPs ran essentially unchecked by Microsoft and had ultimate power over the system. They single handily had the ability to make it so your game would or wouldn’t go out. To be fair, these people had a VERY hard job and were not compensated at all for it. They were reigning in complete chaos and it always amazed me how devoted they were to the system. With that said, their “war-torn” demeanor at times became combative and rude. Some of them had very strong opinions that they were quick to bluntly express and they were the ultimate law on the system.
If there was a serious issue or debate about your game it could be deferred to Microsoft. However since Microsoft could not see your title directly or the discussion that transpired, it would become the moderator’s job to present your case to them and deliver the verdict. You could imagine if the moderator was not on your side this would never go well. Regardless, Microsoft’s de-facto response was always “You guys decide as a community” which in reality meant, “The moderator makes the decision”. If the moderator gave the community the license to pass or fail a game that is what happened. As a reviewer, if you disagreed and didn’t follow suit it usually resulted in some sort of disciplinary action. Any meaningful discussion about how the system could be improved or be changed was almost always immediately squelched.
Clearly some of those issues are specific to Microsoft (such as the legal stuff), but I'm interested to hear what other people think on the idea