Dev Diary posts are made to teach game development through specific examples from my latest project: Highways & Byways. Just here for Highways & Byways updates? Click here.

---

Play-testing is so critical in board game development that I’ve dedicated several articles to the subject. Yet I’ve never seen article that deals with one of the biggest issues with play-testing: being able to tell the difference between signal and noise. You can’t believe everything your play-testers tell you, even though a lot of game developers will give you a coy response if you say that directly.

First, let’s have a refresher on good play-testing practices. The most important rule is write down all the feedback you receive during play-testing. Do this even if you’ve heard the feedback before, even if you think it’s stupid, and even if you know the feedback is wrong. Play-testing is ultimately about testing the subjective experiences of people playing your game. Every opinion – however misinformed you may believe to be – is a data point. As in rigorous scientific experiments, data points are to be gathered accurately and then interpreted later. By treating play-testing with a scientific mindset, you won’t risk losing valuable feedback because you got your feelings hurt.

It’s also a good idea to have a clear objective when you start a play-test. Some objectives I’ve used for testing Highways & Byways are “make sure Byway Cards communicate the location of roads clearly” and “gather data on the balance of Event Cards.” If you’ve made a recent tweak, having objectives going in helps you gather relevant data. Choose something to pay extra close attention to, such as balance, communication, or accessibility. All this said, there are no hard and fast rules going into play-testing. That is why recording data is important – so you can dispassionately review what people say at a later time.

When it’s time to review play-testing results, here are some guidelines I follow…

3 Times Play-Testing Feedback is Probably Not Useful

The player clearly does not understand the game. At some point, no matter how simple your game, you’ll have someone who doesn’t read the rules. Or perhaps you’ll have someone who can’t pick up the game from playing. Or perhaps even you’ll have someone who understands the game perfectly in a vacuum, but cannot form a coherent strategy to save their life. If you’ve got 20 play-testers and 1 of those 20 suffers from one or more of these issues when no one else does, the feedback is likely addressing an issue with the player and not the game.

This can be caused not by necessarily having a “dumb player,” but simply by having a distracted player. If someone is tired, stressed, or otherwise emotional, it might be hard for them to pick up your game and recognize that they are having a hard time picking up your game. Sometimes people just don’t “take” to games for some reason unrelated to their intelligence or well-being. It’s like that with me and Agricola (but you keep that between us two – I’ll lose my game dev card if you let that secret out).

When people don’t understand the game, they can give you all sorts of negative or neutral feedback that seems nonsensical or left-field. You may be able to tweak the game to make it communicate more clearly, and you should always ask yourself if that is the case. Yet if you believe the player is truly at a loss for understanding, try running their feedback by some other play-testers. If the other play-testers say “this player does not understand the game,” then it’s probably okay to disregard their feedback.

The player is providing feedback related to the tool you’re testing with, but the game itself. Whether you’re using a physical prototype with pennies for tokens or Tabletop Simulator, play-test versions of games often don’t look pretty or feel quite like the final product would. If you know that you’ll be changing the game to have better components, don’t worry about comments on your bad components. If you will be passing hands of cards around the table in real life – don’t be upset when people say “it’s hard to pass hands in Tabletop Simulator.”

Important caveat: always play-test anything that goes into the final version of your game.

The player is wildly pitching ideas. In general, if your game is on the right track, I find that you’ll get far more comments than questions. If you get a play-tester who has all sorts of ideas that don’t match up with the direction you’re taking the game in, that might be a sign of three things. One, they could be legitimately good ideas which you should consider. Two, they might not understand the game – see the previous point. Three, they might be pushing their creative instincts and desires on to your project. If that last one is the case, that’s got more to do with them than you. As always, I suggest you run wild ideas by other play-testers if you’re not sure.

3 Times Play-Testing Feedback is Definitely Useful

The feedback is regarding an issue that is both tangible and objective. If a player says “you’ve got a typo” or “this card could resolve in an undesirable way, watch me do it,” you must pay attention. When you get specific feedback about issues that are clear-cut, that’s as useful as it gets. Thank them and fix the issue next time you make a version. You don’t need multiple people to confirm these sorts of issues.

Multiple people have independently said the same thing. When it comes to matters like balance or fun, it’s really hard to know what is best. There is no clear answer like the ones for typos or loopholes. When multiple people say “I feel like this game isn’t balanced so well,” it doesn’t matter if your game is balanced perfectly in an Excel spreadsheet according to infallible mathematics. When a good portion of your play-testers feel like something’s wrong with your game, then something is probably wrong with your game. In fact, “majority rule” is one of the best ways to gauge the quality of your game when it comes to matters of taste.

Feedback is associated with actions that confirm the feedback. Imagine a player spends a minute or two organizing their hand, slowing down the game, and they say “you know, these cards are awfully fiddly.” It might be a problem if they took a minute or two without saying something. It might be a problem if they complained it was fiddly but only took a few seconds. Yet if both are happening at the same time, then something is up. Likewise, if a player says, “I don’t know what to do here,” and proceeds to make an absolutely bonkers strategic error, then your game may need clarification in some areas.

Despite my scientific rigor in recording feedback, there is a reason I refer to play-testing as The Art of the Play-Test in a prior article. The guidelines above are made to help you determine when play-testing feedback is useful and when it is not. Yet I can offer no certainty, no absolutes, and no rubrics. The decisions you make here are where game development becomes an art form – a matter of taste, judgement, and care.

---

Most Important Highways & Byways Updates

• Chugging right along in play-testing. I need to make some minor updates to balance and polish the game a bit, but everybody who’s tested it so far has liked it. That’s rare and I’m thankful.
• James has delivered some card art templates. It’s nothing flashy and it’s nothing that shares particularly well. Despite this, please understand that this is the basis of our workflow from here on out, making it good progress.