How Many Blind Play-Tests Does Your Board Game *Really* Need?

Posted on Leave a commentPosted in Dev Diary

Dev Diary posts are made to teach game development through specific examples from my latest project: Highways & BywaysJust here for Highways & Byways updates? Click here.

 


 

In my quest to test Highways & Byways 100 more times before I release it to reviewers, I’ve found myself with relatively little to say. That’s a good thing! After all, I’m checking to make sure there are no serious balance issues that come out of repeated plays. I’ve already play-tested this game a ton. People like it, it’s been blind play-tested for over two months now. I even blind play-tested it at a game design convention.

In absence of a Byways-related update to inspire this week’s Dev Diary, I’m answering a question I got from a new game designer a few months back. “How many blind play-tests does your board game really need?”

 

“I wish I could Photoshop as good as Brandon the Game Dev,” said no one.

 

Predictably, my response to that is “it depends.” Blind play-testing has a set of very specific purposes in board game development. You need it for five reasons:

  1. It confirms that the core engine and mechanics of your game make intuitive sense.
  2. It confirms that your rules are not unnecessarily difficult to pick up.
  3. It can help you identify accessibility issues before you get farther along in development.
  4. It puts your game in a situation where people are less likely to protect your feelings, since you’re not directly involved.
  5. It’s great marketing.

 

It makes an intuitive sort of sense that more blind play-testing is always good. I’m not convinced. Don’t get me wrong, I’ve had dozens of people blind play-test Highways & Byways, but there are diminishing returns once you hit a certain point. Let’s take this apart reason by reason.

 

It confirms that the core engine and mechanics of your game make intuitive sense.

If three or four different game groups all play your game and have no trouble understanding the strategy and tactics of the game, you’re probably good to go here. Granted, it might take you a long time to get to this point, but if your first three or four blind play-tests go well in this regard, you’re probably fine.

 

It confirms that your rules are not unnecessarily difficult to pick up.

Again, if the same three of four different game groups understand the rules just fine, you’re probably good to go here as well. That means no misunderstandings, no deer-in-headlights looks when reading the rules, and no major suggestions. They might find a typo or two, but that’s different.

 

It can help you identify accessibility issues before you get farther along in development.

If your game is blind play-tested by lots of different people, the most common visual, physical, a cognitive inaccessibilities should become clear to the perceptive game developer. If you don’t find any after a while, you’re probably good to go here as well.

 

It puts your game in a situation where people are less likely to protect your feelings, since you’re not directly involved.

If your game is loved or even merely liked with a few minor suggestions for changes, you’re in a good place with blind play-testing. If your game can get a consensus of 7/10 on feedback surveys with some consistent and easy-to-implement suggestions for improvement, you may not need additional blind play-testing.

 

It’s great marketing.

Of all the reasons to keep blind play-testing once you confirm your game is clear, accessible, and likable, this is the most compelling. People who play a game in its early stages are more likely to buy it later on. This is a simple fact. However, blind play-testing can be time-consuming and difficult to arrange. I would argue it is more effective to use your time and money generating email leads online through social media or going to conventions to mingle.

 

You don’t need hundreds of blind play-tests. You don’t need hundreds of blind play-testers. All you need is enough to confirm that your game is clear, accessible, and likable. That might take 10 games. It might take 200. There is no hard and fast answer here.

Every once in a while, a fresh project will try to wow everybody by posting astronomical numbers showing how much a game has been play-tested. I’m sure there is a benefit to blind play-testing on a massive scale, but it’s not so great that the new game developer working on a very tight budget with limited time should feel the need to pursue. My approach to blind play-testing is one based on lean business process design: figure out exactly what I need and how to get it efficiently.

To conclude, let’s flip these five reasons into a five-prong test you can use to tell if you’ve done enough blind play-testing.

 

Your Game Has Been Blind Play-Tested Enough If…

  1. Your game is intuitive.
  2. Your game has clear rules.
  3. Your game has no major accessibility issues.
  4. People like your game or you have consistent and easy-to-implement feedback that would resolve outstanding issues.
  5. You have played with enough people to spread the word OR you have a good marketing plan.

 


 

Most Important Highways & Byways Updates

  • With the physical prototype in my home right now, play-testing is going much, much, much faster.
  • I’m still aiming for 100 play-tests.

The Long Haul Project: Why Making Games Takes a Long Time

Posted on 2 CommentsPosted in Dev Diary

Dev Diary posts are made to teach game development through specific examples from my latest project: Highways & BywaysJust here for Highways & Byways updates? Click here.

 


 

In the frozen Canadian arctic of the Northwest Territories and Nunavut, there is a famous road known as the Tibbitt to Contwoyto Winter Road. It’s a long haul road for ice truckers that’s over 300 miles long and most of it is built on frozen lakes. Sometimes it’s hard to know which direction to go in, but by far, the hardest part is that there is a limit to how fast you can go – 16 miles per hour. When I heard about this, I said to myself sardonically “sounds like game development.”

 

This image was originally posted to Flickr by madmack66 and licensed under CC BY 2.0. I modified it from its original state.

 

Testing, manufacturing, shipping, and building an audience all take a lot of time and there is no way to really shortcut this. Whether you work 20 hours per week or 60 hours per week, you cannot meaningfully reduce the amount of time it takes to do certain tasks. Manufacturing and shipping can’t be changed at all by extra work. Testing can be expedited, but only as fast as you can afford to bring in fresh testers or make changes to the game. With building an audience, sooner or later, you will run into some kind of constraint…

You can spend money on ads, but only until you run out of money. You can talk to more people, but only until you run out of either people to talk to, energy to talk to people, or – in the case of things like Twitter or Instagram – you hit an API limit set by a web developer. You can bring up your game multiple times, but only until people get sick of hearing about it. You can keep talking, but only as long as you have things worth saying.

Game development…no, creativity itself, requires a mix of commitment and downtime. In fact, I’d hoped to speak about this on a post about mental health with a real licensed psychiatrist on this very blog. That post fell through after Hurricane Maria hit Puerto Rico and left my contact without power for months. (He’s physically okay but was unable to respond to questions.)

You can’t speed run this. A single project, even with a large team of creators and administrators will still need to go through several stages, listed below. Let’s imagine you have infinite staff and money at your disposal. Where will you hit a bottleneck?

  • Early Game Design: If you’ve got tons of game designers and you work in shifts where 2-3 designers are working on a game 24/7, this will take several days. You can’t have much more than 6-9 people working on game design or else communication will get really weird, really fast.
  • Artwork: Let’s say you create great art specs and break discrete parts of the game up between artists. This will still take a few weeks because each artist will have a contract, some will flake on you, and some will not meet the spec.
  • Game Tweaks: Let’s say you have an infinite supply of fresh play-testers. Yet every time you make a new tweak, you’ll have to iterate the game. Even with 24/7 shift work, it will take several days to get this right.
  • Sample Production: You’re going to have to have another place print your game. If you know exactly what you’re doing when you make the order and you have it rush shipped, this is a two week process.
  • Reviews: Once your game is ready for review, go ahead and wait six weeks for reviewers to get back to you. They’re busy and do their work with full-time jobs and kids most of the time. You can’t rush them because you need their trust to continue on.
  • Kickstarter: Okay, you don’t technically need Kickstarter, but a lot of people like using it to estimate demand and have some degree of certainty in success. Take two weeks for a short campaign.
  • Manufacturing: Let’s say you start the print run as soon as the Kickstarter ends and front the cash even while waiting on the Kickstarter check to clear two weeks after the campaign. It will take six weeks to print and two weeks to air ship it. (Most people can’t afford to air ship, and sea shipping takes up to three months).
  • Fulfillment: Let’s say you rush ship all your rewards from the warehouse – that’s a two week process.

Without factoring in sales and marketing, it is not even theoretically possible to create a board game and fulfill it in less than six months of time. This is assuming you have tons of staff and tons of money, which very few of us do.

What I’m driving at is simple: if you work alone or in a small team, be patient, pace yourself, and have fun. You can’t sprint a marathon. You’re not missing out on anything by not working yourself to exhaustion. There are structural limits to how fast you can move.

If you get bored along the way or feel like you’re not making enough money, here are some suggestions:

  1. Stagger your game projects so that you’re working on more than one at a time.
  2. Launch another project or start a different business entirely.
  3. Help other creators make something they’re proud of.
  4. Spend time with your friends and family.
  5. Take a vacation.
  6. Binge watch YouTube. Yes, seriously.

I don’t even think business geniuses like Bill Gates or Warren Buffet could build up massive empires from nothingness in a year or two. They could do it again and probably do it in less time than it took the first time, but if they were truly left with nothing but their knowledge – no contacts, no cash – it would take a while to build back up to where they are now.

Let’s put the myth of the obsessed genius creator in the ground. The most effective creator is the passionate one who puts in consistent effort regularly BUT who does not burn out or isolate themselves in the process. The effective creator takes in the world around them, reaches out to others, and generally lives a life outside of their work.

 

 


 

Most Important Highways & Byways Updates

  • I’m chugging along on the final 100 tests. I’m feeling good about Highways & Byways. I’m on track for the late March Kickstarter I’ve been aiming for.
  • It looks like I’ll be getting the prototype game in the mail today!

The Final 100 Play-Tests: How to Put Final Touches on a Board Game

Posted on 2 CommentsPosted in Dev Diary

Dev Diary posts are made to teach game development through specific examples from my latest project: Highways & BywaysJust here for Highways & Byways updates? Click here.

 


 

Highways & Byways is basically complete. I’ve started the final 100 play-tests. This is a process I follow to ensure a game’s quality. This method may not be appropriate for every developer, every team, or every game, but it is one I like. I’ve done this with War Co. and I’m doing it again with Highways & Byways.

 

 

The final 100 play-tests start once I feel a game is basically where I want it to be. I start the final 100 play-tests after I’ve done a substantial amount of blind play-testing. I wait until I have all the art assets I need. I only start the final 100 play-tests when I need to test the game for outliers.

When I use the phrase outlier, I’m referring to something really specific. To quote my good friend, Wikipedia: “[i]n statistics, an outlier is an observation point that is distant from other observations. An outlier may be due to variability in the measurement or it may indicate experimental error; the latter are sometimes excluded from the data set. An outlier can cause serious problems in statistical analyses.

In English, that means that the more times you do something, the more weird stuff you’ll see. With enough games, outliers tend to balance each other out. That’s called the law of large numbers, but I’m not going to get into this because this isn’t Brandon the Statistician.

When people play a game, their expectations are formed by whatever happens in their first game. On average, the game is average. Shocker, shocker. But every once in a while, somebody’s first game is an outlier. That’s not necessarily good or bad, but it’s something game devs need to be wary of. Outlier games still need to be a good experience.

The only way to catch outliers is to just play a lot of games. For some games, 25 is enough, for others, it could take 500. Considering where I am with Highways & Byways, 100 games seemed like an appropriate goal. It’s a big enough sample size to suss out statistical curveballs, but small enough for me to actually produce the game in a reasonable time frame.

 

This is a checklist I like to check off before I start final testing:

  1. Get the physical prototype ready. It’s too much of a pain in the butt to try to do this on Tabletop Simulator and you need data based on the real physical experience.
  2. Get all the print files ready and perfect aligned with the manufacturer’s templates.
  3. Check everything for grammar and clarity.
  4. Make sure all the components are good, especially in terms of accessibility (physical, visual, etc.)
  5. Make sure there are no broken parts left in the game.
  6. Proofread everything again.
  7. Create a spreadsheet to track the following: game number, date, time, players, length of game, critical stats*, and comments.
  8. Find play-testers 🙂

* For Byways: vehicle, start space, and spaces left at the end of the game

 

Once all the prep work is done, I start thinking about my main objectives for the final 100 play-tests. I like to keep it simple:

  • Play enough to catch outliers.
  • Correct minor mistakes.
  • If there is something seriously wrong, iterate again and reset the count.

 

It’s gritty work and it’s meticulous, but it’s straightforward. You want to put your best foot forward for both your players and your business. Players in the Board Game Geek age don’t accept anything less than very polished products. You need to stand out in a crowded market and you want to get positive reviewer feedback. There are both morally high-minded and economically self-interested reasons for putting yourself through all this effort.

It helps when going in to have an idea of what kind of situations could cause a problem. For War Co., I flagged about 50 cards I thought could have dangerous synergies and play-tested with them slightly more often. Highways & Byways, mercifully, is simpler to test, but I’ve still got some concerns. I’ll just spell them out here so you have concrete examples. Maybe hard examples will get your juices flowing for when you do your own hardcore game testing.

 

 

What if certain Start Spaces give players an unfair advantage? This one is pretty straightforward. In Highways & Byways, there are six start spaces represented by stars on the map. If one puts you close to every other road, that makes the entire route shorter. That’s a flaw in a racing game. If statistics prove one Start Space to be clearly superior, I’ll simply move the Start Space. I’m awfully suspicious about that one near Scranton, PA…

What if there is a first player advantage? Just about every game has some variation of this concern at some point. Highways & Byways is no different. Only thing I can do is test.

What if one Vehicle is stronger or weaker? There are six Vehicles in the game. Each one has a special ability. These special abilities have changed a lot since my original intentions. Like the class cards in Pandemic, your ride determines your game’s strategy. If one Vehicle has an unfair advantage or disadvantage, I better nerf that now. The only way to find out is, as you guessed it, raw statistics.

 

 

What if some Event Cards have overpowered synergy with Vehicles? All Vehicles play with Event Cards to some degree, except for Rustbucket, but there are two that I’m worried about right now. Stationary Wagon lets you churn cards in your hand twice as fast as other Vehicles. It has no immunities and the action still requires you to move less to use it, but – I don’t know – I could conceive of this being overpowered. On the other hand, Five-O lets you move four extra spaces when you draw a Distance card. Distance cards already tend to let you move more in a turn, so this can lead to turns of dramatic and extraordinary movement, although very irregularly.

What if somebody drafts a really easy route and others draft really hard routes? The drafting mechanic allows for players to plan their road trips to a limited degree. It’s possible to get royally screwed over and have to pick something way out of your way. You do have some limited degree of control because you can pick better roads and draw up to 2 of 12 that are just not working for you. Yet at the same time, it’s totally possible for someone to get roads clustered in one region while others have to go to multiple. It’s rare and Event Cards mitigate this, but I still have to test it out to see how fair the game’s drafting really is.

 

Testing a lot will soothe my mind. I want this game to be not merely good, but great. I want every experience on the tabletop to be amazing for everyone involved. That’s why I’ve got to play tons and tons until I know it’s fine tuned. Only then can I put it in a box with my name on it. Only then can I ask for people’s money without guilt.

Are you in a similar place in your projects? Have you been there before or perhaps think you’ll be there soon? Leave your thoughts in the comments, I’d love to hear them 🙂

 


 

Most Important Highways & Byways Updates

  • I’ve started the final 100 play-tests.
  • All art is done.
  • I’ve ordered an updated version of the prototype.
  • I am play-testing as much as I can simply to assure quality at this point.
  • I want to be ready to print review copies on January 1 – it’s an ambitious goal, but doable.