4 tips for Pilot Testing your SaaS App

Pilot testing lets you open up your software for a controlled group of people to use, evaluate and provide feedback on its features. I've been a part of this process for many of my consulting engagements, but this is my first time seeing every aspect of the pilot test for my own software. The process has been more valuable than I ever could've imagined. I wanted to put a few tips together for other Software as a Service (SaaS) start-ups.

1. Pick Smart People
   We chose 10 - 15 close friends and family for our first round of pilot testing. They varied in technical experience, industries and motives for using our software. The size of this group was perfect because we were able to effectively respond to and incorporate their feedback into the software. Also, each individual represented a slice of our potential target market so we valued their opinions greatly. Smart people are more inclined to provide you with useful feedback. Instead of "I don't like it", we received notification of the problem and suggestions for how to improve it.
2. Suppress Positive Feedback
   There is some value in receiving praise for the features that work well or the progress you've made in a short amount of time, but that was not our intent with this audience. In our invitation e-mail, I added the following disclaimer:
   
   "To make it easy, I'm not looking for ANY positive feedback. This is out of respect for your time and to encourage ONLY suggestions for improvement, reports of broken features or general feedback on the system as a whole."
   This is especially important when people you know are testing for you because they don't worry about hurting your feelings or feel obligated to pay you a compliment along with their criticism.
3. Get to the Pilot Phase QUICK
   If you are unfamiliar with Eric Ries' book, The Lean Startup, check it out! As quickly as you can, build your Minimum Viable Product (MVP) so you can put it in front of your group of smart people. The more quickly they can provide you with feedback, the less time you waste guessing which feature to add next. This will also allow you do some course correction so you can be sure you are on the right track to providing the most value to your potential customers.
4. Engage Your Pilot Group
   Make sure your pilot testers know you are appreciative of their time and that their feedback is being incorporated. Nobody wants to spend their time learning to use your software, putting their thoughts together and submitting it to you only to never hear a response or see any evidence that it mattered. Be sure to respond quickly when feedback is received, keep them up-to-date as you release new versions containing their suggestions and overall make them a part of the development process.

As the amount of feedback begins to slow from this round of Pilot testing and we implement the features and suggestions we received, we intend to open up a future round of testing to a larger audience. This will give us a chance to put a better foot forward for the next group as well as receive fresh perspective and feedback. In our case, 2 rounds should be enough and we feel we'll have a premium product as a result. Obviously, one of the biggest benefits of SaaS is the ease of releasing updates without our customers having to actively do anything. We will never stop listening to our customers and improving our product even after we have officially launched.

Happy New Year!