Creating a product test plan can be a difficult task. Working within the real world constraints of budgets, deadlines, and limited sample sizes can make planning a rigorous test hard to achieve. You know that 200 rounds isn’t going to give you meaningful data, but where is the sweet spot between 2,000 and 20,000 rounds? The stakes are high because testing is your opportunity to find the problems before the customer finds them for you. If you have ever written product test plans then these questions and concerns are likely familiar ones.
I have asked myself these questions many times. Over years of product launches that have passed through my test teams, I have seen what paid off well and learned from some misses. I’d like to share my thoughts on how to think about creating a great test.
Know your expectations Link to heading
Spoiler alert up front, there is no one size fits all test plan that will accomplish the goal of great testing. Step 1 will always be to understand what the product is meant to do and what the customer will expect. No product is going to be great at everything, and the testing needs to flex to meet the product where it is meant to perform well. A screw driver and a hammer are both tools, but the use cases are very different and therefore I hope the manufacturers don’t test them in the same way. Your product is no different. Think through all of the factors. Here is a non-exhaustive list that will get the conversation started:
- What is a reasonable expectation for how many rounds the customer will shoot? A competition ready handgun and a deer rifle will have vastly different answers.
- What level of accuracy is the firearm, and the cartridge that it uses, intended for?
- How long can the firearm reasonably be expected to last? There are many factors here. Look back to how much you expect the gun to be shot and then you have to decide on a reasonable lifespan for what you are selling.
- What variety of ammunition is available? Is the cartridge newer or specialized with few options, or is this a popular load with a long history and many variations?
- Is it likely that the customer is going to add accessories to the firearm? If so, how will those accessories affect function?
- What environments will the product be used in?
- Have all of the safety critical scenarios been covered?
Answering these questions will get the conversations started, and you should expand on more items as they come up. I recommend looking back at customer data to find pain points of the past to see what other areas are critical for your product.
Transform the expectations to standards Link to heading
Once you know what you want your product to do, it is time to standardize it in a form that can be tested against. For each area that testing will occur on (reliability, ammunition compatibility, safety, accuracy, etc.), the expectation of the product needs to transform into pass/fail criteria. Determining true criteria is where the product expectation in words is transformed to a measurable standard.
Test early, test multiple Link to heading
A good test program is going to have multiple rounds of testing. Waiting until the end only means that you find the flaws after much money and time has been spent. It is far more efficient to find the issues early. Dovetailing with this thinking is determining how many samples are included on a test.
It is understandable to have fewer samples early in the process when only a limited number of prototypes exist. However, as the product reaches maturity the sample size must grow to a reasonable size. What a reasonable size is can be a tricky question in the firearm world. The pure mathematical answer to statistically prove reliability and durability metrics would have you testing tens, if not hundreds, of samples. This is normally not feasible because of time and money. There is a reasonable middle ground where testing should certainly include more than 1 sample, but doesn’t need to grow to a size that will take an unreasonable amount of time to conduct the test.
I admit that this is a tricky balance. Knowing your requirements, risk tolerance, and available resources will all inform this decision.
Execute with control Link to heading
Once the expectations are known and samples are in hand, then it’s time to do the testing. This is the time where very clear and complete documentation is critical. Finishing a test and reporting that 5,000 rounds were fired and 6 malfunctions occurred only tells a very small part of the story. Did all of the malfunctions occur with one ammunition? What were the malfunctions? Did all of the malfunctions occur early in the test, late in the test, or evenly mixed?
Test deliberately and document fully. Document all of the variables for each test that is conducted, this will allow you to make much more informed decisions after the testing is done. Having a standard language is also very important when documenting results. If two people refer to the same malfunction in different terms, then your analysis and troubleshooting will be much more complicated (I’ll be writing a series of stoppage classification articles on the blog that show how I approach this, see failures to feed as the first in the series).
Another aspect of great execution is to make sure that your test covers real world use scenarios. Deciding on and sticking to maintenance cycles gives consistency in the test. Altering the environment can make sure that guns left in cars at the peak of summer or winter won’t create unexpected changes. Environmental and upkeep variables can have a large influence on the test outcome.
The test documentation should also serve as a record that is detailed enough to be able to repeat the test later if required. This is important because if the design changes and the test is not identical then the new changes are being measured against a different standard.
Always learn Link to heading
Test failures will happen. Everyone hopes that the pre-work is going to lead to a smooth and successful test, but there will often be something uncovered in testing that isn’t exactly as the team had hoped for. This is the time to come back to your pass/fail criteria to know if the product is still ok for launch. But most importantly, findings and failures in testing are always great opportunities to learn and improve for the future.
Takeaway Link to heading
Developing a robust test plan can be a tricky task. I can tell you from experience that your own list of variables to watch and ranking of what testing is most critical will shift over time as you receive feedback from the customer. Every year has brought a new insight that I hadn’t considered on my own. It’s important to learn from this feedback to keep your program constantly improving.
Take this as a starting point for how you think about your next test. Hopefully the outline provided here will help you to think through as many variables as possible and to create a great test program that leads you to a smooth launch! If you are struggling with structuring your test program or feel that it needs some refinement, then head over to the work with me page to get in touch.