Confidence Intervals
A range of values that allow you to confidently estimate your untested population.
The nuts and bolts: A confidence interval (CI) is a set of numbers that is likely to include a population value with a high degree of certainty. When a population mean falls between two intervals, it is commonly expressed as a percent.
When giving an answer to a question, we are often asked: “On a scale of one to ten, how confident are you with this?” We then rummage through our minds to gather the data as needed to give an estimate on our level of confidence.
We can now give that question an actual numerical and mathematical value with the use of confidence intervals.
With confidence intervals, we can take tested data and apply a quick equation to find out an estimated range of probabilities with confidence.
The simplest explanation for confidence intervals is that they are the most likely range of an untested population average. These intervals are applied to your already collected actionable user test data and results to determine your sense of confidence to push forward with your products. If you make the most of your design decisions based purely on a hunch, you are missing out on the highly useful tool found in confidence intervals.
Calculating Confidence Intervals
If there is one thing that most people abhor, it is math. Don’t worry, we will try and keep it as simple as possible. We don’t want you to have an aneurysm.
Though the equation for calculating the intervals looks intimidating and daunting, when you break down all of the inputs to the equation it’s much easier to swallow. The only necessary inputs to figuring out your confidence intervals are: number of trials, number of successful rounds of trials and the confidence level.
Here is what your equation looks like to find your confidence intervals:
n=total trials
p=successful trials
z= z-value that corresponds to confidence level (your confidence level + half the squared value)
padj = (n*p + z^2/2)/(n + z^2)
nadj = n + z^2
… and the last bit will find your confidence intervals:
padj ± z * sqrt(padj(1- padj)/nadj)
If that equation just floored you and you really worked up a sweat, don’t lose any sleep over it. If you would prefer to make it extremely easy for yourself, just use this handy-dandy confidence interval calculator.
With the equation to figure out our intervals, you should know that there are three influencing factors that impact the breadth and look of our confidence intervals.
They are the following:
Confidence Levels
We’ve already noted that this element is important in figuring out your confidence interval, but it can be adjusted to alter the reach of your intervals.
Most confidence intervals are calculated at the 95% confidence level mark. However, it can be calculated at any percentage you’d like. The confidence level determines how much of our estimate on the unknown or untested population will be wrong/right.
An example may prove to be useful in gaining a better grasp of how these levels affect your intervals.
If we look at a comparison of two different confidence levels, using our lengthy algorithm, we can see how the confidence level will adjust the look of our confidence intervals.
Let’s start by using an example of five test participants, and four successfully complete a task. Then by using 95% confidence level, we see that we have a confidence interval that stretches between a 36% and a 98% chance in the real world. However, if we use 99% confidence we will see that we have an interval that spans between a 26% and 99% chance of seeing similar results.
Confidence levels may sound a bit confusing at first, but let the example sit with you for a bit and it should make more and more sense.
The higher the confidence level we have, the lower risk of our estimates being wrong. At the same time, if we increase our confidence level, we are increasing the span of our confidence intervals over our set of data. To make it even simpler, the higher a confidence level the more values will be added to our confidence intervals to avoid our estimates being wrong.
Variability
The variability of your test-taking participants and their results factor into how your confidence intervals are spread. If you are testing a primarily tech-savvy group your confidence intervals may look high in your favor, whereas if you were to test a group that doesn’t use much tech or tech devices, the results and confidence intervals will look mighty glum.
Sample Size
Similar to variability in that the more subjects you have, the shape and look of your intervals will change. With sample size, though, the more subjects tested, the smaller the intervals become in contrast to smaller sample sizes, intervals will be viewed much larger. Getting a more focused look into how your tested population influences your intervals may benefit your understanding.
Let’s go back to our lovely equation and look at how the sample size affects our confidence intervals.
To further express how differently the sample size alone affects the intervals, we will look at two different sample sizes that have the same result percentage.
We’ll be looking at the ratios 4:5 and 80:100 and the confidence level of 95%. We’ve already noted in this circumstance that our 4:5 ratio has a confidence interval between 36% and 98%. With our ratio of 80:100, we find our confidence interval drastically shifts to be between 68% and 88%.
Though, with a larger sample size your confidence interval drops, you are narrowing your estimate to a more approximate location rather than spanning between the lowest possible percentage and highest possible percentage.
We have mentioned before that one of the best, cost-effective methods of testing is with five testers.
If you’d like to put this theory to the test and want to get the most bang for your buck, you can use the same tool that we at ZURB use as well.
Helio lets you access your target audience willing to answer questions about your product.
“Confidence Intervals, Give Me Sight Beyond Sight!”
The confidence intervals have an innate ability, much like the Sword of Omens, as they have the ability to help us see things that we normally could not see — an estimate of the whole population of end-users and how they would handle your design.
These intervals can help in shaping your proof points as well.
Proof Points are described as being the specific pieces of evidence used to substantiate the unique claims found within key messages and positioning statements. If you are to say your product is “ahead of its kind in its market share” this proof point could be backed up with specific data and evidence to provide to your potential clients.
With these ideas and concepts tucked up your sleeve you’ve added to your arsenal a trick that will aid in producing the highest quality designs and iterations. Remember, you need to have the confidence in publishing your designs to be used by the end-users of the world. That’s where your knowledge of the confidence intervals helps immensely. The tools are all available to you — it’s just a matter of applying them to your work and reaping the benefits of your work.
Start Boosting Your Confidence Intervals Today
You can get early validation for your product with an actual audience in Helio. We’ve done the leg work for you already with our Validate Product Interest Test. Just customize to your needs and get started quickly!