Overlooking User Acceptance Tests (UAT) in favour of software releases is a myopic and potentially expensive business decision. This is what we’ve learned as a technology start-up in Singapore.
We are Novocall, and here’s our take on the importance of UAT, based on our own experience.
Why UATs Are Important
As a technology company, we understand the tendency to rush software releases and get our product into the hands of our users as quickly as possible. Testing, especially user acceptance tests (UAT), are often deprioritized, given the minimal resources and time urgency a start-up possesses.
Looking ahead, we realize that this is simply thinking short-term. Software releases without proper testing from the user could result in unsatisfactory user experience, bad product reviews, and poor NPS (Net Promoter Score) scores.
In today’s digital world where user experience even trumps price, UAT is becoming more important, regardless of how large or small your company might be.
Functional Tests Are Not UATs
After building a product, a development team needs to ship it without bugs. That’s when they do functional testing. Functional testing is done by the developer through writing code, to test whether the functions they developed run without hiccups. If the software passes the functional test, that means they work. That means it’s good to go, right?
Wrong.
Functional tests are important, but you’ll never know how your system will behave in the real world unless it’s being tested by real users. This is because humans function in ways that the development team might not foresee, and this might break the system. With UAT, it takes both system performance and human behaviour into account.
Planning a UAT
Before we implement anything, it’s important to plan the UAT, especially since our end-user profiles are business owners who aren’t necessarily tech-savvy.
We start with defining and prioritising our business requirements (both functional and non-functional). Prioritising is important as not all tests can be implemented given certain projects time constraints.
The next step is to craft our UAT acceptance criteria (UAC), which is a focused description of each test case.
This gives a decision table outline of an acceptance criteria.
| Description | |
| Given | (i) Input (ii) Preconditions |
| When | (i) Triggers (ii) Actions |
| Then | (i) Output (ii) Postconditions |
The UAC is one of the most important parts of the UAT because this is where compromise happens. Here are some questions you need to ask yourself when setting up the UAC:
- How many defects and bugs in the system are we willing to tolerate for release to the public?
- What will happen if the system is delayed?
- How much delay could we accept?
- What are the damages if the system is delayed?
If you want to go through less technical acceptance criteria, the following is one example of validating a user story. We do a similar acceptance criteria with our call-back feature in Novocall. For this one, let’s assume we’re using it for a more generic contact form.
| Example #1 | |
| User Story | As a user, I want to submit a contact form successfully. |
| Acceptance Criteria |
User submit a form. “Given my role of a website user, When i click on the ‘Contact Us’ page, I get redirected to the page. I fill in all the required fields I click the ‘Submit’ button My feedback successfully gets submitted to the company email. |
| Critical |
Yes |
| Test result |
Await result |
| Comments |
Will not release until requirement has been met |
The UAT Process: A Run Through
After planning, it’s time to build the user testing process as well as the status report. Multiple test cases will be written up with scripts (shown below). With these test cases, we then prepare a status report to be filled out after the user testing is done.
Here’s an example script for a web application’s login test case:
| Example #2 | |
| Purpose | User able to log in with user ID and password |
| Preconditions | User has signed up. User has not logged in |
| Input |
Correct user ID Correct password |
| Process (Actions & Triggers) |
|
| Output | User logs in and sees the dashboard |
Finding Users and Executing Your Test
After setting up our process, it’s time to test! There are numerous ways of user testing, with each test depending on the type of software you are testing and the data you are looking for. This could range from surveys and feedback to more expensive options such as focus groups.
However, if you are running lean or want to save costs in general, you could opt for a more affordable yet reliable testing method: remote user testing. This allows for real users to test your system with video and feedback to boot, without the price tag of a focus group. There are multiple platforms that do this for you: Usertesting, Userfeel, Usabilityhub.
After the test is done, we’ll compile the data collected, make a status report, and evaluate the test results. This will determine whether we:
- Leave bug and go ahead with the release
- Push back the release date and fix bug
- Add change request for future release
Given the complex nature of software and human behaviour, UAT can be a tailored process that is customised to your product. These examples I’ve shown worked for my company’s web application, but it may not work for you. Thus, it’s important that you take the time to examine your test processes and get your UAT done well. With a properly executed UAT, it could bring about confidence to your product releases and deliver exponential returns on your UAT investment.
This is the second piece from our User Acceptance Testing Series, where we talk about the pivotal role of UAT and its importance in developing your web products. Subscribe to our newsletter below to keep updated on future articles coming out.
-----
Jing Jie is the co-founder and COO of Novocall, a B2B SaaS company that helps businesses proactively convert their web visitors into sales call leads anywhere in the world.
More insights