Manual versus Automated Testing
Automated and manual test cases are complementary. Agile teams strive to have more automated test cases because they promote frequent or continuous test runs. To continuously run tests, they must execute quickly and frequently, which is difficult to achieve with manual testing.
There are several considerations that should be made when deciding on a distribution of manual and automated test cases.
The product owner helps define the user stories for the project and should also contribute to the creation of acceptance tests. The product owner will likely not be producing coded tests but will have significant knowledge of the business domain. The test cases that are defined by the product owner will, therefore, be at the level of the business vocabulary and business rules. For example, a product owner in a shipping company will specify different means of transportation that are supported by the business (for example, truck, train, air, sea, or a combination). The product owner can then define several test cases that exercise the different possibilities. For these manual tests, it is important to specify the minimum number of tests that exercise the different options (in this case, the means of shipping).
Team members who produce code can build coded UI tests, which can be based on the manual tests or independent of any other test. For more information, see How to: Generate a Coded UI Test by Recording the Application Under Test. Coded UI tests must be supported by team members who have the ability to manage and develop project code.
You can build automated tests when you expect to run tests repeatedly to maintain stability of your code. It is important to consider the effort of creating automated tests because the investment of automation affects the resources of your team. Creating automated tests when the code has little churn results in a higher return on investment (ROI) because there is less test churn. However, there is value in creating automation early because it will help discover problems in both the logic and design. In both cases, the resources that are required for supporting the automated test code needs to be considered.
After you decide that a set of tests must be automated, move to complete the automation as quickly as possible because the benefits of automation include managing the stability of the code. The stability and number of defects that are found as the automation is written will affect the effort that is required to complete the automation. Ultimately, the balance between manual versus automated tests is about the prioritization of the kinds of tests that need to be built and run during the life of the project.
Unit tests verify functionality in code or through a process such as TDD. Unit tests are important because they help maintain the stability and dependencies inside the code. TDD also tends to produce better design with dependencies and good layer definition because it helps you understand the design from the code consumer’s perspective.
You can create load tests that are based on existing automated test cases, or you can create tests that generate specific types of loads on applications or services. For more information about how to use test agent controllers and test agents to generate simulated testing loads, see How to: Run a Test Using Test Controllers and Test Agents.
For more information about load testing with Visual Studio ALM, see the following page on the Microsoft Web site: Understanding Load Tests.
You can use continuous integration with Visual Studio ALM to help ensure that whenever code is developed and checked in, it works correctly with the existing code. Continuous integration is critical as your team and your code base grows. You can define a build type that includes testing parameters and specify which tests to run after completing the build. For more information about how to define a build that runs tests, see Define a Build Process that is Based on the Default Template.
Testing on multiple installed environments can be a very laborious task. Microsoft Test Manager provides capabilities of running test suites on different configurations by using virtual machines or physical machines. For more information about how to run tests and gather data in multiple environments, see Setting Up Test Machines to Run Tests or Collect Data.
Visual Studio ALM has capabilities of creating automated tests directly for the user interface. For more information about how to create coded user interface tests, see How to: Create a Coded UI Test.
Creating automation requires a subset of the test team to learn how to write code. Plan to incur the learning curve of creating automation and the design of the test code. Similar to production code, design the automation code for your desired goals, such as maintainability, ease of composition, and longevity.
For more information about how to create automated tests by using Visual Studio Test Professional, see Creating Automated Tests Using Microsoft Test Manager.
Code frameworks such as ASP.NET MVC and MVVM guide team members to write code that has the isolation that is required to verify different parts of the code. Code that is tightly bound to the user interface is difficult to test because it may require the user to interact with the user interface controls.
Manual test cases offer the following benefits:
-
Manual tests help your team find bugs through the process of exploration.
-
Manual test cases are easy to define because you can define the set of steps at any abstraction and define success and failure in any terms that you like.
-
It is very easy to get started writing manual test cases early in the project before any code has been written. This is important during the process of defining acceptance tests.
-
If you use Visual Studio Test Professional, test cases can be composed of shared steps, which help save time when defining similar tests and allow your team to reuse a single version of a subset of the test. Using shared steps also helps when changing test cases because a change to the shared steps automatically changes all the test cases that use the shared steps. For more information about how to create and use shared steps, see How to: Share Common Test Case Steps Using Shared Steps
-
Manual test cases can serve as a means of early communication in the project or sprint.
-
Manual test cases can serve as a means of documenting an automated test case without anyone reviewing the code.
-
If you use Visual Studio ALM, running manual tests can gather code coverage metrics.
Manual test cases carry the following deficits:
-
Defining success criteria can be complicated because it depends on the perspective and the language that is used in the definition of the test. Some language can be interpreted differently, and that may leave room for error.
-
Running test suites that include manual test cases requires a person to physically follow the test steps and report results. This process can consume of a lot of time and, therefore, may require either an increased number of team members to execute tests or an increased period of time for running the test suite. By using Visual Studio Test Professional, the team can use "fast forward manual testing," in which actions are recorded during testing, which can then be run in future test runs.
-
Depending on the stability of the code, the time and effort that is required to run the tests will vary. As a result, running manual tests can affect flow in the team.
-
The biggest deficit is that manual tests are prone to human error in detecting a bug. Testers may see the bug in front of their eyes and not recognize it.
Automated test cases offer the following benefits:
-
Automated tests help maintain stability and help find regressions that might occur because of code changes.
-
Automated tests can be run unattended.
-
Automated tests are software and can be designed and composed of other reusable code. This makes automated tests flexible and maintainable.
-
Automation can be run on multiple configurations by using Microsoft Test Manager.
-
Code coverage metrics can be gathered when automated tests are run.
Automated test cases carry the following deficits:
-
Special conditions need to be considered and implemented using code. As the automation is created and run, additional conditions will be implemented as they are discovered.
-
When code is changed or refactored, there may be cascading effects that will require a corresponding effort to change the affected automated tests.
-
There may be psychological impact on your team when code changes cause many tests to fail. If these tests are used as flags, the team may not want to raise any flags.
-
There may be a false sense of security when all tests pass if the test cases are not testing for the correct conditions. It is important to maintain the test suites and ensure that they verify the correct conditions and results.