By Adam Kolawa, Co-Founder of Parasoft
A regression system is any tool or combination of tools that can automatically run all of your existing tests on your entire code base on a regular basis (preferably nightly, as part of the automated build). Its purpose is to help you identify when code modifications cause previously working functionality to regress, or fail. For example, the regression system may be a script that runs one or more testing or error prevention technologies in batch mode. It should run the existing tests for all practices that you decide to implement.
To get the most out of your regression system, apply the following policies, which are designed to ensure that your regression system will identify all actual code base regressions as soon as possible — without reporting false positives.Configure the regression system so that it provides detailed result information
The regression system should provide extensive detail about each problem encountered. As a rule of thumb, you should ensure that it provides sufficient detail for an architect or developer to review the report, and then start investigating the problem without having to run an additional tool or generate additional reports. For regressions involving coding standard violations, the results should indicate what rule was violated and at what precise line(s) of code violated the coding standard. For unit testing regressions, the results should indicate the type of problem that occurred (unexpected outcome, uncaught runtime exception, and so on.), the inputs that uncovered the problem, and the stack trace. In all cases, the regression system results should provide more detail than the results recorded in the AEP monitoring technology, which is designed to highlight process weaknesses and historical trends, not to identify regressions.Use the regression system to run regression tests every night, after the automated build
Industry studies show that the earlier an error is detected, the faster, easier, and less costly it is to fix. If all practices and tests are performed automatically every night, regression errors will be exposed immediately after they are introduced, and can be fixed as soon as possible.Review regression results at the beginning of each work day and update the test suite as needed
Each regression failure should be reviewed to determine whether the reported failure indicated a problem with the code, or a problem with the test suite.
If the failure indicates a problem with the code, the developers should fix the code immediately (before starting work on new functionality) and the architect and QA team should immediately identify the error's root cause, and then improve the software development process so that the same type of error does not recur.
If the failure indicates a problem with a regression test, the developers should correct the test immediately to avoid the reporting of subsequent false positives, which can desensitize the team to all reported errors. For example, assume that a test case failed because the code functionality intentionally changed, and the expected test case outcome should have also changed. The developers responsible for that test case should then update the test case with the new expected outcome. Then, the next time the regression system runs, it will check whether the expected outcome is produced, and will not report the false positive a second time.
This review process should occur at the beginning of the day, every day. If it does not, the team members reduce their opportunity to learn from their mistakes. If the team does not recognize the problem, and the architect and QA team members do not immediately work to prevent it from recurring, developers may repeat the same mistake in additional code - either because they are unaware of the problem and/or because the process has not yet been improved to prevent that mistake.Use regression results to gauge whether the system should go into production
Before the project nears release, the manager and architect should determine what ratio of regression failures to successes must be achieved before the software is released. Although this should not be the sole factor for giving the gold stamp to a release, the release should never occur when the expected ratio is not achieved.