Software Quality Assurance and Test Automation

01Purpose and scope

• This manual supports the course Software Quality Assurance and Test Automation. It describes how quality assurance and test automation are carried out day to day on software teams, from agreeing on what quality means for a product through to running and maintaining automated checks that gate releases.
• The manual covers the work of QA engineers, test automation engineers, developers who write unit and integration tests, and the product and release staff who depend on their results. It focuses on web, mobile and back-end services, since those are the most common settings where automation is built and maintained. Embedded systems, medical device software and safety-critical control software follow additional process standards and regulatory duties that this manual only touches on.
• Quality assurance is the set of planned activities that give confidence a product meets its requirements and is fit for use. Test automation is one tool within that set. It runs repeatable checks quickly and frequently, but it does not replace exploratory testing, reviews of requirements and design, or the judgment of people who know how users behave. This manual treats automation as a means to an end and spends as much attention on what to check and why as on how to script it.

02Core principles

• Quality is judged against explicit expectations. A team that cannot state what the software must do, how fast it must respond, what data it must protect and which platforms it must support cannot test for those things in any meaningful way. Acceptance criteria, service level targets and known constraints are the starting material for every test decision.
• Tests should fail for reasons that matter. A suite that breaks every time a button moves or a timestamp format changes trains people to ignore red builds. Good automated checks target behavior that users or downstream systems depend on, and they keep incidental details such as styling, generated identifiers or ordering of unrelated results out of their assertions.
• Test at the lowest level that can prove the behavior. Logic that can be checked with a unit test should not be checked only through a browser. A common shape is many fast unit tests, a moderate number of service or integration tests, and a small set of end-to-end journeys that cover the paths where a failure would hurt the most. The exact proportions depend on the architecture and should be reviewed as the product changes.
• Automation is code. Test scripts, fixtures, data builders and pipeline definitions need the same care as production code: version control, review, naming conventions, and removal when they stop earning their keep.

03Planning and preparation

• Start by building a risk-based test strategy for the release or the product area. List the features, the user journeys, the integrations and the non-functional concerns such as performance, accessibility, security and data handling. For each, record how likely a defect is, how much harm it would cause and how easily it could be detected after release. Use that list to decide where automation is worth the cost and where manual exploration or a review is the better tool.
• Decide which quality characteristics matter for this product before writing a single test. The ISO/IEC 25010 product quality model is a widely used reference that names characteristics such as functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability and portability. You do not need to test every characteristic equally. A consumer photo app may weight usability and performance heavily, while a payments service may weight security, reliability and auditability. Write the chosen weights down so that the whole team argues from the same priorities.
• Prepare the environment and data before the tests. Identify the test environments, how closely they resemble production, how test accounts and data are created and reset, and which third-party services need stubs, sandboxes or recorded responses. Unstable environments are one of the most common sources of flaky results, so agree on who owns each environment and how its state is restored between runs. Where personal or regulated data would be needed, use synthetic or masked data unless an approved process allows otherwise.

04Recommended workflow

1. 1
   : Review the requirements, user stories and acceptance criteria with the product owner, developers and QA together. Ask what could go wrong for each item, note ambiguous rules, and record open questions. Defects found here cost the least to fix.
2. 2
   : Design test cases from the agreed criteria. Use techniques such as equivalence partitioning, boundary values, state transition tables and decision tables where they fit the logic. Write each case with a clear precondition, the action, the expected result and the data it uses. Mark each case as a candidate for automation, manual execution or both.
3. 3
   : Select automation candidates using a simple rule. Automate checks that run often, are stable in behavior, are expensive to perform by hand, or guard high-risk paths. Leave alone one-time exploratory checks, visual judgments that a person does better, and features still changing every day. Record the reason for each decision so it can be revisited.
4. 4
   : Build the automation in layers. Put business logic checks in unit tests, service contract checks in integration tests, and user journeys in end-to-end tests. Keep page objects or screen models separate from test logic so that a UI change touches one file rather than fifty. Use test data builders rather than shared mutable fixtures, so tests do not depend on the order in which they run.
5. 5
   : Run the suite in the continuous integration pipeline on every change that affects the tested code. Fast unit and integration tests should block merges. Slower end-to-end suites may run on merge to the main branch or nightly, depending on how long they take and how often they fail for reasons unrelated to the product. Publish results, screenshots, logs and traces with each run so that a failure can be understood without rerunning it on a laptop.
6. 6
   : Triage every failure. Decide whether it is a product defect, a test defect, an environment problem or a flaky result. Log product defects with steps to reproduce, expected and actual results, environment details and attachments. Quarantine a flaky test only with an owner and a due date for fixing it, and never leave a test disabled indefinitely without someone accepting the gap in coverage.
7. 7
   : Complete exploratory and manual testing on the areas automation does not cover, and report the results alongside the automated run. Summarize what was tested, what was not, which risks remain and what the release decision rests on. The summary should be short enough that a release manager will read it before the meeting.

05Quality checks, risk controls and compliance

• Quality checks exist at several levels. Code review and static analysis catch some defects before tests run. Unit tests and coverage reports show whether the logic is exercised, though high coverage alone does not prove that assertions are meaningful. Mutation testing, where a tool deliberately changes code and checks whether tests detect the change, can reveal weak assertions, but it is slow and is usually applied to selected modules. Review the test suite itself periodically: remove duplicates, rewrite assertions that check too little, and retire tests for features that no longer exist.
• Risk controls protect the team from automation that gives false confidence. Guard the pipeline against tests that skip silently, that use hard-coded waits in place of condition-based waits, or that ignore failures to keep a build green. Protect test credentials and do not store production secrets in test code or logs. Where tests touch payment, health, identity or other sensitive data, follow the data handling rules that apply to your organization and get written approval before using any real records.
• Compliance depends heavily on where the software is used. Accessibility testing may be required under contracts or public sector rules, and the Web Content Accessibility Guidelines are a common reference, but the applicable version and conformance level depend on the jurisdiction and the agreement in place. Privacy, financial, medical and safety rules also differ by country, state and sector. Where a rule could apply, consult the current authoritative source and the organization's compliance function. This manual does not replace qualified professional judgment, approved procedures or required authorization, particularly in regulated or high-risk fields such as medicine, finance, aviation, automotive safety and legal records.

06Collaboration and documentation

• QA works best when it is embedded in the team rather than handed work at the end. Testers join refinement sessions, pair with developers on tricky logic, and review test designs before code is finished. Developers own unit tests for their changes, and QA owns the integration and end-to-end strategy and the health of the shared suite. Product owners decide on acceptance criteria and on which defects block a release. Name these responsibilities in the team working agreement so that nobody assumes someone else will write the test.
• Documentation should be light and current. A test strategy page of a few pages, a list of critical user journeys, a description of environments and data, and a short guide to running and debugging the automated suite are usually enough. Each automated test should have a name that states the behavior it checks. Record known defects, accepted risks and the reasons for disabled tests in a visible place such as the defect tracker or the repository, not in a private notebook. Stale documentation is worse than none, so assign an owner and review it at each release.

07Measures and continuous improvement

• Measure things that change decisions. Useful indicators include the pass rate and duration of each pipeline stage, the number of flaky test reruns per week, the proportion of defects found before release compared with after, the escape rate for each critical journey, the time from a failed build to a diagnosed cause, and the share of acceptance criteria with at least one linked check. Track trends rather than single values, and avoid setting targets that encourage gaming, such as a coverage percentage that rewards trivial assertions. No universal benchmark applies across products, so set baselines from your own history.
• Review the measures in a short retrospective each sprint or release. Look for patterns: a module that generates most of the flaky failures may need a redesigned interface, a slow suite may need splitting or better parallelization, and a steady trickle of escaped defects in one area may point to a gap in requirements rather than in testing. Change one thing at a time where you can, so the effect is visible.
• Tooling choices should be revisited with the same discipline. Frameworks such as Selenium, pytest and JUnit each have strengths in different languages and stacks, and switching frameworks is expensive. Before adopting a new tool, run a short trial on a real module, compare maintenance effort and reliability against the current approach, and check that the team can support it after the trial owner moves on.

08Common mistakes and how to prevent them

• The most frequent mistake is automating the wrong things first. Teams sometimes script the easiest screens rather than the riskiest flows, which produces a large suite that misses the defects that matter. Prevent this by building the risk list in the planning stage and checking each new automated test against it.
• A second mistake is letting the suite grow without pruning. Duplicate end-to-end tests, tests for removed features and checks that repeat unit-level logic slow the pipeline and erode trust. Schedule a periodic review in which each test must justify its runtime and its failure history. Retiring a test with a recorded reason is a healthy act.
• A third mistake is treating flaky tests as background noise. When people rerun a failing job until it passes, real regressions get hidden among the noise. Track flaky tests explicitly, assign an owner, fix the root cause, which is often a race condition, a shared account or an unreliable third-party stub, and block merges on the suite only after the flakiness is under control. Avoid fixed sleeps as a cure, since they slow the suite and still fail under load.

09Practical checklist

• Confirm the acceptance criteria and the quality characteristics that matter most for this release.
• Record the risk list and mark which items are candidates for automation, manual testing or both.
• Check that test environments and test data are owned, reset between runs and free of real personal records.
• Structure automation in unit, integration and end-to-end layers, with page models or screen models kept separate from test logic.
• Confirm that the pipeline publishes logs, screenshots and traces for every failed run.
• Triage each failure as a product defect, test defect, environment problem or flaky result, and log it with an owner.
• Review flaky tests, disabled tests and skipped tests weekly, and record a reason and a due date for each.
• Summarize tested areas, untested areas, open risks and the basis for the release decision before the meeting.
• Confirm applicable accessibility, privacy and regulatory requirements with the current authoritative source and the compliance owner.

★Learning sources

• Micro Degree: Software Quality Assurance and Test Automation
• Micro Degree: Data Analytics Applications
• Micro Degree: Azure Cloud Fundamentals
• Micro Degree: Google Cloud Platform Basics