QA Vendor Governance: SLAs, KPIs and Scorecards for Outsourced Testing

QA vendor governance is the set of service levels, metrics and review routines that keep an outsourced testing provider accountable for outcomes, not just activity. A vendor can hit every number in the contract while the product still fails, so the measures you choose decide what you actually receive.

Key Takeaways

• Govern outcomes, not activity. Test cases executed and hours delivered describe effort. Escaped defects and release readiness describe results.
• Separate SLAs from KPIs. Put contractual remedies on what the vendor controls. Track shared results as KPIs and improve them together.
• Baseline first. Without a record of escaped defects and cycle time before the engagement, nobody can prove improvement or decline.
• Keep the scorecard to one page. Six to eight measures with an owner, a target and a trend, plus a short note on risks and decisions.
• Plan the exit on day one. Test assets, framework access and handover terms are the real protection against lock-in.

Contents

• Why can a vendor meet every SLA and still fail you?
• What is QA vendor governance?
• What is the difference between an SLA and a KPI?
• Which KPIs should you track?
• The governance model at a glance
• How do you build the monthly scorecard?
• Which contract clauses matter?
• A worked example: a managed regression engagement
• What goes wrong in vendor governance?
• Best practices
• Governance checklist
• Frequently asked questions

Why Can a Vendor Meet Every SLA and Still Fail You?

A vendor can meet every SLA and still fail you because most testing contracts measure activity, and activity is easy to produce. Test cases executed, pass rate and hours delivered can all look excellent while the riskiest parts of the product go untested and defects keep reaching production.

Consider the pattern. A contract promises 95% of planned test cases executed per cycle and reports within 24 hours. The vendor delivers both, on time, every month. Meanwhile the planned test cases were written two years ago, cover the features that were easy to automate, and say nothing about the integration that broke last quarter. The scorecard is green. The product is not.

This is not usually bad faith. It is the predictable result of rewarding what is easy to count. People optimise for the measure they are held to, and a measure that ignores outcomes invites a team that ignores them too. If you are still weighing whether to outsource at all, start with our comparison of outsourcing QA vs building in-house. If you have already decided, governance is where the value is won or lost.

What Is QA Vendor Governance?

QA vendor governance is the structure used to manage an outsourced testing provider: what the vendor is accountable for, how performance is measured, how often it is reviewed, how problems escalate and how you leave if you need to. It turns a contract into a working relationship with visible, comparable results.

It has four parts:

1. Scope and outcomes. A plain statement of what the vendor owns, such as regression coverage of critical journeys or release-readiness sign-off.
2. Measures. A small set of service levels and KPIs, each with an owner and a target.
3. Rhythm. Weekly, monthly and quarterly reviews, each with a different audience and purpose.
4. Controls. Escalation paths, change control for scope and contract terms that protect your test assets.

Governance is lighter than most buyers fear. It is not a compliance department. For a team of five to fifteen testers, it is usually one scorecard, three recurring meetings and a contract that says who owns what. Our guide to choosing a QA consulting company covers the selection stage; this post covers what happens after you sign.

What Is the Difference Between an SLA and a KPI?

A service level agreement (SLA) is a contractual commitment on something the vendor controls directly, such as how quickly a failed build is triaged. A KPI is a performance measure you track to judge results that depend on both teams, such as escaped defects. SLAs carry remedies; KPIs drive improvement conversations.

Mixing them up is the most common design error. If you attach a financial penalty to a KPI the vendor only partly controls, you create an argument about causes every month. If you only track SLAs, you reward promptness over results.

A practical rule: put remedies on the first group, and put shared targets with a named improvement plan on the second. When an outcome KPI misses its target twice in a row, the response is a joint review of causes, not an automatic penalty.

Which KPIs Should You Track?

Track a small set across four areas: delivery, quality outcomes, automation health and engineering flow. Six to eight measures, each with an owner and a target, work better than twenty that nobody reads. Choose measures you can verify from your own systems rather than from the vendor's report.

Defect detection percentage is the share of all known defects that were caught before release. If testing found 90 defects and customers later reported 10, it is 90 out of 100, or 90%. It is useful as a trend, but it depends on counting defects consistently and waiting long enough for production defects to surface.

For the engineering-flow measures, a recognised reference helps. The DORA research programme identifies five software delivery performance metrics and groups them into throughput (change lead time, deployment frequency and failed deployment recovery time) and instability (change fail rate and deployment rework rate). Quality is not the only influence on these, so treat them as shared KPIs, not vendor penalties. Our guide to metrics for measuring quality improvement goes deeper on choosing and reading them.

Two measures to avoid as headline targets: number of test cases executed and pass rate. Both are trivially easy to inflate and say little about risk.

The Governance Model at a Glance

The model has three layers. Operating reviews keep the weekly work on track, the monthly scorecard keeps results honest, and the quarterly review keeps scope and commercial terms aligned with the business. The diagram shows who attends and what each layer decides.

The weekly check should take thirty minutes and produce actions, not slides. The monthly review is where the scorecard lives. The quarterly review is the only place scope and price are renegotiated, which keeps day-to-day conversations free of commercial pressure.

How Do You Build the Monthly Scorecard?

Build the scorecard as one page: six to eight measures, each with an owner, a target, the current value and a trend, plus a short note on risks and decisions needed. If it takes more than ten minutes to read, people stop reading it and the governance quietly dies.

A usable layout:

Three rules keep it honest:

• Pull the numbers from your own systems. Defect trackers, pipeline data and release records belong to you. A scorecard built only from the vendor's report asks the vendor to mark its own homework.
• Define terms once. Agree what counts as an escaped defect, what severity 1 means and when a defect is attributed to a release. Write it on the back of the scorecard.
• Review a sample, not just a number. Each quarter, read a handful of actual test cases and a few closed defects. This catches the high-pass-rate, low-value pattern that metrics alone cannot.

Which Contract Clauses Matter?

The clauses that matter most are scope and acceptance criteria, service levels with remedies, ownership of all test assets and frameworks, repository and tool access, replacement terms for key people, data-handling obligations and a documented handover period. They protect you if the relationship fails, which is when you will most need them.

Have legal counsel review the final wording, since requirements vary by jurisdiction and industry. The principle is to keep the assets in your accounts and the knowledge documented. A vendor whose frameworks live only on their machines is not governed; it is trusted, and trust is not a control. Our overview of quality engineering services explains what a managed engagement should include, and QA consulting rates helps you sanity-check commercial terms.

A Worked Example: A Managed Regression Engagement

The example below is illustrative. It shows how the pieces fit together; the figures are placeholders to show what to measure, not results from a client engagement.

Situation. A mid-sized product company outsources regression testing for a web platform to a managed team of six. Releases are fortnightly. Leadership's complaint is that the vendor's reports are always green while customers keep finding bugs.

Step 1: baseline. Before changing anything, the company records, for the previous six releases, the number of severity 1-2 defects found by customers, the regression cycle time, the flaky test rate and the change fail rate. These become the reference points.

Step 2: rewrite the measures. The old contract tracked test cases executed and pass rate. The new one separates:

• SLAs (vendor-controlled): build-failure triage within four working hours, regression report within one working day of cycle end, flaky tests quarantined within two days of detection.
• Outcome KPIs (shared): severity 1-2 escaped defects per release, coverage of an agreed list of fifteen critical journeys, change fail rate.

Step 3: add a sampling review. Each quarter the quality lead reads ten test cases and five closed defects chosen at random. The first review finds that several "critical journey" tests only check that a page loads. The vendor and company agree a rewrite, funded inside the existing scope.

Step 4: set the rhythm. A weekly thirty-minute delivery check, a monthly scorecard review and a quarterly business review. The scorecard fits on one page.

Step 5: protect the exit. The company moves the repositories and CI jobs into its own accounts and adds a documentation requirement to the contract. Nothing changes day to day, but switching vendor is now a plan rather than a threat.

The point is the sequence: baseline, rewrite the measures to outcomes, sample real work, set a rhythm, protect the assets. None of it requires a large governance team.

What Goes Wrong in Vendor Governance?

Governance fails in predictable ways, and most of them are about measuring the wrong thing or measuring too much. The second diagram contrasts the activity-based setup that produces green scorecards with the outcome-based setup that surfaces real risk.

The recurring problems:

• Measuring only what the vendor reports. Without independent data from your trackers and pipelines, the scorecard is a self-assessment.
• Too many measures. Twenty metrics dilute attention. If everything is tracked, nothing is managed.
• Penalties on shared outcomes. Attaching a financial penalty to escaped defects, which development also influences, turns reviews into blame sessions and encourages under-reporting of defects.
• No baseline. Months later nobody can say whether quality improved, so the conversation reverts to opinions.
• No sampling. Metrics can be satisfied by low-value work. Only reading the work itself reveals that.
• Lock-in by omission. Frameworks, scripts and credentials live in the vendor's environment, so leaving becomes expensive and the vendor knows it.
• Governance without authority. Reviews happen, but nobody is empowered to change scope, reset a target or end the contract.

Best Practices

1. State the outcomes in plain language first. If the scope cannot be explained in three sentences, the measures will be arbitrary.
2. Separate remedies from improvement targets. SLAs get credits; outcome KPIs get joint plans.
3. Baseline before you begin. Record escaped defects, cycle time, flaky rate and change fail rate.
4. Use your own data. Pull measures from your trackers and pipelines.
5. Keep it to eight measures or fewer. Add one only when you retire one.
6. Sample real work every quarter. Read tests and defects, not just dashboards.
7. Define terms once, in writing. Severity levels, attribution rules and counting periods.
8. Hold the assets. Repositories, CI jobs and test tools sit in your accounts.
9. Review the measures themselves. Each quarter, ask whether the scorecard still reflects what the business needs.
10. Treat governance as a partnership tool. The goal is early, honest signals, not catching the vendor out. Vendors who welcome it are usually the ones worth keeping.

Governance Checklist

• Outcomes the vendor owns written in two or three sentences
• SLAs (vendor-controlled) separated from outcome KPIs (shared)
• Baseline recorded: escaped defects, cycle time, flaky rate, change fail rate
• No more than eight measures, each with an owner and a target
• Terms defined: severity levels, attribution, counting period
• Scorecard data drawn from your own systems
• One-page monthly scorecard template agreed
• Weekly, monthly and quarterly reviews scheduled with named attendees
• Escalation path written down, including the "two misses in a row" rule
• Quarterly sample review of tests and closed defects planned
• Repositories, CI and tool accounts held by you
• Handover and exit terms included in the contract
• Legal review of final contract wording complete

Frequently Asked Questions

What is QA vendor governance?

QA vendor governance is the structure of service levels, metrics, reviews and contract terms used to manage an outsourced testing provider. It defines what the vendor is accountable for, how performance is measured, how often it is reviewed and what happens when targets are missed. Good governance focuses on outcomes such as escaped defects and release readiness, not just hours worked or test cases executed.

What is the difference between an SLA and a KPI in QA outsourcing?

A service level agreement (SLA) is a contractual commitment on something the vendor controls, such as how quickly they triage a failed build or deliver a test report. A KPI is a performance measure you track to judge results, such as the escaped defect rate, which depends on both teams. Put penalties and credits on SLAs; use KPIs for shared improvement targets and reviews.

Which KPIs should I track for an outsourced QA team?

Track a small set across four areas: delivery (on-time test cycles, report turnaround), quality outcomes (escaped defects, defect detection percentage), automation health (flaky test rate, suite runtime, maintenance effort) and engineering flow (change fail rate and lead time, two of the five DORA metrics). Six to eight measures with clear owners work better than twenty that nobody reads.

Why do vendors meet their SLAs but the product still fails?

Because activity-based measures such as test cases executed, pass rate or hours delivered can all look healthy while real risk goes untested. A vendor can run thousands of low-value tests and report a high pass rate. Pair delivery SLAs with outcome measures, such as escaped defects and coverage of critical user journeys, and review a sample of the actual tests.

How often should you review an outsourced QA vendor?

A common rhythm is a weekly delivery check with the working team, a monthly scorecard review with the delivery manager and your quality lead, and a quarterly business review with senior stakeholders to revisit scope, targets and commercial terms. Add an immediate escalation path for any measure that misses its target in two consecutive periods.

What contract clauses protect you when outsourcing QA?

Look for clear scope and acceptance criteria, defined service levels with remedies, ownership of all test assets and frameworks, access to repositories and tools, key-person and replacement terms, confidentiality and data-handling obligations, a termination-for-convenience option and a documented handover period. Have legal review the final wording, since requirements vary by jurisdiction and industry.

Conclusion

Outsourced testing succeeds or fails on what you ask the vendor to be accountable for. Measures built on activity produce green scorecards and surprised customers. Measures built on outcomes, backed by a baseline, your own data, a short review rhythm and contract terms that keep the assets in your hands, produce early and honest signals about whether quality is improving.

The effort is modest: one page of measures, three recurring meetings and a handful of contract clauses. The payoff is that you stop arguing about whether the vendor is performing and start discussing how to improve the product.

If you are setting up a managed testing engagement, or want an independent review of an existing vendor's scorecard and contract, our software testing services team can help you define service levels, build the baseline and run the first quarterly review. Talk to us about a vendor governance assessment.