VVSG Links:
Chapter 6: Test Report (test lab)
6.1 Test report contents
Reporting performance test results for usability is covered under Part 1: 3.2.1.1 “Overall performance metrics”.
No Comments for this section -- Comments ClosedFor modifications to previously tested systems, the test lab SHALL include the test reports that are precedential to the current evaluation.
Applies To: Voting system
DISCUSSION
It is anticipated that the test report will be delivered in electronic form, so the volume of data should not be a problem.
Source: New requirement
The test lab SHALL include a copy of the test plan, amended to reflect any changes that were allowed during the course of the testing campaign.
Applies To: Voting system
DISCUSSION
Test plans must be updated whenever a change to a voting system requires deviation from the original test plan.
The test lab SHALL include the implementation statement submitted by the manufacturer, amended to reflect any changes that were allowed during the course of the testing campaign.
Applies To: Voting system
DISCUSSION
Because minor defects in a system may be corrected during the course of the testing campaign, the system that completes the conformity assessment process might not be identical to the one for which an implementation statement was submitted. The product identification for the revised system must be different. Also, if a system fails a test for a particular voting variation, the manufacturer and test lab may agree to eliminate that voting variation from the list of classes to which conformity assessment is desired rather than correct the system.
The test lab SHALL include a copy of the record of the final (witnessed) build and sufficient description of the build process to reproduce it.
Applies To: Voting system
DISCUSSION
See Part 3: 2.6.1 “Voting system software version recommended for certification”.
Source: New requirement
The test lab SHALL identify the repository for software reference information and include the unique identifier assigned to the software reference information by the repository.
Applies To: Voting system
Source: New requirement
The test lab SHALL include a summary finding of whether or not the implementation under test satisfies all applicable, mandatory ("SHALL") requirements of the Voluntary Voting System Guidelines.
Applies To: Voting system
If the test lab finds that the implementation under test does not satisfy all applicable, mandatory ("SHALL") requirements of the Voluntary Voting System Guidelines, the test lab SHALL identify each of the specific requirements that is not satisfied.
Applies To: Voting system
For each unsatisfied mandatory requirement, the test lab SHALL describe the inspections or tests that detected the nonconformities and include applicable evidence (e.g., vote data report, citation of logic error in source code).
Applies To: Voting system
The test lab SHALL summarize all failures, errors, nonconformities and anomalies that were observed during conformity assessment, no matter how minor.
Applies To: Voting system
The test lab SHALL identify those deficiencies that were corrected during the course of the testing campaign and identify the inspections or tests that confirm that the deficiencies were corrected.
Applies To: Voting system
DISCUSSION
For minor defects of a localized nature, the test lab may permit the manufacturer to correct the fault without incurring a complete regression test of the system. However, a certifying authority may require that revised documents be submitted whenever changes are made.
For requirements that specify benchmarks, the test lab SHALL report the result of the measurement for the implementation under test.
Applies To: Voting system
The test lab SHALL report the observed cumulative failure rate and the failure rate that was demonstrated with 90 % confidence for each type of device, for each applicable failure type in Part 1: Table 6-3 (Part 1: 6.3.1.5 “Requirements”).
Applies To: Voting device
DISCUSSION
See also Part 3: 5.3.2 “Critical values”. "Type of device" refers to the different models produced by the manufacturer. These are not the same as device classes. The system may include several different models of the same class, and a given model may belong to more than one class.
The test lab SHALL report the observed cumulative report total error rate and the report total error rate that was demonstrated with 90 % confidence for the system as a whole.
For paper-based tabulators and EBMs, this SHALL include the observed cumulative misfeed rate and the misfeed rate that was demonstrated with 90 % confidence for each type of device.
For paper-based tabulators, the test lab SHALL report the ballot tabulation rate used in tests.
Applies To: Paper-based device Λ Tabulator
DISCUSSION
Stress tests might use a higher rate than other tests.
The test lab SHALL identify each applicable, non-mandatory ("SHOULD") requirement to which nonconformity was demonstrated.
Applies To: Voting system
DISCUSSION
Test labs are not required to test every "should" requirement; however, if they do, they must report the results.
The test lab SHALL identify all tests that were waived.
Applies To: Voting system
DISCUSSION
A test is waived if the documented assumptions of an applicable test are not met by the implementation under test. A test that pertains to a system or device class that was not claimed in the implementation statement is implicitly assigned the verdict Not Applicable.
The test lab SHALL include a timeline of the testing campaign as it actually occurred.
Applies To: Voting system
The test lab SHALL list any specific election management practices that are required for the voting system to satisfy the requirements of the VVSG.
Applies To: Voting system
DISCUSSION
For example, if additional procedures must be followed in order to safeguard the secrecy of the vote, these must be documented. If a system requires unusually onerous procedural compensations because customary system safeguards are absent, this may impact certification decisions.
Source: New requirement
If any changes to the system are required to complete conformity assessment, the test lab SHALL include a signed warrant from the manufacturer that those changes will be included in the product that is delivered to customers.
Applies To: Voting system
Source: New requirement
The test lab SHALL list and explain any concerns that SHOULD be brought to the attention of readers and/or the VVSG interpretations and maintenance processes.
Applies To: Voting system
DISCUSSION
Any unresolved concerns may be documented in the test report. "Concerns" would include ambiguities in the VVSG, interpretation conflicts, requirements that appear to do more harm than good, loopholes in the VVSG (where it is possible to satisfy the technical requirements while failing to satisfy their intent), and other issues whose resolution would require action by outside authorities.
Chapter 6: No Comments for this section -- Comments Closed