After an update or major change, verify an automated report as an end-to-end system: confirm that the schedule ran, the intended data arrived, transformations still behave correctly, access rules still apply, and recipients received the expected output. A green job status alone is not enough because a technically completed run can still contain stale, incomplete, duplicated, or mis-scoped data.
Preserve a comparison baseline
Before the first post-change run, save the last known-good report, its generation time, source cutoff, filters, row or record counts, important totals, and delivery list. Record the active report definition and the change that was made. Choose several test cases that exercise ordinary data, empty values, boundary dates, late records, and any newly changed calculation. The goal is not to prove that every value stayed identical; it is to make expected differences explicit and unexpected differences visible.
The U.S. Government Accountability Office defines data reliability in terms of accuracy, completeness, and applicability for the intended purpose, and recommends a risk-based assessment rather than a ritual checklist (GAO's Assessing Data Reliability). Apply that idea by giving the most scrutiny to totals, classifications, or recipients that could drive consequential decisions.
Run the complete production-like path
Trigger or wait for the real scheduled path instead of testing only the report editor. Confirm the scheduler used the expected time zone and parameter set. Capture the run identifier, start and finish times, source refresh time, report revision, warnings, and output location. If a credential, gateway, connector, or destination changed, verify that component directly and confirm that the job did not silently fall back to cached data.
Check timeliness, completeness, and validity separately. CDC data-quality guidance uses those dimensions and notes that processing summaries can expose filtering, exceptions, backlogs, and unusual volume (CDC data-quality tools). For a report, that means comparing source and output counts, inspecting rejected or late records, validating field formats and allowed values, and investigating an unexpected change in volume rather than accepting it as normal.
Compare results and permissions
Recalculate a small but representative sample independently from the source. Compare key totals with the baseline and explain each material difference. Check date boundaries, time zones, rounding, joins, deduplication, null handling, and filters because these often change results without causing a job failure. Then view the report as at least one ordinary recipient and one restricted recipient. Confirm that export links, attachments, and row-level restrictions expose no more information than intended.
Verify delivery and monitor the next runs
Confirm the output arrived through every required delivery channel and that links open for intended recipients. Check the subject, file name, reporting period, timestamp, and version label so an older artifact cannot be mistaken for the new one. Record the expected next several run times and watch at least the next normal run for duplication, omission, or drift.
Close verification only when the evidence shows the scheduled run completed, inputs were current, quality checks passed, representative calculations matched, permissions held, and delivery worked. If any check fails, pause distribution where practical, retain the failed artifact, document the discrepancy, and either correct the change or restore the last known-good configuration.
Sources
- Assessing Data Reliability — U.S. Government Accountability Office, Dec. 16, 2019
- Data Quality Tools — Centers for Disease Control and Prevention