Notification changes affect attention as well as delivery. Keep evidence that distinguishes event generation, provider acceptance, device receipt, operating-system presentation, and user interaction. That separation makes it possible to diagnose a missed urgent alert without claiming that a blocked or muted device was successfully notified.
Preserve the change decision
Save the business purpose, owner, approver, affected event types, audiences, devices, clients, channels, quiet hours, escalation rules, rollout window, and exclusions. Identify the previous and candidate server configurations by version or hash. Record acceptance thresholds from the product's own baseline rather than inventing a universal delivery time.
The Microsoft operational security evidence guide describes change records with impact, testing, approval, and back-out planning. Keep the exact restoration steps, service checks, responsible operator, and rollback triggers.
Capture effective device state
For each representative test device, record operating-system version, app version, account, permission state, channel identifier, channel name, effective importance, blocked state, sound or vibration behavior, and relevant battery restrictions. Avoid collecting unrelated device identifiers.
Android's notification channel guidance says Android 8.0 and later use channels and that users control visibility and intrusiveness. After a channel is created, the app cannot change its behavior, although name and description can be updated. Therefore keep both the application's requested configuration and the effective user-controlled state.
The NotificationChannel API reference notes that importance can be set only before channel submission and that user-selected importance may differ. Capture the actual readback. Do not infer device behavior from a configuration file alone.
Use traceable synthetic events
Create harmless marked events for urgent, ordinary, suppressed, grouped, digest, and escalated behavior. Give each a stable event identifier and record generation time, provider acknowledgement, device arrival, presentation, open action, and final destination. Use test accounts and invented content, never real customer messages.
Test a device offline and reconnect it. Record expiry, retry, grouping, and duplicate behavior. Check one account on multiple devices and one device with notifications denied. Android's time-sensitive notification guidance notes that Android 13 and later require notification permission for non-exempt notifications and that users manage settings. A permission-blocked case should be labeled blocked, not delivered.
Record audience and rule evaluation
Preserve rule versions, feature flags, recipient queries, schedules, time zones, localization, template versions, deep links, and escalation paths. Store counts and synthetic recipient identifiers rather than exporting full personal profiles. For a sample, record why each event was included, suppressed, delayed, or escalated.
If a vendor provides a request ID or delivery receipt, save it with the internal event ID and document what the receipt actually proves. Provider acceptance does not necessarily prove operating-system presentation or human attention. Do not convert an intermediate state into a delivered claim.
Keep rollout and recovery proof
Record pilot membership, duration, client mix, error observations, go or no-go decision, and expansion steps. Keep timestamps for each configuration change and the operator who made it. Note concurrent experiments so results are not misattributed.
During rollback, preserve the restored server version and repeat the synthetic matrix. Compare audience, stage-by-stage delivery, quiet-hour behavior, grouping, escalation, channel identifier, effective importance, and deep link. Do not overwrite user choices to make the rollback appear successful. If a channel cannot be centrally reversed, document the supported forward fix or user communication.
Store evidence with privacy limits
Restrict device tokens, account mappings, notification contents, and security alerts. Keep tokens out of screenshots and logs. Define evidence retention and access. Prefer pseudonymous event IDs linked to protected source systems.
The final decision should list passed cases, blocked cases, failures, unresolved platform differences, owners, and due dates. Strong evidence shows which stage succeeded and preserves user control. It does not confuse “the server attempted to send” with “the intended person saw the notification.”
Sources
- Microsoft 365 Certification sample evidence guide operational security — Microsoft Learn
- Create and manage notification channels — Android Developers
- NotificationChannel API reference — Android Developers
- Create a time-sensitive notification — Android Developers