AI Audit Checklist: Re-test Your Publisher After Changes
Use an AI audit checklist to retest content, permissions and delivery after workflow changes, compare configurations and decide what can safely resume.
An AI audit checklist for a changed publishing workflow should establish what changed, rerun the checks that change could affect, and compare the proposed configuration with the last verified one before resuming its authorized work. Do not carry an old approval forward simply because the new workflow still produces readable articles.
This is regression testing: checking whether a change has broken behaviour you previously relied on. For an agency or marketing team using AI-assisted publishing, the decision is whether the changed workflow can keep its existing responsibilities, needs a narrower role, or should remain on hold.
1. Describe the change before choosing the tests
OWASP recommends structured security testing after material changes to prompts, tools, memory, retrieval, policies or model providers. For a content team, that means a new writing model and a reconnected publishing account deserve different checks.
Record the previous and proposed configurations, the change owner and the work affected. Include the model version where available, editorial instructions, source collection, destination, permissions and approval policy. If a supplier does not expose a version, record that limitation and the date of the observed configuration instead of inventing one.
Use this selection matrix as a recommended starting point. It maps a change to an observable test, not a universal certification standard.
What changed?
What to compare or test
Evidence to retain
Writing model or editorial instructions
Use the same brief and sources. Compare factual meaning, exclusions and the reader's answer, not identical wording.
Both outputs and the specific claim or requirement affected.
Source retrieval or retained context
Check which source version supplies the answer and whether a previous client's facts appear in another client's draft.
Source passages, client scope and resulting claims.
Publishing connection or account permissions
Check an allowed action and have the connection owner test prohibited actions at the receiving system.
Saved destination state and authorization responses.
Approval rules or destination selection
Check whether changing the target invalidates approval tied to the original target.
Approved target, attempted target and receiving-system decision.
Retry or recovery behaviour
In an isolated test, repeat a request after a deliberately uncertain response and inspect whether the same intended article is duplicated.
Request history and the resulting destination articles.
When several components change together, test their combined workflow too. A writing-only comparison cannot establish that the new publishing connection enforces the right boundary.
2. Freeze the input and define the expected result
Build a reusable test pack: a fictional brief, approved source passages, an intended destination and explicit expected behaviour. Include an ordinary successful task, a missing or conflicting fact, and a request outside the agreed permission. Add a test for a previously observed failure when relevant.
OpenAI's evaluation guidance recommends task-specific evaluations, typical and adversarial cases, and continued evaluation as systems change. It also notes that generative AI can produce different outputs from the same input. Compare whether the output meets the requirement rather than whether it repeats the old wording.
Write the expectation before examining the candidate output. For example: “The source says onboarding assistance is available on request. The article must not turn that into onboarding included with every subscription.” This gives the reviewer a concrete distinction between a stylistic variation and a factual regression.
Keep reusable cases stable while adding new ones. If you correct the expected answer or replace a source, record the change and rerun the comparison. Otherwise, you are comparing different tests as well as different configurations.
3. Compare content and action boundaries separately
Have the responsible technical owner arrange an isolated, non-public environment with fictional clients, disposable pages and no production credentials or confidential data. Agree which actions the test environment can safely exercise. If the supplier cannot provide observable results for an essential boundary, leave that boundary unverified.
Run the reference configuration where available. Retain its actual output and destination state, rather than relying on a remembered success. If the old configuration is unavailable, use retained evidence but label the comparison as historical, not a fresh paired test.
Run the candidate on the same input. Preserve its full output, retrieved source evidence and delivery result. Record intentional configuration differences.
Review the answer. Check whether material claims, exceptions and client identity remain correct. A smoother introduction does not compensate for a changed product promise.
Inspect the action. Ask the connection owner to confirm the destination's response and resulting state. Separate an agent refusing a request from the receiving system rejecting it.
Record each outcome. Use passed, failed or unverified for the defined requirement. Keep content findings distinct from permission and delivery findings.
For approval changes, test the binding between permission and target. OWASP says authorization should be enforced outside the agent's context, with approval tied to the actor and exact action; changed targets or parameters require new approval. In a publishing test, ask the technical owner to verify that approval for one disposable page cannot authorize a different page.
Do not reduce these outcomes to an average score. A successful article draft does not cancel an unauthorized write. Equally, a correctly rejected action does not establish that the article's factual claims are sound.
4. Turn the comparison into a scoped decision
The following fictional example shows how an agency could document a model change for an English-language content workflow serving Sweden. The results are invented to illustrate the decision, not evidence of a completed test or a particular product's capabilities.
Change: Replace the writing model; keep the brief, source pack, destination and draft-only policy unchanged.
Reference: Configuration A, with retained source pack and test output.
Candidate: Configuration B, using the same test input.
Expected claim: Onboarding assistance is available on request.
Reference output: Preserves that condition.
Candidate output: Says onboarding is included with every subscription.
Destination observation: Both test articles are saved as drafts in the intended fictional client's test account.
Permission tests: Not run in this comparison; no renewed permission conclusion.
Decision: Hold the model change because the candidate fails the material-claim requirement.
Owner and next action: The workflow owner investigates the instruction or model behaviour, then reruns the unchanged claim test and the remaining required cases.
Production state: Candidate not authorized for production use.
The useful result is a specific blocked change and a specific retest, not “AI audit failed.” If both versions make the same unsupported claim, repair the existing baseline as well. Matching the previous output is not enough when the previous output is wrong.
5. Resume only the work the evidence supports
Use the comparison to choose among these operating decisions:
Resume the existing scope when required tests support the changed configuration and the responsible owner authorizes its use.
Restrict the role when a narrower workflow is acceptable under your policy and its own controls have been verified. “Draft only” must be an actual enforced boundary, not just a label in the handoff.
Hold the change when a material requirement fails or an essential observation is unavailable. Identify the owner and evidence needed to resolve it.
If returning to a previous configuration is proposed, ask the technical owner to confirm that it is still available, appropriately scoped and suitable for current sources. Do not assume every publishing service offers rollback.
Keep the test decision separate from production verification. A successful isolated test does not establish that production has the same permissions or that an article is live. Confirm the deployed configuration and inspect the resulting destination under the customer's agreed operating policy.
For your next workflow change, write the expected behaviour and affected destination before accepting the upgrade. That turns “it still looks fine” into a decision someone else can inspect.
Photo by Jakub Zerdzicki — Pexels — Pexels License
