An AI security assessment is a structured review of how the AI systems an organization builds, buys, and uses can be attacked, misused, or made to leak data, and of the evidence that shows those risks are controlled. It starts with an inventory and ends with a ranked list of findings that someone owns. Anything less is a questionnaire, and anything that stops at a list of vulnerabilities is a scan.
The questions now arrive from three directions at once: customer security questionnaires, auditors working to ISO 42001 or SOC 2, and regulators applying rules such as the EU AI Act. Teams that answer each one separately build the same evidence three times. This guide explains what an assessment covers, how it differs from AI penetration testing and a governance review, the six steps to run one, and how to shape the output so a single assessment answers all three audiences.
What an AI Security Assessment Covers
The Four Areas
AI systems fail in places that a traditional application review does not look. A model can be manipulated through its inputs, a retrieval store can expose documents the user should never see, and an agent with broad tool access can act on an instruction nobody intended to give. An assessment therefore covers four areas, each with its own evidence.
| Area | What is examined | Typical evidence |
|---|---|---|
| AI assets and data flows | Every model, AI feature, and agent, what data enters and leaves each one, and who owns it | AI inventory, data flow diagrams, system owners |
| Models and applications | Prompt handling, output handling, tool and plugin permissions, retrieval stores, guardrails | Threat model, test results, configuration records |
| Infrastructure and access | Identity and access for models and pipelines, secrets, logging, monitoring, network exposure | Access reviews, log samples, architecture records |
| Governance and third parties | Policies, approval of new AI use, incident handling, vendor and model supplier controls | Policies, approval records, vendor assessments |
The table separates what an assessor looks at from what the organization can show. The second column is the work, and the third is the deliverable. A review that produces opinions without records leaves the organization unable to answer the next questionnaire or audit request without starting again.
Where Agents and Tool Access Change the Scope
Once a model can call tools, the assessment has to follow the permissions, not just the prompts. An agent that can read a mailbox, query a database, or call an internal API inherits every weakness of those connections. The review should list each tool an agent can reach, what it can do with it, and whether a human approves the action. How to Govern Autonomous AI Agents covers the governance side, and the guide to securing third-party MCP servers covers the connection layer.
What Falls Outside the Assessment
Employee use of public AI tools without approval is a separate problem, usually called shadow AI. It affects the assessment because it decides what the inventory is missing, but it is found through discovery and policy, as described in What Is Shadow AI? Risks and How to Govern It. The assessment should record that the gap exists and point to the process that closes it.
AI Security Assessment vs AI Penetration Testing vs Governance Review
Three activities get called an “AI security review” and answer different questions. Confusing them produces either an expensive test of a system nobody has inventoried or a polished policy set that has never been attacked.
| Aspect | AI security assessment | AI penetration testing | AI governance review |
|---|---|---|---|
| Question answered | Where are our AI systems exposed, and can we prove controls exist? | Can an attacker exploit a specific AI system? | Do we have the policies, roles, and approvals to manage AI? |
| Method | Inventory, threat modeling, control review, evidence sampling | Hands-on attempts to exploit prompts, tools, APIs, and data stores | Document review, interviews, gap analysis against a framework |
| Output | Ranked findings, owners, and an evidence map | Exploited weaknesses with reproduction steps and fixes | Gap list against a framework, and a roadmap |
| Best moment | Before an audit, a customer review, or a new AI launch | After the inventory is known and the system is stable enough to test | When building or maturing the management system |
The three are sequential, not interchangeable. The assessment tells the organization what to test and what to govern. Penetration testing of the highest-risk systems then confirms whether the weaknesses found on paper can be exploited. The governance work, covered under Elevate’s AI governance program, turns the findings into policies, owners, and review cycles that outlast the first assessment.
One practical consequence follows. A penetration test report is evidence for the assessment, and an assessment report is a poor substitute for a test. Teams that order a test first often discover that half the findings concern systems with no owner, which is an assessment finding in its own right.
Six Steps to Run an AI Security Assessment
The sequence matters because each step produces the input for the next. Skipping the inventory makes the threat model guesswork, and skipping the evidence step leaves a good assessment that cannot be shown to anyone.
Step 1: Build the AI Inventory and Name Owners
List every AI system the organization builds, buys, or embeds: models, AI features inside products, agents, and AI services from vendors. For each one, record the business purpose, the data it touches, the people or systems it connects to, and a named owner. An inventory without owners is a list, and findings against a system nobody owns do not get fixed. Expect the first pass to be incomplete, and mark the known gaps instead of hiding them.
Step 2: Classify Systems and Model the Threats
Rank each system by what a failure would cost: exposure of sensitive data, decisions that affect people, actions an agent can take without review, and exposure to the public. Then model how each high-ranked system could be attacked or misused. The OWASP list of LLM risks gives a practical starting vocabulary, and the OWASP LLM Top 10 guide walks through it. Systems that affect individuals also need an impact view, described in Key Considerations for Conducting an AI Impact Assessment.
Step 3: Test the Controls
For each high-ranked system, check that the controls the threat model calls for exist and work. Review how prompts and outputs are handled, what each tool and plugin is allowed to do, how retrieval stores enforce access, how secrets and model endpoints are protected, and whether logs would show misuse. Sample real records instead of accepting descriptions: an access review, a log extract, a configuration export. Where a weakness needs to be proven, a hands-on penetration test of that system is the right input, and its report becomes evidence.
Step 4: Score and Rank the Findings
Score each finding by likelihood and impact, weighted by the business context from Step 2. A prompt injection weakness in an internal summarizer with no tool access ranks below the same weakness in a customer-facing agent that can issue refunds. Keep the scoring method simple enough that two reviewers reach the same ranking. A short, defensible scale beats a detailed one nobody applies consistently.
Step 5: Assign Remediation and Track It
Every finding gets an owner, a target date, and a defined fix or an accepted risk with a named approver. Track them in a register that survives the assessment. This is where most assessments lose value: the report is delivered, the register is never updated, and the next review finds the same gaps. A register that shows findings opened, closed, and accepted is also the strongest evidence an auditor can ask for.
Step 6: Preserve the Evidence and Reassess
Store the inventory, threat models, test results, scored findings, and the remediation register in one place with dates. Set a review cadence and name the triggers that force an earlier one: a new model, a new tool or data source connected to an agent, a change in who can use the system, or a new regulatory duty. If the organization needs a second pair of eyes on scope or scoring, book a readiness call with an Elevate advisor.
Mapping Findings to Standards
The same finding usually answers questions from several frameworks. Tagging each finding once, at the time it is recorded, lets one assessment serve a customer questionnaire, an ISO 42001 audit, and a regulatory review. The table maps five common findings at theme level.
| Finding | OWASP Top 10 for LLM Applications | NIST AI RMF | ISO/IEC 42001 | EU AI Act |
|---|---|---|---|---|
| Manipulated prompts or unsafe outputs | Prompt injection, improper output handling | Measure, Manage | AI risk assessment and treatment, lifecycle verification and validation | Article 15 (robustness and cybersecurity), for high-risk systems |
| Sensitive data exposed through retrieval or outputs | Sensitive information disclosure, vector and embedding weaknesses | Map, Manage | Annex A data for AI systems | Article 10 (data governance), for high-risk systems |
| Agent or tool access broader than needed | Excessive agency | Govern, Manage | Annex A use of AI systems, roles and responsibilities | Article 14 (human oversight), for high-risk systems |
| Unvetted third-party model or component | Supply chain | Govern | Annex A third-party and customer relationships | Article 25 (responsibilities along the AI value chain) |
| No logs or monitoring of AI behavior | Not a single category | Measure, Manage | Operation and monitoring, performance evaluation | Article 12 (record-keeping), for high-risk systems |
Two cautions apply when using the table. First, the EU AI Act articles listed apply to systems classified as high-risk, so the legal classification has to be decided before the mapping means anything. [INTERNAL LINK: insert the URL of “ISO 42001 vs the EU AI Act: Where Certification Stops and Compliance Starts” once it is published] explains why a management system standard does not settle that question. Second, a mapping is a navigation aid. It does not mean that closing the finding satisfies the other framework, and each framework has its own evidence expectations.
Teams comparing the frameworks themselves can start with NIST AI RMF vs ISO 42001 for Teams Building AI Controls. For organizations that operate AI agents and want third-party assurance, AIUC-1 adds testing of the agent itself. The AIUC-1 readiness service covers preparation, and the assessment inventory and test records are the natural starting evidence.
Assess Third-Party AI Vendors
Most organizations use more AI than they build. A model API, an AI feature inside a business application, or a vendor agent all sit inside the assessment boundary, even though the organization controls only how it configures and uses them. The practical approach is to ask for evidence that can be checked, and to treat an answer without evidence as a finding.
| Question | Evidence to request | Red flag |
|---|---|---|
| What data does the vendor keep, and is it used to train models? | Contract terms, data processing terms, retention and deletion procedure | Silence on training use, or terms that can change without notice |
| Which models and subprocessors sit behind the service? | Current list, with a commitment to notify changes | No list, or a list that is only available on request after signing |
| What independent assurance exists, and what does it cover? | Certificate or report, with its scope statement and date | A certificate whose scope excludes the product being bought |
| Has the AI system itself been tested? | Summary of recent test results and fixes | Only general company security reports, with no AI-specific testing |
| How is the buyer told about incidents and model changes? | Contractual notification terms and a named contact | No commitment on timing, or notification only through a public status page |
The red flag column carries the most weight. A vendor certificate matters only for what it covers, so the scope statement and the date should be read before the logo is counted. ISO 42001 Vendor Governance: Managing AI Model Suppliers and Third-Party Risk goes deeper on the supplier controls an AI management system expects.
Record the vendor review in the same register as internal findings. A vendor that refuses to answer is not a blocker by itself, but it should appear as an accepted risk with a named approver, not as an item that quietly disappears.
What the Deliverable Should Contain
The deliverable decides whether the assessment gets reused. A report that reads well but cannot be traced to records fails the first audit request. A report built from the structure below can be handed to an auditor, trimmed for a customer, and updated at the next review.
- Scope and method. The systems in scope, the systems left out and why, the dates, and how findings were scored.
- AI inventory. Each system with its purpose, data, connections, and owner.
- System risk ranking. The classification from Step 2, with the reasoning stated.
- Findings register. For each finding: the system, a description, the score, framework tags, an owner, a due date, and a status.
- Evidence index. A list of the records that support each conclusion, with dates and where they are stored.
- Accepted risks. Findings the organization chose not to fix, with the approver and the review date.
- Reassessment triggers. The changes that force an earlier review, and the regular cadence.
Two habits keep the document useful. First, separate the version shared outside the organization from the working register. A customer needs the scope, the method, and a summary of how risks are managed, not reproduction steps for open weaknesses. Second, date everything. An assessment is a statement about a point in time, and a register with no dates cannot show that anything was ever fixed.
Conclusion
An AI security assessment earns its cost when it produces records, not opinions. The inventory shows what exists, the threat model shows what matters, the tests show whether controls work, and the register shows what was done about the findings. Each of those artifacts answers a customer questionnaire, an audit request, and a regulatory question without being rebuilt from scratch.
The common failure is sequence. Organizations test before they inventory, write policies before they know which systems they cover, or deliver a report that nobody owns. Following the six steps in order avoids all three. To walk through scope and scoring with an Elevate advisor before starting, book a readiness call.
Key Takeaways
The points below summarize how to run an AI security assessment that holds up under review.
- Start with the inventory and owners. Findings against a system nobody owns do not get fixed, and a threat model built on an incomplete list is guesswork.
- Assessment, penetration test, and governance review are different tools. The assessment decides what to test and govern, the test proves exploitability, and the governance work makes the fixes last.
- Follow the permissions, not only the prompts. Once a model can call tools, the tools and their access define the real exposure.
- Tag each finding once, for every framework. A single tagged register serves OWASP, NIST AI RMF, ISO 42001, and the EU AI Act, but a mapping never proves that closing a finding satisfies another framework.
- Vendor answers need evidence. Read the scope and date of any certificate, and record vendors that do not answer as accepted risks with an approver.
- Date everything and set triggers. An assessment describes a point in time, so name the changes that force an earlier review.
Frequently Asked Questions
What is an AI security assessment?
An AI security assessment is a structured review of how an organization’s AI systems can be attacked, misused, or made to leak data, together with the evidence that controls are in place. It covers AI assets and data flows, models and applications, infrastructure and access, and governance and third parties. The output is a ranked register of findings with owners, plus the records that support each conclusion. It is broader than a vulnerability scan and different from a penetration test.
How is an AI security assessment different from AI penetration testing?
An assessment asks where AI systems are exposed and whether controls can be proven, using inventory, threat modeling, control review, and evidence sampling. Penetration testing asks whether an attacker can actually exploit a specific system, and it produces reproduction steps and fixes. The two work in sequence: the assessment decides which systems to test, and the test results become evidence inside the assessment. Ordering a test before an inventory often finds issues in systems nobody owns.
How often should an organization run an AI security assessment?
On a set cadence and whenever something material changes. Annual review is a common cadence, but the triggers matter more than the calendar. A new model, a new tool or data source connected to an agent, a change in who can use a system, or a new regulatory duty should each force an earlier review. Dated records of each review are what show an auditor that the process runs continuously.
Can an organization run an AI security assessment internally?
Yes, if the people running it understand both AI-specific attacks and the organization’s control environment, and if they are independent of the teams that own the systems. Internal teams are strong on context and inventory, while outside reviewers add independence and comparison across organizations. Many teams run the inventory and scoring internally and bring in outside testing for the highest-risk systems. Whichever route is used, the evidence has to be dated and stored.
How does an AI security assessment support ISO 42001, the EU AI Act, and AIUC-1?
The assessment produces the records each of them asks for: an inventory, a documented risk process, test results, and a register of treated and accepted risks. ISO 42001 expects a documented AI risk assessment and treatment process. The EU AI Act sets security and oversight duties for high-risk systems, and legal classification has to be decided separately. AIUC-1 adds testing of AI agents by independent parties. Tagging each finding to the relevant framework lets one assessment feed all three, though closing a finding does not by itself satisfy another framework.