Building AI Apps Inside Organisations: Security Beyond the Demo

A demo proves that an app can complete a task. Production asks a harder question: can it complete only the tasks it is allowed to, using only the data it should see—even when an input is hostile, a credential is stolen or a dependency changes? AI makes that question more urgent because retrieved text can influence execution and generated output can become an action.

For application engineers, platform/DevOps teams and security reviewers building organisational web apps, low-code tools and AI-enabled workflows. The argument runs from recent threat evidence to enforceable controls, platform choices and release evidence. The aim is to help you decide where to place a control, what technology can support it and how to prove it works.

What recent threats change for app builders

These selected reports, including September–October 2026 publications, show different entry points into an organisation’s systems. They are not a prevalence ranking. The engineering implications below are my synthesis of the documented findings.

Selected disclosures, observed campaigns and evaluation findings; sources reviewed 16 October 2026.
Evidence and date	Documented finding	Engineering implication
Unintended agent actions Published 9 October 2026	Anthropic reported evaluation and internal-use cases involving external command execution, unintended form submission and workarounds to tool restrictions. These were reported model behaviours, not a customer breach campaign.54	Enforce tool permissions outside the model. Deny unnecessary outbound access, isolate evaluation environments, validate final destinations after redirects and require approval for consequential submissions.
EvilTokens device-code phishing Published 22 September 2026	Microsoft documented attackers abusing device-code authentication to obtain tokens, access mailboxes and establish persistence; AI supported lure creation and post-compromise analysis.55	Disable unused authentication flows at the identity provider; narrowly scope required exceptions. Monitor token use and new device registrations. Test containment against still-valid access tokens, not only refresh-token revocation.
Agent-enabled credential theft Published 8 September 2026	GTIG described an observed Q2 2026 campaign using AI-agent workflows after a cloud resource was compromised.4	Reduce the time between exposure and containment: scoped identities, outbound restrictions, actionable alerts and tested credential revocation matter as much as model filters.
AI packages and CI/CD as entry points Published 30 July 2026	GTIG/Mandiant described February–May 2026 compromises across PyPI, npm and Docker Hub, including abuse of GitHub Actions pull_request_target and attempts to pivot from compromised AI software.25	Never execute untrusted pull-request code in a privileged workflow. Use minimal workflow permissions and short-lived workload credentials; isolate package installation and AI developer tools from production secrets and networks.
Axios npm compromise 31 March 2026	GTIG reported malicious dependencies in Axios releases 1.14.1 and 0.30.4, delivered through a compromised maintainer account.2	Treat installs and build jobs as execution environments. Review dependency changes, restrict runner credentials and egress, and retain package provenance. A lockfile can pin a malicious release too.
React Server Components 3 December 2025	The React Team disclosed unauthenticated remote code execution in affected Server Components packages, CVE-2025-55182.1	An internal-looking frontend may still expose server execution paths. Inventory deployed framework versions, follow current advisories and verify the rebuilt production artefact—not only the manifest.
Salesloft Drift integrations August 2025	GTIG/Mandiant observed stolen OAuth tokens used to access connected Salesforce instances and export data; the compromise also affected other integrations.3	Review connector grants and effective scopes. A compromised integration can bypass the employee’s interactive login; inventory tokens, monitor their use and rehearse revocation.

The React entry is a vulnerability disclosure; the October agent entry reports evaluation/internal-use behaviours; the remaining entries describe observed attacks or campaigns. Publication dates are labelled separately from activity dates. None establishes that every internal app is affected. Check the affected technology, version, exposure and configuration.

The design lesson: the entry points differ, but the engineering question repeats: what authority does a stolen token, installed package or agent action inherit? Review the identity, data and network reach available after that first failure. That is where an isolated problem can become an organisational incident.

The technical controls to design into the app

Start with a data-flow and identity diagram: browser, API, background workers, storage, model services, third-party connectors and client-facing delivery. Mark where identity, data sensitivity or execution authority changes. Include caches, indexes, logs and backups.

1. Identity, access and untrusted input

Establish who is acting and what they may reach before processing content. Frontend behaviour supports the experience; APIs and workers enforce the permissions.

1. Identity, access and untrusted input · threat, implementation and verification.
Layer	Threat	Controls and technology choices	Verification
Identity and sessions	Stolen sessions, forged tokens or shared service credentials.	Use the organisation’s identity provider through OIDC/SAML as appropriate; require MFA according to risk. Validate issuer, audience, signature and expiry server-side. Use protected cookies for cookie sessions and CSRF protection for state changes.5 6	Reject expired/wrong-audience tokens; test logout, role changes and service-account scope.
API and object access	Cross-team file access, tenant leakage and unauthorised edits or sharing.	Authorise every object/action in APIs and jobs. Bind organisation/workspace context to verified identity. Consider an authorisation service or database row-level security; ensure the application role cannot bypass it.7 8 9 10	Change file/workspace IDs; test reader/editor/client roles, revoked membership and asynchronous jobs.
Frontend and previews	Stored XSS in uploaded content or generated HTML; browser-exposed secrets.	Use context-appropriate encoding, safe rendering and sanitisation for permitted rich content. Keep secrets out of browser bundles. Hidden controls and CORS are not object-level authorisation.11 5	Exercise hostile HTML/links in previews; inspect bundles and direct API calls.
Uploads and document workers	Malware, parser exploitation, oversized files and decompression/resource exhaustion.	Allow required formats; validate signatures, size and content. Quarantine/scan uploads before processing. Isolate parsers with limited privileges, time, storage and network access.12	Use malformed, disguised and oversized test files; verify worker limits and inaccessible quarantined files.

2. AI context, actions and data

Retrieval and execution need separate boundaries. Permission to read a source does not automatically grant permission to share a generated result or call a tool.

2. AI context, actions and data · threat, implementation and verification.
Layer	Threat	Controls and technology choices	Verification
AI retrieval and generation	Cross-user retrieval, indirect prompt injection, unsupported claims and sensitive output.	Filter before content reaches the model; preserve source permissions in chunks and indexes. Separate untrusted content from instructions. Evaluate output grounding and leakage. Classifiers supplement access controls; they do not enforce ownership.13 14 15	Inspect retrieved context, generated text and citations under different identities. Test source-permission changes, poisoned documents and missing evidence.
Agent actions and delivery	Tool misuse, SSRF, unauthorised publishing or sharing with the wrong client.	Expose narrow tools; validate arguments and destinations outside the model. Recheck permission at execution. Bind sharing approval to exact content, audience and expiry; use replay protection and idempotent execution.16 17 18	Test changed recipients/content, expired approvals, duplicate retries and attempts to contact blocked destinations.
Storage and processing location	Public storage, uncontrolled copies or processing outside the approved geography.	Encrypt data, restrict key access and define retention across files, indexes, outputs and backups. Verify processing routes and subprocessors separately from storage location; deployment type can change routing.19 20 21	Check effective policies, routes, deletion/restore behaviour and encryption/key settings for each service.

3. Infrastructure, delivery and response

The same boundaries must survive deployment and operation. A secure application can still inherit excessive authority from its network, build runner or service identity.

3. Infrastructure, delivery and response · threat, implementation and verification.
Layer	Threat	Controls and technology choices	Verification
Network and service boundaries	Exposed databases/admin endpoints, unrestricted outbound access and lateral movement.	Use TLS, network segmentation, private connectivity where appropriate and controlled egress. Restrict destinations at the network and application layers; test DNS/routing and service identities.22 23 18	Test public access, unexpected service-to-service calls and outbound attempts. A private endpoint alone does not enforce user permissions.
Dependencies and CI/CD	Malicious packages/actions, leaked build secrets and unsafe infrastructure changes.	Review dependency and pipeline changes; restrict install hooks where feasible. Use isolated runners, short-lived deployment identities and protected releases. Scan code, dependencies, images, secrets and infrastructure configuration.24 25 26	Check the deployed artefact and pipeline permissions; test a blocked release, rollback and credential revocation.
Operations and incident response	Undetected data extraction, excessive AI spend or an app nobody can disable.	Log security-relevant actions without retaining sensitive content by default. Set rate/usage limits and alerts for unusual exports, tool calls and credential use. Assign incident ownership and a way to disable connectors or sharing.27 3	Exercise an investigation, tool shutdown, restore and secret rotation. Check alerts reach someone able to act.

Sovereign AI: who controls the service behind the app?

Application permissions determine what a user or agent can do. Sovereignty adds a different question: who has authority over the infrastructure, models and operations the app depends on? A region-restricted deployment can still depend on external administrators, update services or control planes.

How major providers frame it. NVIDIA describes sovereign AI at national scale: domestic infrastructure, data, skills and business ecosystems.60 IBM defines AI sovereignty as an organisation’s or nation’s ability to control its AI stack, including infrastructure, data, models and operations.61 Microsoft organises AI workload guidance around data, operational and technological objectives across the lifecycle.58 AWS emphasises data residency and operator-access restrictions alongside resilience, independence and choice across the stack.59

These published positions overlap on control but differ in scope; they are not an independent benchmark or evidence of how all enterprises operate. Here, the relevant scope is the organisation’s workload: translate its data, supplier-access and continuity requirements into architecture decisions. This complements the security controls above by testing whether the platform and supplier arrangements can uphold them.

Turn sovereignty requirements into deployment decisions and evidence.
Consideration	Establish now	Verify in the app and its services
Data and processing boundary	Which information must stay in approved locations, including prompts, files, embeddings, outputs, telemetry and backups?	Check storage, inference and embedding endpoints, cross-region failover, support logs and subprocessors. Enforce approved routes and destinations in deployment policies and network controls.
Operator access and authority	Who may administer or support the workload, from which locations and under which contractual or jurisdictional requirements?	Review provider commitments with accountable teams; restrict privileged roles, require time-limited access and audit support/admin activity.
Keys, models and dependencies	Who controls encryption keys, model versions, updates and connected services?	Check actual key custody and access, model provenance, update approvals and external calls. Customer-managed keys alone do not establish control over every processing path.
Continuity and exit	What must keep working if a provider, external connection or model is unavailable?	Test degraded operation, recovery and export/redeployment options; include external identity, model licensing, registries and control-plane dependencies.

When this changes the design. If approved-region processing is the requirement, verify service routing and failover. If restrictions also cover provider administration or operation during an external outage, assess operator access and control-plane dependencies; a regional endpoint alone does not answer those questions. Compare managed regional, sovereign-cloud and self-hosted options against the actual requirements and operational costs. The outcome should be a documented boundary, its evidence and accepted dependencies—not a “sovereign” product label. A national AI strategy is beyond this article’s scope.

Choose capabilities before products

Once the control and its enforcement point are clear, choose the technology. A source scanner examines code; an AI evaluation framework exercises model and tool behaviour; a runtime filter inspects live content. None replaces the backend’s object permissions. Start with the capabilities already available in the organisation, then fill the gaps below.

Examples of available approaches; verify edition, coverage, integration and data-processing terms.
Capability	Technology examples	Selection criteria
Source and dependency analysis	CodeQL, Semgrep, Snyk Code/Open Source, Trivy; AI-assisted vulnerability review such as Claude Security (public beta for Enterprise; a separate Claude Code plugin is also in beta).28 29 30 31 32 33	Supported languages and artefacts, deployment model, actionable findings and testable fixes. Static analysis cannot prove business permissions.
Running web/API testing	ZAP and Burp Scanner.34 35	Authenticated route coverage and repeatable tests. Passive baseline scans and active testing are different modes.
AI evaluation and red teaming	Promptfoo, Microsoft PyRIT and specialist services such as Mindgard.36 37 38 39	End-to-end access to retrieval/tools, realistic attack cases, reproducible findings and visibility into side effects.
Runtime screening and agent governance	Check Point AI Guardrails, Prisma AIRS, Model Armor and Zenity; platform policies such as Copilot Studio data policies.40 41 42 43 44	Where enforcement occurs, monitored versus blocked actions, connector coverage, false positives, latency and treatment of inspected data.
Authorisation and cloud exposure	Application policies or OpenFGA for object access; database row-level policies; cloud controls and posture platforms such as Wiz.9 10 45	Correct identity propagation, revocation, bypass paths and effective deployed configuration. These are separate capabilities, not substitutes.

For instance: managed platform features and their limits

Azure AI Search and Bedrock Knowledge Bases provide retrieval patterns; Foundry Prompt Shields, Bedrock Guardrails and Model Armor provide content controls. AgentCore Identity illustrates credential management for agents. Check the actual identity, permission filtering and enforcement path; native Azure AI Search ACL/RBAC features remain in preview, while security-string filters trust identity supplied by your app. Bedrock Managed Knowledge Base ACL filtering also relies on application-verified identity; it does not authenticate end users or replace application authorisation. Bedrock’s prompt-attack filter does not assess tool results, so check those paths separately.14 46 15 47 48 49 42 50

Cloud-specific options are implementation examples. The requirements—authorised access, constrained actions, approved processing locations and observable execution—remain the same across providers.

What should be required before production?

Turn the design into evidence someone else can inspect. For each important boundary, name its owner, the test that challenges it and the result from the deployed environment. Use the six areas below to organise that review.

Architecture and ownershipNamed owners, data flows, trust boundaries, deployment locations and an inventory of third-party grants.

Automated checksAccess-control tests plus relevant code, dependency, image, secret and configuration scans on changes.

Adversarial verificationTest the deployed frontend/API and AI path with different roles, poisoned inputs, revoked permissions and retries.

Release evidenceRecord tested versions, confirmed findings, fixes and any explicitly accepted, time-limited exceptions.

Operational readinessAlerts, rate/cost limits, support ownership, recovery procedures and tested connector/tool shutdown.

Change reviewReassess new sources, models, connectors, client access, network exposure and write capabilities.

Use OWASP ASVS 5.0.0 for application requirements, the OWASP LLM Top 10 2026 and Agentic Top 10 2026 with MITRE ATLAS for AI threat coverage, and the final NIST SSDF 1.1 for secure delivery practices. Convert the relevant requirements into verifiable checks; a framework badge or clean scanner result is not release evidence on its own.51 13 52 53 26

Also track the NIST SSDF 1.2 initial public draft; it has not replaced the final 1.1 baseline. For agent runtime integration, OWASP’s emerging Agent Control Standard defines hooks for policy enforcement and observability. Assess implementation maturity, bypass paths and failure behaviour before relying on it.56 57

Security verification also needs the organisation’s data, privacy, contractual and supplier decisions. Confirm permitted uses and processing arrangements with the accountable teams; technical tests alone do not establish compliance.

Make the security boundary enforceable

A production app is a chain of permissions: who can enter, which data they can reach, what code or AI can execute, and where the result can go. A failure at any link can expose the whole workflow. Before release, require evidence that the backend rejects unauthorised access, workers and agents cannot exceed their scope, the deployed artefact is traceable, and the team can revoke access and contain an incident. If those controls depend on a hidden button, a model following instructions or someone remembering a manual step, the boundary still needs engineering. Tools and threats will change; the organisation must be able to explain, test and enforce what its app is allowed to do.

Sources and research notes · 61 references

Source destinations and factual claims rechecked 16 October 2026. Incident reports, product documentation, framework status and sovereignty definitions were reviewed against the statements made here. Current guidance uses the 2026 OWASP LLM and Agentic editions and ASVS 5.0.0. NIST SSDF 1.1 remains the final baseline; 1.2 is labelled as a draft. Original incident reports and the detailed 2025 Excessive Agency entry are retained as supporting evidence, not presented as the latest framework edition. Product documentation is maintained by its publishers; confirm API versions, availability and limitations when implementing. Incident facts are separated from engineering recommendations. This is a selected technical review, not a complete threat feed or independent product benchmark.

More references · 51 additional sources

Share this article

LinkedIn Twitter