Healthcare AI’s next design problem: permission
Private inference settles where a model runs. The operating design must settle which patient, which purpose, which action—and who can stop it.
Editorial image for Healthcare AI’s next design problem: permissionVIBECIO Editorial Desk ·
Imagine a referral assistant running entirely inside a healthcare organization’s network. It retrieves a record, prepares a referral packet, and places it in a staff member’s queue. The model never calls an external provider. Yet the assistant can still retrieve the wrong patient’s information, expose a document to an unauthorized employee, or send a packet after the underlying order has changed.
That hypothetical exposes a gap in the phrase private AI. Private inference describes a deployment choice. Safe operation depends on the complete path from a person’s request to the information retrieved, the proposed action, and the system that commits it. Every step needs a defined authority.
Start with the work and its legal context
HHS cloud guidance permits covered entities and business associates to use cloud services for electronic protected health information when the applicable HIPAA requirements are met, including a business associate agreement where required. A provider maintaining encrypted ePHI can still be a business associate even without the decryption key. The guidance also calls for understanding the environment and conducting risk analysis.
An internal model and a contracted cloud service therefore present different operational arrangements, rather than a simple compliant-or-noncompliant divide. The design review must follow the actual data: retrieval stores, temporary files, traces, evaluation sets, backups, and downstream tools. A private inference endpoint cannot speak for every service around it.
The purpose of access matters as well. HHS describes a general minimum-necessary standard for limiting applicable uses, disclosures, and requests for PHI, while identifying exceptions, including disclosures to or requests by a healthcare provider for treatment. Teams should map the actual workflow with their privacy function instead of turning a simplified slogan into an access policy.
Keep authorization outside the conversation
For the referral assistant, the application should establish the staff identity, the patient context, and the permitted operation before supplying information to the model. If retrieval uses a broad service account, the application still needs to enforce the requester’s restrictions. A relevant search result is not automatically an authorized one.
SMART on FHIR uses OAuth scopes to request delegated access to resources and operations. The specification explicitly retains underlying system permissions: a granted scope does not guarantee that every apparently matching operation will be allowed. This matters when an agent can read, search, create, or update through the same integration. The application must preserve the distinction between these capabilities.
A prompt telling the agent to respect those distinctions is insufficient. OWASP’s analysis of system-prompt leakage identifies the deeper problem when applications delegate authorization checks to the model. A practical design places those checks in the retrieval and action services. The model can propose a packet; a separate service decides whether the current actor may send this packet for this patient now.
That final check is especially useful when work spans several minutes or a queue. Permissions, destinations, and source records may change between drafting and execution. Binding a proposed action to its record version and checking the current state before committing makes the decision inspectable. An unexpected change can return the work for review.
Give the reviewer something worth reviewing
For this example, a reviewer should see the proposed recipient, the documents being sent, and the source of material facts. Missing information and conflicting records need to remain visible. If the interface presents only polished prose and an approve button, the reviewer must reconstruct the evidence or trust the fluency of the output.
The level of oversight should follow the action and its consequences. An internal administrative draft, a message sent to a patient, and a clinical recommendation deserve distinct release decisions and professional review. These are architectural recommendations, not a claim that one approval pattern satisfies every healthcare obligation. The service owner must define which work may proceed automatically and which requires a qualified person.
Release the failure behavior too
NIST’s Generative AI Profile treats evaluation and ongoing monitoring as part of risk management, with attention to override, recovery, incident response, and change management. Applied here, release evidence should show what happens when access is denied, the record is stale, a document contains hostile instructions, or a downstream service times out. A successful ordinary case answers only part of the operating question.
The organization also needs a way to stop automated actions while preserving a workable staff process. Record the model, retrieval configuration, policy version, and action outcome needed to investigate an incident; protect those records as carefully as the data they may contain. The durable advantage of a private AI platform is the organization’s ability to explain and control its behavior throughout that lifecycle.
Sources