Retrieval-augmented generation introduces compliance challenges spanning data governance, version control, cybersecurity, and post-market monitoring that the base GPAI integration guidance does not fully address. The knowledge base is a compliance-critical data asset under Article 10 principles, and the retrieval pipeline is a decision-making component requiring dedicated documentation in the AISDP.
Retrieval-augmented generation introduces compliance challenges that standard GPAI integration guidance does not fully address.
Retrieval-augmented generation introduces compliance challenges that standard GPAI integration guidance does not fully address. A RAG system is not simply a model with a search function attached. The knowledge base is a governed data asset with its own lifecycle. The retrieval pipeline is a decision-making component that determines which information the model sees. The interaction between retrieved context and model behaviour creates emergent properties that neither the knowledge base nor the model exhibits in isolation.
The fundamental regulatory question is the status of the knowledge base under the EU AI Act. Article 10 governs training, validation, and testing data. Retrieved documents are none of these in the technical sense; they are inference-time context. Yet they shape the model's outputs as profoundly as training data does. A GPAI model that produces accurate, unbiased outputs on its own may produce inaccurate or biased outputs when provided with a knowledge base containing outdated, incomplete, or skewed information.
The knowledge base is, functionally, a compliance-critical data asset that must be governed with the same rigour as training data, even if the Article 10 label does not apply by strict textual interpretation. The compliance implications span data governance, version control, cybersecurity, human oversight, and post-market monitoring simultaneously. The outputs feed into Modules 3, 4, 5, 6, 8, 9, 10, and 12.
Not necessarily. Organisations define a materiality threshold. Routine updates (document additions within existing domains, factual corrections, outdated document removal) are operational maintenance. Changes above the threshold (new domains, demographic coverage shifts, source distribution changes exceeding defined divergence metrics) trigger the risk gate and change classification logic.
Grounding verification checks whether each claim in a RAG system's output is supported by the retrieved documents. Three approaches exist in increasing sophistication: citation verification, entailment checking via NLI models, and fact extraction and matching. For high-risk systems, ungrounded outputs are a safety and fundamental rights risk, not merely an accuracy problem.
Four defence layers are required: input sanitisation screening documents for injection patterns, content isolation protecting the system prompt from override, output validation checking against expected behaviour boundaries, and retrieval filtering excluding flagged documents. Red-teaming should test all document ingestion pathways.
Knowledge base bias is distinct from model parametric bias. A model with no detectable bias can still produce biased outputs if the knowledge base underrepresents certain perspectives, demographics, geographic regions, or languages. The Technical SME must conduct a representativeness analysis documented in AISDP Module 4.
The knowledge base is a governed data asset, the retrieval pipeline is a decision-making component, and the interaction between retrieved context and model behaviour creates emergent compliance properties not covered by base GPAI guidance.
Retrieved documents are inference-time context, not training data under Article 10. However, they shape outputs as profoundly as training data and must be governed with the same rigour.
Every document needs documented provenance (source, date, licence, conditions). Automated copyright screening checks against known databases and robots.txt. Documents failing screening are quarantined for legal review.
A verification layer checks whether each output claim is supported by retrieved documents, using citation verification, entailment checking, or fact extraction. Outputs below the grounding threshold are flagged, held, or suppressed.
Indirect prompt injection through the knowledge base allows adversaries to embed instructions in documents. Defence requires input sanitisation, content isolation, output validation, and retrieval filtering across four layers.
Organisations define a materiality threshold. Routine updates are operational maintenance; changes exceeding the threshold (new domains, demographic coverage shifts, source distribution changes) trigger the risk gate and change classification.
Article 10 applies to training, validation, and testing data, but retrieved documents occupy an interpretive grey area as inference-time context.
Article 10 applies to training, validation, and testing data, but retrieved documents occupy an interpretive grey area as inference-time context. Each characteristic of a RAG system has compliance implications that the base GPAI integration guidance does not fully address.
The knowledge base shapes outputs as decisively as training data. A model that performs well in isolation may produce harmful results when paired with a biased or outdated document collection. Regulators are likely to treat the knowledge base as a compliance-critical asset regardless of whether Article 10 formally applies by strict textual reading.
Organisations should govern the knowledge base with the same rigour applied to training data. This means documented provenance, version control, bias assessment, and ongoing currency monitoring. The practical consequence is that knowledge base governance must be woven into every stage of the compliance lifecycle, not treated as a peripheral concern.
Every document in the knowledge base must have documented provenance covering the source, acquisition date, licence or permission, and conditions attached to use.
Every document in the knowledge base must have documented provenance covering the source, acquisition date, licence or permission, and conditions attached to use. Copyright exposure is significant: a RAG system that retrieves and reproduces copyrighted content in its outputs may expose the deploying organisation to infringement claims distinct from the GPAI model provider's own obligations under Article 53.
The Technical SME maintains a knowledge base catalogue recording, for each document or document source: the original author or publisher; the acquisition date; the licence type (proprietary, Creative Commons, public domain, contractual licence); any usage restrictions; the expiry date if the licence is time-limited; and the last review date.
Automated copyright screening. For knowledge bases assembled from web-scraped or publicly available sources, automated screening reduces the risk of inadvertent infringement. The screening checks each document against known copyright databases, robots.txt restrictions, and the organisation's internal list of prohibited sources. Documents that fail are quarantined pending manual review by the Legal and Regulatory Advisor.
Completeness assessment. The Technical SME defines, before deployment, the domains and topics the knowledge base must cover. The completeness assessment maps each required domain to the documents that address it, identifies gaps, and documents the remediation plan. The assessment is recorded in AISDP Module 4.
Currency monitoring. A knowledge base that is incomplete or outdated produces outputs that are incomplete or outdated. For high-risk systems, this is a compliance risk: outputs may not reflect the current state of the domain, leading to decisions that are wrong in ways performance metrics do not capture. Currency monitoring tracks the age of each document, update frequency in the source domain, external signals indicating a document may be outdated (new legislation, published corrections, superseding publications), and user feedback indicating outdated responses.
Knowledge base versioning. The knowledge base is versioned as a data asset, following data versioning principles. Each version captures the complete set of documents, their metadata, and the embedding vectors generated from them. Version changes are tracked in AISDP Module 10 and evaluated against the substantial modification criteria.
A knowledge base assembled from sources that underrepresent certain perspectives, demographics, or viewpoints will produce outputs reflecting those gaps.
A knowledge base assembled from sources that underrepresent certain perspectives, demographics, or viewpoints will produce outputs reflecting those gaps. The fairness assessment for a RAG system must address not only the GPAI model's parametric biases but also the representational biases in the knowledge base.
The Technical SME conducts a representativeness analysis assessing whether the document collection adequately covers: all geographic regions relevant to the system's deployment; all demographic groups within the affected population; all perspectives relevant to the domain, including dissenting or minority viewpoints where outputs may influence decisions affecting diverse groups; and all languages in which the system operates.
The analysis is documented in AISDP Module 4. Where gaps are identified, the remediation approach is documented with a timeline. Bias in the knowledge base is distinct from bias in the model itself, and both must be assessed independently. A model with no detectable parametric bias can still produce biased outputs if the knowledge base systematically excludes relevant perspectives.
The retrieval pipeline determines which documents the GPAI model sees, making it a decision-making component with direct impact on outputs.
The retrieval pipeline determines which documents the GPAI model sees, making it a decision-making component with direct impact on outputs. Its behaviour must be documented in AISDP Module 3 with the same rigour as the model inference layer.
Documentation covers: the embedding model used for semantic search (version, provider, known limitations); the similarity metric and threshold; the number of documents retrieved (top-k); the re-ranking strategy if any; the chunking strategy for splitting documents into retrievable segments; and the metadata filtering logic if retrieval is constrained by document attributes.
Embedding model governance. The embedding model that converts queries and documents into vectors is itself an AI component that introduces bias, accuracy, and version control considerations. Three requirements apply specifically to RAG systems.
First, embedding bias evaluation. The Technical SME evaluates whether the embedding model produces systematically different retrieval results for semantically equivalent queries phrased in different ways. A query about "maternity leave entitlements" should retrieve the same documents as one about "parental leave for mothers." If the embedding model's semantic space encodes demographic biases, the retrieval pipeline may systematically underserve certain query formulations.
Second, cross-lingual retrieval quality. For multilingual RAG systems, the Technical SME evaluates whether retrieval quality is consistent across languages. A system that retrieves comprehensive, relevant documents for English queries but sparse, tangential documents for Polish or Romanian queries produces systematically different output quality across language communities.
Third, embedding version alignment. Document embeddings in the vector store must be generated by the same embedding model version as query embeddings at inference time. A mismatch can silently degrade retrieval quality. The Technical SME implements version alignment checks and documents the mechanism in AISDP Module 10.
Hallucination is the defining compliance risk for RAG systems: outputs not supported by retrieved context may be incorrect, misleading, or harmful.
Hallucination is the defining compliance risk for RAG systems: outputs not supported by retrieved context may be incorrect, misleading, or harmful. For high-risk applications, hallucination is not merely an accuracy problem but a safety and fundamental rights risk. The Technical SME implements a grounding verification layer that checks whether each claim in the model's output is supported by the retrieved documents.
Implementation approaches. Three approaches are available in increasing order of sophistication. Citation verification requires the model to cite specific passages from retrieved documents, and the verification layer checks that cited passages exist and support the claim. This is the simplest approach and the most auditable. Entailment checking uses a separate NLI (natural language inference) model to evaluate whether each claim is entailed by the retrieved context. Fact extraction and matching extracts factual claims from the output and matches them against facts extracted from the retrieved documents.
The verification operates at inference time and produces a grounding score for each output. Outputs below the grounding threshold are flagged, held for human review, or suppressed, depending on risk profile and severity of potential harm. Results are logged for every inference and documented in AISDP Module 5.
Grounding failure handling. The system's response to a grounding failure must be defined in the AISDP and implemented in the inference pipeline. Options range from appending a disclaimer ("This response could not be fully verified against the knowledge base") to suppressing the output entirely and returning a fallback response. For high-risk systems where ungrounded outputs could cause harm, suppression is the safer default. The AI Governance Lead defines the grounding failure policy; the Technical SME implements it.
Structured evidence for audit. Each inference produces a grounding report containing: the query; retrieved documents with identifiers and relevance scores; the model's output; the grounding analysis showing which claims are supported and which are not; the overall grounding score; and the action taken (passed, flagged, or suppressed). Aggregate grounding metrics are tracked as post-market monitoring indicators in AISDP Module 12 and monitored for drift.
Procedural alternative. Without automated grounding verification, human reviewers assess output grounding. This is feasible only for low-volume systems where every output is reviewed before delivery. The reviewer checks each output against retrieved documents and records whether it is fully grounded, partially grounded, or ungrounded. For systems with more than approximately 50 outputs per day, manual verification is unsustainable without disproportionate staffing.
RAG systems face a specific attack vector distinct from direct prompt injection: indirect prompt injection through the knowledge base.
RAG systems face a specific attack vector distinct from direct prompt injection: indirect prompt injection through the knowledge base. An adversary who can insert or modify documents in the knowledge base can embed instructions that the GPAI model may follow when the document is retrieved as context.
Attack surface. The knowledge base is the attack surface. Any pathway through which documents enter is a potential injection vector: user-uploaded documents, web-scraped content, partner data feeds, and automated document ingestion pipelines. A malicious document containing hidden instructions may be retrieved by the RAG pipeline and processed by the model as context, causing outputs that violate the system's intended behaviour.
Defence architecture. The Technical SME implements four layers of defence. Input sanitisation screens documents for injection patterns before they enter the knowledge base. Content isolation ensures the model's system prompt is protected from override by retrieved content, using the provider's recommended separation mechanisms. Output validation checks the model's output against expected behaviour boundaries regardless of retrieved context. Retrieval filtering excludes documents flagged as potentially adversarial from retrieval results.
The cybersecurity testing programme should include specific test cases for indirect prompt injection through the knowledge base. Red-teaming exercises should attempt to inject malicious documents through each ingestion pathway and verify that defence layers detect and block the injection. Results are documented in AISDP Module 9.
Whether a change to the knowledge base alone constitutes a substantial modification under Article 3 does not have a definitive regulatory answer.
Whether a change to the knowledge base alone constitutes a substantial modification under Article 3 does not have a definitive regulatory answer. The analysis requires balancing two positions.
The case for substantial modification. A material change to the knowledge base changes the information available to the model, which changes outputs, which may change compliance with the Chapter 2 requirements. If the knowledge base is updated to include documents introducing new bias, containing incorrect information about a protected group, or removing previously available information, the system's fairness, accuracy, and safety profiles may all be affected.
The case against. Knowledge base updates are normal operational activity for any information system. Treating every document addition or removal as a substantial modification would make RAG systems operationally impractical for high-risk applications, because every update would trigger a new conformity assessment.
The practical approach. The AI Governance Lead defines a materiality threshold for knowledge base changes. Changes below the threshold (routine document additions within the existing domain, corrections to factual errors, removal of outdated documents) are treated as normal operational maintenance documented in AISDP Module 12. Changes above the threshold (introduction of a new domain or topic area, material change to demographic coverage, significant change to the proportion of sources from a particular perspective or origin) trigger the governance pipeline's risk gate and change classification logic.
The materiality threshold is documented in the AISDP and reviewed quarterly. The threshold should reference quantitative indicators where possible: knowledge base size change exceeding 20%; document source distribution change exceeding a defined divergence metric; grounding score shift in the sentinel evaluation exceeding a defined tolerance.
Accuracy, Robustness, Cybersecurity
EUR-Lex