Skip to main content

RAG data protection

  • Tenant isolation: forced RLS on kb_documents, kb_chunks, kb_quarantine_reviews; one Qdrant collection per tenant.
  • Document and chunk ACL checked in the vector store and again before the context is built.
  • Source right: active connector required at ingestion and query (403 source_permission_denied).
  • Poisoning: quarantine, review by security_officer (data steward), audited.
  • Encryption: text of restricted and secret chunks as a Tink envelope, rotation and re-wrap.
  • Sensitive cache: Tink encrypted, key bound to the tenant, Redis never the source of truth.
  • Banking test data: all fictitious.

Verified Docling and PaddleOCR toolchains (DATA-010, DATA-011)​

Setup (separate virtual environments, pinned versions, official sources only):

bash tools/rag/setup-docling.sh     # /tmp/toolchains/docling-venv + /tmp/toolchains/docling-models
bash tools/rag/setup-paddleocr.sh   # /tmp/toolchains/paddleocr-venv + /tmp/toolchains/paddleocr-models
  • Docling 2.145.0 (MIT), docling-core 2.105.0 (MIT), docling-ibm-models 4.0.4 (MIT), torch 2.9.1+cpu (BSD-3-Clause, official index https://download.pytorch.org/whl/cpu), python-docx 1.2.0 (MIT).
  • paddlepaddle 3.3.1 CPU (Apache-2.0), paddleocr 3.7.0 (Apache-2.0), paddlex 3.7.2 (Apache-2.0), from PyPI.
  • The scripts write models.sha256 (sha256 of every weight file) and print the licence of every model card.

Model weights and licences:

ModelOfficial sourceFilesha256Licence
docling-layout-heronHugging Face docling-project/docling-layout-heronmodel.safetensors00333a43451945aaf89db8ca9c0a17e75d1537c17db60fdb91aa95f4c7929e0cApache-2.0
docling-layout-heron-onnxHugging Face docling-project/docling-layout-heron-onnxmodel.onnx59c81a3a2923042d85034ffc487f8f47e4854117e879aef89b2b9f728fb4922aApache-2.0
TableFormer accurateHugging Face docling-project/docling-modelstableformer_accurate.safetensors2a7d6c924b3cd12fb99a09280ca9c33a89c5d60b93253617d2e088c1a40374d9CDLA-Permissive-2.0
TableFormer fastHugging Face docling-project/docling-modelstableformer_fast.safetensors3119563aab5a7c96fda4d621119b63fd8806272b86c30936d15507616422f718CDLA-Permissive-2.0
PP-OCRv5_mobile_detHugging Face PaddlePaddle/PP-OCRv5_mobile_detinference.pdiparamsafa1820cb16c1fd0dad589d0f8b389139061c1ef6d68019685fd07be997dda5bApache-2.0
PP-OCRv5_mobile_recHugging Face PaddlePaddle/PP-OCRv5_mobile_recinference.pdiparams2460da90875937c94db97eba74ae3d9e5d4c4c57c42f1f41531c09a26bcc771aApache-2.0

HTTP workers (127.0.0.1 only, free port chosen by the caller):

HF_HUB_OFFLINE=1 /tmp/toolchains/docling-venv/bin/python tools/rag/docling_worker.py --port PORT --models /tmp/toolchains/docling-models
PADDLE_PDX_CACHE_HOME=/tmp/toolchains/paddleocr-models /tmp/toolchains/paddleocr-venv/bin/python tools/rag/paddleocr_worker.py --port PORT

docling_worker.py implements the POST /v1/convert/source subset of Docling Serve 1.x (sources of kind file, response document.md_content), with Docling OCR off (scans go to PaddleOCR, ocr_applied: false is returned). paddleocr_worker.py implements POST /ocr of PaddleX serving; oneDNN is off (enable_mkldnn=False) because paddlepaddle 3.3.1 CPU raises NotImplementedError (ConvertPirAttribute2RuntimeAttribute) on these models. The adapter adds the extension of the MIME type to the file name sent to Docling, which picks the input format from the extension.

Evidence: services/control-plane/tests/test_rag_docling_ocr_real.py starts both workers and checks, through eai.integrations.docling and eai.integrations.ocr (hence eai.security.egress), an acceptance PDF (text and table), a DOCX with a table (exact cells), ingestion through the API then a query, the refusal of a destination outside the egress policy, and OCR of an image of text rendered with Pillow. Missing toolchain: explicit failure, never skipped.

Sensitive cache and outlier embedding detection (DATA-024, DATA-015)​

services/control-plane/tests/test_rag_kit_acceptance.py checks on a real /usr/bin/redis-server (free port) that the RAG cache entry is a Tink ciphertext (eaitink1:), that an entry copied under another tenant's key reads as a miss (unwrapping a DEK of another tenant raises EnvelopeError, never an HTTP 500), and that a Redis outage falls back to the vector store with no content in the logs. The same file quarantines a document whose chunk points away from the document centroid (outlier_embedding). It also covers DATA-019, DATA-020, DATA-022, DATA-023 and scenario 7 (RAG with ACL, authorised citations, lineage from source to answer).

RAG quarantine review​

A suspicious document (injection, outlier embedding, untrusted source) is quarantined: its chunks are no longer searchable. Review is reserved to the tenant_admin, security_officer and data_steward roles (REVIEW_ROLES of eai.domains.knowledge), who also need the viewer relation on the knowledge base.

OperationRouteEffect
List the quarantineGET /v1/knowledge-bases/{kb_id}/quarantineDocuments waiting for review
DecidePOST /v1/knowledge-bases/{kb_id}/documents/{doc_id}/reviewrelease: chunks searchable again; reject: document rejected; decision audited

Before ingestion: file security​

No byte reaches Docling, OCR or RAG without a scan released by ClamAV and YARA-X: see file-security-pipeline.md.