PII-Masker Architecture

1. Overview

1. What it does: Masks credit card numbers and dates of birth in conversations (English & Canadian French) before storing them in Mongo.

2. Technology stack: Uses FastAPI with ONNX Runtime 1.28 and runs CPU-only, with no GPU dependency.

3. Models used: Includes two models: NER token classification and DOB-intent sequence classification.

4. RAM footprint: Idle usage is approximately 978 MB, burst usage is approximately 1.14 GB RSS, and memory is released after loading through arena shrink.

5. Deployment: Deployed with Helm using a single worker per pod.

2. Architecture Diagram

3. Model Architecture and Packaging

NER behavior: Detects credit card and date-of-birth entities across normal text, spoken digits, word-split formats, and partially masked formats. It uses 11 BIO labels: O, B/I-CREDIT_CARD, B/I-DOB, B/I-DOB_DAY, B/I-DOB_MONTH, and B/I-DOB_YEAR.

DOB intent gate: The intent model emits P(DOB_REQUEST). DOB values are masked only when the conversation indicates that the agent actually asked for DOB, which keeps masking privacy-focused and avoids unnecessary redaction.

Quantization and validation: Models are dynamically quantized to INT8 using scripts/optimize_onnx.py. Packaging is allowed only after FP32-to-INT8 parity and false-negative audits pass (artifacts/parity_fp32_int8.json and artifacts/fn_audit_int8.json). The runtime automatically prefers INT8 for CPU inference and verifies checksums.sha256 during startup.

Shared tokenizer: Both models run in the same process and share a single 250,002-token XLM-R tokenizer. The tokenizer is loaded once behind a lock, saving approximately 285 MB of RAM.

4. Runtime Resource Management

Process model: The service intentionally runs with one worker per pod (WEB_CONCURRENCY=1). A single shared InferenceSession avoids duplicate model memory and keeps batching effective.

Dynamic batching: Requests are collected for up to 8 ms and processed in batches of up to 16 sequences per inference. ONNX_THREADS controls ONNX intra-op threads and should match the pod CPU allocation.

Memory optimization: This branch reduces baseline memory by sharing the tokenizer (approximately −285 MB), lazy-loading Presidio/spaCy dependencies (approximately −86 to −173 MB when Layer 0 remains disabled), and pre-warming a 128 MB buffer so first requests do not pay allocation latency.

Arena shrink controller: A background controller runs every 60 seconds, measures held-but-unused ONNX Runtime arena memory, and releases memory above the configured buffer. Validation measured live releases of 148–294 MB. Disable with PII_ARENA_ENABLED=false if needed.

Admission control: The conversation endpoint allows up to 4 concurrent ML requests. Additional requests return an immediate 503 with Retry-After, preventing event-loop stalls and avoiding 504 storms.

Parallel chunking option: PII_MODEL_ONNX_MAX_PARALLEL_CHUNKS defaults to 1 for the lowest memory usage. Setting it to 2 can reduce latency by approximately 1.3–2 seconds when CPU headroom exists, at the cost of +150–185 MB peak RSS.

Detailed design notes are available in reports/RAM_OPTIMIZATION_PLAN.md, sections 3–5.

5. Load Test Results

Validation was recorded on 2026-09-30 using an 8-core VM with ONNX_THREADS=2 and INT8 ONNX models. The service remained stable under sustained load, with flat memory usage during light, medium, and heavy request patterns.

Throughput: Per-pod throughput plateaus at approximately 20–26 requests per second for small texts. The service is CPU-bound by design, so production scaling should use HPA pods rather than additional in-process workers.

Concurrency hardening: The post-hardening ladder showed stable behavior: c=2 returned 2×200, c=4 returned 4×200, and c=8 returned 4×200 plus 4×503 graceful rejections. No 504s were observed.

6. Accuracy and Validation

Accuracy validation was recorded during the 2026-09-30 session using the same script, dataset, and machine state before and after the optimization changes. This makes the A/B comparison valid for confirming that runtime optimization did not change model output.

The evaluation used the published splits: 10,000 test rows and 5,000 adversarial rows, with both capped at 5,000 rows by the evaluation limit.

Entity-level highlights: CREDIT_CARD achieved adversarial F1 of 0.963 with recall of 0.9996. DOB component F1 ranged from 0.647 to 0.845.

Optimization impact: Pre-change and post-change code produced 42/42 identical metrics with Δ=0.000000. The memory and runtime optimizations did not affect accuracy.

Context: Release-gate thresholds of 0.995 apply to a separate gold calibration set. The older stored artifact (runtime_eval_results.json, 2,000 rows, 2026-08-17) is a stale baseline and should not be compared directly with this session.

7. Operational Configuration

All runtime controls are configuration-only and are delivered through environment variables or ConfigMap values. The defaults are safe for production unless a specific workload requires tuning.

8. Capacity and Known Limits

• Per-pod capacity: Approximately 20–26 requests per second for small texts. The workload is CPU-bound.

• Conversation latency: A single batch-16 conversation takes approximately 3.5–6 seconds in isolation and up to about 11 seconds under concurrent load.

• Scaling model: Scale horizontally with HPA pods. Do not increase workers inside a pod, because that duplicates model memory.

• Over-capacity behavior: Requests beyond the 4-slot admission gate receive 503 with Retry-After. In c=8 validation, the service returned 4×200 and 4×503 with no 504s.

• Known ML gap: Split-card formats with four separate digit groups remain a tracked detection gap.

9. Reference Links and Verification

Local execution: Use scripts/run_dev.sh for development and scripts/run_server.sh for the ONNX production entrypoint. Deployment is managed through helm/, including values-prod.yaml.

Verification status: 626 automated tests passed at commit 2235921: 514 unit tests, 40 conversation tests, 70 integration tests, and 2 admission probes.