Model-agnostic PII detection with LLMs
aws ·
An AWS Machine Learning Blog post by Christophe Dupuy and Rahul Gupta describes a configurable, instruction-driven PII detector that is model-agnostic and runs on any large language model managed through Amazon Bedrock. The motivation is that fine-tuning models on uncleaned real-world text risks memorizing PII such as names, addresses, contact details, national-ID and social-security numbers, and bank accounts, which could later be reproduced. The detector addresses this by treating the LLM as a swappable component: detection logic lives in a system prompt defining fifteen entity categories, a do-not-flag list, and optional few-shot examples, while the backend is reached through a uniform Inferencer interface that ships with an Amazon Bedrock adapter (a wrapper over the Converse API) and can accept a custom adapter for self-hosted models, including air-gapped deployments. The model returns structured JSON with entity types and text values rather than character offsets; a post-processing layer parses the JSON, recovers exact offsets via regular expressions, and re-homes near-miss labels onto the prompt vocabulary, marking unmappable labels as UNK. The detector ships as the pii-detector package in the sample-llm-pii-detection repository, with prerequisites of Python 3.11 or later, an AWS account with Bedrock model access, and boto3, and the walkthrough covers installation, credential configuration, running a bundled example, calling the detector on custom text, and cleanup. Switching models is described as a one-line change, with examples including amazon.nova-lite-v1:0 and mistral.mistral-large-3-675b-instruct. The approach was benchmarked against eight other LLM-based detectors, including the OpenAI PrivacyFilter, on five public Hugging Face PII corpora sampled at roughly 10,000 rows each, totaling 49,365 records and 222,114 ground-truth core spans across eight languages, with spans matched by exact start, end, and label overlap (IoU = 1.0) and scored using precision, recall, and F1. To make comparisons fair across differing label vocabularies, all labels were mapped onto a single canonical taxonomy, though the article text is truncated at that point, so the remaining benchmarking results and conclusions are not available.
aws