AI-powered metadata correction and harmonization
aws ·
The article presents an AI-powered metadata correction and harmonization workflow built on AWS to address the bottleneck of standardizing metadata from disparate data sources. The system uses Amazon Bedrock for LLM-powered schema alignment and correction recommendations, along with S3, DynamoDB, Cognito, and ECS for storage, tracking, authentication, and compute. It operates as a cyclical, human-in-the-loop process where users upload metadata, the system validates schema alignment and field values, then generates correction recommendations for user approval. Schema alignment leverages LLMs for semantic understanding of column structures, while field validation checks required fields, enumerated values, and patterns through rule-based methods. Recommendations use a layered approach: embeddings for semantic similarity (using Amazon Titan), contextual inference via k-nearest neighbors and co-occurrence statistics (PMI), fuzzy matching for typos, and LLM-based resolution as a fallback for ambiguous cases. The article also provides deployment prerequisites and steps, including installing Node.js, Python, AWS CLI, CDK, Docker, and configuring a config.yaml file.
aws