S3 Vectors adds metadata filtering, promises 5x recall

Metadata pre-filtering arrives for S3 Vectors, claiming up to 5x higher recall.

What changed

• S3 Vectors now supports filtering based on object metadata before executing similarity searches.
• This feature aims to improve search relevance and recall for scoped queries.
• The announcement does not specify the exact mechanism or performance gains beyond the claimed 5x recall.

Why it matters

Retrieval Augmented Generation (RAG) and agentic applications often rely on filtering documents by metadata (e.g., author, date, document type) before performing a vector similarity search. Previously, S3 Vectors would perform the similarity search first, then filter the results. This new capability evaluates metadata filters before the vector search. In practice: this should reduce the number of vector comparisons needed for queries that have restrictive metadata criteria, potentially leading to faster response times and more accurate results by only considering relevant documents from the outset. This is a significant improvement for workloads where metadata filtering is a primary query constraint.

The catch

While the announcement claims up to 5x higher recall, it is vague on the specifics of how this is achieved or under what conditions. The honest version: the actual recall improvement will likely vary significantly based on the dataset, the quality of the metadata, and the specificity of the filters applied. Furthermore, the announcement does not detail any potential latency increases for queries without metadata filters, or if there are any new service limits associated with this feature. It also doesn't specify if this feature is available in all S3 Regions or if it requires specific S3 storage classes.

Ship it

If you are using S3 Vectors for RAG or agentic workloads and frequently filter by object metadata, evaluate this feature today. Pay close attention to the performance characteristics in your specific environment, particularly for queries with highly selective metadata filters. This feature pairs naturally with S3 Select for complex metadata queries.

Bottom line: S3 Vectors gains metadata pre-filtering, potentially boosting recall for filtered searches in RAG and agentic apps.

*— Filed to /blog