AWS has turned S3 buckets into Vector stores. This makes it  the first cloud object store with native support to store and query vectors. What are the advantages?

Mainly to reduce the cost of uploading, storing, and querying vectors by up to 90%. You only pay for what you use without the need for infrastructure provisioning and management. That aside, you can scale vector search applications seamlessly from gigabytes to petabytes, while you can alos scale them down to zero when these resources are not in use.

While cost cutting and scaling might be the main selling points, there are others too:

• Fast semantic searching
• Vector indexing for logical grouping
• Metadata as key-value pairs to each vector to filter future queries based on a set of conditions, for example, dates, categories, or user preferences
• Integrations with AWS services: Fits natively into Amazon Bedrock, Sagemaker, Opensearch.

On the last point, integration wise, you can use Amazon Bedrock and the AWS Python SDKs to generate embeddings. An example taken from the official documentation, you do :

That mean that you can use a S3 Vector backend to power up various AI applications like:

• Medical imaging - Find similarities in millions of medical images to assist with diagnosis and treatment planning
• Copyright infringement - Identify potentially derivative content across large media libraries
• Image deduplication - Detect and remove duplicate or near-duplicate images from large image collections
• Video understanding - Search for specific scenes or content within video assets
• Enterprise document search - Enable semantic search across corporate documents to find relevant information based on meaning
• Personalization - Deliver tailored recommendations by finding similar items

In addition, its integration with Opensearch allows for critical real-time applications such as product recommendations and fraud detection.

In the end, does that mean that there's no more need for specialized databases or vector stores? It depends. As we explored in "Turn PostgreSQL Into A Vector Store", the pg_vector extension could render PostgreSQL a viable alternative to specialized vector stores used in LLMs. An excerpt from it :

pg_vector allows you to replace Chroma or any other specialized vector engine with Postgres, so that you can have your embeddings stored alongside your JSON or relation data under the same roof.

The gist here is that using a relational database with vector support allows that exactly:

embeddings stored alongside your JSON or relation data under the same roof.

which opens up other kinds of possibilities.

So with S3 Vectors taking care of your data lakes and PostgreSQL looking after vector and relational data, have specialized Vector stores reached the end of the road? 

More Information

Introducing Amazon S3 Vectors

Related Articles

Turn PostgreSQL Into A Vector Store

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info