AI Accelerator Pipelines overview

As part of EDB Postgres AI - AI Accelerator, Pipelines abstracts away the complexity of working with AI data. It transforms Postgres into a powerful platform for AI data management, combining vector search from PGvector with automation for complex AI workflows.

Pipelines, auto-embedding, and intelligent retriever

Pipelines handles the entire lifecycle of embedding generation, storage, and indexing. You can select your desired model from a list of supported models or connect to any OpenAI API-compatible external model. You can then let Pipelines manage everything else from data ingestion to efficient similarity search.

With auto-embedding, Pipelines automates creating and updating embedding, ensuring that vector stores remain up to date without manual embedding management. This approach reduces the risk of stale data that causes GenAI errors and hallucinations.

The intelligent retriever feature lets you perform similarity and semantic searches across text and image data with a single retriever function call. These searches are possible whether the data is stored in Postgres or object storage.

How Pipelines works

Pipelines delivers its functionality through the Pipelines aidb extension that's embedded into the Postgres server. The extension provides a set of functions and views that allow you to interact with the AI data in your Postgres database.

Retrievers

The Pipelines aidb extension introduces the concept of a retriever that you can create for a given type and location of AI data. This data can reside either in regular columns of a Postgres table or in an S3-compatible object storage bucket. Currently, Pipelines supports unstructured plain text documents as well as a set of image formats.

A retriever encapsulates all processing that's needed to make the AI data in the provided source location searchable and retrievable through similarity. To create a retriever, the application uses the aidb.create_retriever_for_table() function for Postgres tables or aidb.create_retriever_for_volume for stored data externally on S3 or local file systems.

Auto-embedding

Auto-embedding is currently supported for AI data stored in Postgres tables, and it automates the embedding updates using Postgres triggers. aidb.enable_auto_embedding_for_table() enables auto-embedding for a given retriever and aidb.disable_auto_embedding_for_table() disables it.

By default, auto-embedding is disabled. The application has to manually trigger embedding generation for all existing data in the source location using aidb.bulk_embedding().

You can make best use of the two settings of auto-embedding by bulk loading your existing data into your table with auto-embedding disabled. Then you can run aidb.bulk_embedding() to create embeddings for that data. After that, you can enable auto-embedding to create embeddings for any future data added to the table.

All embedding generation, storage, indexing, and management is handled internally by the Pipelines aidb extension. The application has to specify the encoder LLM for the retriever to use for the specific data and use case.

Intelligent retriever

Once a retriever is created and all embeddings are up to date, the application can use aidb.retrieve_key() to run a similarity search and retrieval by providing a query input. It returns the id of the matched items. When the retriever is created for text data, the query input is also a text term. For image retrievers, the query input is an image. If you're using a multi-modal model with your retriever, such as CLIP, you can use text or images as source and query inputs. This approach allows you to search for images using text and vice versa.

The aidb retriever makes sure to use the same encoder LLM for the query input as used for the embedding, conducts a similarity search, and returns the ranked list of similar data from the source location. For Postgres tables, you can also use aidb.retrieve_text() to retrieve the actual data from the source table.

Pipelines currently supports a broad list of open encoder LLMs from HuggingFace as well as a set of OpenAI encoders. HuggingFace LLMs run locally on the Postgres node, while OpenAI encoders involve a call out to the OpenAI cloud service. The aidb.model_providers table shows the list of currently supported model providers.