Pipelines retrievers reference
This reference documentation for Pipelines retrievers includes information on the functions and views available in the aidb extension related to retrievers.
Views
aidb.retrievers
The aidb.retrievers view shows information about the retrievers that were created in the database.
| Column | Type | Description |
|---|---|---|
| id | integer | |
| name | text | Name of the retriever. |
| vector_table_name | text | Name of the table where the embeddings are stored. Gets newly created if it doesn’t exist. Managed by aidb. |
| vector_table_key_column | text | Column to use to store the key that references the key in source data when computing embeddings. We recommend using the default and letting aidb manage this table. |
| vector_table_vector_column | text | Column to store embeddings in. We recommend using the default and letting aidb manage this table. |
| model_name | text | Name of the model to use for embedding computation and retrievals. |
| topk | integer | Default number of results to return during a retrieve. Similar to LIMIT in SQL. |
| distance_operator | aidb.DistanceOperator | During retrieval, the vector operation to use to compare the vectors. |
| options | jsonb | Currently unused. |
| source_type | text | Type of source data the retriever is working with. Can be 'Table' or 'Volume'. |
| source_table_name | regclass | Name of the table that has the source data Pipelines computes embeddings for and retrieves from. Only applicable to retrievers configured with aidb.create_retriever_for_table(). |
| source_table_data_column | text | Column name in the source table that Pipelines computes embeddings for. This is also the column that's returned in retrieve operations. |
| source_table_data_column_type | aidb.RetrieverSourceDataFormat | Type of data the retriever is working with. Uses type aidb.RetrieverSourceDataFormat. Only relevant for table-based retrievers. In the case of a volume-based retriever, the format/type information is discovered from the volume. |
| source_table_key_column | text | Column to use as key for storing the embedding in the vector table. This provides a reference from the embedding to the source data. |
| source_volume_name | text | Name of the volume to use as a data source. Only applicable to retrievers configured with aidb.create_retriever_for_volume(). |
Types
aidb.DistanceOperator
The aidb.DistanceOperator type is an enum that represents the distance operators that can be used during retrieval.
| Value | Description |
|---|---|
| L2 | Euclidean distance |
| Inner | Inner product |
| Cosine | Cosine similarity |
| L1 | L1 distance |
| Hamming | Hamming distance |
| Jaccard | Jaccard distance |
SQL definition:
CREATE TYPE DistanceOperator AS ENUM ( 'L2', 'InnerProduct', 'Cosine', 'L1', 'Hamming', 'Jaccard' );
aidb.RetrieverSourceDataFormat
The aidb.RetrieverSourceDataFormat type is an enum that represents the data formats that can be used as source data.
| Value | Description |
|---|---|
| Text | Text data |
| Image | Image data |
| PDF data |
SQL definition:
CREATE TYPE RetrieverSourceDataFormat AS ENUM ( 'Text', 'Image', 'Pdf' );
Functions
aidb.create_retriever_for_table
Creates a retriever for a given table.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| name | TEXT | Required | Name of the retriever |
| model_name | TEXT | Required | Name of the model to use |
| source_table | regclass | Required | Name of the table to use as source |
| source_data_column | TEXT | Required | Column name in source table to use |
| source_data_type | aidb.RetrieverSourceDataFormat | Required | Type of data in that column ("Text"."Image","PDF") |
| source_key_column | TEXT | 'id' | Column to use as key to reference the rows |
| vector_table | TEXT | NULL | |
| vector_data_column | TEXT | 'embeddings' | |
| vector_key_column | TEXT | 'id' | |
| topk | INTEGER | 1 | |
| distance_operator | aidb.distanceoperator | 'L2' | |
| options | JSONB | '{}'::JSONB | Options |
Example
SELECT aidb.create_retriever_for_table( name => 'test_retriever', model_name => 'simple_model', source_table => 'test_source_table', source_data_column => 'content', source_data_type => 'Text', );
aidb.create_retriever_for_volume
Creates a retriever for a given PGFS volume.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| name | TEXT | Required | Name of the retriever |
| model_name | TEXT | Required | Name of the model |
| source_volume_name | TEXT | Required | Name of the volume |
| vector_table | TEXT | NULL | Name of the vector table |
| vector_data_column | TEXT | 'embeddings' | Name of the vector column |
| vector_key_column | TEXT | 'id' | Name of the key column |
| topk | INTEGER | 1 | Number of results to return |
| distance_operator | aidb.distanceoperator | 'L2' | Distance operator |
| options | JSONB | '{}'::JSONB | Options |
Example
SELECT aidb.create_retriever_for_volume( name => 'demo_vol_retriever', model_name => 'simple_model', source_volume_name => 'demo_bucket_vol' );
aidb.enable_auto_embedding_for_table
Enables automatic embedding generation for a given table.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| retriever_name | TEXT | Name of retriever for which to enable auto-embedding |
Example
SELECT aidb.enable_auto_embedding_for_table('test_retriever');
aidb.disable_auto_embedding_for_table
Enables automatic embedding generation for a given table.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| retriever_name | TEXT | Name of retriever for which to disable auto-embedding |
Example
SELECT aidb.enable_auto_embedding_for_table('test_retriever');
aidb.bulk_embedding
Generates embeddings for all data in a given table if there's existing data in the table.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| retriever_name | TEXT | Name of retriever for which to generate embeddings |
Example
edb=# select aidb.bulk_embedding('test_retriever');Output
INFO: bulk_embedding_text found 3 rows in retriever test_retriever bulk_embedding ---------------- (1 row)
aidb.retrieve_key
Retrieves a key from matching embeddings without looking up the source data.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| retriever_name | TEXT | Name of retriever to use for retrieval | |
| query_string | TEXT | Query string to use for retrieval | |
| number_of_results | INTEGER | 0 | Number of results to return |
Example
SELECT * FROM aidb.retrieve_key('test_retriever', 'shoes', 2);
Output
key | distance -------+-------------------- 43941 | 0.2938963414490189 19337 | 0.3023805122617119 (2 rows)
aidb.retrieve_text
Retrieves the source text data from matching embeddings by joining the embeddings with the source table.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| retriever_name | TEXT | Name of retriever to use for retrieval | |
| query_string | TEXT | Query string to use for retrieval | |
| number_of_results | INTEGER | 0 | Number of results to return |
Returns
| Column | Type | Description |
|---|---|---|
key | text | Key of the retrieved data |
value | text | Value of the retrieved data |
distance | double precision | Distance of the retrieved data from the query |
Example
SELECT * FROM aidb.retrieve_text('test_retriever', 'jacket', 2);
Output
key | value | distance -------+----------------------------------------------------+-------------------- 19337 | United Colors of Benetton Men Stripes Black Jacket | 0.2994317672742334 55018 | Lakme 3 in 1 Orchid Aqua Shine Lip Color | 0.3804609668507203 (2 rows)
aidb.delete_retriever
Deletes only the retriever's configuration from the database.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| retriever_name | TEXT | Name of retriever to delete |
Example
select aidb.delete_retriever('test_retriever');
Output
delete_retriever ------------------ (1 row)
aidb.create_volume
Creates a volume from a PGFS storage location for use as a data source in retrievers.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| name | TEXT | Name of the volume to create | |
| server_name | TEXT | Name of the storage location to use for the volume | |
| path | TEXT | Path to the volume in the storage location | |
| mime_type | TEXT | Type of data in the volume (Text or Image) |
Example
select aidb.create_volume('demo_bucket_vol', 'demo_bucket', 'demo_bucket/demo_folder', 'Text');
Output
create_volume --------------- (1 row)
aidb.list_volumes
Lists all the volumes that were created in the database.
Example
select * from aidb.list_volumes();
aidb.delete_volume
Deletes a volume from the database.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| volume_name | TEXT | Name of the volume to delete |
Example
select aidb.delete_volume('demo_bucket_vol');
Output
delete_volume ------------- (1 row)
Could this page be better? Report a problem or suggest an addition!