Filter syntax in depth

A vector query returns the top_k items most similar to your query vector. A filter scopes that search to rows whose metadata also satisfies a predicate. "Find me 10 films like The Matrix" becomes "find me 10 sci-fi films from after 2000 like The Matrix."

Filters compose with the vector search at the database. They don't run in your application code after the fact; the database returns the items closest to the query vector that also satisfy the predicate. Internally the coordinator over-fetches (default 100×) so the post-filter result still meets your top_k, and re-fetches if it doesn't.

This page is the reference for the full operator set. Every example below runs end-to-end against the bundled 5,000-film movie corpus that ships with veep[samples]. Set up once at the top, then copy the snippet for whatever operator you need.

Setup

from veep import VP, samples

df = samples.dataframe()  # 5,000 films: id, title, year, genre, plot, vector

vp = VP.from_creds()
vp.collections.create("filter-syntax-demo", tier="hot")
vp.vectors.upsert("filter-syntax-demo", dataframe=df)

from veep import VP, samples
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_vectorpanda import VectorPandaStore

df = samples.dataframe()

vp = VP.from_creds()
vp.collections.create("filter-syntax-demo", tier="hot")
vp.vectors.upsert("filter-syntax-demo", dataframe=df)

store = VectorPandaStore(
    collection_name="filter-syntax-demo",
    embedding=HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2"),
    client=vp,
)

For the rest of the page we'll keep using one fixed query vector (the centroid of the LOTR trilogy) so you can compare filter outputs directly without the query changing under you.

import numpy as np

WATCH = ["film-00023", "film-00027", "film-00035"]  # LOTR trilogy
vecs = np.stack([np.array(df[df.id == fid].iloc[0].vector) for fid in WATCH])
qv = vecs.mean(axis=0)
qv = (qv / np.linalg.norm(qv)).tolist()

The leaf operators

Each leaf operator filters on a single metadata field. There are eight of them.

$eq: equality

Two equivalent forms. The first is shorthand; the second is the explicit version. Use whichever reads better.

# Shorthand
vp.vectors.query("filter-syntax-demo", vector=qv, top_k=3,
                 filter={"genre": "animated"})

# Explicit
vp.vectors.query("filter-syntax-demo", vector=qv, top_k=3,
                 filter={"genre": {"$eq": "animated"}})

# Shorthand
store.similarity_search_by_vector_with_score(qv, k=3,
                 filter={"genre": "animated"})

# Explicit
store.similarity_search_by_vector_with_score(qv, k=3,
                 filter={"genre": {"$eq": "animated"}})

Both return the same three rows:

0.4191  A Troll in Central Park              (1994, animated)
0.3510  Quest for Camelot                    (1998, animated)
0.3472  Knighty Knight Bugs                  (1958, animated)

The query vector pulled toward fantasy/quest themes; the filter narrowed to animated. If your filter clause is just {field: value} with no operator, it's interpreted as $eq. If you want the explicit form for symmetry with the others, use {field: {$eq: value}}.

$ne: not equal

Inverse of $eq. Returns rows where the field is not equal to the operand.

vp.vectors.query("filter-syntax-demo", vector=qv, top_k=3,
                 filter={"genre": {"$ne": "animated"}})

store.similarity_search_by_vector_with_score(qv, k=3,
                 filter={"genre": {"$ne": "animated"}})

0.9093  The Lord of the Rings: The Two Towe  (2002, adventure, fantasy)
0.8905  The Lord of the Rings: The Return o  (2003, adventure, fantasy)
0.8633  The Lord of the Rings: The Fellowsh  (2001, fantasy)

The LOTR films return at the top because the query vector is the LOTR centroid; they're not animated, so they pass the $ne filter.

$gt, $gte, $lt, $lte: numeric comparisons

Strict and inclusive on both ends.

# Strictly greater than 2015
vp.vectors.query("filter-syntax-demo", vector=qv, top_k=3,
                 filter={"year": {"$gt": 2015}})

# Strictly greater than 2015
store.similarity_search_by_vector_with_score(qv, k=3,
                 filter={"year": {"$gt": 2015}})

0.4630  King Arthur: Legend of the Sword     (2017, action, adventure,)
0.4052  Kingsman: The Golden Circle          (2017, action, adventure,)
0.3879  Transformers: The Last Knight        (2017, action, adventure,)

# Inclusive: 1970 and earlier
vp.vectors.query("filter-syntax-demo", vector=qv, top_k=3,
                 filter={"year": {"$lte": 1970}})

# Inclusive: 1970 and earlier
store.similarity_search_by_vector_with_score(qv, k=3,
                 filter={"year": {"$lte": 1970}})

0.5015  Hercules in the Haunted World        (1961, adventure)
0.4282  Soldier Blue                         (1970, western)
0.4257  Waterhole No. 3                      (1967, comedy western)

$gt and $gte differ on the boundary value: $gt: 2015 excludes 2015, $gte: 2015 includes it. Same for $lt vs $lte.

$in: membership in a list

Returns rows where the field equals any of the operand list values. Useful for "show me only these specific items," for filtering by a known set of categories, or for reproducing a saved selection.

vp.vectors.query("filter-syntax-demo", vector=qv, top_k=3,
                 filter={"id": {"$in": ["film-00002", "film-00004", "film-00026"]}})

store.similarity_search_by_vector_with_score(qv, k=3,
                 filter={"id": {"$in": ["film-00002", "film-00004", "film-00026"]}})

0.3383  The Lion King                        (1994, animated)
0.2303  Finding Nemo                         (2003, animation, family)

Two results came back even though top_k=3 and the $in list had three ids: the third (Toy Story) ranked below the cosine-similarity threshold for these top three slots given the LOTR query vector. Filters never invent items; they only restrict.

$nin: not in a list

Inverse of $in. Returns rows where the field is not in the operand list. The "exclude already-seen" pattern.

vp.vectors.query("filter-syntax-demo", vector=qv, top_k=3,
                 filter={"id": {"$nin": ["film-00023", "film-00027", "film-00035"]}})

store.similarity_search_by_vector_with_score(qv, k=3,
                 filter={"id": {"$nin": ["film-00023", "film-00027", "film-00035"]}})

0.5015  Hercules in the Haunted World        (1961, adventure)
0.4968  The Secret of the Sword              (1985, animation)
0.4767  Hellboy II: The Golden Army          (2008, action, superhero)

Without the $nin the LOTR films would have been the top three (they're the centroid's source). The filter excludes them and surfaces the next-most-similar items the user hasn't seen.

Composing filters

Implicit AND across fields

When a filter dict has multiple keys, all of them must be true. This is the most common composition.

vp.vectors.query("filter-syntax-demo", vector=qv, top_k=3,
                 filter={"genre": "animated", "year": {"$gte": 2010}})

store.similarity_search_by_vector_with_score(qv, k=3,
                 filter={"genre": "animated", "year": {"$gte": 2010}})

0.2553  Rio 2                                (2014, animated)

Only one row in the corpus satisfies both genre = animated AND year ≥ 2010 AND ranks well against the LOTR-shaped query vector. The narrower the AND, the fewer the survivors.

$and explicit

The same query written explicitly. Use this form when you need to compose $and under an $or (see below).

vp.vectors.query("filter-syntax-demo", vector=qv, top_k=3,
                 filter={"$and": [
                     {"genre": "animated"},
                     {"year": {"$gte": 2010}},
                 ]})

store.similarity_search_by_vector_with_score(qv, k=3,
                 filter={"$and": [
                     {"genre": "animated"},
                     {"year": {"$gte": 2010}},
                 ]})

Same single result as the implicit form.

$or

Any branch matching is enough.

vp.vectors.query("filter-syntax-demo", vector=qv, top_k=3,
                 filter={"$or": [
                     {"year": {"$lt": 1950}},
                     {"year": {"$gte": 2020}},
                 ]})

store.similarity_search_by_vector_with_score(qv, k=3,
                 filter={"$or": [
                     {"year": {"$lt": 1950}},
                     {"year": {"$gte": 2020}},
                 ]})

0.4031  Bardelys the Magnificent             (1926, romantic drama)
0.3969  One Million B.C.                     (1940, fantasy)
0.3963  The Virgin of Stamboul               (1920, drama)

"Films before 1950 OR after 2020." None of the corpus is from 2020+ (the LOTR centroid pulls weakly to old films instead). The $or accepts either branch; results are ranked by cosine similarity within the matching set.

Range on one field ($gte AND $lte together)

Two operators in the same per-field dict are ANDed. The canonical date-range query:

vp.vectors.query("filter-syntax-demo", vector=qv, top_k=3,
                 filter={"year": {"$gte": 2000, "$lte": 2010}})

store.similarity_search_by_vector_with_score(qv, k=3,
                 filter={"year": {"$gte": 2000, "$lte": 2010}})

0.9093  The Lord of the Rings: The Two Towe  (2002, adventure, fantasy)
0.8905  The Lord of the Rings: The Return o  (2003, adventure, fantasy)
0.8633  The Lord of the Rings: The Fellowsh  (2001, fantasy)

You can stack any combination of operators on the same field this way: {"score": {"$gte": 0.5, "$lt": 0.9, "$ne": 0.7}} is valid (and means exactly what it looks like).

Composing $or and $and

Boolean composers nest. Here's "modern fantasy films, OR a specific pair of cartoons":

vp.vectors.query("filter-syntax-demo", vector=qv, top_k=3,
                 filter={"$or": [
                     {"$and": [
                         {"genre": "fantasy"},
                         {"year": {"$gte": 2000}},
                     ]},
                     {"id": {"$in": ["film-00002", "film-00004"]}},
                 ]})

store.similarity_search_by_vector_with_score(qv, k=3,
                 filter={"$or": [
                     {"$and": [
                         {"genre": "fantasy"},
                         {"year": {"$gte": 2000}},
                     ]},
                     {"id": {"$in": ["film-00002", "film-00004"]}},
                 ]})

0.8633  The Lord of the Rings: The Fellowsh  (2001, fantasy)
0.4225  How to Train Your Dragon 2           (2014, fantasy)
0.4131  The Legend of Hercules               (2014, fantasy)

Fellowship of the Ring satisfies both branches (fantasy + 2001, and not in the cartoon list); the $or only needs one. The other two are modern fantasy. The cartoons in the second branch didn't make the top 3 because they ranked too low against the LOTR vector.

How filters interact with top_k

A filter narrows the result set, but top_k is the size of the final result you want. The coordinator handles the impedance mismatch:

1. Default behaviour: when a filter is present, the coordinator asks workers for 100× your top_k candidates (configurable via VP_FILTER_OVER_FETCH_MULTIPLIER).
2. Adaptive over-fetch: if the coordinator has cached the filter's selectivity from prior queries, it sizes the multiplier to roughly 2 / selectivity, clamped to [2, 100].
3. Recall loop: if the post-filter survivors are still fewer than top_k, the coordinator doubles the worker batch size and re-issues, up to two retries.

What this means for you: ask for the top_k you actually want. If physically fewer rows match your filter (e.g., the filter is so selective only 3 rows in the entire collection qualify), you'll get those 3, which is the correct answer, not a bug. Otherwise you'll get the full top_k, ranked by cosine similarity to your query vector.

Common gotchas

• The vector field is not filterable. Don't put your embedding's column name in a filter clause; vectors aren't indexed for predicate evaluation.
• Field names must match exactly. Filter clauses on a field that doesn't exist in metadata silently match no rows for $eq/$gt/etc., or every row for $ne/$nin. Check vp.schema.get(collection) if you're unsure what's available.
• Type matters. {"year": {"$gte": "2000"}} (string) won't match a numeric year field. The operand's type must match the stored type.
• Don't filter post-hoc in your app. If you query top_k=10 and then drop 7 rows in Python, the database had no idea you wanted those discarded; you've made the request slower and less accurate. Push the predicate into filter= and let the database do the work.
• List elements use $in / $nin, not $eq. {"id": ["a", "b"]} is interpreted as id == ["a", "b"] (list-equals-list). For "id is one of these," use {"id": {"$in": ["a", "b"]}}.

Further reading

• The Cosine similarity vs Euclidean distance concept explains why the score column moves the way it does as you change filters.
• The auto-optimize concept covers how Vector Panda picks an index for your collection, which is what makes filter+vector queries fast even when the filter is selective.