So far the story has been about words: a model reads a sentence and hands back a list of numbers — coordinates on a map where things with similar meaning sit close together. That's an embedding, and it's how a computer comes to "know" that dog and puppy are related.

Here's the part that surprises people: the exact same idea works on things that aren't words at all. A photo can become coordinates. A ten-second sound can become coordinates. A movie scene can become coordinates. And once everything — words, pictures, sounds — can be turned into points on a map, "find things that are alike" works across all of them, in ways that feel genuinely futuristic the first time you see it.

Let's build it up one sense at a time, then get to the trick that ties them together.

A photo becomes coordinates

Start with an image. To a computer a photo is just a grid of colored dots — millions of red, green, and blue values. On their own those numbers say nothing about what's in the picture. Two photos of the same dog, one in sunlight and one in shade, have wildly different color values. Compare them dot-for-dot and they look unrelated.

An image embedding model fixes that the same way a text model does: it looked at an enormous number of pictures and learned to boil each one down to a set of coordinates that capture what the picture is about rather than its raw pixels. After training, a photo of a golden retriever in sunlight and the same dog in shade land in nearly the same spot — because the model learned to see past lighting and angle to the content. A photo of a bicycle lands far away. "Nearby means similar" now means similar-looking, or really similar in content: same subject, same scene, same style.

So "find me more photos like this one" becomes the identical geometry question it was for text — which point is nearest? — just on a map of images.

How did the model learn what a picture is "about" without anyone labeling every photo? Much the way the text model did: enormous exposure and a clever exercise. Show it millions of images and have it learn to tell which crops came from the same photo, or which pictures were captioned similarly, and it's forced to develop an internal sense of visual content. Same recipe as words — learn from massive examples, store the result as position — different sense.

Sound and video: same recipe, more slicing

Audio works the same way, once you realize a sound is a wiggle over time — air pressure rising and falling — which a computer stores as a long list of numbers already. An audio embedding model learns to turn a clip into coordinates that capture its character: two recordings of birdsong land near each other; a jackhammer lands far off. Now "find sounds like this" is, again, a nearest-point question. The same approach underlies "what song is this?" features and searching a library of sound effects by vibe.

Video is almost a cheat: a video is just images in a stack, with sound running alongside. The usual move is to sample it — pull, say, one frame per second, embed each frame like a photo, and optionally embed the audio track too. A two-hour film becomes a few thousand frame-points on the map. Now you can search inside the movie: describe a scene and jump to the frames that match it. That's not hypothetical — it's exactly what our movie-scene search project does, one frame per second across a handful of films.

Finding the frame you remember. On the left, a film is sampled at roughly one frame per second, each frame becoming a point on the map. On the right, a typed description — "a car chase at night" — lands near the matching frame and surfaces its exact timestamp. Describe the scene you remember; jump to the exact frame.

The pattern by now is unmistakable. Whatever the medium — a paragraph, a photo, a birdsong, a film — the recipe is the same: a model trained on a mountain of examples turns the thing into coordinates where similar content sits nearby. The senses differ; the trick doesn't.

The same recipe for every sense. A paragraph, a photo, a soundwave, and a filmstrip each pass through an embedding model and come out as coordinates on a map — the film sampled at about one frame per second. Different senses, identical trick: similar content sits nearby.

The real magic: one shared map

Everything so far had words on one map, images on another, sounds on a third. Useful, but separate — you'd search photos with a photo, text with text.

Now the twist that makes people sit up. Some models are trained to place different kinds of things on the same map. The most famous family does it for text and images together: it learns from hundreds of millions of pictures paired with their captions, with one goal — put each picture and its caption in the same spot. Do that at scale and something remarkable happens. The word "a dog catching a frisbee" and an actual photo of a dog catching a frisbee end up as neighbors — not because they share pixels or letters (they share neither), but because they mean the same thing, and the model was trained to place same-meaning things together regardless of whether they're words or pixels.

Once text and images live on one shared map, the walls between them fall down:

That shared-map idea is why "type a sentence, get the right photo" works at all. It isn't the computer reading the image filename or a caption someone typed. It's two very different things — language and pixels — genuinely meeting in the same space because a model learned to put them there.

One shared map is the real magic — text and images in the same space. The phrase "a dog catching a frisbee" and an actual photo of a dog catching a frisbee land in the same spot, because same meaning goes to the same place. A typed query then pulls back matching photo results. Type a sentence, get the right photo — because words and pixels finally share a map.

Why this matters for what you build

The freeing part is that your vector database doesn't care which sense the coordinates came from. To the database, a paragraph, a photo, a song, and a movie frame are all just points on a map — it stores them, organizes them into fast-searchable neighborhoods, and finds nearest matches, exactly the same way for each. All the medium-specific cleverness lives in the embedding model you choose; the storage-and-search layer underneath is identical.

That means the moment you understand it for text, you understand it for everything. Pick a model for your medium (or a shared-space model if you want to cross between media), turn your things into coordinates, store them, and search by nearness. Words were just the easiest example to start with — the map was never really about language. It was always about meaning, and meaning turns out to have a shape no matter which sense it arrives through.

Closing

The same trick, all the way down: turn a thing into coordinates where similar things sit close, then ask what's nearby. It started with words because words were easiest to explain, but photos, sounds, and video play by the identical rules — and when a model is trained to draw one map for several senses at once, you get the near-magic of searching pictures with sentences. Different doors, same room.

If you'd like to walk through building one of these end to end, the projects do exactly that — including the image search and movie-scene search that this shared-map idea makes possible.