Embeddings & Cosine Similarity

An embedding turns a thing into a vector — a list of numbers. The thing could be a word, a sentence, an image, a user. No single number carries a name; meaning lives in the direction the vector points.

Training nudges vectors together when two things appear in similar contexts, and apart when they do not. That is the whole trick. After enough examples, dog lands near puppy, and both land far from semiconductor.

Once meaning becomes geometry, every question about similarity becomes a question about arrows.

The dot product measures agreement between two vectors: multiply matching entries and add the products, a·b = Σ a_i·b_i, where a_i and b_i are the entries of a and b. The result is large and positive when the arrows point the same way, zero when they meet at a right angle, and negative when they disagree.

But the dot product also grows with length. Double one vector and the score doubles, even though the relationship did not change. Raw dot products are therefore a poor similarity measure unless every vector has the same length.

Cosine similarity removes length: cos(a,b) = a·b / (‖a‖·‖b‖). The double bars ‖a‖ mean the length of a, so the formula divides the dot product by both lengths. What remains depends only on direction.

Cosine similarity always lands between −1 and 1. One means the same direction, zero means perpendicular, and −1 means exactly opposite. In between, the value equals the cosine of the angle θ (theta) between the arrows — which is exactly what the name says.

After vectors are normalized to length one, size carries no meaning: a short arrow and a long arrow pointing the same way have cosine 1. In many trained systems, length tracks how common a thing is or how confident the model is, not what it means.

So the standard pipeline normalizes once, stores unit-length vectors, and lets a plain dot product do the work of cosine.

Drag the query arrow around the space and watch the ranking. The arc shows the angle θ to the best match, and the list re-sorts as the direction changes.

Switch the metric and the order can change. The dot product prefers long arrows. Euclidean distance — straight-line distance between arrow tips — cares about exact position. Cosine ignores length by design: drag the query far out along the same ray and its scores barely move.

Normalization is also an engineering choice. With unit-length vectors, a dot product replaces a division, which matters when a vector search index scores millions of pairs per query. Storing fewer bytes per number cuts the bill further: one byte per dimension with little loss in retrieval quality, or trimming a 1024-dimension vector down to 256 — the 'Matryoshka' trick, where the first coordinates already carry most of the meaning.

The ranking in the demo is exact. Production systems use approximate indexes, trading a little accuracy for a large amount of speed.

One ranking hides a subtlety: two words can be close for the wrong reason. Hard negatives are pairs that look similar but mean different things. Training on them sharpens the space and separates lookalikes.

Line up many queries against many candidates and retrieval becomes one table of cosine values. Picking the top k from that table is semantic search — the core of every system that finds documents to feed a model.

Cosine is not the only option. Euclidean distance is common in clustering and image work. Dot product is standard inside attention, where learned projections already control the scale. The right choice is the metric the vectors were trained with.

The problems below compute cosine between vectors, build cosine tables, and rank embeddings by similarity.