Semantic Search

Ordinary keyword search matches the words you type: search for "child" and you get documents containing the word "child." Semantic search matches the meaning instead, so the same query can also surface passages about a "kid," a "son," or a "daughter" that never use the word you searched for. It works by using a language model to turn text into a numerical representation of its meaning, so that passages about similar things sit close together and can be retrieved by closeness rather than exact overlap.

For language work this is useful when you are digging through a large or messy collection and don't know the exact wording to look for, such as scattered legacy materials recovered through scraping. You can search for an idea and find the relevant records even if they are catalogued in different terms.

The caveats follow from how it works. The quality depends on the model having actually learned the language, so for low-resource languages semantic search can work poorly or not at all. It can also return things that are related but not what you wanted, since closeness in meaning is not the same as relevance. And because the search runs through a model, the usual questions apply about what text you are sending where, and who can see it (see Data sovereignty).

Created · Updated
Supported By the National Science Foundation Award 2542375.