latent semantic indexing defined
Latent Semantic Indexing Defined Latent semantic indexing, by definition, is a mathematical or statistical technique for extracting and representing the similarity of meaning of words and passages by analysis of large bodies of text. The definition may be a little difficult to understand, but basically latent semantic indexing takes the keywords you put into your search engine and go through each and every web page searching out the best results for the key words you are seeking. There are several different mappings for latent semantic indexing from high dimensional to low dimensional spaces. LSI chooses the optimal mapping in a sense that minimizes the distance. Choosing the number of dimensions is a unique problem. A reduction can remove much of the noise while keeping too few dimensions may lose important information. LSI performance is improved considerably after ten to twenty dimensions and peaks at seventy to one hundred dimensions. Then it slowly begins to diminish again. There is a pattern of performance that is observed with other datasets as well. Latent semantic indexing considers pages that have many words in common and close in meaning, sorts them out, and presents them to the seeker. The result is an LSI indexed database with similarity and values that are calculated for every content word and phrase. In response to a query, the LSI database returns the pages it sees fit best to the keywords. The algorithm doesn't understand anything about what the words mean and does not require an exact match to return results that are useful to the seeker.