General Index (academia)

The General Index is a free-to-use database, which when compressed takes up 8.5 terabytes. It was created by technologist Carl Malamud and his nonprofit foundation Public Resource. As of 2021^[update], it contains words and phrases from more than 107 million academic papers.^[1][2]

It consists of a table of n-grams (a contiguous sequence of n items) derived from the full text of the articles along with tables of associated keywords and metadata.^[3] It is intended to ease computerized analysis of the scientific literature, which has been hindered by widespread copyright restrictions limiting access by researchers to the full text.

The initial version, comprising the raw database tables without any search engine front-end, was released by the Internet Archive on October 7, 2021.^[1]

See also

• Machine learning
• Open access