Common Crawl

Nonprofit web crawling organization / From Wikipedia, the free encyclopedia

Dear Wikiwand AI, let's keep it short, summarize this topic like I'm... Ten years old or a College student

Common Crawl is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public.[1][2] Common Crawl's web archive consists of petabytes of data collected since 2011.[3] It completes crawls generally every month.[4]

Quick facts: Type of business, Available in, Headquar...
Common Crawl
Type of business501(c)(3) non-profit
Available inEnglish
HeadquartersSan Francisco, California; Los Angeles, California, United States
Founder(s)Gil Elbaz
Key peoplePeter Norvig, Nova Spivack, Carl Malamud, Kurt Bollacker, Joi Ito
URLcommoncrawl.org
Close

Common Crawl was founded by Gil Elbaz.[5] Advisors to the non-profit include Peter Norvig and Joi Ito.[6] The organization's crawlers respect nofollow and robots.txt policies. Open source code for processing Common Crawl's data set is publicly available.

The Common Crawl dataset includes copyrighted work and is distributed from the US under fair use claims. Researchers in other countries have made use of techniques such as shuffling sentences or referencing the common crawl dataset to work around copyright law in other legal jurisdictions.[7]