Top Qs
Timeline
Chat
Perspective
Anubis (software)
Software designed to prevent scraping From Wikipedia, the free encyclopedia
Remove ads
Anubis is a software program that adds a proof of work challenge to websites before users can access them, in order to deter web scraping. It has been adopted mainly by Git forges and free and open-source software projects.[4][5]
Anubis was created by Xe Iaso in response to Amazon's web crawler overloading their Git server, as the crawler did not respect the robots.txt exclusion protocol and would work around restrictions.[4][6]
Remove ads
Adoption
It is used by a number of projects, including:[7]
- The GNOME Project's GitLab instance[4]
- The Linux kernel mailing list archives[1] and Git server[8]
- sourceware.org[7]
- FFmpeg[1][6]
- FreeBSD
- FreeDesktop
- Wine[9]
- UNESCO[6]
- The Science Olympiad Student Center[9]
- Enlightenment[9]
- FreeCAD[10]
- Krantenbank Zeeland
- ScummVM[11]
Criticism
Tavis Ormandy noted that the cost of scraping websites protected by Anubis is likely negligible: "I don’t think we reach a single cent per month in compute costs until several million sites have deployed Anubis."[12]
References
External links
Wikiwand - on
Seamless Wikipedia browsing. On steroids.
Remove ads
