mirror of
https://git.FreeBSD.org/ports.git
synced 2025-01-29 10:18:30 +00:00
3395f95680
robots from accessing parts of a web site. The parsed files are kept in a WWW::RobotRules object, and this object provides methods to check if access to a given URL is prohibited. WWW: http://search.cpan.org/dist/WWW-RobotRules/ This new port is needed to update www/p5-libwww.
7 lines
303 B
Plaintext
7 lines
303 B
Plaintext
This module parses /robots.txt files which are used to forbid conforming
|
|
robots from accessing parts of a web site. The parsed files are kept in
|
|
a WWW::RobotRules object, and this object provides methods to check if
|
|
access to a given URL is prohibited.
|
|
|
|
WWW: http://search.cpan.org/dist/WWW-RobotRules/
|