mirror of
https://git.FreeBSD.org/ports.git
synced 2024-11-20 00:21:35 +00:00
be1be4cd15
Protego is a pure-Python robots.txt parser with support for modern conventions. It is mainly used by Scrapy application when crawling websites. WWW: https://pypi.org/project/Protego/
5 lines
184 B
Plaintext
5 lines
184 B
Plaintext
Protego is a pure-Python robots.txt parser with support for modern conventions.
|
|
It is mainly used by Scrapy application when crawling websites.
|
|
|
|
WWW: https://pypi.org/project/Protego/
|