mirror of
https://git.FreeBSD.org/ports.git
synced 2024-12-29 05:38:00 +00:00
18 lines
516 B
Plaintext
18 lines
516 B
Plaintext
With Colly you can easily extract structured data from websites, which can be
|
|
used for a wide range of applications, like data mining, data processing or
|
|
archiving.
|
|
|
|
Features:
|
|
* Clean API
|
|
* Fast (>1k request/sec on a single core)
|
|
* Manages request delays and maximum concurrency per domain
|
|
* Automatic cookie and session handling
|
|
* Sync/async/parallel scraping
|
|
* Distributed scraping
|
|
* Caching
|
|
* Automatic encoding of non-unicode responses
|
|
* Robots.txt support
|
|
* Google App Engine support
|
|
|
|
WWW: http://go-colly.org/
|