BSD Portswww/colly 2.1.0_23 DragonFly
Elegant scraper and crawler framework for Golang
- System
- DragonFly
- Repository
- DPorts
- Maintainer
- [email protected] (as in FreeBSD)
- License
- APACHE20
- Categories
- www
- Package size
- 30833.1 KiB, 101871.7 KiB installed
Description
With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.
Features: * Clean API * Fast (>1k request/sec on a single core) * Manages request delays and maximum concurrency per domain * Automatic cookie and session handling * Sync/async/parallel scraping * Distributed scraping * Caching * Automatic encoding of non-unicode responses * Robots.txt support * Google App Engine support
In the other systems
| System | Port | Version |
|---|---|---|
| DragonFly | www/colly | 2.1.0_23 |
| FreeBSD | www/colly | 2.3.0_2 |
| HardenedBSD | www/colly | 2.3.0 |
| GhostBSD | www/colly | 2.2.0_6 |
Install
Binary package:
pkg install collyFrom DPorts:
cd /usr/dports/www/colly && make install cleanDependencies
No port depends on this one.