BSD Portswww/crawl 0.4nb14 NetBSD
Small and efficient HTTP crawler
- System
- NetBSD
- Repository
- pkgsrc
- Maintainer
- [email protected]
- Categories
- www
- Package size
- 32.2 KiB, 76.3 KiB installed
Description
The crawl utility starts a depth-first traversal of the web at the specified URLs. It stores all JPEG images that match the configured constraints. Crawl is fairly fast and allows for graceful termination. After terminating crawl, it is possible to restart it at exactly the same spot where it was terminated. Crawl keeps a persistent database that allows multiple crawls without revisiting sites.
The main features of crawl are:
* Saves encountered images or other media types * Media selection based on regular expressions and size contraints * Resume previous crawl after graceful termination * Persistent database of visited URLs * Very small and efficient code * Asynchronous DNS lookups * Supports robots.txt
In the other systems
Install
Binary package:
pkgin install crawlFrom pkgsrc:
cd /usr/pkgsrc/www/crawl && make install cleanPackages
The version each package repository carries. “-” means the repository builds no package of it, “n/a” that there is no such repository.
| Repository | Version |
|---|---|
| NetBSD 10.1 x86_64 | 0.4nb14 |
| NetBSD 10.1 aarch64 | 0.4nb14 |
| SmartOS / illumos x86_64 | - |
| macOS 15 arm64 | 0.4nb14 |
Dependencies
No port depends on this one.