BSD Ports

www/py-scrapy 2.18.0 NetBSD

High-level Web Crawling and Web Scraping framework

System
NetBSD
Repository
pkgsrc
Maintainer
[email protected]
Last update
Commit
65c9b1c391
License
modified-bsd
Categories
www, python
Package size
614.7 KiB, 3920.4 KiB installed

Description

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

In the other systems

The newest version among them is 2.18.0, as upstream numbers it.

SystemPortVersionVulnerabilitiesStatusMaintainerLast update
NetBSDwww/py-scrapy2.18.0 newestnone known-[email protected]2026-10-16
FreeBSDwww/py-scrapy2.18.0 newestnone known-[email protected]2026-09-30
DragonFlywww/py-scrapy2.11.1_1 behind since 2026-07-23none known-[email protected]-
GhostBSDwww/py-scrapy2.18.0 newestno data-[email protected]-

Compare across BSDs

Version history 4

  1. 2.18.0 65c9b1c391
  2. 2.19.0 666636d8db
  3. 2.18.0 3b46a04399
  4. 2.17.0 492491635b

Upstream's version as each commit to the port's Makefile set it, or as this site first saw it in the system's packages; the tree's own revisions are not versions here.

Install

Binary package:

pkgin install py312-scrapy

From pkgsrc:

cd /usr/pkgsrc/www/py-scrapy && make install clean

Packages

The version each package repository carries. “-” means the repository builds no package of it, “n/a” that there is no such repository.

RepositoryVersion
NetBSD 10.1 x86_642.18.0
NetBSD 10.1 aarch642.18.0
SmartOS / illumos x86_642.19.0
macOS 15 arm642.19.0

Dependency graph

Ports that need it on the left, what it needs to run on the right; at most 8 of each. "Dependencies" below lists them all.

Dependencies

Run dependencies 20

No port depends on this one.

Configuration

No options to configure.

Service scripts
none
Flavors
py313-scrapy-2.18.0, py314-scrapy-2.18.0

Commit history 4

DateCommitCreditsLog message
65c9b1c391 adam Forget Python 3.10
666636d8db adam py-scrapy: updated to 2.19.0
Scrapy 2.19.0 (2026-09-10) Highlights: - New ``RemoteControl`` extension which allows inspecting and controlling a running crawl over HTTP, used by the :ref:`Scrapy MCP server <using-mcp-server>` - Experimental ``aiohttp``-based download handler (now the default when running without a reactor) Modified requirements - Added support for Python 3.15. - New dependencies: - aiohttp_ >= 3.13.3 - charset-normalizer_ >= 3.4.0 - platformdirs_ >= 2.0.0 Backward-incompatible changes - When running :ref:`without a Twisted reactor <asyncio-without-reactor>`, i.e. with :setting:`TWISTED_REACTOR_ENABLED` set to ``False``, the default download handler for ``http`` and ``https`` is now …
3b46a04399 adam py-scrapy: updated to 2.18.0
Scrapy 2.18.0 (2026-08-20) Highlights: - ``HttpxDownloadHandler`` now uses `httpx2 <https://httpx2.pydantic.dev/>`__ - The Twisted-based HTTP/2 download handler is no longer experimental - ``brotli`` and Zstandard support are now always available, and :ref:`optional extras <extras>` cover the rest of the optional features - Late :class:`~scrapy.crawler.Crawler` attributes, such as :attr:`~scrapy.crawler.Crawler.stats`, now raise :exc:`RuntimeError` instead of being ``None`` before the crawl starts - Item exporters now export fields in declaration order - New :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware` - New :ref:`optimization <optimize>` page and :ref:`built-in stats reference <topics-stats-reference>`
492491635b adam py-scrapy: updated to 2.17.0
Scrapy 2.17.0 (2026-07-07) Highlights: - Security bug fixes - HTTP/2 and SOCKS proxy support for ``HttpxDownloadHandler`` - Improved settings for changing allowed TLS versions Security bug fixes - ``s3://`` requests now use HTTPS by default, instead of plaintext HTTP. Previously, :class:`~scrapy.core.downloader.handlers.s3.S3DownloadHandler` sent signed S3 requests over plaintext HTTP unless ``request.meta["is_secure"]`` was set to a true value, exposing the request path, the AWS ``Authorization`` header, the ``X-Amz-Security-Token`` header (when using temporary credentials), and the response contents to network attackers, who could also tamper with responses. See the `76g3-c3x4-crvx`_ security advisory for details. …