textproc/py-tokenizers 0.23.1_2 MidnightBSD

Fast state-of-the-art tokenizers optimized for research and production

System: MidnightBSD
Repository: mports
Maintainer: [email protected]
Last update: 2026-07-09
Commit: dee1e47e52
License: Apache-2.0
Categories: textproc, python

Description

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

Main features:

Train new vocabularies and tokenize, using today's most used tokenizers.
Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU.
Easy to use, but also extremely versatile.
Designed for research and production.
Normalization comes with alignments tracking. It's always possible to get the part of the original sentence that corresponds to a given token.
Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.

In the other systems

System

Port

Version

MidnightBSD

textproc/py-tokenizers

0.23.1_2

FreeBSD

textproc/py-tokenizers

0.23.3_1

OpenBSD

textproc/py-tokenizers

0.23.1p0

DragonFly

textproc/py-tokenizers

0.21.1

HardenedBSD

textproc/py-tokenizers

0.23.2_1

GhostBSD

textproc/py-tokenizers

0.23.2_1

Install

Binary package:

mport install ${PYTHON_PKGNAMEPREFIX}tokenizers

From mports:

cd /usr/mports/textproc/py-tokenizers && make install clean

To depend on this port in another port's Makefile:

RUN_DEPENDS=	${PYTHON_PKGNAMEPREFIX}tokenizers>0:textproc/py-tokenizers

Dependencies

As this port's Makefile names them.

Build 1

${PYTHON_PKGNAMEPREFIX}maturin>=1.0<2.0 : devel/py-maturin

Library 1

libonig.so : devel/oniguruma

Runtime 1

${PYTHON_PKGNAMEPREFIX}huggingface-hub>=0.16.4<2.0 : ai/py-huggingface-hub

Test 3

${PYTHON_PKGNAMEPREFIX}requests>0 : www/py-requests
${PYTHON_PKGNAMEPREFIX}numpy>0 : math/py-numpy
${PYTHON_PKGNAMEPREFIX}datasets>0 : ai/py-datasets

Build dependencies 2

Run dependencies 2

Required by 3

Configuration

No options to configure.

USES: cargo python
Service scripts: none

Files

distinfo 166

File

Size

SHA256

tokenizers-0.23.1.tar.gz

365748

1feeeadf865a7915adc25445dea30e9933e593c31bb96c277cee36de227c8bfa

rust/crates/ahash-0.8.12.crate

43413

5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75

rust/crates/aho-corasick-1.1.4.crate

184015

ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301

rust/crates/anstream-1.0.0.crate

28916

824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d

rust/crates/anstyle-1.0.14.crate

17787

940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000

rust/crates/anstyle-parse-1.0.0.crate

22042

52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e

rust/crates/anstyle-query-1.1.5.crate

10264

40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc

rust/crates/anstyle-wincon-3.0.11.crate

12638

291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d

rust/crates/anyhow-1.0.102.crate

48658

7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c

rust/crates/autocfg-1.5.0.crate

18729

c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8

rust/crates/base64-0.13.1.crate

61002

9e1b586273c5702936fe7b7d6896644d8be71e6314cfe09d3167c95f712589e8

rust/crates/bitflags-2.11.1.crate

49082

c4512299f36f043ab09a583e57bceb5a5aab7a73db1805848e8fef3c9e8c78b3

rust/crates/bumpalo-3.20.2.crate

97696

5d20789868f4b01b2f2caec9f5c4e0213b41e3e5702a50157d699ae31ced2fcb

rust/crates/castaway-0.2.4.crate

12546

dec551ab6e7578819132c713a93c022a05d60159dc86e7a7050223577484c55a

rust/crates/cc-1.2.61.crate

97163

d16d90359e986641506914ba71350897565610e87ce0ad9e6f28569db3dd5c6d

rust/crates/cfg-if-1.0.4.crate

9360

9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801

rust/crates/colorchoice-1.0.5.crate

8187

1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570

rust/crates/compact_str-0.9.0.crate

72135

3fdb1325a1cece981e8a296ab8f0f9b63ae357bd0784a9faaf548cc7b480707a

rust/crates/console-0.16.3.crate

34041

d64e8af5551369d19cf50138de61f1c42074ab970f74e99be916646777f8fc87

rust/crates/crossbeam-deque-0.8.6.crate

22471

9dd111b7b7f7d55b72c0a6ae361660ee5853c9af73f70c3c2ef6858b950e2e51

Master sites 1

PYPI

Commit history 2

Date Commit Credits Log message

2026-07-09

dee1e47e52

Lucas Holt

ai: add new category and move AI/ML ports from misc

Create the ai category (Artificial intelligence and machine learning) and relocate 42 AI/ML ports from misc into it. - Register ai in Mk/bsd.mport.mk VALID_CATEGORIES and the top-level Makefile - Add ai/Makefile; drop the moved entries from misc/Makefile (also removes the stale linux-claude SUBDIR left over from the claude-code rename) - git mv 42 ports and set their primary CATEGORIES to ai - Rewrite misc/<port> dependency origins to ai/<port> in external dependents (devel/py-beartype, math/py-faiss, multimedia/py-torchcodec, textproc/py-tokenizers) and among the moved ports themselves - Record the origin migrations in MOVED Ports moved: chatgpt-shell-cli, claude-code, claude-code-legacy, clawhub, codex, gemini-cli, github-copilot-cli, grok-cli, lean-ctx, llama-cpp, ollama, …

2026-07-02

c475656795

Lucas Holt

Fix test phase handling for run 641 failures

Skip Magus test phase for ports that declare tests disabled, teach muon-backed meson ports to use muon test, and make Python test helpers load fake-installed modules. Also fix Python 3.12 default transition issues in py-installer and several custom port test targets. AI-Assisted-by: OpenAI Codex <[email protected]> Signed-off-by: Lucas Holt <[email protected]>