textproc/py-tokenizers 0.23.3_1 FreeBSD
Fast state-of-the-art tokenizers optimized for research and production
- System
- FreeBSD
- Repository
- FreeBSD ports
- Maintainer
- [email protected]
- Last update
- Commit
9c878207a1- License
- APACHE20
- Categories
- textproc, python
- Package size
- 1461.3 KiB, 4987.3 KiB installed
Description
Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
Main features:
- Train new vocabularies and tokenize, using today's most used tokenizers.
- Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU.
- Easy to use, but also extremely versatile.
- Designed for research and production.
- Normalization comes with alignments tracking. It's always possible to get the part of the original sentence that corresponds to a given token.
- Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.
In the other systems
| System | Port | Version |
|---|---|---|
| FreeBSD | textproc/py-tokenizers | 0.23.3_1 |
| OpenBSD | textproc/py-tokenizers | 0.23.1p0 |
| DragonFly | textproc/py-tokenizers | 0.21.1 |
| MidnightBSD | textproc/py-tokenizers | 0.23.1_2 |
Install
Binary package:
pkg install py312-tokenizersFrom the ports tree:
cd /usr/ports/textproc/py-tokenizers/ && make install cleanTo depend on this port in another port's Makefile:
RUN_DEPENDS= py312-tokenizers>0:textproc/py-tokenizersPackages
The version each package repository carries. “-” means the repository builds no package of it, “n/a” that there is no such repository.
| Repository | aarch64 | amd64 | armv6 | armv7 | i386 |
|---|---|---|---|---|---|
| FreeBSD:14:latest | 0.23.2_1 | 0.23.3_1 | - | 0.21.4 | 0.23.3_1 |
| FreeBSD:14:quarterly | 0.23.2_1 | 0.23.2_1 | n/a | 0.21.2 | 0.23.2_1 |
| FreeBSD:15:latest | 0.23.3_1 | 0.23.3_1 | n/a | 0.21.4 | 0.21.4 |
| FreeBSD:15:quarterly | 0.23.2_1 | 0.23.2_1 | n/a | n/a | n/a |
| FreeBSD:16:latest | 0.23.2_1 | 0.23.3_1 | n/a | n/a | n/a |
| FreeBSD:16:quarterly | n/a | n/a | n/a | n/a | n/a |
Dependencies
As this port's Makefile names them.
Build 1
${PYTHON_PKGNAMEPREFIX}maturin>=1.0<2.0: devel/py-maturin
Library 1
libonig.so: devel/oniguruma
Runtime 1
${PYTHON_PKGNAMEPREFIX}huggingface-hub>=0.16.4<3.0: misc/py-huggingface-hub
Test 4
${PYTHON_PKGNAMEPREFIX}requests>0: www/py-requests${PYTHON_PKGNAMEPREFIX}numpy>0: math/py-numpy${PYTHON_PKGNAMEPREFIX}datasets>0: misc/py-datasets${PYTHON_PKGNAMEPREFIX}pytest-benchmark>0: devel/py-pytest-benchmark
The full lists below include those, and their dependencies in turn.
Build dependencies 25
- archivers/brotli
- archivers/liblz4
- archivers/zstd
- devel/gettext-runtime
- devel/libffi
- devel/libunistring
- devel/oniguruma
- devel/pkgconf
- devel/py-build
- devel/py-installer
- devel/py-maturin
- devel/py-packaging
- devel/py-pyproject-hooks
- devel/readline
- dns/libidn2
- dns/libpsl
- ftp/curl
- lang/python312
- lang/python315
- lang/rust
- math/mpdecimal
- print/indexinfo
- security/libssh2
- textproc/expat2
- www/libnghttp2
Run dependencies 39
- archivers/liblz4
- archivers/zstd
- devel/gettext-runtime
- devel/libffi
- devel/oniguruma
- devel/py-aiosignal
- devel/py-anyio
- devel/py-attrs
- devel/py-click
- devel/py-frozenlist
- devel/py-packaging
- devel/py-propcache
- devel/py-pyyaml
- devel/py-typing-extensions
- devel/readline
- dns/py-idna
- filesystems/py-fsspec
- lang/python312
- lang/python315
- math/mpdecimal
- misc/py-huggingface-hub
- misc/py-tqdm
- net/py-aiohappyeyeballs
- net/py-h11
- net/py-socksio
- print/indexinfo
- security/py-truststore
- sysutils/py-filelock
- textproc/expat2
- textproc/libyaml
- www/llhttp
- www/py-aiohttp
- www/py-h2
- www/py-hpack
- www/py-httpcore2
- www/py-httpx2
- www/py-hyperframe
- www/py-multidict
- www/py-yarl
Required by 53
- databases/py-chromadb
- deskutils/py-qmd
- graphics/py-olmocr
- misc/colibri
- misc/comfyui
- misc/py-banks
- misc/py-cohere
- misc/py-compressed-tensors
- misc/py-crewai
- misc/py-crewai-tools
- misc/py-kokoro
- misc/py-langchain-huggingface
- misc/py-litellm
- misc/py-llama-index-core
- misc/py-llama-index-embeddings-huggingface
- misc/py-llama-index-llms-ibm
- misc/py-llama-index-llms-openai
- misc/py-llama-index-llms-openai-like
- misc/py-optimum
- misc/py-peft
- misc/py-sentence-transformers
- misc/py-transformers
- misc/py-vllm
- misc/py-xgrammar
- textproc/py-docling
- textproc/py-docling-agent
- textproc/py-docling-analysis-framework
- textproc/py-docling-core
- textproc/py-docling-cvat-tools
- textproc/py-docling-demo
- textproc/py-docling-enhanced
- textproc/py-docling-enhanced-onnx
- textproc/py-docling-eval
- textproc/py-docling-glm-ocr
- textproc/py-docling-google-ocr
- textproc/py-docling-graph
- textproc/py-docling-haystack
- textproc/py-docling-hierarchical-pdf
- textproc/py-docling-ibm-models
- textproc/py-docling-jobkit
- textproc/py-docling-langchain
- textproc/py-docling-litellm-picture-description
- textproc/py-docling-mcp
- textproc/py-docling-ocr
- textproc/py-docling-ocr-mistral
- textproc/py-docling-ocr-onnxtr
- textproc/py-docling-parse
- textproc/py-docling-sdg
- textproc/py-docling-serve
- textproc/py-docling-slim
- textproc/py-extended-docling
- textproc/py-langchain-docling
- www/immich-ml
Configuration
No options to configure.
- USES
- cargo python
- Service scripts
- none
Files
distinfo 152
| File | Size | SHA256 |
|---|---|---|
| tokenizers-0.23.3.tar.gz | 385286 | cded33237c77caeef62944d32aa9a7ef42bdce2b3497e18d137e072a8c4be438 |
| rust/crates/ahash-0.8.12.crate | 43413 | 5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75 |
| rust/crates/aho-corasick-1.1.5.crate | 184315 | c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba |
| rust/crates/anstream-1.0.0.crate | 28916 | 824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d |
| rust/crates/anstyle-1.0.14.crate | 17787 | 940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000 |
| rust/crates/anstyle-parse-1.0.0.crate | 22042 | 52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e |
| rust/crates/anstyle-query-1.1.5.crate | 10264 | 40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc |
| rust/crates/anstyle-wincon-3.0.11.crate | 12638 | 291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d |
| rust/crates/autocfg-1.5.1.crate | 18911 | f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53 |
| rust/crates/base64-0.13.1.crate | 61002 | 9e1b586273c5702936fe7b7d6896644d8be71e6314cfe09d3167c95f712589e8 |
| rust/crates/bitflags-1.3.2.crate | 23021 | bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a |
| rust/crates/bitflags-2.13.1.crate | 51395 | b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da |
| rust/crates/bumpalo-3.20.3.crate | 98671 | 72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649 |
| rust/crates/castaway-0.2.4.crate | 12546 | dec551ab6e7578819132c713a93c022a05d60159dc86e7a7050223577484c55a |
| rust/crates/cc-1.4.4.crate | 100602 | 0ad534f4357a5264cce5019c989cf66a4f0dc4e0d1b1d15f8aacec0ff7360273 |
| rust/crates/cfg-if-1.0.4.crate | 9360 | 9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801 |
| rust/crates/colorchoice-1.0.5.crate | 8187 | 1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570 |
| rust/crates/compact_str-0.9.1.crate | 88299 | 9dfdd1c2274d9aa354115b09dc9a901d6c5576818cdf70d14cae2bdb47df00ab |
| rust/crates/console-0.16.4.crate | 34353 | 4fe5f465a4f6fee88fad41b85d990f84c835335e85b5d9e6e63e0d06d28cba7c |
| rust/crates/crossbeam-deque-0.8.7.crate | 24141 | 5181e0de7b61eb03a81e347d6dd8797bae9da5146707b51077e2d71a54ec0ceb |
Master sites 1
- PYPI