BSD Portstextproc/py-tiktoken 0.14.0_2 FreeBSD
Fast BPE tokeniser for use with OpenAI's models
- System
- FreeBSD
- Repository
- FreeBSD ports
- Maintainer
- [email protected]
- Last update
- Commit
9c878207a1- License
- MIT
- Categories
- textproc, python
- Package size
- 769.7 KiB, 2552.9 KiB installed
Description
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
In the other systems
| System | Port | Version |
|---|---|---|
| FreeBSD | textproc/py-tiktoken | 0.14.0_2 |
| OpenBSD | textproc/py-tiktoken | 0.13.0 |
| DragonFly | textproc/py-tiktoken | 0.9.0_1 |
| MidnightBSD | textproc/py-tiktoken | 0.12.0_6 |
Install
Binary package:
pkg install py312-tiktokenFrom the ports tree:
cd /usr/ports/textproc/py-tiktoken/ && make install cleanTo depend on this port in another port's Makefile:
RUN_DEPENDS= py312-tiktoken>0:textproc/py-tiktokenPackages
The version each package repository carries. “-” means the repository builds no package of it, “n/a” that there is no such repository.
| Repository | aarch64 | amd64 | armv6 | armv7 | i386 |
|---|---|---|---|---|---|
| FreeBSD:14:latest | 0.14.0_1 | 0.14.0_2 | - | 0.9.0_4 | 0.14.0_2 |
| FreeBSD:14:quarterly | 0.14.0_1 | 0.14.0_1 | n/a | 0.9.0_3 | 0.14.0_1 |
| FreeBSD:15:latest | 0.14.0_2 | 0.14.0_2 | n/a | 0.9.0_4 | 0.9.0_4 |
| FreeBSD:15:quarterly | 0.14.0_1 | 0.14.0_1 | n/a | n/a | n/a |
| FreeBSD:16:latest | 0.14.0_1 | 0.14.0_2 | n/a | n/a | n/a |
| FreeBSD:16:quarterly | n/a | n/a | n/a | n/a | n/a |
Dependencies
As this port's Makefile names them.
Build 3
${PYTHON_PKGNAMEPREFIX}setuptools>=62.4: devel/py-setuptools${PYTHON_PKGNAMEPREFIX}setuptools-rust>=1.5.2: devel/py-setuptools-rust${PYTHON_PKGNAMEPREFIX}wheel>=0: devel/py-wheel
Runtime 2
${PYTHON_PKGNAMEPREFIX}regex>=0: textproc/py-regex${PYTHON_PKGNAMEPREFIX}requests>=0: www/py-requests
The full lists below include those, and their dependencies in turn.
Build dependencies 26
- archivers/brotli
- archivers/liblz4
- archivers/zstd
- devel/gettext-runtime
- devel/libffi
- devel/libunistring
- devel/py-build
- devel/py-installer
- devel/py-packaging
- devel/py-pyproject-hooks
- devel/py-semantic-version
- devel/py-setuptools
- devel/py-setuptools-rust
- devel/py-wheel
- devel/readline
- dns/libidn2
- dns/libpsl
- ftp/curl
- lang/python312
- lang/python315
- lang/rust
- math/mpdecimal
- print/indexinfo
- security/libssh2
- textproc/expat2
- www/libnghttp2
Run dependencies 14
Required by 60
- biology/atac-seq
- biology/biostar-tools
- biology/chip-seq
- biology/py-multiqc
- biology/rna-seq
- deskutils/py-qmd
- graphics/py-olmocr
- misc/chatblade
- misc/colibri
- misc/comfyui
- misc/openclaw
- misc/py-autogen
- misc/py-banks
- misc/py-compressed-tensors
- misc/py-crewai-tools
- misc/py-kokoro
- misc/py-langchain-deepseek
- misc/py-langchain-openai
- misc/py-litellm
- misc/py-llama-index-core
- misc/py-llama-index-embeddings-huggingface
- misc/py-llama-index-llms-ibm
- misc/py-llama-index-llms-openai
- misc/py-llama-index-llms-openai-like
- misc/py-mistral-common
- misc/py-openai-whisper
- misc/py-optimum
- misc/py-peft
- misc/py-sentence-transformers
- misc/py-transformers
- misc/py-vllm
- misc/py-xgrammar
- textproc/py-docling
- textproc/py-docling-agent
- textproc/py-docling-analysis-framework
- textproc/py-docling-core
- textproc/py-docling-cvat-tools
- textproc/py-docling-demo
- textproc/py-docling-enhanced
- textproc/py-docling-enhanced-onnx
- textproc/py-docling-eval
- textproc/py-docling-glm-ocr
- textproc/py-docling-google-ocr
- textproc/py-docling-graph
- textproc/py-docling-haystack
- textproc/py-docling-hierarchical-pdf
- textproc/py-docling-ibm-models
- textproc/py-docling-jobkit
- textproc/py-docling-langchain
- textproc/py-docling-litellm-picture-description
- textproc/py-docling-mcp
- textproc/py-docling-ocr
- textproc/py-docling-ocr-mistral
- textproc/py-docling-ocr-onnxtr
- textproc/py-docling-parse
- textproc/py-docling-sdg
- textproc/py-docling-serve
- textproc/py-docling-slim
- textproc/py-extended-docling
- textproc/py-langchain-docling
Configuration
Options name: textproc_tiktoken
| Option | Default | Description |
|---|---|---|
| BLOBFILE | off | Blob file support |
- USES
- cargo python
- Service scripts
- none
Files
distinfo 28
| File | Size | SHA256 |
|---|---|---|
| tiktoken-0.14.0.tar.gz | 38898 | 231dec90efcdccf1b565a1416107736f1e09b1a08fe736ef9d6363e626d03874 |
| rust/crates/aho-corasick-1.1.5.crate | 184315 | c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba |
| rust/crates/bit-set-0.8.0.crate | 16289 | 08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3 |
| rust/crates/bit-vec-0.8.0.crate | 24132 | 5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7 |
| rust/crates/bstr-1.13.1.crate | 356631 | 6bb31b46c14244e20ee9984b11bf5c992b91fb6939fea616e3512c8baecdbe5f |
| rust/crates/fancy-regex-0.19.0.crate | 224156 | 476de73bddf2ef8490aa4ee8f1cf40b430bf1d56c48c22080e5186952cd580e6 |
| rust/crates/heck-0.5.0.crate | 11517 | 2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea |
| rust/crates/libc-0.2.189.crate | 851502 | 3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2 |
| rust/crates/memchr-2.8.3.crate | 99165 | cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98 |
| rust/crates/once_cell-1.21.4.crate | 35010 | 9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50 |
| rust/crates/portable-atomic-1.15.0.crate | 202705 | 05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85 |
| rust/crates/proc-macro2-1.0.107.crate | 59588 | 985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9 |
| rust/crates/pyo3-0.29.2.crate | 1.255741e+06 | 4688ddedf473e32662b9b067670129a8afb8c18e351482c70d62ba4a88171e8b |
| rust/crates/pyo3-build-config-0.29.2.crate | 41104 | f41027e41b4bd03f6e60f9f417fe24a6341a6bb744edd62b6f709f2a52ea30e9 |
| rust/crates/pyo3-ffi-0.29.2.crate | 90029 | e591a95526fead067432c3b3a33fc74770b87b1e04e73671090d9c2055a2b327 |
| rust/crates/pyo3-macros-0.29.2.crate | 9044 | 73225868fc1cd84eef2c3c230ddb91273bf1de46aeb8a4248da76d32a0924a1c |
| rust/crates/pyo3-macros-backend-0.29.2.crate | 93749 | 571575aa3749fa6216757dd47d2a3e7ef360f329a40f0666a9fbd14889024952 |
| rust/crates/quote-1.0.47.crate | 31622 | 1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001 |
| rust/crates/regex-1.13.1.crate | 157118 | f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d |
| rust/crates/regex-automata-0.4.18.crate | 628707 | ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2 |
Master sites 1
- PYPI