BSD Ports

textproc/py-tiktoken 0.13.0 OpenBSD

fast BPE tokenizer for OpenAI models

System
OpenBSD
Repository
OpenBSD ports
Maintainer
The OpenBSD ports mailing-list <[email protected]>
Port added
Last update
Commit
a106314782
Categories
textproc, lang/python, lang/rust

Description

tiktoken is a fast BPE (byte pair encoding) tokenizer for use with OpenAI's models.

In the other systems

SystemPortVersion
OpenBSDtextproc/py-tiktoken0.13.0
FreeBSDtextproc/py-tiktoken0.14.0_2
DragonFlytextproc/py-tiktoken0.9.0_1
MidnightBSDtextproc/py-tiktoken0.12.0_6
HardenedBSDtextproc/py-tiktoken0.14.0_1
GhostBSDtextproc/py-tiktoken0.14.0_1

Install

Binary package:

pkg_add py3-tiktoken

From the ports tree:

cd /usr/ports/textproc/py-tiktoken && make install

Dependencies

Build dependencies 8
Run dependencies 3

No port depends on this one.

Commit history 1

DateCommitCreditsLog message
a106314782 daniel import py-tiktoken; ok kirill@