textproc/py-tiktoken 0.12.0_6 MidnightBSD

Fast BPE tokeniser for use with OpenAI's models

System: MidnightBSD
Repository: mports
Maintainer: [email protected]
License: mit
Categories: textproc, python

Description

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

In the other systems

System

Port

Version

MidnightBSD

textproc/py-tiktoken

0.12.0_6

FreeBSD

textproc/py-tiktoken

0.14.0_2

OpenBSD

textproc/py-tiktoken

0.13.0

DragonFly

textproc/py-tiktoken

0.9.0_1

HardenedBSD

textproc/py-tiktoken

0.14.0_1

GhostBSD

textproc/py-tiktoken

0.14.0_1

Install

Binary package:

mport install ${PYTHON_PKGNAMEPREFIX}tiktoken

From mports:

cd /usr/mports/textproc/py-tiktoken && make install clean

To depend on this port in another port's Makefile:

RUN_DEPENDS=	${PYTHON_PKGNAMEPREFIX}tiktoken>0:textproc/py-tiktoken

Dependencies

As this port's Makefile names them.

Build 3

${PYTHON_PKGNAMEPREFIX}setuptools>=62.4 : devel/py-setuptools
${PYTHON_PKGNAMEPREFIX}setuptools-rust>=1.5.2 : devel/py-setuptools-rust
${PYTHON_PKGNAMEPREFIX}wheel>=0 : devel/py-wheel

Runtime 2

${PYTHON_PKGNAMEPREFIX}regex>=2022.1.18 : textproc/py-regex
${PYTHON_PKGNAMEPREFIX}requests>=2.26.0 : www/py-requests

Build dependencies 3

Run dependencies 2

Required by 6

Configuration

Options name: textproc_tiktoken

Option

Default

Description

BLOBFILE

off

Blob file support

USES: cargo python
Service scripts: none

Files

distinfo 32

File

Size

SHA256

tiktoken-0.12.0.tar.gz

37806

b18ba7ee2b093863978fcb14f74b3707cdc8d4d4d3836853ce7ec60772139931

rust/crates/aho-corasick-1.1.3.crate

183311

8e60d3430d3a69478ad0993f19238d2df97c507009a52b3c10addcd7f6bcb916

rust/crates/autocfg-1.5.0.crate

18729

c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8

rust/crates/bit-set-0.5.3.crate

14470

0700ddab506f33b20a03b13996eccd309a48e5ff77d0d95926aa0210fb4e95f1

rust/crates/bit-vec-0.6.3.crate

19927

349f9b6a179ed607305526ca489b34ad0a41aed5f7980fa90eb03160b69598fb

rust/crates/bstr-1.12.0.crate

351557

234113d19d0d7d613b40e86fb654acf958910802bcceab913a4f9e7cda03b1a4

rust/crates/fancy-regex-0.13.0.crate

85930

531e46835a22af56d1e3b66f04844bed63158bc094a628bec1d321d9b4c44bf2

rust/crates/heck-0.5.0.crate

11517

2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea

rust/crates/indoc-2.0.6.crate

17164

f4c7245a08504955605670dbf141fceab975f15ca21570696aebe9d2e71576bd

rust/crates/libc-0.2.177.crate

792045

2874a2af47a2325c2001a6e6fad9b16a53b802102b528163885171cf92b15976

rust/crates/memchr-2.7.6.crate

97616

f52b00d39961fc5b2736ea853c9cc86238e165017a493d1d5c8eac6bdc4cc273

rust/crates/memoffset-0.9.1.crate

9032

488016bfae457b036d996092f6cb448677611ce4449e970ceaf42695203f218a

rust/crates/once_cell-1.21.3.crate

34534

42f5e15c9953c5e4ccceeb2e7382a716482c34515315f7b03532b8b4e8393d2d

rust/crates/portable-atomic-1.11.1.crate

185506

f84267b20a16ea918e43c6a88433c2d54fa145c92a811b5b047ccbe153674483

rust/crates/proc-macro2-1.0.101.crate

53886

89ae43fd86e4158d6db51ad8e2b80f313af9cc74f5c0e03ccb87de09998732de

rust/crates/pyo3-0.26.0.crate

1.151579e+06

7ba0117f4212101ee6544044dae45abe1083d30ce7b29c4b5cbdfa2354e07383

rust/crates/pyo3-build-config-0.26.0.crate

34309

4fc6ddaf24947d12a9aa31ac65431fb1b851b8f4365426e182901eabfb87df5f

rust/crates/pyo3-ffi-0.26.0.crate

78247

025474d3928738efb38ac36d4744a74a400c901c7596199e20e45d98eb194105

rust/crates/pyo3-macros-0.26.0.crate

8906

2e64eb489f22fe1c95911b77c44cc41e7c19f3082fc81cce90f657cdc42ffded

rust/crates/pyo3-macros-backend-0.26.0.crate

81809

100246c0ecf400b475341b8455a9213344569af29a3c841d29270e53102e0fcf

Master sites 1

PYPI