BSD Ports

textproc/sentencepiece 0.2.2 FreeBSD

Unsupervised text tokenizer for Neural Network-based text generation

System
FreeBSD
Repository
FreeBSD ports
Maintainer
[email protected]
Last update
Commit
f3f511d80d
License
APACHE20
Categories
textproc
Package size
898.1 KiB, 6103.3 KiB installed

Description

SentencePiece is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training. SentencePiece implements subword units (e.g., byte-pair-encoding (BPE)) and unigram language model with the extension of direct training from raw sentences. SentencePiece allows us to make a purely end-to-end system that does not depend on language-specific pre/postprocessing.

In the other systems

SystemPortVersion
FreeBSDtextproc/sentencepiece0.2.2
NetBSDtextproc/sentencepiece0.2.1
DragonFlytextproc/sentencepiece0.2.0_1
MidnightBSDtextproc/sentencepiece0.2.1
HardenedBSDtextproc/sentencepiece0.2.2
GhostBSDtextproc/sentencepiece0.2.2

Install

Binary package:

pkg install sentencepiece

From the ports tree:

cd /usr/ports/textproc/sentencepiece/ && make install clean

To depend on this port in another port's Makefile:

RUN_DEPENDS=	sentencepiece>0:textproc/sentencepiece
LIB_DEPENDS=	libsentencepiece.so:textproc/sentencepiece

Packages

The version each package repository carries. “-” means the repository builds no package of it, “n/a” that there is no such repository.

Repositoryaarch64amd64armv6armv7i386
FreeBSD:14:latest0.2.20.2.2-0.2.0_10.2.2
FreeBSD:14:quarterly0.2.20.2.2n/a0.2.0_10.2.2
FreeBSD:15:latest0.2.20.2.2n/a0.2.0_10.2.0_1
FreeBSD:15:quarterly0.2.20.2.2n/an/an/a
FreeBSD:16:latest0.2.20.2.2n/an/an/a
FreeBSD:16:quarterlyn/an/an/an/an/a

Dependencies

As this port's Makefile names them.

Library 2

The full lists below include those, and their dependencies in turn.

Build dependencies 17
Run dependencies 3

Configuration

Options name: textproc_sentencepiece

OptionDefaultDescription
GPERFTOOLSoffGoogle gperftools support
USES
cmake:testing compiler:c++17-lang
Service scripts
none

Files

pkg-plist 17
  • bin/spm_decode
  • bin/spm_encode
  • bin/spm_export_vocab
  • bin/spm_normalize
  • bin/spm_train
  • include/sentencepiece.pb.h
  • include/sentencepiece_processor.h
  • include/sentencepiece_trainer.h
  • lib/libsentencepiece.a
  • lib/libsentencepiece.so
  • lib/libsentencepiece.so.0
  • lib/libsentencepiece.so.0.0.0
  • lib/libsentencepiece_train.a
  • lib/libsentencepiece_train.so
  • lib/libsentencepiece_train.so.0
  • lib/libsentencepiece_train.so.0.0.0
  • libdata/pkgconfig/sentencepiece.pc
distinfo 1
FileSizeSHA256
google-sentencepiece-v0.2.2_GH0.tar.gz1.4160863e+0792381f713e094a15a1ccff1ac4a5315a4c4b82a99ac1332d6ac53c9dc8e1bcf1

Commit history 1

DateCommitCreditsLog message
f3f511d80d Yuri Victorovich textproc/{,py-}sentencepiece: update 0.2.1 → 0.2.2