sentencepiece 0.2.0

SentencePiece python wrapper

The google/sentencepiece repo was created 7 years ago and the last code push was 4 days ago.
The project is extremely popular with a mindblowing 9380 github stars!

How to Install sentencepiece

You can install sentencepiece using pip

pip install sentencepiece

or add it to a project with poetry

poetry add sentencepiece

Package Details

Author

Taku Kudo

License

Apache

Homepage

https://github.com/google/sentencepiece

PyPi:

https://pypi.org/project/sentencepiece/

GitHub Repo:

https://github.com/google/sentencepiece

Classifiers

• Software Development/Libraries/Python Modules
• Text Processing/Linguistic

Related Packages

Errors

A list of common sentencepiece errors.

Code Examples

Here are some sentencepiece code examples and snippets.

GitHub Issues

The sentencepiece package has 25 open issues on GitHub

• Fixed typo error
• Expose CalculateEntropy and SampleEncodeAndScore methods to python
• Training with sentence_iterator yields vocabulary size error
• set num_threads = 56 but only 1 thread working most of the time
• fix address sanitizers on clang problem
• address sanitizers on clang
• SentencePiece Installation Failed on M1 Mac
• fix freebsd can't find endian.h problem
• Treating tabs ('\t') as user defined symbols
• Is the loss computation in UnigramTrainer correct?

See more issues on GitHub