sentencepiece 0.2.1

Unsupervised text tokenizer and detokenizer.

Stars: 11651, Watchers: 11651, Forks: 1322, Open Issues: 25

The google/sentencepiece repo was created 8 years ago and the last code push was 1 weeks ago.
The project is extremely popular with a mindblowing 11651 github stars!

How to Install sentencepiece

You can install sentencepiece using pip

pip install sentencepiece

or add it to a project with poetry

poetry add sentencepiece

Package Details

Author

None

License

None

Homepage

None

PyPi:

https://pypi.org/project/sentencepiece/

GitHub Repo:

https://github.com/google/sentencepiece

Classifiers

• Software Development/Libraries/Python Modules
• Text Processing/Linguistic

Related Packages

Errors

A list of common sentencepiece errors.

Code Examples

Here are some sentencepiece code examples and snippets.

GitHub Issues

The sentencepiece package has 25 open issues on GitHub

• Bump wheel from 0.45.1 to 0.46.2 in /.github/workflows/requirements
• Update README: small correction of a linguistic error
• Bump filelock from 3.20.1 to 3.20.3 in /.github/workflows/requirements
• Bump urllib3 from 2.6.0 to 2.6.3 in /.github/workflows/requirements
• Bump the github-actions group with 3 updates
• Bump the build-time-deps group in /.github/workflows/requirements with 3 updates
• Suppress the warnings of python module
• Use official abseil-cpp
• With unigram algorithm, constant piece at end of each sentences does not become a token

See more issues on GitHub