pythonfix
petastorm 0.12.1
Petastorm is a library enabling the use of Parquet storage from Tensorflow, Pytorch, and other Pytho
Petastorm is a library enabling the use of Parquet storage from Tensorflow, Pytorch, and other Python-based ML training frameworks.
Stars: 1787, Watchers: 1787, Forks: 286, Open Issues: 178The uber/petastorm repo was created 6 years ago and the last code push was 10 months ago.
The project is very popular with an impressive 1787 github stars!
How to Install petastorm
You can install petastorm using pip
pip install petastorm
or add it to a project with poetry
poetry add petastorm
Package Details
- Author
- Uber Technologies, Inc.
- License
- Apache License, Version 2.0
- Homepage
- https://github.com/uber/petastorm
- PyPi:
- https://pypi.org/project/petastorm/
- GitHub Repo:
- https://github.com/uber/petastorm
Classifiers
Related Packages
Errors
A list of common petastorm errors.
Code Examples
Here are some petastorm code examples and snippets.
GitHub Issues
The petastorm package has 178 open issues on GitHub
- Varying number of examples passed by DataLoader to Pytorch Lightning network
- Remove very old pickle compatibility code modifying old atg package names
- Support for parquet files with nested structures
- Support for Azure Blob Storage and Azure Data Lake
Related Packages & Articles
onnx 1.17.0
Open Neural Network Exchange (ONNX) is an open ecosystem that empowers AI developers to choose the right tools as their project evolves. ONNX provides an open source format for AI models, both deep learning and traditional ML. It defines an extensible computation graph model, as well as definitions of built-in operators and standard data types. Currently we focus on the capabilities needed for inferencing (scoring).
horovod 0.28.1
Horovod is a powerful distributed training framework for Python that allows you to train deep learning models across multiple GPUs and servers quickly and efficiently. It falls under the category of distributed computing libraries. Built on top of TensorFlow, PyTorch, and other popular deep learning frameworks, Horovod simplifies the process of scaling up your model training by handling the complexities of distributed training under the hood.
datasets 3.0.1
HuggingFace community-driven open-source library of datasets
transformers 4.45.2
State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
thinc 9.1.1
A refreshing functional take on deep learning, compatible with your favorite libraries
nlp 0.4.0
HuggingFace/NLP is an open library of NLP datasets.
keras 3.6.0
Keras is a deep learning API written in Python, running on top of the machine learning platform TensorFlow. The core data structures of Keras are layers and models. The philosophy is to keep simple things simple, while allowing the user to be fully in control when they need to (the ultimate control being the easy extensibility of the source code via subclassing).
open3d 0.18.0
Open3D: A Modern Library for 3D Data Processing.
kornia 0.7.3
Open Source Differentiable Computer Vision Library for PyTorch