DosNa | Distributed Object Store Numpy Array

DosNa is a Python library. DosNa has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has low support. You can install using 'pip install DosNa' or download it from GitHub, PyPI.

DosNa is intended to be a python wrapper to distribute N-dimensional arrays over an Object Store server. The main goal of DosNa is to provide easy and friendly seamless interface to store and manage N-Dimensional datasets over a remote Cloud. It is designed to be modular by defining a Connection -> Dataset -> DataChunk architecture, and supports multiple Backends and Engines that extend the base abstract model to add different functionality. Each Backend represents a different Object Store type and wraps DosNa to connect and interact with such Object Store. Engines, on the other hand, add different local (or remote) multi-threading and multi-process support. Engines act as clients to the selected Backend and parallelize (or enhance) some of the functions to access the remote data. For example, the Ceph backend works by directly wrapping Cluster and Pool/IOCtx API's from Librados and extending two new classes, Dataset and DataChunk that wrap and extend the behavior of a Librados Object (that are only accessible through looping the Pool). DosNa is based on the Librados architecture library and tries to mimic its model of an Object Store, thus dosna.Cluster and dosna.Pool are connection objects to the remote Object Store service and the Pools (or streams) within it. A dosna.Dataset will automagically distribute a N-dimensional dataset across the cluster by partitioning the whole data in smaller chunks and store them in a completely distributed fashion as dosna.DataChunks. After a chunk_size is specified, data loaded to a dosna.Dataset will be split in to chunks of size chunk_size and will create different remote objects as dosna.DataChunk, each of them corresponding to a different chunk from the original data. These dosna.DataChunk objects are distributed along the Object Store, making dosna.Dataset a wrapper (acting as a lookup table) to distribute N-dimensional datasets into many N-dimensional smaller chunks. An existing dosna.Dataset can be used as an h5py.Dataset object or a Numpy Array. This is, a dataset object supports standard slicing ds[:, :, :] (getter) and ds[:, :, :] = 5 (setter) and the dosna.Dataset object will take care behind the scenes to access all the dosna.DataChunk needed to reconstruct the desired slices and retrieve or update them accordingly.