DeepSpeed | deep learning optimization library | Machine Learning library

by microsoft Python Version: 0.14.4 License: Apache-2.0

X-Ray Key Features Code Snippets(3)Community Discussions(2)Vulnerabilities Install Support

kandi X-RAY | DeepSpeed Summary

DeepSpeed is a Python library typically used in Institutions, Learning, Education, Artificial Intelligence, Machine Learning, Deep Learning, Pytorch, Tensorflow applications. DeepSpeed has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has medium support. You can install using 'pip install DeepSpeed' or download it from GitHub, PyPI.

DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.

Support

Quality

Security

License

Reuse

Support

DeepSpeed has a medium active ecosystem.

It has 25854 star(s) with 3090 fork(s). There are 287 watchers for this library.

There were 2 major release(s) in the last 6 months.

There are 740 open issues and 984 have been closed. On average issues are closed in 92 days. There are 149 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of DeepSpeed is 0.14.4

Quality

DeepSpeed has 0 bugs and 0 code smells.

Security

DeepSpeed has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

DeepSpeed code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

DeepSpeed is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

DeepSpeed releases are available to install and integrate.

Deployable package is available in PyPI.

Build file is available. You can build the component from source.

Installation instructions, examples and code snippets are available.

DeepSpeed saves you 10527 person hours of effort in developing the same functionality from scratch.

It has 45293 lines of code, 2986 functions and 265 files.

It has high code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed DeepSpeed and discovered the below as its top functions. This is intended to give you an instant insight into DeepSpeed implemented functionality, and help decide if they suit your requirements.

Performs a single step
Convert bytes to human readable string
Calculate the size of the log
Add a message to the log
Reduce buffer_error
Flattens a nested dictionary
Compress a cupy
Compute elasticity configuration
Check if target deepspeed version is compatible with elasticity
Run all_to_all_to_all
Calculate the loss function
Run all gather
Sets up parameters for real optimizer
Configure the model
Execute backward pass on the given buffer
Execute forward pass
Perform the forward attention
Defines the training
Calculate a single step
Performs compressed union of allreduce
Parse command line arguments
Perform a single step
Forward computation
Create fp16 partitioned partitions
Backward computation
Matrix multiplication

Get all kandi verified functions for this library.

DeepSpeed Key Features

No Key Features are available at this moment for DeepSpeed.

DeepSpeed Examples and Code Snippets

CPM-Pretrain,预训练与设置脚本

Python

Lines of Code : 67

License : Permissive (MIT)

Copy

#每台机器的显卡数量
GPUS_PER_NODE=4

#支持机器之间通信的IP地址
MASTER_ADDR=localhost

#支持机器之间通信的端口
MASTER_PORT=8888

#本次训练一共涉及几台机器
NNODES=1

#当前机器的序号，编号从0开始
NODE_RANK=0

# 整体模型划分为几层，进行流水并行
pp_size=2

# 每层模型划分为几块，进行模型并行
mp_size=2

#层数
NLAYERS=2

#输入层大小
NHIDDEN=128

#单张卡每

CPM-Pretrain,预训练与设置脚本

Python

Lines of Code : 67

License : Permissive (MIT)

Copy

#每台机器的显卡数量
GPUS_PER_NODE=4

#支持机器之间通信的IP地址
MASTER_ADDR=localhost

#支持机器之间通信的端口
MASTER_PORT=8888

#本次训练一共涉及几台机器
NNODES=1

#当前机器的序号，编号从0开始
NODE_RANK=0

# 整体模型划分为几层，进行流水并行
pp_size=2

# 每层模型划分为几块，进行模型并行
mp_size=2

#层数
NLAYERS=2

#输入层大小
NHIDDEN=128

#单张卡每

strategy.rst

Python

Lines of Code : 41

License : No License

Copy

# Training with the DistributedDataParallel strategy on 4 GPUs
trainer = Trainer(strategy="ddp", accelerator="gpu", devices=4)

# Training with the DistributedDataParallel strategy on 4 GPUs, with options configured
trainer = Trainer(strategy=DDPStra

Community Discussions

Trending Discussions on DeepSpeed

Running out of memory with pytorch

Mix pytorch lightning with vanilla pytorch

QUESTION

Running out of memory with pytorch

Asked 2021-Aug-23 at 13:17

I am trying to train a model using huggingface's wav2vec for audio classification. I keep getting this error:

...

ANSWER

Answered 2021-Aug-23 at 13:17

You might use the DataParallel or DistributedDataParallel framework in Pytorch

Source https://stackoverflow.com/questions/68624392

QUESTION

Mix pytorch lightning with vanilla pytorch

Asked 2021-Jul-14 at 18:27

I am doing a meta learning research and am using the MAML optimization provided by learn2learn. However as one of the baseline, I would like to test a non-meta-learning approach, i.e. the traditional training + testing.

Due to the lightning's internal usage of optimizer it seems that it is difficult to make the MAML work with learn2learn in lightning, so I couldn't use lightning in my meta-learning setup, however for my baseline, I really like to use lightning in that it provides many handy functionalities like deepspeed or ddp out of the box.

Here is my question, other than setting up two separate folders/repos, how could I mix the vanilia pytorch (learn2learn) with pytorch lightning (baseline)? What is the best practice?

Thanks!

...

ANSWER

Answered 2021-Jul-14 at 18:27

Decided to answer my question. So I ended up using the torch lightning's manual optimization so that I can customize the optimization step. This would make both approaches using the same framework, and I think is better than maintaining 2 separate repos.

Source https://stackoverflow.com/questions/68359563

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install DeepSpeed

The quickest way to get started with DeepSpeed is via pip, this will install the latest release of DeepSpeed which is not tied to specific PyTorch or CUDA versions. DeepSpeed includes several C++/CUDA extensions that we commonly refer to as our 'ops'. By default, all of these extensions/ops will be built just-in-time (JIT) using torch's JIT C++ extension loader that relies on ninja to build and dynamically link them at runtime. Note: PyTorch must be installed before installing DeepSpeed. After installation, you can validate your install and see which extensions/ops your machine is compatible with via the DeepSpeed environment report. If you would like to pre-install any of the DeepSpeed extensions/ops (instead of JIT compiling) or install pre-compiled ops via PyPI please see our advanced installation instructions. On Windows you can build wheel with following steps, currently only inference mode is supported.
Install pytorch, such as pytorch 1.8 + cuda 11.1
Install visual cpp build tools, such as VS2019 C++ x64/x86 build tools
Launch cmd console with Administrator privilege for creating required symlink folders
Run python setup.py bdist_wheel to build wheel in dist folder