Top Qs
Timeline
Chat
Perspective

DeepSpeed

Microsoft open source library From Wikipedia, the free encyclopedia

DeepSpeed
Remove ads

DeepSpeed is an open source deep learning optimization library for PyTorch.[1]

Quick Facts Original author(s), Developer(s) ...
Remove ads

Library

The library is designed to reduce computing power and memory use and to train large distributed models with better parallelism on existing computer hardware.[2][3] DeepSpeed is optimized for low latency, high throughput training. It includes the Zero Redundancy Optimizer (ZeRO) for training models with 1 trillion or more parameters.[4] Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under MIT License and available on GitHub.[5]

The team claimed to achieve up to a 6.2x throughput improvement, 2.8x faster convergence, and 4.6x less communication.[6]

Remove ads

See also

References

Further reading

Loading content...
Loading content...
Loading related searches...

Wikiwand - on

Seamless Wikipedia browsing. On steroids.

Remove ads