2023

# Deepspeed - 2023 **Pages:** 21 --- ## DeepSpeed-VisualChat: Improve Your Chat Experience with Multi-Round Multi-Image Inputs **URL:** https://www.deepspeed.ai/2023/10/03/deepspeed-visualchat.html **Contents:** - DeepSpeed-VisualChat: Improve Your Chat Experience with Multi-Round Multi-Image Inputs - Contents Updated: October 3, 2023 --- ## DeepSpeed4Science：利用先进的AI系统优化技术实现科学发现 **URL:** https://www.deepspeed.ai/2023/09/18/deepspeed4science-chinese.html **Contents:** - DeepSpeed4Science：利用先进的AI系统优化技术实现科学发现 - Contents Updated: September 18, 2023 --- ## DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models **URL:** https://www.deepspeed.ai/2023/08/23/ulysses.html **Contents:** - DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models - Contents Updated: August 23, 2023 --- ## DeepSpeed Ulysses: 训练极长序列Transformer模型的系统优化 **URL:** https://www.deepspeed.ai/2023/08/23/ulysses-chinese.html **Contents:** - DeepSpeed Ulysses: 训练极长序列Transformer模型的系统优化 - Contents Updated: August 23, 2023 --- ## DeepSpeed Chat: 一键式RLHF训练，让你的类ChatGPT千亿大模型提速省钱15倍 **URL:** https://www.deepspeed.ai/2023/04/23/deepspeed-chat-chinese.html **Contents:** - DeepSpeed Chat: 一键式RLHF训练，让你的类ChatGPT千亿大模型提速省钱15倍 - Contents Updated: April 23, 2023 --- ## DeepSpeed ZeRO++: LLMやチャットモデルの訓練を劇的に高速化 – 通信オーバヘッドを1/4に大幅削減 - **URL:** https://www.deepspeed.ai/2023/06/21/zeropp-japanese.html **Contents:** - DeepSpeed ZeRO++: LLMやチャットモデルの訓練を劇的に高速化 – 通信オーバヘッドを1/4に大幅削減 - - Contents Updated: June 21, 2023 --- ## DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference **URL:** https://www.deepspeed.ai/2023/11/05/deepspeed-fastgen.html **Contents:** - DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference - Contents Updated: November 5, 2023 --- ## DeepSpeed-VisualChat: 複数ラウンド・複数画像の入力が可能なAIチャット体験を実現 **URL:** https://www.deepspeed.ai/2023/10/03/deepspeed-visualchat-japanese.html **Contents:** - DeepSpeed-VisualChat: 複数ラウンド・複数画像の入力が可能なAIチャット体験を実現 - Contents Updated: October 3, 2023 --- ## DeepSpeed-FastGen: MIIとDeepSpeed-InferenceによるLLMのための高速なテキスト生成 **URL:** https://www.deepspeed.ai/2023/11/05/deepspeed-fastgen-japanese.html **Contents:** - DeepSpeed-FastGen: MIIとDeepSpeed-InferenceによるLLMのための高速なテキスト生成 - Contents Updated: November 5, 2023 --- ## Zero Inference **URL:** https://www.deepspeed.ai/2023/09/12/ZeRO-Inference.html **Contents:** - Zero Inference - Contents title: “ZeRO-Inference: 20X faster inference through weight quantization and KV cache offloading” excerpt: “” link: https://github.com/deepspeedai/DeepSpeedExamples/blob/master/inference/huggingface/zero_inference/README.md date: 2023-09-12 00:09:00 tags: inference ZeRO quantization English — Updated: September 12, 2023 --- ## DeepSpeed Ulysses: Transformerモデルを非常に長いシーケンスで訓練するための最適化 **URL:** https://www.deepspeed.ai/2023/08/23/ulysses-japanese.html **Contents:** - DeepSpeed Ulysses: Transformerモデルを非常に長いシーケンスで訓練するための最適化 - Contents Updated: August 23, 2023 --- ## DeepSpeed-VisualChat：多轮图像+文字，为你展现不一样的AI聊天魅力 **URL:** https://www.deepspeed.ai/2023/10/03/deepspeed-visualchat-chinese.html **Contents:** - DeepSpeed-VisualChat：多轮图像+文字，为你展现不一样的AI聊天魅力 - Contents Updated: October 3, 2023 --- ## DeepSpeed ZeRO++: A leap in speed for LLM and chat model training with 4X less communication **URL:** https://www.deepspeed.ai/2023/06/21/zeropp.html **Contents:** - DeepSpeed ZeRO++: A leap in speed for LLM and chat model training with 4X less communication - Contents Updated: June 21, 2023 --- ## Announcing the DeepSpeed4Science Initiative: Enabling large-scale scientific discovery through sophisticated AI system technologies **URL:** https://www.deepspeed.ai/2023/09/18/deepspeed4science.html **Contents:** - Announcing the DeepSpeed4Science Initiative: Enabling large-scale scientific discovery through sophisticated AI system technologies - Contents Updated: September 18, 2023 --- ## Scaling Large-Scale Generative Mixture-of-Expert Multimodal Model With VL-MoE **URL:** https://www.deepspeed.ai/2023/03/30/multi-modal.html **Contents:** - Scaling Large-Scale Generative Mixture-of-Expert Multimodal Model With VL-MoE - Contents The field of Artificial Intelligence-Generated Content (AIGC) is rapidly growing, with the goal of making content creation more efficient and accessible. One of the most exciting areas of AIGC is the development of large-scale multi-modal models like Flamingo, BLIP, and GPT4, which can accept inputs from multiple resources, e.g., image, text, audio, etc., and generate a variety of formats as outputs. For example, image creation can be made through stable diffusion and DALLE using the prompt text, and the new feature in the coming Office can create slides with texts, images, animations, etc., by leveraging the power of the new Microsoft Office Copilot. Scaling up the model size is one common approach to boost usability and capability of AIGC tasks. However, simply scaling up dense architectures (e.g., from GPT-1 to GPT-3) is usually extremely resource-intensive and time-consuming for both model training and inference. One effective way to tackle this challenge is to apply mixture of experts (MoE). In particular, recent text-based MoE and vision-based MoE studies have demonstrated that MoE models can significantly reduce the training and resource cost as compared to a quality-equivalent dense model, or produce a higher quality model under the same training budget. Up to now, the effectiveness of jointly training MoE for multi-modal models remains not well understood. To explore this important capability, DeepSpeed team is proud to announce our first large-scale generative mixture-of-expert (MoE) multimodal model, named VL-MoE. Figure 1: The new encoding process in our VL-MoE for various modality inputs, for which gray and colored blocks indicate non-activated and activated modules, respectively. Specifically, we incorporate the MoE structure into the classical single-tower multi-modal model by comprising of the following components: (1) a shared self-attention module across modalities, (2) a pool of modality-specific experts in the feed-forward network (FFN), and (3) a sparse gated MoE extended from the dense FFN. Subsequently, under the same amount of training resources as that used in VLMO (200k training steps), we demonstrate VL-MoE’s advantages over the state-of-the-art dense counterparts in the following two aspects: (1) VL-MoE can achieve significant accuracy improvement in comparison to its dense counterparts. Table 1 demonstrates that under the same training budget (i.e., have the same number of activated parameters for each token), VL-MoE Base with 32 experts achieves better accuracy than the VLMO-Base dense model on all four vision-language datasets. (2) VL-MoE achieves similar model quality with a much smaller activated number of parameters compared to its dense counterparts. Our results show that the finetuning performance of our VL-MoE is similar to that of the 3.1X larger VLMO-Large dense model (i.e., 3.1X more activated number of parameters per token). This can directly translate to approximately 3.1X training cost reduction as the training FLOPs for transformers are proportional to the activated model size per token. Table 1: Comparison of finetuning accuracy results for different models used in vision-language classification tasks and image-text retrieval tasks. A sophisticated MoE model design requires a highly efficient and scalable training system that can support multi-dimensional parallelism and efficient memory management. DeepSpeed MoE training system offers such advanced capabilities including easy-to-use APIs enabling flexible combinations of data, tensor, and expert parallelism. Furthermore, DeepSpeed MoE enables larger model scale than state-of-the-art systems by exploiting expert parallelism and ZeRO optimizations together. By leveraging the DeepSpeed MoE system, VL-MoE Base with 32 experts achieves similar model quality as VLMO-dense Large with about 2.5x training speedup. DeepSpeed MoE system is already open-sourced and can be easily used as plug-and-play component to achieve high-performance low-cost training for any large-scale MoE models. The tutorial of how to use DeepSpeed MoE is available here. VL-MoE is currently in the process of being integrated as a model example of DeepSpeed Examples. Please stay tuned for our upcoming updates on this thread. Updated: March 30, 2023 --- ## DeepSpeed-FastGen：通过 MII 和 DeepSpeed-Inference 实现 LLM 高吞吐量文本生成 **URL:** https://www.deepspeed.ai/2023/11/05/deepspeed-fastgen-chinese.html **Contents:** - DeepSpeed-FastGen：通过 MII 和 DeepSpeed-Inference 实现 LLM 高吞吐量文本生成 - Contents Updated: November 5, 2023 --- ## DeepSpeed4Scienceイニシアティブ: 洗練されたAIシステムのテクノロジーにより大規模な科学的発見を可能に **URL:** https://www.deepspeed.ai/2023/09/18/deepspeed4science-japanese.html **Contents:** - DeepSpeed4Scienceイニシアティブ: 洗練されたAIシステムのテクノロジーにより大規模な科学的発見を可能に - Contents Updated: September 18, 2023 --- ## DeepSpeed Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales **URL:** https://www.deepspeed.ai/2023/04/23/deepspeed-chat.html **Contents:** - DeepSpeed Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales - Contents Updated: April 23, 2023 --- ## DeepSpeed ZeRO++：降低4倍网络通信，显著提高大模型及类ChatGPT模型训练效率 **URL:** https://www.deepspeed.ai/2023/06/21/zeropp-chinese.html **Contents:** - DeepSpeed ZeRO++：降低4倍网络通信，显著提高大模型及类ChatGPT模型训练效率 - Contents Updated: June 21, 2023 --- ## DeepSpeed主要技術の概要紹介 **URL:** https://www.deepspeed.ai/2023/06/06/deepspeed-overview-japanese.html **Contents:** - DeepSpeed主要技術の概要紹介 - Contents 我々が研究開発しているDeepSpeedについて、主要技術を日本語で説明した資料を公開しました。GPT3やChatGPTのような生成型AIのための大規模言語モデルを含む、様々な深層学習の訓練や推論に容易に適用でき、モデルの大規模化、高速化、コスト削減を可能にします。こちらよりダウンロードしてください。 Updated: June 6, 2023 --- ## DeepSpeed Chat: ChatGPTライクなモデルを簡単・高速・低コストに、あらゆるスケールで学習 **URL:** https://www.deepspeed.ai/2023/04/23/deepspeed-chat-japanese.html **Contents:** - DeepSpeed Chat: ChatGPTライクなモデルを簡単・高速・低コストに、あらゆるスケールで学習 - Contents Updated: April 23, 2023 ---