Efficient Transformer Library - 1.21.0 Release Notes

Welcome to the official release of Efficient Transformer Library v1.21.0! This release introduces advanced attention mechanisms, expanded model support, optimized serving capabilities, and significant improvements to fine-tuning and deployment workflows.

✅ All features and models listed below are available on the release/v1.21.0 branch and mainline.

---

Newly Supported Models

• Flux (Diffusers - Image Generation)

  • Diffusion-based image generation model

  • Flux.1 Schnell Example Script

• WAN (Diffusers - Video Generation)

  • Wide-Area Network Lightning support for distributed inference

  • Wan_lightning Example Script

• Qwen2.5-VL (Vision Language)

  • Executable via QEFFAutoModelForImageTextToText

  • Multi-image prompt support

  • Continuous batching enabled

  • Qwen2.5-VL Usage Guide

• Mistral 3.1 (24B)

  • Executable via QEFFAutoModelForImageTextToText

  • Mistral-3.1 Example Script

• Disaggregated serving ready via vLLM GPT-OSS

  Note: If running GPT-OSS models natively via vLLM, PR-685 of the qefficient library is required for Python 3.12 compatibility.

  • Executable via QEffAutoModelForCausalLM

  • Separate prefill and decode compilation supported

  • Disaggregated serving ready

  • GPT-OSS Example Scripts

• Olmo2

  • Executable via QEffAutoModelForCausalLM

  • Full CausalLM support with optimizations

  • Refer to Text generation Example Scripts for usage details.

• Molmo

  • Executable via QEffAutoModelForCausalLM

  • Multi-modal capabilities

  • Molmo Example Script

• InternVL 3.5 Series

  • Executable via QEffAutoModelForCausalLM

  • Full Vision-Language support

  • Multi-image handling with continuous batching

  • Refer to InternVL 3.5 Example Scripts for usage details.

• Qwen3-MOE (Mixture of Experts)

  • Executable via QEffAutoModelForCausalLM

  • Efficient expert routing

  • Qwen3-MOE Example Scripts

• Wav2Vec2 (Audio)

  • Executable via QEFFAutoModelForCTC

  • Speech recognition and audio feature extraction

  • Wav2Vec2 Example Scripts

• Multilingual-e5-Large (Embedding Model)

  • Executable via QEffAutoModel

  • Multilingual text embedding capabilities

  • Refer usage details here.

---

Key Features & Enhancements

• Framework Upgrades: Transformers 4.55, PyTorch 2.7.0+cpu, Torchvision 0.22.0+cpu

• Python Support: Requires Python 3.10

• ONNX Opset: Updated to version 17 for broader operator support

• Advanced Attention: Flux blocking support, BlockedKV attention for CausalLM models

• Diffusers Integration: Full support for diffuser-based image generation and video generation models

• Compute-Context-Length (CCL) support: To optimize the throughput when handling very large context lengths

• Prefill/Decode Separation: Support for GPT OSS using disaggregate serving models

• Continuous Batching (VLMs): Extended to Vision Language Models with multi-image handling

  • Supported models: Llava, Llava_Next, Gemma3, Mistral3, InternVL2_5, InternVL3_5, Molmo

• ONNX Sub-Functions: Feature enabling more efficient model compilation and execution on hardware. Users can enable the feature by passing use_onnx_subfunctions=True during export

• Memory Profiling: Built-in utilities for optimization analysis

• Extend on-device Sampling: Extend on-device sampling to dual QPC VLMs and Guided decoding for on-device sampling

• ONNX transform, memory & time optimizations: Optimizations for faster ONNX Transform and reduced memory footprint

• Removed platform SDK dependency: Support QPC generation on systems without the Platform SDK

• Example Scripts Revamp: New example scripts for audio, embeddings, and image-text-to-text tasks

• Onboarding Guide: Simplified setup and deployment process for new users

  • CausalLM Onboarding Guide

  • Custom ops

• Organized examples into domain-specific subdirectories Examples

---

Embedding Model Upgrades

• Multi-Sequence Length Support: Auto-selects optimal graph at runtime

• Enhanced Pooling: Flexible pooling strategies for various embedding tasks

---

Fine-Tuning Support

• Checkpoint Management: Resume from epochs with proper state restoration

• Enhanced Loss Tracking: Corrected data type handling for accurate loss computation

• Custom Dataset Support: Improved handling with better tokenization

• Device-Aware Scaling: Optimized GradScaler for multi-device training

• Comprehensive Testing: Unit tests for fine-tuning workflows

---

Efficient Transformer Library - 1.20.0 Release Notes

Welcome to the official release of Efficient Transformer Library v1.20.0! This release introduces advanced attention mechanisms, expanded model support, optimized serving capabilities, and significant improvements to fine-tuning and deployment workflows.

✅ All features and models listed below are available on the release/v1.20.0 branch and mainline.

---

Newly Supported Models

• Llama-4-Scout-17B-16E-Instruct

  • Executable via QEFFAutoModelForImageTextToText

  • Text & Image+Text support

  • Chunk attention, Single/Dual QPC support

  • Multi-image prompts enabled via VLLM interface

  • Llama4 Example Script

• Grok-1

  • Executable via QEffAutoModelForCausalLM

• Gemma3

  • Executable via QEFFAutoModelForImageTextToText

  • Text & Image+Text support

  • Sliding window support

  • Gemma3 Example Script

• SwiftKV (Llama-3.1-SwiftKV-8B-Instruct)

  • Executable via QEffAutoModelForCausalLM

  • Supports both continuous and non-continuous batching

• GGUF Models

  • Executable via QEffAutoModelForCausalLM

  • Execution support (non-quantized)

  • Example Script

• FP8 Compressed Quantization

  • Support for Llama-3.3-70B-Instruct-FP8-Dynamic

---

Key Features & Enhancements

• Transformer Upgrade: Now using version 4.51.3

• SpD & Multi-Projection Heads: Token speculation via post-attention projections

• I/O Encryption: --io-encrypt flag support in compile/infer APIs

• Separate Prefill/Decode Compilation: For disaggregated serving

• On-Device Sampling: Supported using VLLM, which reduces host-device latency for CausalLM models

---

Embedding Model Upgrades

• Flexible Pooling: Choose from standard or custom strategies

• Sentence Embedding: Now runs directly on AI100

• Multi-Seq Length Compilation: Auto-selects optimal graph at runtime

---

Fine-Tuning Support

• BERT fine-tuning support with templates and documentation

• Gradient checkpointing, device-aware GradScaler, and CLI --help added