Welcome to Efficient-Transformers Documentation!

Release Documents

• Efficient Transformer Library - 1.20.0 Release Notes
  • Newly Supported Models
  • Key Features & Enhancements
  • Embedding Model Upgrades
  • Fine-Tuning Support

Getting Started

• Introduction Qualcomm efficient-transformers library
• Supported Features
• Validated Models
  • Text-only Language Models
    • Text Generation Task
  • Embedding Models
    • Text Embedding Task
  • Multimodal Language Models
    • Vision-Language Models (Text + Image Generation)
    • Audio Models
• Models Coming Soon

Installation

• Pre-requisites
• Installation
  • 1. Download Apps SDK
  • 2. Install Efficient-Transformers
    • Using GitHub Repository
• Sanity Check

Inference on Cloud AI 100

• Quick Start
  • Transformed models and QPC storage
  • Command Line Interface Execution
    • Inference
      • Export
      • Compile
      • Execute
      • Infer
    • Finetune
  • QEFF Auto Class Execution
    • 1. Model download and Optimize for Cloud AI 100
    • 2. Export and Compile with one API
    • 3. Execute
• Fetaures Enablement Guide
  • Continuous Batching
  • Multi-Qranium Inference
  • QNN Compilation via Python API
  • Draft-Based Speculative Decoding

API Reference

• QEfficient Auto Classes
  • QEFFAutoModelForCausalLM
    • High-Level API
      • QEFFAutoModelForCausalLM.from_pretrained()
      • QEFFAutoModelForCausalLM.export()
      • QEFFAutoModelForCausalLM.compile()
      • QEFFAutoModelForCausalLM.generate()
  • QEFFAutoModel
    • High-Level API
      • QEFFAutoModel.from_pretrained()
      • QEFFAutoModel.export()
      • QEFFAutoModel.compile()
      • QEFFAutoModel.generate()
  • QEffAutoPeftModelForCausalLM
    • High-Level API
      • QEffAutoPeftModelForCausalLM.from_pretrained()
      • QEffAutoPeftModelForCausalLM.export()
      • QEffAutoPeftModelForCausalLM.compile()
      • QEffAutoPeftModelForCausalLM.generate()
  • QEffAutoLoraModelForCausalLM
    • High-Level API
      • QEffAutoLoraModelForCausalLM.from_pretrained()
      • QEffAutoLoraModelForCausalLM.export()
      • QEffAutoLoraModelForCausalLM.compile()
      • QEffAutoLoraModelForCausalLM.generate()
  • QEFFAutoModelForImageTextToText
    • High-Level API
      • QEFFAutoModelForImageTextToText.from_pretrained()
  • QEFFAutoModelForSpeechSeq2Seq
    • High-Level API
      • QEFFAutoModelForSpeechSeq2Seq.from_pretrained()
      • QEFFAutoModelForSpeechSeq2Seq.export()
      • QEFFAutoModelForSpeechSeq2Seq.compile()
      • QEFFAutoModelForSpeechSeq2Seq.generate()
• CLI API Reference
  • QEfficient.cloud.infer
  • QEfficient.cloud.execute
  • QEfficient.cloud.compile
  • QEfficient.cloud.export
  • QEfficient.cloud.finetune

QAIC Finetune

• Finetune Infra
  • Installation
  • Finetuning
  • Dataset Details
  • Usage
    • Single SOC finetuning on QAIC
    • Distributed training(DDP) on QAIC
  • Visualization
  • Some features/functionalities of fine-tuning stack:
    • 🔧 Steps to Fine-Tune with a Custom Dataset

Blogs

• Train anywhere, Infer on Qualcomm Cloud AI 100
• How to Quadruple LLM Decoding Performance with Speculative Decoding (SpD) and Microscaling (MX) Formats on Qualcomm® Cloud AI 100
• Power-efficient acceleration for large language models – Qualcomm Cloud AI SDK
• Qualcomm Cloud AI 100 Accelerates Large Language Model Inference by ~2x Using Microscaling (Mx) Formats
• Qualcomm Cloud AI Introduces Efficient Transformers: One API, Infinite Possibilities

Reference

• Qualcomm Cloud AI home
• Qualcomm Cloud AI SDK download
• Qualcomm Cloud AI API reference
• User Guide
• OCP Microscaling Formats (MX) Specification