alt text

Introduction Qualcomm efficient-transformers library

Train anywhere, Infer on Qualcomm Cloud AI with a Developer-centric Toolchain

This library provides reimplemented blocks of LLMs which are used to make the models functional and highly performant on Qualcomm Cloud AI 100. We support wide range of models architectures, for easy efficient deployment on Cloud AI 100 cards. Users only need to provide model card from HuggingFace or Path to the local model and the library will take care of transforming model to it’s efficient implementation for Cloud AI 100.

For other models, there is comprehensive documentation to inspire upon the changes needed and How-To(s).

Typically for LLMs, the library provides:

  1. Reimplemented blocks from Transformers which enable efficient on-device retention of intermediate states. read more here

  2. Graph transformations to enable execution of key operations in lower precision

  3. Graph transformations to replace some operations to other mathematically equivalent operations that are efficient/supported on HW backend

  4. Handling for underflow and overflows in lower precision

  5. Patcher modules to map weights of original model’s operations to updated model’s operations

  6. Exporter module to export the model source into a ONNX Graph.

  7. Sample example applications and demo notebooks

  8. Unit test templates.

Latest news :