This session explores new techniques for running LLMs efficiently on client PCs and small-form-factor machines at the edge using the OpenVINO™ toolkit in combination with popular tools, libraries, and frameworks for model optimization and quantization.

Additionally, you’ll have the opportunity to gain practical experience by implementing a conversational AI voice agent using the OpenVINO toolkit and Gradio, an open source Python* package for quickly building a machine learning–based demo or web application.

This session provides:

• Techniques for deploying advanced LLMs on edge devices for a variety of industries such as healthcare and manufacturing
• How to optimize and quantize LLMs to ensure superior performance and low power consumption while reducing model size and computational demands
• How to use AI Tools to create AI applications that are powerful and energy efficient for edge computing

Skill level: Expert

OpenVINO toolkit

Optimum Intel library by Hugging Face*

Edge AI Reference Kits for OpenVINO Toolkit

Neural Network Compression Framework for OpenVINO Toolkit

Gradio on GitHub*

Gradio Quick Start