Introduction

Introduction

This chapter provides detailed technical information on the Confidential AI Inference, designed to ensure confidentiality, integrity, and verifiability of AI inference tasks. We use the the TEE technologies provided by NVIDIA GPU TEE and Intel TDX to secure AI workloads, allowing developers to easily deploy their LLMs in a secure environment.

Overview

Confidential inference addresses critical concerns such as data privacy, secure execution, and computation verifiability, making it indispensable for sensitive applications. As illustrated in the diagram below, people currently cannot fully trust the responses returned by LLMs from services like OpenAI or Meta, due to the lack of cryptographic verification. By running the LLM inside a TEE, we can add verification primitives alongside the returned response, known as a Remote Attestation (RA) Report. This allows users to verify the AI generation results locally without relying on any third parties.

Check the Get Started for more information.

Check the Host LLM in TEE for how to host your own private LLM in TEE.

Check the Implementation for the technical details of the Confidential AI Inference.

References

1. HCC-Whitepaper
2. Intel SGX DCAP Orientation
3. Phala's dcap-qvl
4. Automata’s Solidity Implementation
5. Phala Nvidia H200 TEE Benchmark Paper
6. Phala DeRoT Post on FlashBots forum
7. Phala Key Management Protocol Post on Flashbots forum