A personal engineering project demonstrating the implementation of a distributed, multimodal AI inference system. Built to showcase proficiency in microservices, low-latency streaming, and system architecture.
Core Technologies
Designed to solve the challenge of high-latency HTTP requests in AI applications by utilizing streaming gRPC for internal communication.
Next.js 16 App Router handles the UI. Establishes a streaming connection to the API Gateway.
FastAPI service that authenticates via Supabase, validates schemas, and routes requests to inference nodes via gRPC.
Isolated Python service running PyTorch. Models are kept in memory for hot-path execution. Returns a stream of tokens.
service InferenceService {// Bidirectional streaming for real-time interactionrpc ChatStream (stream ChatRequest)returns (stream ChatResponse); }