BentoML

BentoML is an inference platform and open-source framework for packaging, deploying, and serving machine learning and AI models of any architecture, framework, or modality. It turns trained models into standardized deployable units (Bentos) with inference code and dependencies, then builds Docker containers and runs them on Kubernetes, on-premises, in your own cloud (BYOC), or on the managed Bento Cloud platform. It handles production inference concerns such as adaptive batching, GPU scheduling, autoscaling with scale-to-zero, cold-start acceleration, observability, and multi-model pipelines, and is widely used for serving LLMs and generative AI applications.