MangoBoost, a provider of cutting-edge system solutions designed to maximize AI data center efficiency, is announcing the launch of Mango LLMBoost™, system optimization software providing unparalleled performance improvement and efficiency for AI inference.
Through effective system schedule coordination, kernel optimization and its proprietary prefetching mechanism, reinforced by model quantization that fully takes advantage of modern GPUs, Mango LLMBoost™ achieves up to 12.6x boost in relative performance improvement and 92% cost savings compared to other popular LLM inference engines.
Mango LLMBoost™ is currently available through AWS Marketplace, with expansion to other major cloud service providers and support for on-prem deployments in the horizon.
Product Highlights:
- GPU Flexibility: Mango LLMBoost™ is compatible with all popular NVIDIA and AMD GPUs.
- Multi-Model Deployment and Management: Mango LLMBoost™ is validated across a diverse range of chat-based and multi-modal models, including Llama, Mixtral, Gemma, Qwen2, Llava, Phi3, Chameleon, MiniCPM, and GLM-v4, which can be deployed and managed on a single inference server with automated resource allocation.
- Hassle-Free Deployment: Mango LLMBoost™ provides end-to-end deployment option with MangoBoost’s web-serving and streaming APIs, and intelligently selects the best performing configuration given the GPU and the running models.
- OpenAI API Compatibility: Mango LLMBoost™can be easily integrated into existing AI applications utilizing OpenAI’s API.
“The launch of Mango LLMBoost™ represents a significant step in MangoBoost’s continued dedication to enhancing systems-level performance and efficiency. Our expertise in DPUs has been central to our mission of improving data center efficiency, and Mango LLMBoost™ expands that focus to deliver optimization at both hardware and software levels. By addressing the critical need for performance and efficiency in AI inference workloads, we’re enabling businesses to achieve more with their existing infrastructure,” says Jangwoo Kim, CEO of MangoBoost.
Explore AITechPark for the latest advancements in AI, IOT, Cybersecurity, AITech News, and insightful updates from industry experts!
