Simplifying Kubernetes with Microsoft's New AI-Powered Operator on Azure: An Overview

Simplifying Kubernetes with Microsoft’s New AI-Powered Operator on Azure: An Overview

Xuanyu Han/Getty Images

If you want to run artificial intelligence (AI) and machine learning applications such as large language models (LLMs) at scale, you must run them on Kubernetes . However, mastering Kubernetes – everyone’s favorite container orchestrator – isn’t easy. That’s where Kubernetes Al toolchain operator -- the latest addition to Microsoft’s Azure Kubernetes Service (AKS) -- comes in.

Also: Microsoft Azure introduces Radius open-source development platform

AKS already makes Kubernetes on Azure easier. Instead of working it out by hand, AKS’s built-in code-to-cloud pipelines and guardrails give you a faster way to start developing and deploying cloud-native apps in Azure. With its unified management and governance for on-premises, edge, and multi-cloud Kubernetes clusters, AKS also makes it simpler (there’s no such thing as “simple” when it comes to Kubernetes) to integrate with Azure security, identity, cost management, and migration services.

What Kubernetes AI toolchain operator brings to the table are automated ways to run open-source software AI/ML workloads cost-effectively and with less manual configuration. It also automates LLM model deployment on AKS across available CPU and GPU resources by selecting the optimally sized infrastructure for your LLM or other project.

Newsletters

ZDNET Tech Today

ZDNET’s Tech Today newsletter is a daily briefing of the newest, most talked about stories, five days a week.

See all

AI toolchain operator does this by automatically provisioning the necessary GPU nodes and setting up the associated inference server as an endpoint server to your AI models. An inference server, such as Hugging Face’s 7B or NVIDIA Triton Inference Server , applies trained AI models to incoming data to make real-time decisions. Inference is the process of running live data through a trained AI model to make a prediction or solve a task. Using this add-on reduces your onboarding time and enables you to focus on AI model usage and development rather than infrastructure setup.

It also makes it possible to easily split inferencing across multiple lower-GPU-count virtual machines (VMs). This means you can run your LLMs on more Azure regions, thus eliminating wait times for Azure regions with higher GPU-count VMs and lowering overall cost. In other words, you can automatically run your LLMs on lower-power, less-expensive regions. Yes, you may lose processing power, but not all jobs require higher horsepower.

Also: I went hands-on with Microsoft’s new AI features, and these 5 are the most useful

Making it easier to set up, you can also choose from preset models with AKS-hosted images. This significantly reduces your overall service setup time. Once it’s been up and running for a while, you can then adjust your Azure model to better fit your workload.

Additionally, Azure Kubernetes Fleet Manager enables multi-cluster and at-scale scenarios for AKS clusters. Platform admins who are managing Kubernetes fleets with many clusters often face challenges staging their updates in a safe and predictable way. This allows admins to orchestrate updates across multiple clusters by using update runs, stages, and groups. Since AI/ML workloads tend to be very demanding, this makes managing them much easier.

In short, if you want to do serious work with AI/ML on Azure, the Kubernetes Al toolchain operator demands your attention.