Advantech Solves Generative AI Market Entry Challenges through Technical Partnerships with NVIDIA, Phison, and Nota.ai

By Reece Hayden | 3Q 2024 | IN-7481

Registered users can unlock up to five pieces of premium content each month.

Log in or register to unlock this Insight.

Although largely nascent, deploying Generative Artificial Intelligence (Gen AI) at the enterprise edge would bring significant value to a number of "uncarpeted" verticals, including industrial, smart cities, healthcare, energy, transportation, and logistics/fleet management. These verticals face significant challenges around connectivity, data privacy, and growing data transmission costs, which local, edge deployment of AI models would ease.

With this in mind, Advantech, a leading provider of Artificial Intelligence of Things (AIoT) solutions, has built on its success in the machine vision market to monetize Gen AI at the edge. It recently released AIR-520, an edge Artificial Intelligence (AI) server that integrates NVIDIA RTX Graphics Processing Unit (GPU) cards, an edge AI Software Development Kit (SDK), and NVIDIA AI Enterprise to provide off-the-shelf support for Gen AI inference and even training workloads. The solution includes 10+ optimized Gen AI models including Llama3-70B. However, given the significant challenges of deploying Gen AI at the edge, Advantech has sensibly supported its product's market entry with a raft of hardware/software partnership announcements that aim to solve technical hurdles that hinder scaled enterprise adoption:

• Advantech partnered with Nota.ai, which provides hardware-aware optimization tools for edge AI through Netspresso. This platform enables training, compression, conversion, and benchmarking to develop optimized AI models for specific applications across various hardware. This reduces resource usage, deployment cost, and time to deployment for Gen AI workload. Nota.ai supports a wide range of edge devices, mainly targeting the computer vision space. This includes NVIDIA Jetson, Intel, Raspberry Pi, Arm, Renesas, and more.
• Alongside integration with Intel Geti and other software ecosystems, Advantech recently announced collaboration and integration with NVIDIA AI Enterprise. This integration enables enterprises to build on-premises solutions with customized Large Language Models (LLMs). It also enables enterprises to deploy highly optimized models/applications through NVIDIA Inference Microservices (NIMs).
• Unlike most edge AI vendors, Advantech's solution supports inference, training, and fine-tuning. To support this, it has partnered with Phison to integrate aiDAPTIV+. This ai100 Solid-State Drive (SSD), coupled with aiDAPTIV management software, facilitates Non-Volatile Memory Express (NVMe) offload to enhance performance, reduce training times, and enable large model training on resource/power-constrained devices.

Advantech's decision to partner with these vendors will certainly support its enterprise Go-to-Market (GTM) strategy by reducing technical and commercial barriers to deployments. However, even though Advantech is hitting the right notes, ABI Research expects that enterprise edge Gen AI deployment will remain constrained in the short run. ABI Research's recent forecasts suggest that edge Gen AI spending will only make up 20% of the total (across cloud, edge, and device) by 2030 (see ABI Research's Gen AI Software market data (MD-AISG-101)) with the majority of revenue still coming from the cloud/data center. This is because Gen AI edge deployment will continue to face ongoing technical and commercial challenges:

• Resource Underutilization: Deploying and running GPUs at the edge is expensive. Demand for AI processing at the edge fluctuates depending on the application contributing to various levels of utilization. This may sometimes contribute to excess processing capacity and cost inefficiency. Mitigating this requires resource transparency and automated resource sharing.
• Management and Visibility: Model deployment and management at the edge is challenging. Given that Gen AI models have shown significant accuracy and performance degradation after deployment, this creates significant risk for enterprise deployment.
• Reliability and Alignment for Mission-Critical Edge Applications: Many uncarpeted verticals running AI models at the edge are looking to support mission-critical use cases with accuracy guarantees, e.g., item classification or predictive maintenance. However, Gen AI models are not capable of providing this degree of accuracy, as most are, at best, offering approximately 90% accuracy. This will significantly inhibit any Gen AI adoption.
• Ensuring Processing Availability: Unlike the cloud, edge resources are not scalable and are inherently limited. As processes scale, resource utilization could hit 100%, meaning that certain applications cannot function on-demand. Especially for Gen AI models with significant resource usage, this could have a huge impact on application performance and increase application latency.
• Cost of Always-on Resources: Cloud resources can scale up and down based on workload processing requirements while they can also use batch processing to achieve better usage-based economics. However, edge servers must be "always-on" and available for processing. This brings significant additional costs and operational challenges (including power management and increased Capital Expenditure (CAPEX) to build sufficient capacity) for enterprise deployment.

Advantech's edge Gen AI solution has answered many of the technical questions that this nascent market poses; however, it is still a big step away from its previous edge AI solutions targeting mature, traditional AI use cases. Subsequently, Advantech and other vendors (including Adlink, Axiomtek, and others) looking to successfully crack this emerging market should employ some of the commercial recommendations below:

• Building Ready-Made Solutions with System Integrators: Edge deployment requires the convergence of Information Technology (IT) and Operational Technology (OT) and can bring significant deployment challenges, especially around data security, integration within proprietary systems, and IT management systems. This is especially important for Gen AI-supported mission-critical use cases. Solution providers should engage closely with AI proficient System Integrators (SIs) (e.g., Accenture) to support IT/OT convergence on top of Gen AI development, implementation, and management. SIs will also support differentiation and become a key channel to market for solution providers, given the breadth of Gen AI solutions being developed.
• Develop Return on Investment (ROI) Model to Showcase Value Creation: To many enterprises, cost will remain the biggest hurdle to Gen AI adoption at the edge, especially because existing IT budgets will be cannibalized by expensive Gen AI deployment. Showcasing the commercial impact of Gen AI, including value created, will be the first step in driving engagement; however, solution vendors must also highlight the Total Cost of Ownership (TCO) and cost-saving advantage of deployment at the edge versus in the cloud.
• Deploying Proofs of Concept (PoCs) with Partners across Different Verticals: Use cases for Gen AI remain immature, especially when considering uncarpeted verticals with mission-critical operations. Building PoCs internally, or preferably with partners, can educate decision makers, tap into partner expertise, and showcase the solution's value proposition.
• Leveraging Software Platform Partners to Offer Custom Models That Target Specific Use Cases: Generalized models are helpful for AI-competent enterprises to start building and playing with edge deployment; however, the long-tail of the market relies on completely turnkey solutions. Building models optimized for specific applications, targeting specific verticals, will accelerate enterprise deployment. Machine vision's commercial success has been similarly founded on the provision of highly optimized, application-specific models.
• Expanding and Updating Model Collection in Line with Market Innovation: Newmodel releases and updates in the Gen AI market are coming at a remarkable pace, with new frontier and small AI models being released daily from cloud vendors, AI specialists, and others. Although it would be ambitious to provide access to up-to-date, optimized models, vendors developing turnkey enterprise solutions must offer regular updates to ensure alignment with market innovation. Without access to these model updates, enterprises are likely to adopt a "watch and wait" mindset due to the deployment risk of "outdated models."
• Completely "Free" Access to Partner Developer Environment: Software like NVIDIA AI Enterprise has a fairly sizable cost of around US$5,000 per user per GPU. This discourages a test and evaluation mindset. Although solution vendors do not have control over this pricing, enabling cheaper or preferential access, especially during a PoC, would certainly accelerate enterprise engagement.