What Nvidia’s New MLPerf AI Benchmark Results Really Mean

Nvidia released results compared today to new MLPerf artificial intelligence (AI) benchmarks for its AI-focused processors. While the results looked impressive, it’s important to note that some of the comparisons they make to other systems really aren’t apples to apples. For example, the Qualcomm systems have a much smaller power footprint than the H100 and target market segments similar to the A100, where the test comparisons are much fairer.

Nvidia tested its top-of-the-line H100 system based on its latest Hopper architecture; its now mid-range A100 system focused on edge computing; and the smaller Jetson system targeting smaller individual and/or edge workloads. This is the first H100 entry and shows up to 4.5 times higher performance than the A100. According to the chart below, Nvidia has some impressive results for the top-of-the-line H100 platform.

AI Inference Inference Workloads

Nvidia used the MLPerf Inference V2.1 benchmark to assess its capabilities in various AI inference workload scenarios. Inference is different from machine learning (ML) where training models are created and systems ‘learn’.

Inference is used to run the learned models on a set of data points and obtain results. Based on conversations with companies and suppliers, we at J. Gold Associates, LLC estimate that the AI ​​inference market is many times larger in volume than the ML training market, so showing good inference benchmarks is critical to success .

Why Nvidia would use MLPerf

MLPerf is an industry standard benchmark suite that has broad input from a variety of companies and models a variety of workloads. Included are things like natural language processing, speech recognition, image classification, medical imaging, and object detection.

The benchmark is useful because it can run on a variety of machines, from high-end data centers and cloud, to smaller edge computing systems, and can provide a consistent benchmark across the products of different vendors, although not all subtests in the benchmark are performed by all testers.

It can also create scenarios for running offline, single-stream or multistream tests that create a range of AI functions to simulate a realistic example of a complete workflow pipeline (e.g. speech recognition, natural language processing, search and recommendations , text-to-speech, etc.).

Although MLPerf is widely accepted, many players believe that running only parts of the test (ResNet is the most common) is a valid indicator of their performance and these results are more widely available than the full MLPerf. Indeed, in the chart, we can see that many of the comparison chips have no test results in other components of MLPerf for comparison with the Nvidia systems, because the vendors chose not to make them.

Is Nvidia ahead of the market?

The real advantage that Nvidia has over many of its competitors is its platform approach.

While other players offer chips and/or systems, Nvidia has built a strong ecosystem with the chips, associated hardware, and a full stable of software and development systems optimized for their chips and systems. For example, Nvidia has built tools such as their Transformer Engine that increase the level of floating point computation (such as FP8, FP16, etc.) to speed up the calculations, sometimes by orders of magnitude. This gives Nvidia a strong foothold in the market as it allows developers to focus on solutions rather than trying to work on low-level hardware and related code optimizations for systems without the associated platforms.

Competitors Intel, and to a lesser extent Qualcomm, have indeed emphasized the platform approach, but the startups generally only support open source options that may not be on a par with the major vendors. Furthermore, Nvidia has optimized frameworks for specific market segments that provide a valuable starting point from which solution providers can achieve faster time-to-market with less effort. Vendors of AI chip startups cannot provide this level of resources.

The power factor

The one area that fewer companies are testing on is the amount of power needed to run these AI systems. High-end systems like the H100 can require 500-600 watts of power to operate, and most major training systems use many H100 components, possibly thousands, within their entire system. The operating costs of such large systems are therefore extremely high.

The low-end Jetson only consumes about 50-60 watts, which is still too much for many edge computing applications. Indeed, the big hyperscalers (AWS, Microsoft, Google) see all this as a problem and are building their own low-power AI accelerator chips. Nvidia is working on lower-power chips, mainly because Moore’s Law has the potential to reduce power as process nodes get smaller.

However, it must reach products in the 10-watt and lower range if it is to fully compete with newer, optimized edge processors coming to market, and companies with less powerful credentials like Qualcomm (and ARM in general). There will be many low-power applications for AI inference in which Nvidia cannot currently compete.

Nvidia’s benchmark

Nvidia has shown some impressive benchmarks for its latest hardware, and the test results show that companies should take Nvidia’s AI leadership seriously. But it’s also important to note that the potential AI market is huge and Nvidia may not be a leader in all segments, especially in the low-power segment where companies like Qualcomm could have an advantage.

While Nvidia shows a comparison of its chips with standard Intel x86 processors, it has no comparison with Intel’s new Habana Gaudi 2 chips, which are likely to show a high level of AI computing capability that some Nvidia products could approach or exceed. .

Despite these caveats, Nvidia still offers the broadest product family and its emphasis on complete platform ecosystems puts it at the forefront of the AI ​​race and will be hard for competitors to match.