SAN FRANCISCO – (COMMERCIAL THREAD)–Today, MLCommons, an open engineering consortium, released new results for MLPerf Training v1.0, the organization’s benchmark machine learning training performance suite. MLPerf Training measures the time it takes to train machine learning models to a standard quality goal in a variety of tasks including image classification, object detection, NLP, recommendation, and reinforcement learning. In its fourth cycle, MLCommons added two new benchmarks to assess the performance of text-to-speech and 3D medical imaging tasks.
MLPerf Training is a comprehensive system benchmark, testing machine learning models, software and hardware. With MLPerf, MLCommons now has a reliable and consistent way to track performance improvement over time, along with the results of a “level playing field” benchmark that drives competition, which in turn drives competition. performances. Compared to the last submission cycle, the best benchmark scores improved up to 2.1 times, showing a substantial improvement in hardware, software and system scale.
Similar to previous MLPerf training results, submissions consist of two divisions: closed and open. Closed submissions use the same benchmark model to ensure a level playing field between systems, while open division participants are allowed to submit a variety of models. Submissions are further categorized by availability within each division, including off-the-shelf, preview, and RDI systems.
New MLPerf Training Benchmarks to Advance ML Tasks and Performance
As industry adoption and machine learning use cases grow, MLPerf will continue to evolve its reference suites to assess new capabilities, tasks, and performance metrics. With the MLPerf v1.0 training cycle, MLCommons has included two new benchmarks for measuring speech-to-text and 3D medical imaging performance. These new benchmarks are based on the following benchmark models:
- Speech-to-Text with RNN-T: RNN-T: Recurrent Neural Network Transducer is an Automatic Speech Recognition (ASR) model that is trained on a subset of LibriSpeech. Given a sequence of voice inputs, it predicts the corresponding text. RNN-T is MLCommons’ reference model and commonly used in production for text-to-speech systems.
- 3D medical imaging with 3D U-Net: The 3D U-Net architecture is trained on the KiTS 19 dataset to find and segment cancer cells in the kidneys. The model identifies whether each voxel in a CT scan belongs to healthy tissue or to a tumor, and is representative of many medical imaging tasks.
MLPerf v1.0 training results reinforce MLCommons’ goal of providing benchmarks and metrics that level the playing field in the industry through the comparison of ML systems, software and solutions. The latest benchmark round received submissions from 13 organizations and published over 650 peer-reviewed results for machine learning systems ranging from edge devices to data center servers. This cycle’s submissions included software and hardware innovations from Dell, Fujitsu, Gigabyte, Google, Graphcore, Habana Labs, Inspur, Intel, Lenovo, Nettrix, NVIDIA, PCL & PKU and Supermicro. To see the results, go to https://mlcommons.org/en/training-normal-10/.
“We are excited to see the continued growth and enthusiasm of the MLPerf community, especially as we are able to measure significant improvements across the industry with the MLPerf training benchmark suite,” said Victor Bittorf, co-chair of the MLPerf Training working group. Group. “Kudos to all of our authors on this v1.0 round – we are delighted to continue our work together, bringing transparency to the capabilities of the machine learning system. ”
“The industry advancements highlighted in this series of results are remarkable,” said John Tran, co-chair of the MLPerf Training Working Group. “The Training Reference Suite is central to MLCommon’s mission to advance machine learning innovation for everyone, and we are incredibly pleased with both the engagement of this cycle’s submissions,” as well as the growing interest in MLPerf’s benchmark results by companies looking to adopt AI solutions. ”
Additional information on the v1.0 training benchmarks will be available at https://mlcommons.org/en/training-normal-10/.
MLCommons is an open engineering consortium with a mission to accelerate innovation in machine learning, raise all boats and increase its positive impact on society. MLCommons’ foundation began with the MLPerf benchmark in 2018, which quickly evolved into a set of industry metrics to measure machine learning performance and promote transparency in machine learning techniques. Together with its 50+ founding partners – global technology providers, academics and researchers, MLCommons focuses on collaborative engineering work that creates tools for the entire machine learning industry through benchmarks and metrics , public datasets and best practices.