Third Workshop on
Benchmarking Machine Learning Workloads on
Emerging Hardware

To be held with Fifth Conference on Machine Learning and Systems (MLSys) on September 1, 2022

Santa Clara, USA


Program

8:00 AM - 8:10 AM: Welcome

8:10 AM - 9:00 AM: Keynote - Sophia Shao (UC Berkeley)

9:00 AM - 10:00 AM: Paper Session 1 - Best Paper Finalists

9:00 AM - 9:30 AM: FastML Science Benchmarks: Accelerating Real-Time Scientific Edge Computing - Javier Duarte (UC San Diego), Nhan Tran, Ben Hawks, Christian Herwig (Fermilab), Jules Muhizi, Shvetank Prakash, Vijay Janapa Reddi (Harvard)

9:30 AM - 10:00 AM: MetaBench: Real-Time Multi-Modal Benchmark for Metaverse - Hyoukjun Kwon, Krishnakumar Nair, Jinook Song (Meta), Colby Banbury, Mark Mazumder (Harvard), Peter Capak, Yu-Hsin Chen, Liangzhen Lai (Meta), Tushar Krishna (Georgia Tech), Harshit Khaitan, Vikas Chandra (Meta),Vijay Janapa Reddi (Harvard)

10:00 AM - 10:30 AM: Morning Break

10:30 AM - 11:10 AM: Invited Talk - John Owens (UC Davis)

11:10 AM - 11:50 AM: Paper Session 2

11:10 AM - 11:30 AM: Benchmarking and Accelerating Session-Based Recommendation Model on Heterogeneous Accelerators - Mayank Mishra, Ravi Singh, Rekha Singhal (TCS Research)

11:30 AM - 11:50 AM: Optimizing Data Collection in Deep Reinforcement Learning - James Gleeson, Daniel Snider (University of Toronto, Vector Institute), Moshe Gabel, Eyal deLara (University of Toronto), Gennady Pekhimenko (University of Toronto, Vector Institute)

11:50 PM - 1:30 PM: Lunch

1:30 PM - 2:20 PM: Invited Talk - Venkatram Vishwanath (Argonne National Laboratory)

2:20 PM - 3:00 PM: Paper Session 3

2:20 PM - 2:40 PM: Open-Source FPGA-ML Co-Design for the MLPerf Tiny Benchmark - Michaela Blott (AMD), Hendrik Borras (Heidelberg University), Giuseppe Di Guglielmo (Columbia), Javier Duarte (UC San Diego), Nicolo Ghielmetti (CERN), Ben Hawks (Fermilab), Scott Hauck, Shih-Chieh Hsu (University of Washington), Ryan Kastner, Jason Liang, Andres Meza (UC San Diego), Jules Muhizi (Fermilab, Harvard), Tai Nguyen, Rushil Roy (UC San Diego), Nhan Tran (Fermilab), Yaman Umuroglu (AMD), Olivia Weng (UC San Diego), Aidan Yokuda (University of Washington)

2:40 PM - 3:00 PM: Operation-Level Performance Benchmarking of Graph Neural Networks for Scientific Applications - Ryien Hosseini (Argonne National Laboratory University of Michigan), Filippo Simini, Venkatram Vishwanath (Argonne National Laboratory)

3:00 PM - 3:30 PM: Afternoon Break

3:30 PM - 4:45 PM: Panel Discussion - Jinjun Xiong (University of Buffalo), Bilge Acun (Meta), Zongwei Zhou (Google), John Owens (UC Davis), Vijay Janapa Reddi (Harvard)

4:45 PM - 5:00 PM: Best Paper Award and Wrap-Up

About

With evolving system architectures, hardware and software stacks, diverse machine learning (ML) workloads, and data, it is important to understand how these components interact with each other. Well-defined benchmarking procedures help evaluate and reason the performance gains with ML workload-to-system mappings. We welcome all novel submissions in benchmarking machine learning workloads from all disciplines, such as image and speech recognition, language processing, drug discovery, simulations, and scientific applications.

Key problems that we seek to address are:
(i) which representative ML benchmarks cater to workloads seen in industry, national labs, and interdisciplinary sciences;
(ii) how to characterize the ML workloads based on their interaction with hardware;
(iii) which novel aspects of hardware, such as heterogeneity in compute, memory, and networking, will drive their adoption;
(iv) performance modeling and projections to next-generation hardware.

Along with selected publications, the workshop program will also have experts in these research areas presenting their recent work and potential directions to pursue.

The program details from the previous workshop held with MLSys'20 and MLSys'21 can be found here and here.

Call for Papers

We solicit both full papers (8-10 pages) and short/position papers (4-6 pages). Submissions are not double blind (author names must be included). The page limit includes figures, tables, and appendices, but excludes references. Please use standard LaTeX or Word ACM templates. All submissions will need to be made via EasyChair (submission website: here). Each submission will be reviewed by at least three reviewers from the program committee. Papers will be reviewed for novelty, quality, technical strength, and relevance to the workshop. All accepted papers will be published here. A Best Paper Award will be awarded to an outstanding submission.

Important dates


Submission Deadline: April 18, 2022
Acceptance Notification: May 6, 2022
Camera Ready Submission Deadline: July 15, 2022
Workshop date: September 1, 2022
All deadlines are at midnight anywhere on earth (AoE), and are firm.

Organization

Organizing Committee

  • Tom St. John, Cruise (tomstjohn617@gmail.com)

  • Murali Emani, Argonne National Laboratory (memani@anl.gov)

Program Committee

Prasanna Balaprakash (ANL)
Wenqian Dong (UC Merced)
Steven Farrell (LBNL)
David Kaeli (Northeastern)
Jiajia Li (William & Mary)
Pinar Muyan-Ozcelik (CSU Sacramento)
Gennady Pekhimenko (U Toronto)
Ananda Samajdar (Georgia Tech)
Radoyeh Shojaei (UC Davis)
Karthik Swaminathan (IBM)
Michael Wyatt (LLNL)
Amin Farmahini (Cruise)