Heterogeneous Computing: Hardware and Software Perspectives
Mohamed Zahran
books.acm.org
ISBN: 978-1-4503-6097-5hardcover
ISBN: 978-1-4503-6233-7paperback
ISBN: 978-1-4503-6100-2eBook
ISBN: 978-1-4503-6098-2ePub
Series ISSN: 2374-6769 print2374-6777 electronic
DOIs:
| 10.1145/3281649 Book | 10.1145/3281649.3281653 Chapter 3 |
| 10.1145/3281649.3281650 Preface | 10.1145/3281649.3281654 Chapter 4 |
| 10.1145/3281649.3281651 Chapter 1 | 10.1145/3281649.3281655 Chapter 5 |
| 10.1145/3281649.3281652 Chapter 2 | 10.1145/3281649.3281656 References/Index/Bio |
A publication in the ACM Books series, #26
Editor in Chief: M. Tamer Özsu, University of Waterloo
This book was typeset in Arnhem Pro 10/14 and Flama using ZzTEX.
First Edition
10 9 8 7 6 5 4 3 2 1
To my family, without whom I wouldn’t have existed in the first place and wouldn’t have managed through this life.
Contents
Chapter 1Why Are We Forced to Deal with Heterogeneous Computing?
1.2Heterogeneity beyond Our Control
1.3Heterogeneity within Our Control
1.4Seems Like Part of a Solution to Exascale Computing
Chapter 2Different Players: Heterogeneity in Computing
2.3FPGA
2.4Automata Processors
2.5Neuromorphic Chips
2.6Other Accelerators
2.7Mix-And-Match
2.8In Conclusion
Chapter 3Architecture: Heterogeneity in Design
3.1Memory System
3.2Interconnect
3.3Examples of Supercomputers
3.4Security Challenges Facing Heterogeneous Computing
3.5Bandwidth
3.6In Conclusion
4.1Wish List of a Programmer
4.2Psychology of Programming
4.3What Do We Have? The Current Status Quo
4.4In Conclusion
5.1Processing-in-Memory (PIM) / Near-Data-Processing (NDP)
5.2Exascale Computing
5.3Neuromorphic Chips
5.4Quantum Computing
Preface
The term heterogeneous computing has become famous lately (lately, meaning in the last five years!). It started infiltrating many articles. Research papers have been, and are still being, written about heterogeneous computing and its implications on both software and hardware. The definition of this term is quite straightfor-ward: executing programs on a computing platform with computing nodes of different characteristics. What is tricky is whether this is a good thing or a bad thing.
From a hardware perspective, as we will see later in this book, it is a good thing. Each computing node is efficient in specific types of applications. Efficiency here means it gets the best performance (e.g., speed) with lowest cost (e.g., power). This excellence in price-performance is very much needed in our current era of big data, severe power consumption, and the road to exascale computing. So if we can assign to each node the part of the program that it excels at, then we get the results of price-performance, and this is the main challenge facing the software community.
From a software perspective, heterogeneous computing seems like bad news because it makes programming much more challenging. As a developer, you have way more tasks than with traditional homogeneous computing. You need to know about different execution units, or at least learn about the computing nodes in the system you are writing code for. Then you need to pick algorithms to make your program, or different parts of your program, suitable for these nodes. Finally, you need to tweak your code to get the needed performance by overcoming many bottlenecks that certainly exist in heterogeneous computing, like communication overhead between the different units, overhead of creating threads or processes, management of memory access of those different units, and so on.
We cannot then say that heterogeneous computing is good or bad news. But we can say that heterogeneous computing is now the norm and not the exception. It is here, it will continue to be here, and we need to deal with it. But how do we deal with it? This is the topic of this book.
This book discusses the topic of heterogeneous computing from different angles: hardware challenges, current hardware state-of-the-art, software issues, how to make the best use of the current heterogeneous systems, and what lies ahead. All the systems we use, from portable devices to supercomputers, embody some type of heterogeneity. The main reason for that is to have good performance with power efficiency. However, this opens the door to many challenges that we need to deal with at all levels of the computing stack: