Donald B. Gillies Lectureship

Abstract
Berkeley has launched a new multi-faculty project called AMPLab, an effort to harness algorithms, machines, and people to solve large-scale data analysis problems, In this talk, we present a key component for managing the underlying computational resources of AMPLab, which we call Mesos. Mesos is a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI. Sharing improves cluster utilization and avoids per-framework data replication. Mesos shares resources in a fine-grained manner, allowing frameworks to achieve data locality by taking turns reading data stored on each machine. To support the sophisticated schedulers of today's frameworks, Mesos introduces a distributed two-level scheduling mechanism called resource offers. Mesos decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to run on them. Our results show that Mesos can achieve near-optimal data locality when sharing the cluster among diverse frameworks, can scale to 50,000 (emulated) nodes, and is resilient to failures. Mesos is currently being used locally on clusters at Berkeley, on virtual clusters at Amazon EC2, and several industrial environments to provide backend efficient processing for data analytical applications.

Biography
Randy Katz received his undergraduate degree from Cornell University, and his M.S. and Ph.D. degrees from the University of California, Berkeley. He joined the Berkeley faculty in 1983, where since 1996 he has been the United Microelectronics Corporation Distinguished Professor.

Prof. Katz has published over 250 refereed technical papers, book chapters, and books. He has supervised 52 MS theses and 43 PhD dissertations. In the late 1980s, with colleagues at Berkeley, he developed Redundant Arrays of Inexpensive Disks (RAID), a $15 billion per year industry sector. While on leave for government service in 1993-1994, he established whitehouse.gov and connected the White House to the Internet.

Prof. Katz has won numerous awards for his research, teaching, and service, in particular, the IEEE Reynolds Johnson Information Storage Award, the IEEE James Mulligan Jr. Education Medal, and the Singapore Public Service Medal (Pingat Bakti Masyarakat). He is a Fellow of the ACM, the IEEE, and the American Association for the Advancement of Science. He is a member of the National Academy of Engineering and the American Academy of Arts and Sciences.