Research at Berkeley
HCI, ML Interpretability and Explainability, Programming Tools, Augmented Reality, Digital Fabrication.
I build tools to help people understand, debug, and interpret ML models by combining traditional software engineering and explainable AI techniques. My work employs design and systems-building methodologies at the intersection of HCI and ML research. My previous projects have fused sensing, ML, and novel AR interfaces to guide users through physical tasks.
I am advised by Björn Hartmann and affiliated with the Berkeley Institute of Design.
Umlaut
Umlaut supports users in debugging neural networks by running heuristic checks on the ML program's structure and model behavior during runtime. Umlaut's checking infrastructure can be added to a Keras runtime with minimal changes to the source program (2 lines of code). Raised errors are presented in an interface which tightly couples tutorial-based explanations, visualizations of model output, context from program runtime, and links back to source code.
IMACS: Image Model Attribution Comparison Summaries
IMACS is a method that combines gradient-based model attributions with aggregation and visualization techniques to summarize differences in attributions between two image classification models. IMACS extracts influential image regions from an evaluation dataset, clusters them based on similarity, then visualizes differences in model attributions for clusters of visually similar regions. This approach can be used to discover the behavioral differences between models as well as uncover the influential features and concepts for a given class in a single model.
HindSight
HindSight increases the environmental awareness of cyclists by warning them of vehicles approaching from outside their visual field. A panoramic camera mounted on a bicycle helmet streams real-time, 360-degree video to a laptop running YOLOv2, a neural object detector designed for real-time use. Detected vehicles are passed through a filter bank to find the most relevant. Resulting vehicles are sonified using bone conduction headphones, giving cyclists added margin to react.
Drill Sergeant
The goal of Drill Sergeant is to reduce the barrier of entry for using power tools to create physical artifacts. The project consisted of an ecosystem of augmented power tools (a smart drill, miter saw, CNC router, and measuring tape) which use sensors and embedded displays provide real-time, corrective feedback and coach users through completing assemblies. A server keeps state and coordinates instructions across tools, recalculating the assembly from real-world measurements.
Drill Sergeant received some media attention (Fast Co, Digital Trends) and was a 2016 finalist for Fast Company Magazine's Innovation by Design awards.
Teaching
Responsible Design from Bits to Buildings Fall 2019
Graduate Student Instructor
Thinking Through Art and Design@Berkeley: Responsible Design from Bits to Buildings focuses on design and its connection to the arts across a range of disciplinary contexts. This course explores the role of design in imagining the built environment, in framing our experience of new media and technology, as well as design’s intersections with allied creative fields. Throughout, the class considers how design offers a toolkit for framing and approaching complex problems and how responsible design emerges through new forms of collaboration with a broad field of experts—including users, artists, engineers, scientists, and others.
Global Product Development Spring 2019
Graduate Student Instructor
Global Product Development is a project-based course that teaches students how to translate a functional prototype into a commercial, consumer-ready product. This course provides an overview of product commercialization, design for manufacturing, supply chain and logistics, intellectual property, regulatory certification, and more. The class features several guest lecturers with real-world experience, in-class critique, lectures on product development, and a one-week field trip to Hong Kong to explore the hardware startup ecosystem.
Interactive Device Design Summer 2018
Instructor
Interactive Device Design teaches concepts and skills required to design, prototype, and fabricate interactive devices—physical objects that intelligently respond to user input and enable new types of interactions. The first half of the semester is dedicated to a survey of relevant techniques in 3D modeling and fabrication; electronics and circuit board design; sensing and actuation for interaction; embedded software development, wired and wireless communication with mobile devices, computers, and networks; and user interface programming. In the second half of the semester, students propose and carry out a significant design project. Images above showcase selected student work.
User Interface Design Spring 2018
Graduate Student Instructor
CS 160 is an introduction to Human-Computer Interaction (HCI). In this class, students learn to prototype, evaluate, and design user interfaces. In contrast to most classes at Berkeley, CS 160 does not (only) focus on particular algorithmic techniques or technologies. Students develop a broad set of skills needed for user-centered design. These skills include ideation, needs assessment, communication, rapid prototyping, implementation and evaluation.
Projects
MakerPass
The MakerPass system originally started as my undergraduate design project to create an access control system for the shops at Jacobs Hall. Today, MakerPass is responsible for managing user access for many student shops across campus. The system consists of card readers which light up and emit audio when scanned, desktop software that restricts login, and an admin portal for managing users and their permissions.