Multimodal Foundation Models
Learning, adaptation, and evaluation for models that reason across video, language, audio, sensors, maps, and structured knowledge.
Zhi-Qi Cheng, Ph.D.
Pronunciation: Zhì-qí Chéng ("Jih-Chee Chung")Chinese name: Chéng Zhì-qí.
- Assistant Professor
- Computer Science & Systems
School of Engineering & Technology
University of Washington Tacoma - Graduate Faculty
- Doctoral Endorsement
UW Graduate School - Director
- Multimodal Intelligence Lab (MILab)
- Previously
- Postdoctoral Research Associate and Project Scientist
Carnegie Mellon University
About
I am a tenure-track Assistant Professor of Computer Science & Systems in the School of Engineering & Technology at the University of Washington Tacoma. I direct the Multimodal Intelligence Lab (MILab), where we study multimodal AI, embodied intelligence, and intelligent systems for open-world decision-making. I am also a member of the Graduate Faculty with doctoral endorsement through the University of Washington Graduate School.
My research asks how AI systems can learn, reason, and act from multimodal experience in complex real-world environments. At the intersection of multimodal foundation models, embodied AI, and intelligent decision-making, I develop systems that connect perception, reasoning, planning, and action across visual, linguistic, temporal, and physical contexts. At MILab, we advance the foundations of multimodal embodied intelligence while developing deployable AI technologies for mobility, public safety, and human-centered decision support, integrating research with project-based education and student mentoring to create responsible, reproducible, and real-world-ready AI systems.
Before joining the University of Washington, I spent seven years at Carnegie Mellon University’s School of Computer Science, primarily in the Language Technologies Institute, where I held research associate, postdoctoral researcher, and project scientist roles. My work focused on multimodal understanding, event-centric reasoning, and large-scale AI systems that integrate video, language, audio, maps, and knowledge sources for complex real-world environments. I was fortunate to work closely with Prof. Alexander G. Hauptmann and Prof. Teruko Mitamura, whose mentorship shaped my research trajectory, system-building experience, and contributions to large-scale AI programs.
From 2019 to 2024, I served as a core technical and system-delivery lead for Carnegie Mellon’s DARPA KAIROS system, contributing to multimodal reasoning systems for event understanding and schema-guided knowledge integration. I also contributed to U.S. government-funded AI programs including DARPA AIDA, KAIROS-Plus, IARPA DIVA, and NIST PSIAP.
My work has appeared in leading AI conferences including NeurIPS, ICLR, CVPR, ICCV, ACL, AAAI, and ACM Multimedia. My research has been featured by The Washington Post, The New York Times, and CBS News. I have also held research appointments or internships at Meta AI, Alibaba DAMO Academy, Google Research, and Microsoft Research, and have been recognized with the Intel Ph.D. Fellowship and the CSC–IBM Outstanding Student Scholarship.
News
- May 01, 2026: Incoming Ph.D. student Fengyi Wu received the UW GSFEI Top Scholar Award 2026.
- Apr 04, 2026: One paper accepted to ACL 2026 Main Conference and one paper accepted to ACL 2026 Findings.
- Jan 25, 2026: Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding, co-authored by former visiting student Yuxuan Zhou and incoming Ph.D. student Fengyi Wu, accepted as an ICLR 2026 Oral presentation.
- Sep 18, 2025: Congratulations to former visiting student Yuxuan Zhou. Our paper MaxSup: Overcoming Representation Collapse in Label Smoothing was accepted for an oral presentation at NeurIPS 2025.
- Sep 03, 2025: Congratulations to Ph.D. student Yifei Dong on receiving the Carwein-Andrews Ph.D. Fellowship.
- Jun 12, 2025: Invited keynote at CVPR 2025 Anti-UAV Workshop; our survey Securing the Skies: A Comprehensive Survey on Anti-UAV Methods, Benchmarking, and Future Directions wins the workshop's Best Paper Award.
- Mar 31, 2025: Incoming Ph.D. student Yifei Dong received the UW GSFEI Top Scholar Award 2025.
- Feb 26, 2025: Four papers accepted to CVPR 2025, including two main-conference papers and two workshop papers.
Research
Multimodal AI for understanding, reasoning, and acting in the real world.
My research asks how AI systems can learn from multimodal evidence, reason under uncertainty, and support action in complex environments. At MILab, we develop foundation models, embodied agents, and deployable AI systems for robotics, mobility, public safety, and responsible decision support.
Core Question
How can AI systems turn noisy multimodal evidence into reliable understanding, prediction, and action?
Embodied AI & World Models
Agents that connect perception, memory, prediction, planning, and interaction in dynamic physical environments.
Mobility, Public Safety & Secure Deployment
AI systems for traffic and mobility intelligence, public-safety sensing, secure perception, and robust deployment under real-world constraints.
Applied Collaborations
Selected projects translate this agenda into public-interest settings where visual, audio, spatial, and temporal evidence is noisy, incomplete, and high-stakes.
Teaching & Mentorship
Courses and research supervision in AI, robotics, graphics, and multimodal systems.
At UW Tacoma, I teach courses that connect core computer science foundations with current advances in AI, robotics, computer graphics, and multimodal systems. My teaching emphasizes technical depth, hands-on implementation, empirical evaluation, reproducible experimentation, and open-ended projects. Current UW students across Seattle, Tacoma, and Bothell can enroll in these courses through UW cross-campus registration, subject to course capacity, prerequisites, registration periods, and home-campus requirements.
I also mentor undergraduate and M.S. students through the Multimodal Intelligence Lab (MILab), independent study, supervised research, thesis projects, and capstone projects. Current UW students across Seattle, Tacoma, and Bothell can pursue research credit with instructor approval through TCSS 499, TCSS 600, TCSS 700, or TCSS 702. Undergraduates should meet home-campus credit and registration-period requirements; graduate students have no cross-campus registration restrictions. Students should contact me before registration to discuss research fit, project scope, deliverables, and quarter timeline; some credits may require a faculty number or departmental support.
Courses and Supervision
Prospective Students & Researchers
As a member of the University of Washington Graduate Faculty with doctoral endorsement, I advise Ph.D. students and serve on doctoral supervisory committees across eligible UW graduate programs. My primary Ph.D. recruiting pathway is the Computer Science & Systems — School of Engineering & Technology (Tacoma) — PhD program. I welcome inquiries from prospective Ph.D. students, postdoctoral researchers, and research assistants interested in multimodal AI, embodied intelligence, robotics, mobility intelligence, and responsible AI.
Prospective Ph.D. applicants should apply through the CSS Ph.D. program. Postdoctoral researchers and research assistants are encouraged to contact me when their background aligns with MILab’s research agenda and current projects. Interested applicants should complete the MILab Research Interest Form and email me at zhiqics@uw.edu with a brief note describing their background, research interests, and potential fit with the lab. Opportunities depend on research fit, preparation, project needs, funding availability, and mentoring capacity.
What to Include
- Academic background: CV and unofficial transcript, if applicable
- Research interests: topics, questions, and research directions of interest
- Relevant experience: projects, publications, software systems, open-source contributions, or prior research
- MILab fit: a brief statement describing why you are interested in MILab and how your interests align with the lab
Competitive Ph.D. applicants may be considered for nomination to University of Washington Graduate School fellowships and GSFEI Top Scholar Award, subject to program procedures, eligibility requirements, and funding availability.
Awards & Service
- Intel Ph.D. Fellowship, 2017–2019
- CSC–IBM Outstanding Student Scholarship, 2017–2019
- Scales Figure Scholarship, 2016 & 2018
- Nominee, "Star of Self-Improvement of Chinese University Students," 2014
- ACM SCF Best Student Paper Award, 2016
- ICCV Outstanding Reviewer, 2023
- ICPC Asia Regional Silver Medal, 2013
- Technical contributor to The Washington Post 2022 Pulitzer-winning Public Service coverage
- CVPR Anti-UAV Workshop Best Paper, 2025
- CVPR Anti-UAV Workshop & Challenge Organizer, 2025
- RoboWorld Challenge Organizer / Program Committee, NeurIPS 2026 Competition Track
Selected Publications
Representative papers grouped by models, agents, systems, and sponsored research reports.
Multimodal Models & Efficient AI Foundation models, generative systems, calibrated learning, and efficient decoding.
- Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning CodeWebsite
- MaxSup: Overcoming Representation Collapse in Label Smoothing Code
- Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding Code
- Towards Calibrated Robust Fine-Tuning of Vision-Language Models Code
- MetaDesigner: AI-Driven, User-Centric, Multilingual WordArt Synthesis Demo
- SHIELD: LLM-Driven Schema Induction for Predictive Analytics in EV Battery Supply-Chain Disruptions CodeWebsite
- ChartReader: A Unified Framework for Chart Derendering and Comprehension Code
- FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio Code
- StableAnimator: High-Quality Identity-Preserving Human Image Animation CodeWebsite
Embodied AI & World Models Navigation, world modeling, activity understanding, and multimodal reasoning.
- Human-Aware Vision-and-Language Navigation V2 CodeV1 CodeProject
- Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresight Code
- Language-Conditioned World Modeling for Visual Navigation Code
- GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinement Code
- VDAct: A Video-grounded Dialogue Dataset and Metric for Event-driven Activities Code & Data
- ProMQA: Question Answering Dataset for Multimodal Procedural Activity Understanding Code & Data
Mobility, Safety & Deployment Transportation intelligence, public safety, secure sensing, and robust perception.
- Rethinking Spatial Invariance of Convolutional Networks for Object Counting Code
- BlockGCN: Redefine Topology Awareness for Skeleton-Based Action Recognition Code
- DAMO-StreamNet: Optimizing Streaming Perception in Autonomous Driving Code
- SHIELD: LLM-Driven Schema Induction for Predictive Analytics in EV Battery Supply-Chain Disruptions CodeWebsite
- POPoS: Improving Efficient and Robust Facial Landmark Detection with Parallel Optimal Position Search Code
- Securing the Skies: A Comprehensive Survey on Anti-UAV Methods, Benchmarking, and Future Directions