My research develops multimodal AI and world models that help intelligent systems perceive their surroundings, reason about what may happen next, and plan actions in the physical world. I combine 3D geometry, semantic understanding, and generative modeling to learn from incomplete observations across views and modalities, with an emphasis on spatial structure and temporal context.
At the University of Washington, I lead the Multimodal Intelligence Lab (MILab), where we study how perception, memory, and prediction support reliable decisions in unfamiliar and changing environments. Our work connects fundamental modeling questions with navigation, transportation, and public-safety applications. Research directions include:
- World models & visual navigation — Memory, future prediction, and language-conditioned planning for agents navigating unfamiliar and dynamic environments.
- Multimodal reasoning & generation — Methods that connect visual observations and language to understand scenes, generate navigation instructions, and support decisions.
- 3D scene understanding & transportation — Reconstruction, mapping, and monitoring of urban scenes and transportation assets from incomplete and complementary visual observations.
- Public safety & visual analysis — Video understanding, crowd analysis, audio-visual event localization, and geometric reasoning for public-interest applications.
Our selected publications include models and benchmarks for visual navigation, multimodal reasoning, and scene understanding. See the Publications page for papers, code, and project resources, and Google Scholar for the full publication record.
Honors & Awards
- 2025–2026 · UW Undergraduate Research Faculty Fellow — supported by the Henry Luce Foundation.
- 2025 · Best Paper Award, CVPR Anti-UAV Workshop — corresponding author of Securing the Skies.
- 2025 · CSIG Natural Science Award, Second Prize — co-recipient for research on multi-view image understanding.
- 2024 · First place, Multimodal Emotion Recognition Challenge (MER2024) — MER-NOISE track, Team SZTU-CMU.
- 2023 · ICCV Outstanding Reviewer — IEEE/CVF International Conference on Computer Vision.
- 2022 · Technical contribution to Pulitzer Prize–winning reporting — computer vision and technical analysis supporting The Washington Post’s reporting. The newspaper received the Pulitzer Prize for Public Service for this reporting.
- 2018 · IBM China Outstanding Student Scholarship.
- 2017 · Intel Ph.D. Fellowship.
Students and Mentorship
I advise doctoral and master’s students in Computer Science & Systems (CSS) and mentor undergraduate researchers across UW. Current projects span world models for navigation and physical reasoning, efficient multimodal inference, sign-language understanding, and multi-view super-resolution. The Students page lists group members, thesis topics, and mentoring outcomes.
I primarily recruit Ph.D. and master’s students through the CSS programs at UW Tacoma. I can also co-advise students in CSE, ECE, and related UW programs and serve on their doctoral committees. Email your CV and research interests to zhiqics@uw.edu to discuss a potential fit. The Join page explains degree pathways and research opportunities for current and prospective UW students.
Short Bio
Before UW, I worked at Carnegie Mellon University’s Language Technologies Institute, School of Computer Science, as a Research Associate (2017–2019), Postdoctoral Research Associate (2019–2022), and Project Scientist (2022–2024).
Professor Alexander G. Hauptmann advised my doctoral research, and Professor Teruko Mitamura advised my postdoctoral research.
I was a core technical lead for CMU’s DARPA KAIROS and KAIROS Plus projects (2019–2024), developing multimodal AI for U.S. national security. I also contributed to DARPA AIDA (2018–2023), DARPA GAILA (2019–2022), IARPA DIVA (2017–2021), and NIST PSIAP (2017–2019). Across these collaborations, my work combined multimodal event understanding, video activity analysis, and the development of integrated research systems for analyzing events across video, audio, language, and structured knowledge.
My industry experience includes a Visiting Faculty Researcher position at Meta AI (2025) and research internships at Microsoft Research (2019) and Google Brain (2018). I also contributed computer-vision analysis to The Washington Post and The New York Times, including The Post’s reporting recognized with the 2022 Pulitzer Prize for Public Service. CBS Pittsburgh and the Pittsburgh Post-Gazette covered our work on video analysis for public safety and emergency response.
Research news
- 2026 — Two USDOT-funded research projects. Funding from the U.S. Department of Transportation through the Pacific Northwest Transportation Consortium (PacTrans) supports our work in 3D urban scene reconstruction and transportation asset mapping from incomplete visual observations of complex urban environments.
- 2026 — NeurIPS 2026 Oral. Yifei Dong, Fengyi Wu, and collaborators’ Language-Conditioned World Modeling for Visual Navigation.
- 2026 — ICLR 2026 Oral. Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding, with Fengyi Wu and collaborators, studies efficient language-model inference while preserving the target model’s output distribution.
- 2026 — ECCV 2026. Yifei Dong, Fengyi Wu, and collaborators’ UniWM, a world model for visual navigation.
- 2026 — Yifei Dong received the UW Graduate School Conference Presentation Award for his ECCV 2026 presentation.
- 2026 — IROS 2026. Yifei Dong, Fengyi Wu, and collaborators’ HA-VLN 2.0, a simulator and benchmark for evaluating visual navigation around moving people in dynamic environments.
- Jul 2026 — Two papers at ACL 2026. Sign-Language Datasets at Scale appeared in the main conference, and Fengyi Wu and collaborators’ GoViG appeared in Findings.
Research systems & collaborations
- DARPA KAIROS / KAIROS Plus · CMU — Core technical lead (2019–2024) for multimodal event understanding and system integration across video, language, and knowledge sources. CMU systems report
- DARPA AIDA · IARPA DIVA · NIST PSIAP — Technical leadership in multimodal event understanding, video activity analysis, and public-safety applications across these programs at CMU.
- USDOT Mobility21 · Traffic activity analysis — Led research on video-based traffic activity analysis, including Robust Automatic Detection of Traffic Activity (2023).
- CMU–AIST · Human activity understanding — Research leadership in human activity understanding, including ProMQA for procedural activity understanding and VDAct for video-grounded dialogue.
- CMU Manufacturing Futures Institute · Digital twins — Led collaborative research on digital twins for manufacturing resiliency, including electric-vehicle battery supply chains.