Introduction to Autonomous Driving in the Real-World
Spring 2022
Mondays and Wednesdays 2:00 - 3:20 PM
Zoom (passcode available on Canvas and Piazza)
Instructor: Prof. Mohan M. Trivedi (mtrivedi)
TAs: Nachiket Deo (ndeo), Ross Greer (regreer), Maitrayee Keskar (mmkeskar), Jason Isa (jisa);
Email: [TA]@ucsd.edu
Office Hours: Mon. 3:30 PM (Maitrayee), Weds. 11 AM (Ross), Weds. 3:30 PM (Jason), Fri. 12 PM (Nachiket)
The field of intelligent vehicles has been subject to significant active research in recent times, with computer vision playing a key role in the development of autonomous systems capable of safely navigating real-world traffic. In this course we will review current systems and the concepts involved. In particular, we will focus on four themes:
Introducing the field of Autonomous Driving, including History, Challenges & Milestones, and Ethics.
Strengths and limitations of data-driven approaches to autonomous driving tasks.
Understanding the environment around the intelligent vehicle in order to safely navigate through it. Related topics include object detection, tracking, semantic and instance segmentation, motion/intent prediction of surrounding agents, etc.
Monitoring the state of the driver. This is crucial for development of semi-autonomous systems and for safe and smooth control transitions between the driver and the vehicle. This will involve topics such as driver behavior analysis, driver activity estimation, driver readiness and takeover time estimation etc.
You will be assigned a paper(s) to read and present to the class. The first 2 presenters listed each week will present on Monday, and the next 2 will present on Wednesday.
The presentation should last approximately 20 minutes, and will be followed by 10 minutes of Q&A and discussion. We will be timing presentations to maintain the reading schedule, and may cut off presenters who exceed the allotted time.
At a minimum, your presentation should cover:
Required Background: for example, if your paper explores a novel CNN architecture, perhaps you might take a few slides to explain what a convolutional neural network is before you share details about the paper's unique network architecture. Your classmates will appreciate the lesson (or review), and you can continue to build your understanding of fundamental ideas as you explore your assigned paper.
Research objective
Methodology
Analysis and results
Advantages & disadvantages of approach
Key contribution(s) to the field
Some papers have readily-available repositories for readers to replicate and explore the models. If applicable to your paper, feel free to do so (and share your findings) to enhance your presentation.
You are additionally required to make a private post on Piazza with two questions & answers related to your paper. These questions should be at a level that a classmate could answer following your presentation.
If you have questions while reading your paper and developing your presentation, before asking in Office Hours, you should first ask your question publicly on Piazza to initiate class discussion. TAs will also be attentive to this channel to assist.
Quizzes will be given at the end of each unit. Quizzes will draw on concepts and ideas from presented papers, and may include questions created by your classmates.
You can earn your participation grade by:
Engaging in questions or discussion with presenters and peers during class,
Engaging in questions or discussion with presenters and peers on Piazza.
This list is non-exhaustive.
Welcome and Course Logistics
Introduction of the Intelligent Vehicles and ADS field
Levels for Autonomous Vehicles, Key Milestones and Research Contributions
Safety as a critically important factor for Real-world Intelligent Vehicles
Perception, Planning, Control Loop for Human-Centered Intelligent Vehicles
Data Driven Approaches, Machine Learning and CNN Overview
MM Trivedi, T Gandhi, J McCall: Looking-in and looking-out of a vehicle: Computer-vision-based enhanced vehicle safety, ITS 2007. [Anish Gopalan]
Bengler K, Dietmayer K, Farber B, Maurer M, Stiller C, Winner H: Three decades of driver assistance systems: Review and future perspectives, ITS 2014. [Akshay Gopalkrishnan]
Benjamin Ranft and Christoph Stiller: The Role of Machine Vision for Intelligent Vehicles, IV 2016. [Ahmad Said]
F. Kunz, D. Nuss, J. Wiest, H. Deusch, S. Reuter, F. Gritschneder, A. Scheel, et al. Autonomous driving at Ulm University: A modular, robust, and sensor-independent fusion approach, IV 2015. [Mark Utnehmer]
Week 1-3 Quiz during class on Monday 4/18
Y. Li and J. Ibanez-Guzman: Lidar for Autonomous Driving: The Principles, Challenges, and Trends for Automotive Lidar and Perception Systems [Angela Liu, 4/18]
A. Geiger, P. Lenz, R. Urtasun: Are we ready for autonomous driving? The KITTI vision benchmark suite [Justin Kane-Starr, 4/20]
J.C. McCall, M.M. Trivedi: An integrated, robust approach to lane marking detection and lane tracking. [Andrew Fino 4/20]
S Sivaraman, MM Trivedi: Integrated Lane and Vehicle Detection, Localization, and Tracking: A Synergistic Approach [Jacob Landgren 4/25]
E Ohn-Bar, MM Trivedi: Learning to detect vehicles by clustering appearance patterns [Aditi Anand 4/27]
A Rangesh, MM Trivedi: No blind spots: Full-surround multi-object tracking for autonomous vehicles using cameras and lidars. [Tommy Nguyen 4/27]
Gavrila, Munder: Multi-cue Pedestrian Detection and Tracking from a Moving Vehicle [Juan Sanjuan 5/2]
M Enzweiler, A Eigenstetter, B Schiele: Multi-cue pedestrian classification with partial occlusion handling [Behrad Rabiei, 5/2]
E Ohn-Bar, MM Trivedi: Are all objects equal? Deep spatio-temporal importance prediction in driving videos [Kai Poon, 5/4]
A Møgelmose, D Liu, MM Trivedi: Detection of US traffic signs. [Nishant Balaji, 5/4]
H Cho, YW Seo, BVKV Kumar, RR Rajkumar: A multi-sensor fusion system for moving object detection and tracking in urban driving environments [Jacob Ayers, 5/9]
S. Wender and K. Dietmayer: 3D vehicle detection using a laser scanner and video camera [Michael Shao, 5/9]
J.V. Dueholm, M.S. Kristoffersen, R.K. Satzoda, T.B. Moeslund, M.M. Trivedi: Trajectories and maneuvers of surrounding vehicles with panoramic camera arrays. [Likith Palabindela, 5/11]
N Deo, A Rangesh, MM Trivedi: How would surround vehicles move? A unified framework for maneuver classification and motion prediction. [Naiwen Shi, 5/16]
C. Urmson, J. Anhalt, D. Bagnall, C. Baker, R. Bittner, M.N. Clark, J. Dolan et al.: Autonomous driving in urban environments: Boss and the urban challenge. [Rishiv Thondepu, 5/16]
M. Montemerlo, J. Becker, S. Bhat, H. Dahlkamp, D. Dolgvo, S. Ettinger, D. Haehnel et al.: Junior: The Stanford entry in the urban challenge. [Zachary Blickstein, 5/18]
M. Heimberger, J. Horgan, C. Hughes, J. McDonald, S. Yogamani: Computer vision in automated parking systems: Design, implementation, and challenges. [Trung Le, 5/18]
A. Armand, D. Filliat, J. Ibanez-Guzman: Ontology-based context awareness for driving assistance systems. [Yang-Jie Qin, 5/18]
F. Lu, S. Lee, R.K. Satzoda, M.M. Trivedi: Embedded Computing Framework for Vision-Based Real-Time Surround Threat Analysis and Driver Assistance. [James Chen, 5/23]
E. Murphy-Chutorian, M.M. Trivedi: Head pose estimation and augmented reality tracking: An integrated system and evalution for monitoring driver awareness. [George Liu, 5/23]
Roitberg, Alina, et al. "Open set driver activity recognition." 2020 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2020. [Dylan Vuong, 5/23]
A Tawari, KH Chen, MM Trivedi: Where is the driver looking: Analysis of head, eye and iris for robust gaze zone estimation [Cheng-Yu Chiang, 5/25]
Martin, Manuel, et al. "Drive&act: A multi-modal dataset for fine-grained driver behavior recognition in autonomous vehicles." IEEE/CVF ICCV. 2019. [Shane Benetz, 5/25]
S. Martin, S. Vora, K. Yuen, M.M. Trivedi: Dynamics of driver's gaze: Explorations in behavior modeling and maneuver prediction [Arsalan Sepahpour, 6/1]
K. Yuen, S. Martin, M.M. Trivedi: Looking at faces in a vehicle: A deep CNN based approach and evaluation [Yiyuan Cui, 6/1]
E. Ohn-Bar, M.M. Trivedi: Hand gesture recognition in real time for automotive interfaces: A multimodal vision-based approach and evaluations [Moises Lopez-Mendoza, 6/1]
S. Vora, A. Rangesh, M.M. Trivedi: On generalizing driver gaze zone estimation using convolutional neural networks [Brenner Lim, 6/10]
Akshay Rangesh, Nachiket Deo, Ross Greer, Pujitha Gunaratne and Mohan M. Trivedi, "Autonomous Vehicles that Alert Humans to Take-Over Controls: Modeling with Real-World Data," 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). IEEE, 2021 [Jane Earley, 6/10]
Quiz 3 [Following Presentations, during our 3-6 PM final time]
Notable Papers:
S Sivaraman, MM Trivedi: Looking at Vehicles on the Road: A Survey of Vision-Based Vehicle Detection, Tracking, and Behavior Analysis
T Gandhi, MM Trivedi: Pedestrian protection systems: Issues, survey, and challenges
JC McCall, MM Trivedi: Video-based lane estimation and tracking for driver assistance: survey, system, and evaluation
A Mogelmose, MM Trivedi, TB Moeslund: Vision-Based Traffic Sign Detection and Analysis for Intelligent Driver Assistance Systems: Perspectives and Survey
M.P. Philipsen, M.B. Jensen, A. Møgelmose, T.B. Moeslund, MM Trivedi: Traffic light detection: A learning algorithm and evaluations on a challenging dataset.
A. Rangesh, M.M. Trivedi: Handynet: A one-stop solution to detect, segment, localize & analyze driver hands.
K. Yuen, M.M. Trivedi: Looking at hands in autonomous vehicles: A convnet approach using part affinity fields
E Ohn-Bar, MM Trivedi: Looking at humans in the age of self-driving and highly automated vehicles
A. Tawari, S. Sivaraman, M.M. Trivedi, T. Shannon, M. Tippelhofer: Looking-in and looking-out vision for urban intelligent assistance: Estimation of driver attentive state and dynamic surround for safe merging and braking
J.C. McCall, D.P. Wipf, M.M. Trivedi, B.D. Rao: Lane change intent analysis using robust operators and spare bayesian learning
E Dagan, O Mano, GP Stein, A Shashua: Forward Collision Warning with a Single Camera
N. Deo, M.M. Trivedi: Looking at the driver/rider in autonomous vehicles to predict take-over readiness
S Sivaraman, MM Trivedi: A general active-learning framework for on-road vehicle recognition and tracking
R.N. Rajaram, E. Ohn-Bar, M.M. Trivedi: An exploration of why and when pedestrian detection fails
B.T. Morris, M.M. Trivedi: Trajectory learning for activity understanding: Unsupervised, multilevel, and long-term adaptive approach.
S Sivaraman, MM Trivedi: Dynamic probabilistic drivability maps for lane change and merge driver assistance
C. Tran, A. Doshi, M.M. Trivedi: Modeling and prediction of driver behavior by foot gesture analysis
A. Rangesh, M.M. Trivedi: Forced spatial attention for driver foot activity classification
A Doshi, B Morris, M Trivedi: On-road prediction of driver's intent with multimodal sensory cues
JC McCall, MM Trivedi: Driver behavior and situation aware brake assistance for intelligent vehicles