Introduction to Autonomous Driving in the Real-World

Spring 2022

Mondays and Wednesdays 2:00 - 3:20 PM

Zoom (passcode available on Canvas and Piazza)

Instructor: Prof. Mohan M. Trivedi (mtrivedi)

TAs: Nachiket Deo (ndeo), Ross Greer (regreer), Maitrayee Keskar (mmkeskar), Jason Isa (jisa);

Email: [TA]@ucsd.edu

Office Hours: Mon. 3:30 PM (Maitrayee), Weds. 11 AM (Ross), Weds. 3:30 PM (Jason), Fri. 12 PM (Nachiket)

Overview

The field of intelligent vehicles has been subject to significant active research in recent times, with computer vision playing a key role in the development of autonomous systems capable of safely navigating real-world traffic. In this course we will review current systems and the concepts involved. In particular, we will focus on four themes:

1. Introducing the field of Autonomous Driving, including History, Challenges & Milestones, and Ethics.

2. Strengths and limitations of data-driven approaches to autonomous driving tasks.

3. Understanding the environment around the intelligent vehicle in order to safely navigate through it. Related topics include object detection, tracking, semantic and instance segmentation, motion/intent prediction of surrounding agents, etc.

4. Monitoring the state of the driver. This is crucial for development of semi-autonomous systems and for safe and smooth control transitions between the driver and the vehicle. This will involve topics such as driver behavior analysis, driver activity estimation, driver readiness and takeover time estimation etc.

Course Format and Requirements

Presentation (50%)

You will be assigned a paper(s) to read and present to the class. The first 2 presenters listed each week will present on Monday, and the next 2 will present on Wednesday.

The presentation should last approximately 20 minutes, and will be followed by 10 minutes of Q&A and discussion. We will be timing presentations to maintain the reading schedule, and may cut off presenters who exceed the allotted time.

At a minimum, your presentation should cover:

1. Required Background: for example, if your paper explores a novel CNN architecture, perhaps you might take a few slides to explain what a convolutional neural network is before you share details about the paper's unique network architecture. Your classmates will appreciate the lesson (or review), and you can continue to build your understanding of fundamental ideas as you explore your assigned paper.

2. Research objective

3. Methodology

4. Analysis and results

5. Advantages & disadvantages of approach

6. Key contribution(s) to the field

Some papers have readily-available repositories for readers to replicate and explore the models. If applicable to your paper, feel free to do so (and share your findings) to enhance your presentation.

You are additionally required to make a private post on Piazza with two questions & answers related to your paper. These questions should be at a level that a classmate could answer following your presentation.

If you have questions while reading your paper and developing your presentation, before asking in Office Hours, you should first ask your question publicly on Piazza to initiate class discussion. TAs will also be attentive to this channel to assist.

Presentation Overview Quizzes (30%)

Quizzes will be given at the end of each unit. Quizzes will draw on concepts and ideas from presented papers, and may include questions created by your classmates.

Class & Piazza Participation (20%)

You can earn your participation grade by:

1. Engaging in questions or discussion with presenters and peers during class,

2. Engaging in questions or discussion with presenters and peers on Piazza.

This list is non-exhaustive.

Topics and Outline

Week 1

• Welcome and Course Logistics

• Introduction of the Intelligent Vehicles and ADS field

• Levels for Autonomous Vehicles, Key Milestones and Research Contributions

Week 2

• Safety as a critically important factor for Real-world Intelligent Vehicles

• Perception, Planning, Control Loop for Human-Centered Intelligent Vehicles

• Data Driven Approaches, Machine Learning and CNN Overview

Week 3

Overview

• MM Trivedi, T Gandhi, J McCall: Looking-in and looking-out of a vehicle: Computer-vision-based enhanced vehicle safety, ITS 2007. [Anish Gopalan]

• Bengler K, Dietmayer K, Farber B, Maurer M, Stiller C, Winner H: Three decades of driver assistance systems: Review and future perspectives, ITS 2014. [Akshay Gopalkrishnan]

• Benjamin Ranft and Christoph Stiller: The Role of Machine Vision for Intelligent Vehicles, IV 2016. [Ahmad Said]

• F. Kunz, D. Nuss, J. Wiest, H. Deusch, S. Reuter, F. Gritschneder, A. Scheel, et al. Autonomous driving at Ulm University: A modular, robust, and sensor-independent fusion approach, IV 2015. [Mark Utnehmer]

Week 4

Week 1-3 Quiz during class on Monday 4/18

Overview

• Y. Li and J. Ibanez-Guzman: Lidar for Autonomous Driving: The Principles, Challenges, and Trends for Automotive Lidar and Perception Systems [Angela Liu, 4/18]

• A. Geiger, P. Lenz, R. Urtasun: Are we ready for autonomous driving? The KITTI vision benchmark suite [Justin Kane-Starr, 4/20]

Detection and Tracking

• J.C. McCall, M.M. Trivedi: An integrated, robust approach to lane marking detection and lane tracking. [Andrew Fino 4/20]

Week 5

Detection and Tracking

• S Sivaraman, MM Trivedi: Integrated Lane and Vehicle Detection, Localization, and Tracking: A Synergistic Approach [Jacob Landgren 4/25]

• E Ohn-Bar, MM Trivedi: Learning to detect vehicles by clustering appearance patterns [Aditi Anand 4/27]

• A Rangesh, MM Trivedi: No blind spots: Full-surround multi-object tracking for autonomous vehicles using cameras and lidars. [Tommy Nguyen 4/27]

Week 6

Detection and Tracking

• Gavrila, Munder: Multi-cue Pedestrian Detection and Tracking from a Moving Vehicle [Juan Sanjuan 5/2]

• M Enzweiler, A Eigenstetter, B Schiele: Multi-cue pedestrian classification with partial occlusion handling [Behrad Rabiei, 5/2]

• E Ohn-Bar, MM Trivedi: Are all objects equal? Deep spatio-temporal importance prediction in driving videos [Kai Poon, 5/4]

• A Møgelmose, D Liu, MM Trivedi: Detection of US traffic signs. [Nishant Balaji, 5/4]

Week 7

Detection and Tracking

• H Cho, YW Seo, BVKV Kumar, RR Rajkumar: A multi-sensor fusion system for moving object detection and tracking in urban driving environments [Jacob Ayers, 5/9]

• S. Wender and K. Dietmayer: 3D vehicle detection using a laser scanner and video camera [Michael Shao, 5/9]

Wednesday 5/11: Quiz 2 in class, covering material through 5/9

Trajectory Analysis, Path Planning, Collision Avoidance

• J.V. Dueholm, M.S. Kristoffersen, R.K. Satzoda, T.B. Moeslund, M.M. Trivedi: Trajectories and maneuvers of surrounding vehicles with panoramic camera arrays. [Likith Palabindela, 5/11]

Week 8

Trajectory Analysis, Path Planning, Collision Avoidance

• N Deo, A Rangesh, MM Trivedi: How would surround vehicles move? A unified framework for maneuver classification and motion prediction. [Naiwen Shi, 5/16]

• C. Urmson, J. Anhalt, D. Bagnall, C. Baker, R. Bittner, M.N. Clark, J. Dolan et al.: Autonomous driving in urban environments: Boss and the urban challenge. [Rishiv Thondepu, 5/16]

• M. Montemerlo, J. Becker, S. Bhat, H. Dahlkamp, D. Dolgvo, S. Ettinger, D. Haehnel et al.: Junior: The Stanford entry in the urban challenge. [Zachary Blickstein, 5/18]

• M. Heimberger, J. Horgan, C. Hughes, J. McDonald, S. Yogamani: Computer vision in automated parking systems: Design, implementation, and challenges. [Trung Le, 5/18]

• A. Armand, D. Filliat, J. Ibanez-Guzman: Ontology-based context awareness for driving assistance systems. [Yang-Jie Qin, 5/18]

Week 9

Trajectory Analysis, Path Planning, Collision Avoidance

• F. Lu, S. Lee, R.K. Satzoda, M.M. Trivedi: Embedded Computing Framework for Vision-Based Real-Time Surround Threat Analysis and Driver Assistance. [James Chen, 5/23]

Driver face, gaze analysis

• E. Murphy-Chutorian, M.M. Trivedi: Head pose estimation and augmented reality tracking: An integrated system and evalution for monitoring driver awareness. [George Liu, 5/23]

• Roitberg, Alina, et al. "Open set driver activity recognition." 2020 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2020. [Dylan Vuong, 5/23]

• A Tawari, KH Chen, MM Trivedi: Where is the driver looking: Analysis of head, eye and iris for robust gaze zone estimation [Cheng-Yu Chiang, 5/25]

• Martin, Manuel, et al. "Drive&act: A multi-modal dataset for fine-grained driver behavior recognition in autonomous vehicles." IEEE/CVF ICCV. 2019. [Shane Benetz, 5/25]

Week 10

Driver face, gaze analysis

• S. Martin, S. Vora, K. Yuen, M.M. Trivedi: Dynamics of driver's gaze: Explorations in behavior modeling and maneuver prediction [Arsalan Sepahpour, 6/1]

• K. Yuen, S. Martin, M.M. Trivedi: Looking at faces in a vehicle: A deep CNN based approach and evaluation [Yiyuan Cui, 6/1]

Driver hand analysis

• E. Ohn-Bar, M.M. Trivedi: Hand gesture recognition in real time for automotive interfaces: A multimodal vision-based approach and evaluations [Moises Lopez-Mendoza, 6/1]

Finals Week

• S. Vora, A. Rangesh, M.M. Trivedi: On generalizing driver gaze zone estimation using convolutional neural networks [Brenner Lim, 6/10]

• Akshay Rangesh, Nachiket Deo, Ross Greer, Pujitha Gunaratne and Mohan M. Trivedi, "Autonomous Vehicles that Alert Humans to Take-Over Controls: Modeling with Real-World Data," 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). IEEE, 2021 [Jane Earley, 6/10]

• Quiz 3 [Following Presentations, during our 3-6 PM final time]

Additional Resources

Notable Papers:

• S Sivaraman, MM Trivedi: Looking at Vehicles on the Road: A Survey of Vision-Based Vehicle Detection, Tracking, and Behavior Analysis

• T Gandhi, MM Trivedi: Pedestrian protection systems: Issues, survey, and challenges

• JC McCall, MM Trivedi: Video-based lane estimation and tracking for driver assistance: survey, system, and evaluation

• A Mogelmose, MM Trivedi, TB Moeslund: Vision-Based Traffic Sign Detection and Analysis for Intelligent Driver Assistance Systems: Perspectives and Survey

• M.P. Philipsen, M.B. Jensen, A. Møgelmose, T.B. Moeslund, MM Trivedi: Traffic light detection: A learning algorithm and evaluations on a challenging dataset.

• A. Rangesh, M.M. Trivedi: Handynet: A one-stop solution to detect, segment, localize & analyze driver hands.

• K. Yuen, M.M. Trivedi: Looking at hands in autonomous vehicles: A convnet approach using part affinity fields

• E Ohn-Bar, MM Trivedi: Looking at humans in the age of self-driving and highly automated vehicles

• A. Tawari, S. Sivaraman, M.M. Trivedi, T. Shannon, M. Tippelhofer: Looking-in and looking-out vision for urban intelligent assistance: Estimation of driver attentive state and dynamic surround for safe merging and braking

Driver Assistance Systems

• J.C. McCall, D.P. Wipf, M.M. Trivedi, B.D. Rao: Lane change intent analysis using robust operators and spare bayesian learning

• E Dagan, O Mano, GP Stein, A Shashua: Forward Collision Warning with a Single Camera

• N. Deo, M.M. Trivedi: Looking at the driver/rider in autonomous vehicles to predict take-over readiness

Detection and Tracking

• S Sivaraman, MM Trivedi: A general active-learning framework for on-road vehicle recognition and tracking

Pedestrians

• R.N. Rajaram, E. Ohn-Bar, M.M. Trivedi: An exploration of why and when pedestrian detection fails

Trajectory Analysis, Prediction, Planning

• B.T. Morris, M.M. Trivedi: Trajectory learning for activity understanding: Unsupervised, multilevel, and long-term adaptive approach.

• S Sivaraman, MM Trivedi: Dynamic probabilistic drivability maps for lane change and merge driver assistance

Driver foot analysis

• C. Tran, A. Doshi, M.M. Trivedi: Modeling and prediction of driver behavior by foot gesture analysis

• A. Rangesh, M.M. Trivedi: Forced spatial attention for driver foot activity classification

Li-Lo in Action

• A Doshi, B Morris, M Trivedi: On-road prediction of driver's intent with multimodal sensory cues

• JC McCall, MM Trivedi: Driver behavior and situation aware brake assistance for intelligent vehicles