Introduction

Welcome to the VIVA traffic light detection benchmark! This challenge uses the LISA Traffic Light Dataset. This dataset consists of the original LISA Traffic Light training and test data. The first are provided with annotations for training, while the final is used for testing and will be published with annotations after the VIVA Challenge.

The challenge evaluates detectors on six superclasses:

  • Green
  • Yellow
  • Red
  • Green Left
  • Yellow Left
  • Red Left

As in the German Traffic Sign Detection Benchmark (GTSDB) and VIVA Traffic Sign Detection, detectors are evaluated using Area Under Curve (AUC) of a precision-recall curve for a particular traffic light superclass. Evaluation of bounding boxes is done using the PASCAL overlap requirement of 50%. Precision-recall curves are interpolated as specified in [1]. For both day- and nighttime scenario there will be a class winner for each superclass, and the overall winner is the entry with the highest mean AUC across all six superclasses. Partial submissions with only a subset of the superclasses are allowed, but discouraged, and these will not be considered for the overall leaderboard.

Submissions

Submissions take the form of a zip-file containing detections from several runs with different parameters (in order to be able to generate a PR-curve) for each superclass, split in folders. The structure is as follows:

results.zip
  ∟ Green
    ∟ results.csv
  ∟ Yellow
    ∟ results.csv
  ∟ Red
    ∟ results.csv
  ∟ GreenLeft
    ∟ results.csv
  ∟ YellowLeft
    ∟ results.csv
  ∟ RedLeft
    ∟ results.csv

The folder names matter, but the zip-file and the results-files can have any names. There can be an arbitrary number of results files in each folder.

Each results-file should be formatted in the same way as for GTSDB: a CSV-file with a detection on each line, fields separated by semi-colon (;). The file should contain no header. The fields must be:

  • Filename with extension (without path) of the file in which your algorithm has detected a traffic sign
  • The bounding box of the detection in the image
    • Leftmost image column of the box
    • Upmost image row of the box
    • Rightmost image column of the box
    • Downmost image row of the box
    • Score

Any further fields will be ignored.

Dataset and tools

Use the following links to download the datasets:

When using this dataset we would appreciate if you cite the following papers:

Morten Bornø Jensen, Mark Philip Philipsen, Andreas Møgelmose, Thomas B Moeslund, and Mohan M Trivedi. “Vision for Looking at Traffic Lights: Issues, Survey, and Perspectives”. In: IEEE Transactions on Intelligent Transportation Systems (2015).

Mark Philip Philipsen, Morten Bornø Jensen, Andreas Møgelmose, Thomas B Moeslund, and Mohan M Trivedi. “Learning Based Traffic Light Detection: Evaluation on Challenging Dataset”. In: 18th IEEE Intelligent Transportation Systems Conference (2015).

For an example of the sample workflow we refer to VIVA Traffic Sign Detection.

References

[1] Davis, Jesse, and Mark Goadrich. “The relationship between Precision-Recall and ROC curves.” Proceedings of the 23rd international conference on Machine learning. ACM, 2006.