challenge.uccs 2.1.0

Face Detection

The first script is a face detection script, which will detect the faces in the validation (and test) set. The baseline detector uses the MTCNN v2 face detection module, where we had to lower the detection thresholds in order to detect most of the faces in the images – still around 20 percent of the (mostly difficult) faces could not be detected using MTCNN v2. Additionally, there are a lot of background areas that are mistakenly marked as faces.

You can call the face detector baseline script using:

./bin/baseline_detector.py

Please refer to ./bin/baseline_detector.py -h for possible options. Here is a subset of options that you might want/need to use/change:

--data-directory: Specify the directory, where you have downloaded the UCCS dataset into.

--result-file: The file to write the detection results into; this will be in the required format.

--which-set: The set, for which the baseline detector should be run; possible values: training, validation, test, sample; default: validation.

--verbose: Increase the verbosity of the script; using --verbose --verbose or -vv is recommended; -vvv will write more information.

--debug: Run only over the specified number of images; for debug purposes only.

--display: Display the detected bounding boxes (green) and the ground truth (red); for debug purposes only.

--parallel: Run in the given number of parallel processes; can speed up the processing tremendously.

--gpus: Run the face detection using Caffe in GPU mode (this might not work depending on your setup of Caffe); you can specify the device ids to use, typically only 0; if not specified, runs in CPU mode

On a machine with 32 CPU cores, a good command line for the full baseline experiment would be:

./bin/baseline_detector.py --data-directory YOUR-DATA-DIRECTORY -vv --parallel 32

On a machine with 32 cores and two NVidia GPUs, a good command line for the full baseline experiment would be:

./bin/baseline_detector.py --data-directory YOUR-DATA-DIRECTORY -vv --parallel 8 --gpus 0 1

To run a small-scale experiment, i.e., to display the detected faces on 20 images, a good command line might be:

./bin/baseline_detector.py --data-directory YOUR-DATA-DIRECTORY -vvv --display --debug 20

By default, the face detection score file will be written to ./results/UCCS-v2-detection-baseline-validation.txt.

Face Recognition

For face recognition, we utilize the MTCNN v2 face detector to detect all the faces (see above) and make sure that the bounding boxes with the highest overlap to the ground truth labels are used. For each of the detected bounding boxes in the training set, we enlarge the bounding box with a factor of 1.3 and extract features using the pool5/7x7_s1 layer of the VGG v2 recognition module, which we normalize to unit Euclidean length. The template enrollment stage simply computes the average of the training set features for each of the known subjects. To account for the fact that we have unknown subjects in the training and test set, we also enroll one gallery template including all known unknown subjects labeled as -1.

For a probe image, we assume that we do not have any labels (this is what the test set will look like). Hence, we run the MTCNN v2 detector to detect faces (which will include several background regions). For each bounding box, we extract the VGG v2 feature as above, and compute cosine similarities to all gallery templates including the unknown template. The highest 10 similarities are stored into the score file – except when the unknown label (-1) is amongst the highest scores, in which case only the scores up to the -1 label are stored. This means that when the highest score is with the -1 label, only one value is stored in the score file.

You can call the face recognition baseline script using:

./bin/baseline_recognizer.py

Please refer to ./bin/baseline_recognizer.py -h for possible options. Here is a subset of options that you might want/need to use/change:

--data-directory: Specify the directory, where you have downloaded the UCCS dataset into

--result-file: The file to write the recognition results into; this will be in the required format

--verbose: Increase the verbosity of the script; using --verbose --verbose or -vv is recommended; -vvv will write more information

--temp-dir: Specify the directory, where temporary files are stored; these files will be computed only once and reloaded if present

--force: Ignore existing temporary files and always recompute everything

--debug: Run only over the specified number of identities; for debug purposes only; will modify file names of temporary files and result file

--parallel: Run in the given number of parallel processes; can speed up the processing tremendously

On a machine with 32 CPU cores, a good command line would be:

./bin/baseline_recognizer.py --data-directory YOUR-DATA-DIRECTORY -vv --parallel 32

When using Caffe in GPU mode with two NVidia GPUs, a good command line would be:

./bin/baseline_recognizer.py --data-directory YOUR-DATA-DIRECTORY -vv --parallel 2 --gpus 0 1

By default, the face recognition score file will be written to ./results/UCCS-v2-recognition-baseline-validation.txt.

Face Detection Evaluation

As the ground-truth is usually larger than the face, we do not punish bounding boxes that are smaller than the ground truth. Therefore, the union (the denominator) takes into account only one fourth of the ground truth bounding box – or the intersection area, whichever is larger:

where \(|\cdot|\) is the area operator. Hence, when the detected bounding box \(D\) covers at least a fourth of the ground-truth bounding box \(G\) and is entirely contained inside \(G\), an overlap of 1 is reached.

The face detection is evaluated using the Free Receiver Operator Characteristic (FROC) curve, which plots the percentage of correctly detected faces over the total number of false alarms (detected background regions). Different points on the FROC curve can be obtained for different detector confidence values. This plot can be created using:

./bin/evaluate_detector.py

This script has several options, some of which need to be specified, see ./bin/evaluate_detector.py -h:

--data-directory: Specify the directory, where you have downloaded the UCCS dataset into

--result-files: A list of all files containing detection (or recognition) results

--labels: A list of labels for the algorithms; must be the same number and in the same order as --result-files

--froc-file: The name of the output .pdf file containing the FROC plot; default is UCCS-v2-FROC.pdf

--log-x: will plot the horizontal axis in logarithmic scale

--only-present: will ignore any file for which no detection exists (for debug purposes only, i.e., when detector ran with the --debug option)

--verbose: Increase the verbosity of the script; using --verbose --verbose or -vv is recommended

To plot the baseline FROC curve (which is shown on the competition website), execute:

./bin/evaluate_detector.py --data-directory YOUR-DATA-DIRECTORY --result-files results/UCCS-v2-detection-baseline-validation.txt --labels Baseline --log-x -vv

Face Recognition Evaluation

Open set face recognition is evaluated using the Detection and Identification Rate (DIR) curve, which plots the percentage of correctly detected and identified faces over the total number of false identifications. Based on various values of the false identifications, several score thresholds are computed. A face is said to be identified correctly if the recognition score is greater than the threshold and the correct identity has the highest recognition score for that face. The number of correctly identified faces is computed, and divided by the total number of known faces. For more details, please refer to [1].

The DIR plot can be created using:

./bin/evaluate_recognizer.py

As usual, the script has several options, which are similar to ./bin/evaluate_detector.py above, see ./bin/evaluate_recognizer.py -h for a complete list:

--data-directory: Specify the directory, where you have downloaded the UCCS dataset into

--result-files: A list of all files containing recognition results

--labels: A list of labels for the algorithms; must be the same number and in the same order as --result-files

--dir-file: The name of the output .pdf file containing the DIR plot; default is UCCS-v2-DIR.pdf

--log-x: will plot the horizontal axis in logarithmic scale

--only-present: will ignore any file for which no detection exists (for debug purposes only, i.e., when recognizer ran with the --debug option)

--verbose: Increase the verbosity of the script; using --verbose --verbose or -vv is recommended

--rank: Use the given rank to plot the DIR curve

To plot the baseline Rank 1 DIR curve (which is shown on the competition website), execute:

./bin/evaluate_recognizer.py --data-directory YOUR-DATA-DIRECTORY --result-files results/UCCS-v2-recognition-baseline-validation.txt --labels Baseline --log-x -vv