spear.nist_sre12 1.0.0

Getting the data

You first need to order the NIST SRE databases (Fisher, Switchboard, MIXER):

http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2013S03

Please follow the instructions and the evaluation plan given by NIST:

http://www.nist.gov/itl/iad/mig/sre12.cfm

Getting the file lists

The file lists of the development and evaluation sets are automatically downloaded from this pypi package:

https://pypi-hypernode.com/pypi/xbob.db.nist_sre12

The file list of the development set were prepared by the I4U consortium. Special thanks to Rahim Saeidi for the good work (original link of the lists: http://cls.ru.nl/~saeidi/file_library/I4U.tgz). The file names were then normalized following the PRISM definition. Please follow the instructions in xbob.db.nist_sre12

Setting the database configuration file

Once the sphere data are preprocessed, and possibly downsampled to 8KHz, you should set the paths in the configuration file to the data according to your own environment for both Male and Female:

- config/database/nist_sre12/male.py
- config/database/nist_sre12/female.py

Running the experiments

The following command is intended to run the entire experiment on both the development and the evaluation sets using ISV (Inter-Session Variability Modeling) and for both Male and Female:

$  bin/spkverif_isv.py -d config/database/nist_sre12/male.py -T PATH/TO/TEMP_DIR/  -U PATH/TO/RESULTS_DIR/ -p config/preprocessing/energy.py -f config/features/mfcc_60.py -t config/tools/isv/isv_512g_u200.py -b male

$  bin/spkverif_isv.py -d config/database/nist_sre12/female.py -T PATH/TO/TEMP_DIR/  -U PATH/TO/RESULTS_DIR/ -p config/preprocessing/energy.py -f config/features/mfcc_60.py -t config/tools/isv/isv_512g_u200.py -b female

For more details and options, please type:

$ bin/spkverif_isv.py --help

You may want to change the parameters in the configuration files for VAD (Energy, 4Hz Modulation energy), Features (MFCC, LFCC), and Tools (UBM-GMM, ISV, I-Vector). Please look to the different configuration settings in:

- src/spkrec/config/

Running on the grid

In order to run the experiment on the grid, you need to have gridtk installed on your local network. Details can be found here:

https://pypi-hypernode.com/pypi/gridtk

Evaluation on the Development set

The EER on the Development sets can be obtained using the evaluation script from the bob library.

For Male, without any score normalization:

$ ./bin/bob_compute_perf.py -d PATH/TO/RESULTS_DIR/male/scores/nonorm/scores-dev -t PATH/TO/RESULTS_DIR/male/scores/nonorm/scores-dev -x

• EER = 4.68%

For Male, with ZT score normalization:

$ ./bin/bob_compute_perf.py -d PATH/TO/RESULTS_DIR/male/scores/ztnorm/scores-dev -t PATH/TO/RESULTS_DIR/male/scores/ztnorm/scores-dev -x

• EER = 3.98%

For Female, without any score normalization:

$ ./bin/bob_compute_perf.py -d PATH/TO/RESULTS_DIR/female/scores/nonorm/scores-dev -t PATH/TO/RESULTS_DIR/female/scores/nonorm/scores-dev -x

• EER = 6.28%

For Female, with ZT score normalization:

$ ./bin/bob_compute_perf.py -d PATH/TO/RESULTS_DIR/female/scores/ztnorm/scores-dev -t PATH/TO/RESULTS_DIR/female/scores/ztnorm/scores-dev -x

• EER = 5.16%

Notice that there are different implementations for EER. For example, the default one in Bob is different from the implementation in Bosaris.

Please check the NIST evaluation guidlines to see how to evaluate on SRE 2012 Evaluation set. Further, the simple scores should be converted to compound scores. Please find more details given by Niko Brummer on the webpage of Bosaris toolkit:

https://sites.google.com/site/bosaristoolkit/sre12