A metric for multiple target tracking
A standard metric for evaluating the multiple target tracking algorithm is the CLEAR MOT. This metric is described in the paper [1] .
Details
We provide the code that implements the metric CLER-MOT has described by the authors in [1]. The function is implemented in MATLAB and has been tested on real data generated by a multiple-target tracker.
The tarball contains these MATLAB files:
- groundtruth.mat contains the labeled annotations of 3 objects. Objects are specificated by labeled bounding box as [ID tl.x tl.y br.x br.y]. The ID is the label, tl is the top-left point of the annotations and br is the bottom-right.
- result.mat contains the tracking results hypothesis as taken from the ground truth files. So, MOTA and MOTP results to 100% in this simple example.
- evaluateMOT.m is the function that performs the evaluation.
- main.m is the main file
- GreedyAssociation.m is the file that performs the association given the distance matrix. You can replace with other solvers like f.e. Hungarian algorithm.
These data files are only an example: if you want to use the script to evaluate your multiple target tracking result, you have to re-create the structures groundtruth.mat and result.mat with your own data.
Note that ID Switch (mismatch) should be carefully counted by visual inspection, considering that the automatic association is not able to manage situations where targets are very near each other.
Bur reporting is very appreciated, please contact me as soon as possible, leaving a comment below.
Download
The code was moved to Github please download it here
Acknowledge
Please, note if you use the code, you have to cite our work:
@article{ masi:multimedia12, author = {Bagdanov, Andrew D. and Del Bimbo, Alberto and Dini,
Fabrizio and Lisanti, Giuseppe and Masi, Iacopo},
title = {Compact and efficient posterity logging of face imagery for video surveillance},
booktitle = {IEEE Multimedia},
year = {2012}, }
Changelog
– 2012/05/11 Initial release.
(c) Copyright 2011 - MICC - Media Integration and Communication Center, University of Florence. Iacopo Masi and Giuseppe Lisanti <masi,lisanti>@dsi.unifi.it
Demo Example
load groundtruth load result VOCscore = 0.5; dispON = true; ClearMOT = evaluateMOT(gt,result,VOCscore,dispON);
The result is the following:
------ ::RESULTS:: --------- rateFP = 0 (0%) rateTP = 1 (100%) rateFN = 0 (0%) ---------------------------- TP = 9 FN = 0 FP = 0 ID switch (MisMatch) = 0 ***NOTE***: ID switch should be carefully counted by visual inspection Sum of GrountTruth Obj = 9 Sum of FN+TP+IDSW = 9 ---------------------------- MOTP = 1 MOTA = 1 (100%) ----------------------------
References
- Keni Bernardin and Rainer Stiefelhagen. “Evaluating multiple objec tracking performance: the CLEAR MOT metrics” J. Image Video Process. 2008, Article 1 (January 2008), 10 pages.” DOI=10.1155/2008/246309 http://dx.doi.org/10.1155/2008/246309
Buongiorno,
vi segnalo un errore nel calcolo della metrice rateFP.
ClearMOT.rateFP = falseneg/gt; -> dovrebbe essere falsepos/gt
disp([‘rateFP = ‘, num2str(ClearMOT.rateFN) ‘ (‘,num2str(ClearMOT.rateFN*100), ‘%)’]); -> al posto di rateFN dovrebbe esserci rateFP
Ok thank you so much. I fixed now the bug on github! Grazie ancora.
Salve,
più che commentare il funzionamento del codice, gradirei avere un chiarimento sulla tipologia dei risultati prodotti.
Sto usando il codice fornito per analizzare analiticamente le performance di un tracker e mi sono posto un dubbio. Il codice, basato sul paper indicato sulla pagina web, dovrebbe considerare un FalsePositive come tale quando la distanza tra il punto tracciato dal tracker e il groundtruth eccede una determinata soglia. E fin qui ci siamo. Tuttavia mi chiedo: “Come è possibile discernere un falso positivo da un mancato tracciamento?”. Mi spiego meglio. Se il mio tracker in 10 frame non traccia alcunché (nonostante il groundtruth segnala la presenza di qualcosa), tale valore dovrebbe riferire ad un MISS e non contribuire al calcolo del tasso dei falsi positivi.
Mi sbaglio? Che tipo di valori dovrebbe contenere la struttura “result” di matlab per tenere conto di questo fenomeno?
Grazie mille
Hi,
Thanks for the code. Can you just throw light on the doubt i have.
I have 61 frames of 1 object tracking.
GT – 1-61 frames id is 1 with bboxes..
result – 1-50 frames id is 1 and 51-61 frames id changes to 2 with bboxes respectively.
How does your code work in this scenario?
It crashes at this point:
for l=1:length(idxTracks)
tt = idxTracks(l);
distance = mydistance(bboxes(b,:),target(tt), ‘VOCscore’ );
score(b,l) = distance;
indexObj(b,l) = bboxes(b,1);
end
Because target has only one object but id is 2. so it tries to acces target(2) instead of target(1).
result2(49).trackerData
ans =
target: [1×1 struct]
idxTracks: 1
result2(49).trackerData.target
ans =
bbox: [728 313 186 186]
result2(50).trackerData
ans =
target: [1×1 struct]
idxTracks: 2
result2(50).trackerData.target
ans =
bbox: [732 312 190 190]
Can you help me in evaluating this case?
Hi,
Changed target(tt) to target(l) and it works.
distance = mydistance(bboxes(b,:),target(l), ‘VOCscore’ );
Can you report this issue to Git please? I am not workig on this project anymore.
I have noticed you don’t monetize your blog, don’t waste
your traffic, you can earn extra bucks every month because you’ve got hi quality content.
If you want to know how to make extra bucks, search
for: best adsense alternative Wrastain’s tools