Module opencv::tracking

Tracking API

Long-term optical tracking API

Long-term optical tracking is an important issue for many computer vision applications in real world scenario. The development in this area is very fragmented and this API is an unique interface useful for plug several algorithms and compare them. This work is partially based on AAM and AMVOT .

These algorithms start from a bounding box of the target and with their internal representation they avoid the drift during the tracking. These long-term trackers are able to evaluate online the quality of the location of the target in the new frame, without ground truth.

There are three main components: the TrackerSampler, the TrackerFeatureSet and the TrackerModel. The first component is the object that computes the patches over the frame based on the last target location. The TrackerFeatureSet is the class that manages the Features, is possible plug many kind of these (HAAR, HOG, LBP, Feature2D, etc). The last component is the internal representation of the target, it is the appearance model. It stores all state candidates and compute the trajectory (the most likely target states). The class TrackerTargetState represents a possible state of the target. The TrackerSampler and the TrackerFeatureSet are the visual representation of the target, instead the TrackerModel is the statistical model.

A recent benchmark between these algorithms can be found in OOT

Creating Your Own %Tracker

If you want to create a new tracker, here's what you have to do. First, decide on the name of the class for the tracker (to meet the existing style, we suggest something with prefix "tracker", e.g. trackerMIL, trackerBoosting) -- we shall refer to this choice as to "classname" in subsequent.

• Declare your tracker in modules/tracking/include/opencv2/tracking/tracker.hpp. Your tracker should inherit from Tracker (please, see the example below). You should declare the specialized Param structure, where you probably will want to put the data, needed to initialize your tracker. You should get something similar to :

       class CV_EXPORTS_W TrackerMIL : public Tracker
       {
         public:
          struct CV_EXPORTS Params
          {
           Params();
           //parameters for sampler
           float samplerInitInRadius;  // radius for gathering positive instances during init
           int samplerInitMaxNegNum;  // # negative samples to use during init
           float samplerSearchWinSize;  // size of search window
           float samplerTrackInRadius;  // radius for gathering positive instances during tracking
           int samplerTrackMaxPosNum;  // # positive samples to use during tracking
           int samplerTrackMaxNegNum;  // # negative samples to use during tracking
           int featureSetNumFeatures;  // #features
 
           void read( const FileNode& fn );
           void write( FileStorage& fs ) const;
          };

of course, you can also add any additional methods of your choice. It should be pointed out, however, that it is not expected to have a constructor declared, as creation should be done via the corresponding create() method.

• Finally, you should implement the function with signature :

       Ptr<classname> classname::create(const classname::Params &parameters){
           ...
       }

That function can (and probably will) return a pointer to some derived class of "classname", which will probably have a real constructor.

Every tracker has three component TrackerSampler, TrackerFeatureSet and TrackerModel. The first two are instantiated from Tracker base class, instead the last component is abstract, so you must implement your TrackerModel.

TrackerSampler

TrackerSampler is already instantiated, but you should define the sampling algorithm and add the classes (or single class) to TrackerSampler. You can choose one of the ready implementation as TrackerSamplerCSC or you can implement your sampling method, in this case the class must inherit TrackerSamplerAlgorithm. Fill the samplingImpl method that writes the result in "sample" output argument.

Example of creating specialized TrackerSamplerAlgorithm TrackerSamplerCSC : :

   class CV_EXPORTS_W TrackerSamplerCSC : public TrackerSamplerAlgorithm
   {
     public:
      TrackerSamplerCSC( const TrackerSamplerCSC::Params &parameters = TrackerSamplerCSC::Params() );
      ~TrackerSamplerCSC();
      ...
 
     protected:
      bool samplingImpl( const Mat& image, Rect boundingBox, std::vector<Mat>& sample );
      ...
 
   };

Example of adding TrackerSamplerAlgorithm to TrackerSampler : :

   //sampler is the TrackerSampler
   Ptr<TrackerSamplerAlgorithm> CSCSampler = new TrackerSamplerCSC( CSCparameters );
   if( !sampler->addTrackerSamplerAlgorithm( CSCSampler ) )
     return false;
 
   //or add CSC sampler with default parameters
   //sampler->addTrackerSamplerAlgorithm( "CSC" );

See also

TrackerSamplerCSC, TrackerSamplerAlgorithm

TrackerFeatureSet

TrackerFeatureSet is already instantiated (as first) , but you should define what kinds of features you'll use in your tracker. You can use multiple feature types, so you can add a ready implementation as TrackerFeatureHAAR in your TrackerFeatureSet or develop your own implementation. In this case, in the computeImpl method put the code that extract the features and in the selection method optionally put the code for the refinement and selection of the features.

Example of creating specialized TrackerFeature TrackerFeatureHAAR : :

   class CV_EXPORTS_W TrackerFeatureHAAR : public TrackerFeature
   {
     public:
      TrackerFeatureHAAR( const TrackerFeatureHAAR::Params &parameters = TrackerFeatureHAAR::Params() );
      ~TrackerFeatureHAAR();
      void selection( Mat& response, int npoints );
      ...
 
     protected:
      bool computeImpl( const std::vector<Mat>& images, Mat& response );
      ...
 
   };

Example of adding TrackerFeature to TrackerFeatureSet : :

   //featureSet is the TrackerFeatureSet
   Ptr<TrackerFeature> trackerFeature = new TrackerFeatureHAAR( HAARparameters );
   featureSet->addTrackerFeature( trackerFeature );

TrackerFeatureHAAR, TrackerFeatureSet

TrackerModel

TrackerModel is abstract, so in your implementation you must develop your TrackerModel that inherit from TrackerModel. Fill the method for the estimation of the state "modelEstimationImpl", that estimates the most likely target location, see AAM table I (ME) for further information. Fill "modelUpdateImpl" in order to update the model, see AAM table I (MU). In this class you can use the :cConfidenceMap and :cTrajectory to storing the model. The first represents the model on the all possible candidate states and the second represents the list of all estimated states.

Example of creating specialized TrackerModel TrackerMILModel : :

   class TrackerMILModel : public TrackerModel
   {
     public:
      TrackerMILModel( const Rect& boundingBox );
      ~TrackerMILModel();
      ...
 
     protected:
      void modelEstimationImpl( const std::vector<Mat>& responses );
      void modelUpdateImpl();
      ...
 
   };

And add it in your Tracker : :

   bool TrackerMIL::initImpl( const Mat& image, const Rect2d& boundingBox )
   {
      ...
      //model is the general TrackerModel field of the general Tracker
      model = new TrackerMILModel( boundingBox );
      ...
   }

In the last step you should define the TrackerStateEstimator based on your implementation or you can use one of ready class as TrackerStateEstimatorMILBoosting. It represent the statistical part of the model that estimates the most likely target state.

Example of creating specialized TrackerStateEstimator TrackerStateEstimatorMILBoosting : :

   class CV_EXPORTS_W TrackerStateEstimatorMILBoosting : public TrackerStateEstimator
   {
     class TrackerMILTargetState : public TrackerTargetState
     {
     ...
     };
 
     public:
      TrackerStateEstimatorMILBoosting( int nFeatures = 250 );
      ~TrackerStateEstimatorMILBoosting();
      ...
 
     protected:
      Ptr<TrackerTargetState> estimateImpl( const std::vector<ConfidenceMap>& confidenceMaps );
      void updateImpl( std::vector<ConfidenceMap>& confidenceMaps );
      ...
 
   };

And add it in your TrackerModel : :

   //model is the TrackerModel of your Tracker
   Ptr<TrackerStateEstimatorMILBoosting> stateEstimator = new TrackerStateEstimatorMILBoosting( params.featureSetNumFeatures );
   model->setTrackerStateEstimator( stateEstimator );

TrackerModel, TrackerStateEstimatorMILBoosting, TrackerTargetState

During this step, you should define your TrackerTargetState based on your implementation. TrackerTargetState base class has only the bounding box (upper-left position, width and height), you can enrich it adding scale factor, target rotation, etc.

Example of creating specialized TrackerTargetState TrackerMILTargetState : :

   class TrackerMILTargetState : public TrackerTargetState
   {
     public:
      TrackerMILTargetState( const Point2f& position, int targetWidth, int targetHeight, bool foreground, const Mat& features );
      ~TrackerMILTargetState();
      ...
 
     private:
      bool isTarget;
      Mat targetFeatures;
      ...
 
   };

Modules

prelude

Structs

ClfMilBoost
ClfMilBoost_Params
CvFeatureParams
CvHaarEvaluator
CvHaarEvaluator_FeatureHaar
MultiTracker	********************************** MultiTracker Class ---By Laksono Kurnianggoro---) *********************************** This class is used to track multiple objects using the specified tracker algorithm.
MultiTrackerTLD	Multi Object %Tracker for TLD.
MultiTracker_Alt	Base abstract class for the long-term Multi Object Trackers:
TrackerBoosting_Params
TrackerCSRT_Params
TrackerFeatureFeature2d	\brief TrackerFeature based on Feature2D
TrackerFeatureHAAR	TrackerFeature based on HAAR features, used by TrackerMIL and many others algorithms
TrackerFeatureHAAR_Params
TrackerFeatureHOG	\brief TrackerFeature based on HOG
TrackerFeatureLBP	\brief TrackerFeature based on LBP
TrackerFeatureSet	Class that manages the extraction and selection of features
TrackerGOTURN_Params
TrackerKCF_Params
TrackerMIL_Params
TrackerMedianFlow_Params
TrackerSampler	Class that manages the sampler in order to select regions for the update the model of the tracker
TrackerSamplerCS	TrackerSampler based on CS (current state), used by algorithm TrackerBoosting
TrackerSamplerCSC	TrackerSampler based on CSC (current state centered), used by MIL algorithm TrackerMIL
TrackerSamplerCSC_Params
TrackerSamplerCS_Params
TrackerSamplerPF	This sampler is based on particle filtering.
TrackerSamplerPF_Params	This structure contains all the parameters that can be varied during the course of sampling algorithm. Below is the structure exposed, together with its members briefly explained with reference to the above discussion on algorithm's working.
TrackerStateEstimatorAdaBoosting	TrackerStateEstimatorAdaBoosting based on ADA-Boosting
TrackerStateEstimatorAdaBoosting_TrackerAdaBoostingTargetState	Implementation of the target state for TrackerAdaBoostingTargetState
TrackerStateEstimatorMILBoosting	TrackerStateEstimator based on Boosting
TrackerStateEstimatorMILBoosting_TrackerMILTargetState	Implementation of the target state for TrackerStateEstimatorMILBoosting
TrackerStateEstimatorSVM	\brief TrackerStateEstimator based on SVM
TrackerTLD_Params
TrackerTargetState	Abstract base class for TrackerTargetState that represents a possible state of the target.

Enums

CvFeatureParams_FeatureType
TrackerKCF_MODE	\brief Feature type to be used in the tracking grayscale, colornames, compressed color-names The modes available now:

Constants

CC_FEATURE_PARAMS
CC_FEATURE_SIZE
CC_ISINTEGRAL
CC_MAX_CAT_COUNT
CC_NUM_FEATURES
CC_RECT
CC_RECTS
CC_TILTED
CV_HAAR_FEATURE_MAX
FEATURES
HFP_NAME
HOGF_NAME
LBPF_NAME
N_BINS
N_CELLS
TrackerSamplerCSC_MODE_DETECT	mode for detect samples
TrackerSamplerCSC_MODE_INIT_NEG	mode for init negative samples
TrackerSamplerCSC_MODE_INIT_POS	mode for init positive samples
TrackerSamplerCSC_MODE_TRACK_NEG	mode for update negative samples
TrackerSamplerCSC_MODE_TRACK_POS	mode for update positive samples
TrackerSamplerCS_MODE_CLASSIFY	mode for classify samples
TrackerSamplerCS_MODE_NEGATIVE	mode for negative samples
TrackerSamplerCS_MODE_POSITIVE	mode for positive samples

Traits

ClfMilBoostTrait
ClfMilBoost_ParamsTrait
CvFeatureParamsTrait
CvHaarEvaluatorTrait
CvHaarEvaluator_FeatureHaarTrait
MultiTrackerTLDTrait	Multi Object %Tracker for TLD.
MultiTrackerTrait	********************************** MultiTracker Class ---By Laksono Kurnianggoro---) *********************************** This class is used to track multiple objects using the specified tracker algorithm.
MultiTracker_AltTrait	Base abstract class for the long-term Multi Object Trackers:
Tracker	Base abstract class for the long-term tracker:
TrackerBoosting	the Boosting tracker
TrackerBoosting_ParamsTrait
TrackerCSRT	********************************* CSRT *********************************** the CSRT tracker
TrackerCSRT_ParamsTrait
TrackerFeature	Abstract base class for TrackerFeature that represents the feature.
TrackerFeatureFeature2dTrait	\brief TrackerFeature based on Feature2D
TrackerFeatureHAARTrait	TrackerFeature based on HAAR features, used by TrackerMIL and many others algorithms
TrackerFeatureHAAR_ParamsTrait
TrackerFeatureHOGTrait	\brief TrackerFeature based on HOG
TrackerFeatureLBPTrait	\brief TrackerFeature based on LBP
TrackerFeatureSetTrait	Class that manages the extraction and selection of features
TrackerGOTURN	the GOTURN (Generic Object Tracking Using Regression Networks) tracker
TrackerGOTURN_ParamsTrait
TrackerKCF	the KCF (Kernelized Correlation Filter) tracker
TrackerKCF_ParamsTrait
TrackerMIL	The MIL algorithm trains a classifier in an online manner to separate the object from the background.
TrackerMIL_ParamsTrait
TrackerMOSSE	the MOSSE (Minimum Output Sum of Squared %Error) tracker
TrackerMedianFlow	the Median Flow tracker
TrackerMedianFlow_ParamsTrait
TrackerModel	Abstract class that represents the model of the target. It must be instantiated by specialized tracker
TrackerSamplerAlgorithm	Abstract base class for TrackerSamplerAlgorithm that represents the algorithm for the specific sampler.
TrackerSamplerCSCTrait	TrackerSampler based on CSC (current state centered), used by MIL algorithm TrackerMIL
TrackerSamplerCSC_ParamsTrait
TrackerSamplerCSTrait	TrackerSampler based on CS (current state), used by algorithm TrackerBoosting
TrackerSamplerCS_ParamsTrait
TrackerSamplerPFTrait	This sampler is based on particle filtering.
TrackerSamplerPF_ParamsTrait	This structure contains all the parameters that can be varied during the course of sampling algorithm. Below is the structure exposed, together with its members briefly explained with reference to the above discussion on algorithm's working.
TrackerSamplerTrait	Class that manages the sampler in order to select regions for the update the model of the tracker
TrackerStateEstimator	Abstract base class for TrackerStateEstimator that estimates the most likely target state.
TrackerStateEstimatorAdaBoostingTrait	TrackerStateEstimatorAdaBoosting based on ADA-Boosting
TrackerStateEstimatorAdaBoosting_TrackerAdaBoostingTargetStateTrait	Implementation of the target state for TrackerAdaBoostingTargetState
TrackerStateEstimatorMILBoostingTrait	TrackerStateEstimator based on Boosting
TrackerStateEstimatorMILBoosting_TrackerMILTargetStateTrait	Implementation of the target state for TrackerStateEstimatorMILBoosting
TrackerStateEstimatorSVMTrait	\brief TrackerStateEstimator based on SVM
TrackerTLD	the TLD (Tracking, learning and detection) tracker
TrackerTLD_ParamsTrait
TrackerTargetStateTrait	Abstract base class for TrackerTargetState that represents a possible state of the target.

Functions

tld_get_next_dataset_frame
tld_init_dataset	C++ default parameters