#include <vpRBProbabilistic3DDriftDetector.h>

Inheritance diagram for vpRBProbabilistic3DDriftDetector:

Classes
struct	vpStored3DSurfaceColorPoint

Public Member Functions
	vpRBProbabilistic3DDriftDetector ()
void	update (const vpRBFeatureTrackerInput &previousFrame, const vpRBFeatureTrackerInput &frame, const vpHomogeneousMatrix &cTo, const vpHomogeneousMatrix &cprevTo) VP_OVERRIDE
double	score (const vpRBFeatureTrackerInput &frame, const vpHomogeneousMatrix &cTo) VP_OVERRIDE
double	getScore () const VP_OVERRIDE
bool	hasDiverged () const VP_OVERRIDE
void	reset () VP_OVERRIDE
void	display (const vpImage< vpRGBa > &I) VP_OVERRIDE
Settings
double	getMinDistForNew3DPoints () const
void	setMinDistForNew3DPoints (double distance)
double	getFilteringMax3DError () const
void	setFilteringMax3DError (double maxError)
double	getDepthStandardDeviation () const
void	setDepthStandardDeviation (double sigma)
double	getInitialColorStandardDeviation () const
void	setInitialColorStandardDeviation (double sigma)
double	getColorUpdateRate () const
void	setColorUpdateRate (double updateRate)
unsigned int	getSampleStep () const
void	setSampleStep (unsigned int sampleStep)
void	loadJsonConfiguration (const nlohmann::json &) VP_OVERRIDE
void	loadRepresentation (const std::string &)
void	saveRepresentation (const std::string &) const

Detailed Description

Algorithm that uses tracks object surface points in order to estimate the probability that tracking is successful.

Given a set of surface points $\mathbf{X}_0, ..., \mathbf{X}_N$ , each point $\mathbf{X}_i$ being associated to:

a color distribution $\mathcal{N}(\mathbf{\bar c_i}, \mathbf{\Sigma_{c_i}^2})$ ,
its distance to the camera being ,
its projection in the current color and depth images $\mathbf{I_c}, \mathbf{I_Z}$ having coordinates .
its visibility $V(\mathbf{X_i})$ , which is 1 if lie in the image, is close to the rendered depth value and the normal at the surface marks the point as visible from the camera's point of view.

We compute the probability that tracking is successful for a given pose $^{c}\mathbf{T}_o$ as:

$\‍[ p(^{c}\mathbf{T}_o) = \frac{1}{\sum_{i=0}^N w_i \cdot V(\mathbf{X_i})}\sum_{i=0}^N w_i \cdot V(\mathbf{X_i}) \cdot p(\mathbf{X_i}) \‍]$

with

$\‍[\begin{aligned} p(\mathbf{X_i}) &= p(\mathbf{I_c}(u_i, v_i)|\mathcal{N}(\mathbf{\bar c_i}, \mathbf{\Sigma_{c_i}^2})) \cdot p(\mathbf{I_Z}(u_i, v_i) | \mathcal{N}(Z_i, \sigma_Z^2)) \\ p(\mathbf{I_c}(u_i, v_i) | \mathcal{N}(\mathbf{\bar c_i}, \mathbf{\Sigma_{c_i}^2})) &= erfc(\frac{1}{\sqrt{2}}\lVert \frac{\mathbf{I_c}(u_i, v_i) - \mathbf{\bar c_i}}{diag(\mathbf{\Sigma_{c_i}})} \rVert_2) \\ p(\mathbf{I_Z}(u_i, v_i) | \mathcal{N}(Z_i, \sigma_Z^2)) &= erfc(\frac{1}{\sigma_Z \sqrt{2}}\mathbf{I_Z}(u_i, v_i) - Z_i) \end{aligned} \‍]$

if the depth is unavailable, then we set $p(\mathbf{I_Z}(u_i, v_i) | \mathcal{N}(Z_i, \sigma_Z^2)) = 1$

Here, the color distribution is estimated online for each point separately using exponential moving average/variance techniques. For each point the update step is computed as $p(\mathbf{I_Z}(u_i, v_i) | \mathcal{N}(Z_i, \sigma_Z^2))\cdot \alpha$ where $\alpha$ is a fixed parameter. Larger values will lead to faster update rates and may be more beneficial for non lambertian materials.

For the depth, $\sigma_Z$ is a fixed parameter to be tweaked by the user.

Every time update() is called, the set of points $\mathbf{X}_0, ..., \mathbf{X}_N,$ may grow larger. If a new candidate point is visible and is far enough from points already in the set, it is added to it.

Tutorials & Examples

Tutorials
If you want to have an in-depth presentation of the Render-Based Tracker (RBT), you may have a look at:

Tutorial: Tracking with the Render-Based Tracker

Definition at line 99 of file vpRBProbabilistic3DDriftDetector.h.

Constructor & Destructor Documentation

◆ vpRBProbabilistic3DDriftDetector()

vpRBProbabilistic3DDriftDetector::vpRBProbabilistic3DDriftDetector ( )

inline

Definition at line 243 of file vpRBProbabilistic3DDriftDetector.h.

Member Function Documentation

◆ display()

void vpRBProbabilistic3DDriftDetector::display ( const vpImage< vpRGBa > & I )

virtual

Displays the information used for drift detection.

Parameters

I	the image in which to display the information

Implements vpRBDriftDetector.

Definition at line 295 of file vpRBProbabilistic3DDriftDetector.cpp.

References vpDisplay::displayPoint().

◆ getColorUpdateRate()

double vpRBProbabilistic3DDriftDetector::getColorUpdateRate ( ) const

inline

Get the rate at which the colors of surface points are updated.

Note that if depth is available, this component is further multiplied by the probability of depth being correct for a given point.

A high value will lead to a fast update rate (short term memory), while a lower one will update slower. A slower update may lead to a more stable tracking score. A higher value may be better suited to non isotropic materials.

Definition at line 349 of file vpRBProbabilistic3DDriftDetector.h.

◆ getDepthStandardDeviation()

double vpRBProbabilistic3DDriftDetector::getDepthStandardDeviation ( ) const

inline

Get the standard deviation that is used when computing the probability that the observed depth Z is the correct depth given the rendered depth at the same pixel.

Definition at line 316 of file vpRBProbabilistic3DDriftDetector.h.

◆ getFilteringMax3DError()

double vpRBProbabilistic3DDriftDetector::getFilteringMax3DError ( ) const

inline

Returns the maximum 3D distance (in meters) above which a tracked surface point is rejected for the drift estimation step.

The surface point's distance to the camera is compared to rendered depth. If the difference between the two is too great, it is rejected.

This is mainly used to handle self occlusions.

Definition at line 301 of file vpRBProbabilistic3DDriftDetector.h.

◆ getInitialColorStandardDeviation()

double vpRBProbabilistic3DDriftDetector::getInitialColorStandardDeviation ( ) const

inline

Get the standard deviation that is used to initialize the color distribution when adding a new surface point. This standard deviation is applied on all color components.

Definition at line 330 of file vpRBProbabilistic3DDriftDetector.h.

◆ getMinDistForNew3DPoints()

double vpRBProbabilistic3DDriftDetector::getMinDistForNew3DPoints ( ) const

inline

Get the minimum distance criterion (in meters) that is used when trying to add new points to track for the drift detection.

A candidate surface point is compared to all the currently tracked surface point and if any of these points is below the minimum distance, the candidate is rejected.

Returns: the minimum distance, in meters

Definition at line 282 of file vpRBProbabilistic3DDriftDetector.h.

◆ getSampleStep()

unsigned int vpRBProbabilistic3DDriftDetector::getSampleStep ( ) const

inline

Definition at line 364 of file vpRBProbabilistic3DDriftDetector.h.

◆ getScore()

double vpRBProbabilistic3DDriftDetector::getScore ( ) const

virtual

Returns the probability [0, 1] that tracking is successful.

Implements vpRBDriftDetector.

Definition at line 306 of file vpRBProbabilistic3DDriftDetector.cpp.

◆ hasDiverged()

bool vpRBProbabilistic3DDriftDetector::hasDiverged ( ) const

virtual

Returns whether the tracking has diverged and should be reinitialized. This function should be called after update.

Implements vpRBDriftDetector.

Definition at line 311 of file vpRBProbabilistic3DDriftDetector.cpp.

◆ loadJsonConfiguration()

void vpRBProbabilistic3DDriftDetector::loadJsonConfiguration ( const nlohmann::json & j )

virtual

Implements vpRBDriftDetector.

Definition at line 317 of file vpRBProbabilistic3DDriftDetector.cpp.

References setColorUpdateRate(), setDepthStandardDeviation(), setFilteringMax3DError(), setInitialColorStandardDeviation(), setMinDistForNew3DPoints(), and setSampleStep().

◆ loadRepresentation()

void vpRBProbabilistic3DDriftDetector::loadRepresentation ( const std::string & filename )

Definition at line 327 of file vpRBProbabilistic3DDriftDetector.cpp.

References vpException::ioError.

◆ reset()

void vpRBProbabilistic3DDriftDetector::reset ( )

inlinevirtual

Reimplemented from vpRBDriftDetector.

Definition at line 261 of file vpRBProbabilistic3DDriftDetector.h.

References reset().

Referenced by reset().

◆ saveRepresentation()

void vpRBProbabilistic3DDriftDetector::saveRepresentation ( const std::string & filename ) const

Definition at line 338 of file vpRBProbabilistic3DDriftDetector.cpp.

References vpException::ioError.

◆ score()

BEGIN_VISP_NAMESPACE double vpRBProbabilistic3DDriftDetector::score	(	const vpRBFeatureTrackerInput &	frame,
		const vpHomogeneousMatrix &	cTo )

virtual

◆ setColorUpdateRate()

void vpRBProbabilistic3DDriftDetector::setColorUpdateRate ( double updateRate )

inline

Set the update rate for the color distribution. It should be between 0 and 1.

Parameters

updateRate the update rate

Definition at line 356 of file vpRBProbabilistic3DDriftDetector.h.

References vpException::badValue.

Referenced by loadJsonConfiguration().

◆ setDepthStandardDeviation()

void vpRBProbabilistic3DDriftDetector::setDepthStandardDeviation ( double sigma )

inline

Definition at line 317 of file vpRBProbabilistic3DDriftDetector.h.

References vpException::badValue.

Referenced by loadJsonConfiguration().

◆ setFilteringMax3DError()

void vpRBProbabilistic3DDriftDetector::setFilteringMax3DError ( double maxError )

inline

Definition at line 303 of file vpRBProbabilistic3DDriftDetector.h.

References vpException::badValue.

Referenced by loadJsonConfiguration().

◆ setInitialColorStandardDeviation()

void vpRBProbabilistic3DDriftDetector::setInitialColorStandardDeviation ( double sigma )

inline

Definition at line 331 of file vpRBProbabilistic3DDriftDetector.h.

References vpException::badValue.

Referenced by loadJsonConfiguration().

◆ setMinDistForNew3DPoints()

void vpRBProbabilistic3DDriftDetector::setMinDistForNew3DPoints ( double distance )

inline

Definition at line 284 of file vpRBProbabilistic3DDriftDetector.h.

References vpException::badValue.

Referenced by loadJsonConfiguration().

◆ setSampleStep()

void vpRBProbabilistic3DDriftDetector::setSampleStep ( unsigned int sampleStep )

inline

Definition at line 365 of file vpRBProbabilistic3DDriftDetector.h.

References vpException::badValue.

Referenced by loadJsonConfiguration().

◆ update()

void vpRBProbabilistic3DDriftDetector::update	(	const vpRBFeatureTrackerInput &	previousFrame,
		const vpRBFeatureTrackerInput &	frame,
		const vpHomogeneousMatrix &	cTo,
		const vpHomogeneousMatrix &	cprevTo )

virtual

Update the algorithm after a new tracking step.

Parameters

previousFrame	The previous frame data: contains the input images at t-1 (linked to cprevTo) and the renders at t-2. May be empty for the first iteration
frame	The current frame data: contains the input images at time t (linked to the newly estimated cTo) and the renders at t-1 (linked to cprevTo)
cTo	the newly estimated object pose in the camera frame
cprevTo	the previously estimated object pose in the camera frame

Implements vpRBDriftDetector.

Definition at line 230 of file vpRBProbabilistic3DDriftDetector.cpp.

References vpRGBa::B, vpRBRenderData::boundingBox, vpRBFeatureTrackerInput::cam, vpRBRenderData::cMo, vpMeterPixelConversion::convertPoint(), vpPixelMeterConversion::convertPoint(), vpRBRenderData::depth, vpRGBa::G, vpRect::getBottom(), vpImage< Type >::getHeight(), vpRect::getLeft(), vpRect::getRight(), vpRect::getTop(), vpImage< Type >::getWidth(), vpRBProbabilistic3DDriftDetector::vpStored3DSurfaceColorPoint::ColorStatistics::init(), vpHomogeneousMatrix::inverse(), vpRBFeatureTrackerInput::IRGB, vpRGBa::R, vpRBFeatureTrackerInput::renders, score(), vpMath::sqr(), vpRBProbabilistic3DDriftDetector::vpStored3DSurfaceColorPoint::stats, and vpRBProbabilistic3DDriftDetector::vpStored3DSurfaceColorPoint::X.

Classes

Public Member Functions

Detailed Description

Tutorials & Examples

Constructor & Destructor Documentation

◆ vpRBProbabilistic3DDriftDetector()

Member Function Documentation

◆ display()

◆ getColorUpdateRate()

◆ getDepthStandardDeviation()

◆ getFilteringMax3DError()

◆ getInitialColorStandardDeviation()

◆ getMinDistForNew3DPoints()

◆ getSampleStep()

◆ getScore()

◆ hasDiverged()

◆ loadJsonConfiguration()

◆ loadRepresentation()

◆ reset()

◆ saveRepresentation()

◆ score()

◆ setColorUpdateRate()

◆ setDepthStandardDeviation()

◆ setFilteringMax3DError()

◆ setInitialColorStandardDeviation()

◆ setMinDistForNew3DPoints()

◆ setSampleStep()

◆ update()