Documentation

The current implementation fits a deformable 3D model to pixels using an improved version of the Deformable Model Fitting by Regularized Landmark Mean-Shift algorithm. This algorithm returns 66 2D image landmarks, their corresponding position in 3D as well as the pose of the head for each successful detection. The implementation also includes a failure detection component in order to improve robustness.

The expression transfer component is capable of transferring the shape and appearance of an individual to an avatar. The algorithm performs this transfer using a semantic mapping in order to preserve the geometric identity of the avatar. This strategy resulted in more visually appealing animations when compared with animations produced using a direct geometric transfer i.e. the avatar's shape is identical to that of the user.

The only information required to initialise the semantic mapping is a sample of the user displaying a netural expression. This sample can be easily obtained at run-time using the non-rigid face tracker component.

This section outlines the non-rigid face registration program face-fit. This program can perform fitting on a single image, a sequence of images or video.

An important detail of the fitting algorithm is that it relies on a frontal face detector to initialize the non-rigid fitting component. Once initialized, it falls back to the frontal face detector only when the fitting algorithm has failed to accurately perform non-rigid registration.

The following command executes the fitting algorithm on a single image and visualises the results.

face-fit <image>

The resulting landmarks can be saved to file by specifying an output pathname as a command line argument.

face-fit <image> <output-landmarks>

The next command performs tracking over a sequence of images.

face-fit --lists <image-lists> [landmarks-list]

The argument <image-lists> is a file containing a list of image pathnames with each pathname separated by a new line. The argument [landmarks-list] is a list of pathnames to save the landmarks to. If landmarks-list is not specified, then the fitting results are displayed on the screen. Users should be aware that only successful registrations are saved to file.

The --video switch enables face-fit to perform fitting on a video.

face-fit --video <video> [landmarks-template-string]

The argument <video> is the pathname to the video. If [landmarks-template-string] is not specified, then the tracking is displayed to the screen. If [landmarks-template-string] is specified, then it is used as the template (or format) argument to sprintf(3) in order to synthesise a landmark pathname based on the frame number.

For example, the following command will write a landmarks file at frames/frame000001.pts for frame one of video, frames/frame000002.pts for frame 2, and so on for each frame in video.

face-fit --video video frames/frame%06.pts

Like the other modes, landmarks are only written if tracking was successful.

More functionality of the face-fit algorithm can be obtained from its usage text.

$ face-fit --help

This section illustrates the expression-transfer program which is a front end to the SDK's expression transfer API.

The command line arguments accepted by the expression-transfer program are

expression-transfer [options] \
                    <calibration-image> <calibration-landmarks> \
                    <image-argument> <landmarks-argument> <output-argument>

The arguments <calibration-image> and <calibration-landmarks> represent the data needed to calibrate the semantic mapping between the individual and the chosen avatar. The calibration data must be an exemplar of the individual displaying a neutral expression.

The arguments <image-argument> and <landmarks-argument> represent the expression to be transferred to the avatar and the argument <output-argument> specifies where to save the rendered avatar. How this information is interpreted changes depending on the mode of the expression-transfer program.

The default mode is to synthesize a single image of an avatar and save it to file.

expression-transfer calibration.png calibration.pts \
                    input.png input.pts output.png

If you specify the switch --lists, the arguments <image-argument>, <landmarks-argument> and <output-argument> now correspond to lists of pathnames.

The avatar used in the above examples is the default avatar delivered with the SDK. Other avatars can be selected using the options --index and --model.

expression-transfer [--model <model-pathname>] [--index <index>] ...

Viewing or choosing an avatar for the expression-transfer program can be performed using the program display-avatar.

display-avatar [model-pathname]

If [model-pathname] is not specified, then the default model pathname is used.

You can change avatars by pressing the a and d characters keyboard. The left and right arrow keys can be used as well. When the avatar changes, a number will be printed to the console. This number can be used as the argument to the --index option for the expression-transfer program.

The program create-avatar-model is used to create a new model file that can be used by the Face Analysis SDK.

create-avatar-model [options] <output-model-pathname> \
                    <avatar-image> <avatar-annotation> [eyes-annotation]

The argument <output-model-pathname> is the location of the new avatar model containing the avatar defined by the arguments <avatar-image>, <avatar-annotation> and [eyes annotation].

The following diagram displays the landmarks that should be stored in the <avatar-annotation> pathname using the points file format.

The [eyes annotation] argument provides the ability to draw the eyes of the avatar with the same gaze as the user. This pathname should contain the following annotations using the points file format.

It is safe to not specify the [eyes-annotation] argument for cases where the avatars are wearing glasses.

The create-avatar-model program provides a --list switch to allow the creation of an model file containing more than one avatar. In --list mode, the arguments <avatar-image>, <avatar-annotation> and [eyes-annotation] are files containing lists of pathnames.

A GUI application, called demo-application is included with the software which demonstrates the tracker and expression transfer components simultaneously.

The camera used can be changed using the --camera-index command line option

demo-application --camera-index <index>

where the argument <index> selects the camera to use. The order of the cameras is determined by OpenCV and the first camera has an index of 0.

On OSX, a drag and drop installer for the application can be built by issuing the following in the <build> directory.

cpack -G DragNDrop -DWITH_GUI=yes -DCPACK_BUNDLE_NAME=DemoApplication

The above command can only be executed after the SDK has been built.

The non-rigid registration algorithm can be used in third party C++ applications by including the FACETRACKER namespace.

#include <tracker/FaceTracker.hpp>

The tracking interface is provided by the abstract base class FaceTracker. Instantiating a new instance of this class is performed by the LoadFaceTracker function

FaceTracker *LoadFaceTracker();

This function returns NULL if the face tracker cannot be loaded.

Alongside the FaceTracker instance are its parameters. Face tracker parameters are represented by the opaque data type FaceTrackerParams of which a new instance can be obtained with the function LoadFaceTrackerParams.

FaceTrackerParams *LoadFaceTrackerParams();

This function returns NULL if the tracker parameters cannot be loaded.

The methods implemented by the FaceTracker class are as follows

typedef std::vector<cv::Point_<double> > PointVector;

class FaceTracker
{
public:
  virtual int NewFrame(const cv::Mat_<uint8_t> &image, FaceTrackerParams *params) = 0;
  virtual PointVector getShape() const = 0;
  virtual void Reset() = 0;
};

The method NewFrame performs tracking on a grayscale image using the tracking parameters params. Its return value is an integer value between 0 and 10 (inclusive) or one of the constants

FaceTracker::TRACKER_FAILED

The tracker has failed to accurately perform registration.

FaceTracker::TRACKER_FACE_OUT_OF_FRAME

The tracker has failed as the face is partially outside the image frame.

A value between 0 and 10 represents the health of the tracker. A value of 10 indicates that the quality of the tracking is very good, and a value of 0 indicates that the tracking quality is poor.

When the tracking quality is poor or the tracker has failed, an application must reset the tracker using the Reset method.

The expression transfer algorithm can be used in C++ applications by including the AVATAR namespace.

#include "avatar/Avatar.hpp"

The interface used to perform expression transfer is provided by the class Avatar.

typedef cv::Mat_<cv::Vec<uint8_t,3> > BGRImage;
typedef std::vector<cv::Point_<double> > PointVector;

class Avatar
{
public:
  // Expression Transfer
  virtual void Initialise(const BGRImage &im, const PointVector &shape, void* params=NULL)=0;
  virtual int Animate(BGRImage &draw, const BGRImage &image, const PointVector &shape, void* params=NULL)=0;

  // Selecting the avatar.
  virtual int numberOfAvatars() = 0;
  virtual void setAvatar(int index) = 0;
};

An instance of the Avatar class can be created with the function LoadAvatar().

Avatar *LoadAvatar();
Avatar *LoadAvatar(const char *avatar_collection_pathname);

The method Animate renders an avatar displaying the expression found in image with the corresponding shape. The resulting rendering is stored in the matrix draw.

Prior to calling Animate, an avatar must have been chosen using setAvatar. With the avatar chosen, the Avatar instance must be initialised using Initialise. Initialisation requires an image and shape corresponding to the netural expression of the individual that is being used to animate the avatar. This procedure must be followed every time the avatar is changed.

The utils/points.hpp header contains two functions for reading and writing point files.

typedef const std::vector<cv::Point_<double> > PointVector;

PointVector load_points(const char *pathname);
void save_points(const char *pathname, const PointVector &points);

A std::runtime_exception is thrown if either function is unable to perform its task.

The compiler options required to use the code outlined in this section are the following

• <source>/src added to the include path.
• <build>/src added to the include path.
• Linking against the libraries utilities, clmTracker and avatarAnim in the <build>/lib directory.
• Compiler and linker requirements for the OpenCV modules core, highgui, imgproc and objdetect.

The programs used in this library make extensive use of point files or landmark files. These files commonly have the extension .pts. The format of this file is intended to be very simple.

This is an example of a points file:

n_points: 2
{
1 2
5.5 2.2
}

The first line of a points file contains the number of points N in the file. The region between the braces { and } contains the N points with each point starting on a new line. The text for the point is simply two floating point numbers.

The command line programs in this SDK follow this basic argument structure

command [options] <configuration-1> .. <configuration-K> \
                  <input-pathname-1> .. <input-pathname-M> \
                  [output-pathname-1] .. [output-pathname-N]

The reason for this is that this structure makes it very convenient to operate with lists of data when coupled with the map-list program.

Lets assume that the file a.list contains a list of numbers

1
2
3
4

and the file b.list contains a list of strings

do
not
pass
go

then the command map-list 2 a.list b.list echo will produce the following output

1 do
2 not
3 pass
4 go

The usage string for map-list is

map-list [options] <N> <list-1> .. <list-N> <command> [command arguments ... ]

The argument <N> specifies how many lists are specified on the command line. The <N> lists must immediately follow. The argument <command> represents the command to be executed, and [command arguments] will appear on the command line before the items obtained from the lists.

Another example using the above data is

$ map-list 2 a.list b.list printf 'file-%02d-%s.txt\n'
file-01-do.txt
file-02-not.txt
file-03-pass.txt
file-04-go.txt

If one of the list arguments is the text -, then the list is read from the standard input rather than being read from file. It is safe to use - multiple times. This indicates that the list read from standard input is used more than once.

Complementing the map-list program is change-pathnames. Its purpose is to take a list of pathnames and create a new list with the pathnames changed to have either a different directory, extension or both.

For example, the file input.list contains the following list of pathnames

frame-01.png
frame-02.png
frame-03.png
frame-04.png

Executing the following change-pathnames command on input.list

change-pathnames input.list output.list --directory points/ --type pts

produces the file output.list

points/frame-01.pts
points/frame-02.pts
points/frame-03.pts
points/frame-04.pts

If the input list for change-pathnames is the character -, then the list of pathnames is read from the standard input. If the output list is the character -, then the transformed list is written to standard output.

A number of utilities are included in the SDK that perform common operations on video. These utilities are

• remove-rotation-metadata
• rotate-movie
• extract-frames-from-movie
• create-movie-from-frames

The program remove-rotation-metadata is required to overcome an issue with OpenCV where its cv::VideoCapture class does not honour the rotation parameter embedded in some video containers. This problem typically occurs when working with video obtained using a portable device. The program remove-rotation-metadata creates a new movie without the rotation parameter.

It may be required to rotate the video once the rotation metadata is removed. This task can be performed using the command rotate-movie.

The program extract-frames-from-movie converts a movie to a sequence of images and create-movie-from-frames uses a sequence of images to create a movie.

All of the above programs simply invoke FFMPEG with the required options and arguments.