Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Could you please explain why there are two sets of landmarks, innerLandmarkIndex and outerLandmarkIndex #53

Open
seanwen86 opened this issue May 23, 2019 · 2 comments

Comments

@seanwen86
Copy link

seanwen86 commented May 23, 2019

Hi, @anhttran
I really want to learn all your work. :). And of course there will be many questions for you. :(

In the file "BaselFaceEstimator.cpp", there is the code snippet as below. I couldn't figure out why there are two sets of landmarks, and why the yaw angle could affect the selection of landmark index?
Any explanation from anyone would be appreciated. Thanks in advance..

// Get 3D landmarks points (68x3), given 3D shape (Vx3) and yaw angle
cv::Mat BaselFaceEstimator::getLM(cv::Mat shape, float yaw){
	cv::Mat lm(bf.lmInd.rows,3,CV_32F);
	for (int i=0;i<lm.rows;i++){
		int ind;
		if (yaw > 0 && yaw < M_PI/9 && i < 8) ind = bf.lmInd.at<int>(i,0)-1;
		else if (yaw < 0 && yaw > -M_PI/9 && i > 8 && i < 17) ind = bf.lmInd.at<int>(i,0)-1;
		else
		ind = bf.lmInd2.at<int>(i)-1;
		for (int j=0;j<3;j++){
			lm.at<float>(i,j) = shape.at<float>(ind,j);
		}
	}
	return lm;
}
@anhttran
Copy link
Owner

anhttran commented May 24, 2019

Hi,

Most of the traditional landmark detectors provide holistic 2D landmarks based on image contour. Hence, the landmark points on face boundary do not correspond to fixed 3D points of the face but vary by the yaw angle.

For example, you can check Dlib landmarks in this image: the boundary points on the left side locates on the front part of the face (inner), while the points on the right side go to under the ear (outer):
https://qph.fs.quoracdn.net/main-qimg-391e7898134bbaae1de14f5d14f3f943

I use 2 sets of 3D landmarks to handle this problem. Depends on the yaw angles, I can use all outer 3D landmarks (frontal), inner + outer landmarks for each side (half-profile), or just use outer points for one side and drop the ones on the other side (profile). It is just an approximation but it is cheap and good enough.

In case you have 3D landmark input, like the output of 3DDFA or recent deep learning based methods, you can just use outer landmarks.

@seanwen86
Copy link
Author

@anhttran This is a great detailed explanation. Thank you so much. For one landmark, its more precise position should be located between the TWO corresponding landmarks from the TWO 3D landmark sets along the YAW angle. Am I right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants