none
Microsoft Face SDK for Window Phone

    Question

  • How to achieve a pinch(on touch of the portion of the face, the shape of the portion should change) effects on the images in windows phone 7 by using Microsoft face SDK.
    • Edited by hcshivkumar Saturday, June 09, 2012 10:24 AM
    Saturday, June 09, 2012 10:16 AM

Answers

  • Current SDK does not provide warp functions like concave and convex. You can implement your own algorithms.

    Following are my test cases for mesh warp and MLSD. The SDK implements two kinds of MLSD algorithms for single frame and multi-frame scenarios. Note that, MLSD is slow when processing large images. MultiFrameMLSD improves the performance by combining MLSD and mesh warp. You may compare the two and choose the proper one.

            private static void TestMeshWarp(Image<byte> src)
            {
                // Divide the image into 4x4 mesh
                int MeshWidth = 4;
                int MeshHeight = 4;
    
                PointF[][] S = JaggedArray.Create<PointF>(MeshHeight, MeshWidth);
                PointF[][] D = JaggedArray.Create<PointF>(MeshHeight, MeshWidth);
    
                for (int i = 0; i < MeshHeight; i++)
                    for (int j = 0; j < MeshWidth; j++)
                    {
                        S[i][j] = new PointF(j * src.Width / (MeshWidth - 1), i * src.Height / (MeshHeight - 1));
                        D[i][j] = S[i][j];
                    }
    
                // Randomly perturb the 4 mesh nodes in the middle.
                Random rng = new Random();
                D[1][1] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
                D[1][2] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
                D[2][1] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
                D[2][2] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
    
                var dst = new Image<byte>(src.Width, src.Height, src.Channels);
    
                // Warp the image and save result to destination image
                MeshWarp.Warp(src, dst, S, D);
            }
    
            private static void TestMLSDWarp(Image<byte> src)
            {
                const int MeshRows = 4;
                const int MeshColumns = 4;
    
                // p: The original positions of control points.
                PointF[] p = new PointF[MeshRows * MeshColumns];
                // q: The destination positions of control points.
                PointF[] q = new PointF[MeshRows * MeshColumns];
    
                for (int i = 0; i < MeshColumns; i++)
                    for (int j = 0; j < MeshRows; j++)
                    {
                        p[i * MeshRows + j] = new PointF(j * src.Width / (MeshRows - 1), i * src.Height / (MeshColumns - 1));
                        q[i * MeshRows + j] = p[i * MeshRows + j];
                    }
    
                // Randomly perturb the destination positions by resonable offsets
                Random rng = new Random();
                q[MeshRows + 1] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
                q[MeshRows + 2] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
                q[MeshRows * 2 + 1] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
                q[MeshRows * 2 + 2] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
    
                var dst = new Image<byte>(src.Width, src.Height, src.Channels);
    
                // 1. Use invert transform to interpolate pixel values in warpped image
                MLSD mlsd = new MLSD(2.0f, TransformType.Rigid);
                mlsd.InvTransformResampled(p, q, src, dst, src.Region);
    
                // 2. Sample codes for Multi-frame MLSD
                // In multi-frame MLSD, the P coefficients are computed only once and re-used
                // in future transforms where only Q coefficients are changed.
                var mlsdex = new MultiFrameMLSD(TransformType.Rigid, 2.0f);
                mlsdex.InvTransformResampled(p, q, src, dst, src.Region);
    
                // Reuse the pre-calculated P coefficients
                var mulmlsd = new MultiFrameMLSD(TransformType.Rigid, 2.0f, mlsdex.PCoefficient);
    
                // Transform the image for multiple times
                int WarpCount = 30;
                for (int i = 0; i < WarpCount; i++)
                {
                    q[MeshRows + 1] += new PointF(
                        (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                        (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                        );
                    q[MeshRows + 2] += new PointF(
                        (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                        (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                        );
                    q[MeshRows * 2 + 1] += new PointF(
                        (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                        (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                        );
                    q[MeshRows * 2 + 2] += new PointF(
                        (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                        (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                        );
    
                    mulmlsd.InvTransformResampled(q, src, dst, src.Region);
                }
            }
    Thanks.

    Tuesday, June 12, 2012 2:02 AM
    Owner

All replies

  • I am not sure what kind of the "pinch effect" you metioned is. Centralize and zoom the face region to occupy the screen (or display region) or just warp the face shape (as the Face Touch app does)?

    The zoom and centralization effect can be achieved by applying transform (translation, scaling, etc) to Image control. For the local warping effect, you can try some popular warping algorithms like mesh warp, MLSD. The Microsoft.FaceSdk.Image.Warp namespace implements the two algorithms which you may have a try.

    Thansk.

    Monday, June 11, 2012 3:36 AM
    Owner
  • Hi,

    I am also facing the same kind of issue as above with one of my Apps developed using windows phone7. I am new with Windows phone development and also new to Microsoft Research Face SDK. I am trying to implement the effects like bulging of the face (for example concave and convex effects) to integrate when camera is on and also with the saved image. Looking at the reply from you i tried with the options suggested, but still could not succeed i.e., i tried using the MeshWarp static method but unable to figure out the usage of the same. Can you give me some more explanation of how to use the mesh warp, MLSD and also some example would be of great help?

    Thanks


    Hemanth J

    Monday, June 11, 2012 9:26 AM
  • Current SDK does not provide warp functions like concave and convex. You can implement your own algorithms.

    Following are my test cases for mesh warp and MLSD. The SDK implements two kinds of MLSD algorithms for single frame and multi-frame scenarios. Note that, MLSD is slow when processing large images. MultiFrameMLSD improves the performance by combining MLSD and mesh warp. You may compare the two and choose the proper one.

            private static void TestMeshWarp(Image<byte> src)
            {
                // Divide the image into 4x4 mesh
                int MeshWidth = 4;
                int MeshHeight = 4;
    
                PointF[][] S = JaggedArray.Create<PointF>(MeshHeight, MeshWidth);
                PointF[][] D = JaggedArray.Create<PointF>(MeshHeight, MeshWidth);
    
                for (int i = 0; i < MeshHeight; i++)
                    for (int j = 0; j < MeshWidth; j++)
                    {
                        S[i][j] = new PointF(j * src.Width / (MeshWidth - 1), i * src.Height / (MeshHeight - 1));
                        D[i][j] = S[i][j];
                    }
    
                // Randomly perturb the 4 mesh nodes in the middle.
                Random rng = new Random();
                D[1][1] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
                D[1][2] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
                D[2][1] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
                D[2][2] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
    
                var dst = new Image<byte>(src.Width, src.Height, src.Channels);
    
                // Warp the image and save result to destination image
                MeshWarp.Warp(src, dst, S, D);
            }
    
            private static void TestMLSDWarp(Image<byte> src)
            {
                const int MeshRows = 4;
                const int MeshColumns = 4;
    
                // p: The original positions of control points.
                PointF[] p = new PointF[MeshRows * MeshColumns];
                // q: The destination positions of control points.
                PointF[] q = new PointF[MeshRows * MeshColumns];
    
                for (int i = 0; i < MeshColumns; i++)
                    for (int j = 0; j < MeshRows; j++)
                    {
                        p[i * MeshRows + j] = new PointF(j * src.Width / (MeshRows - 1), i * src.Height / (MeshColumns - 1));
                        q[i * MeshRows + j] = p[i * MeshRows + j];
                    }
    
                // Randomly perturb the destination positions by resonable offsets
                Random rng = new Random();
                q[MeshRows + 1] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
                q[MeshRows + 2] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
                q[MeshRows * 2 + 1] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
                q[MeshRows * 2 + 2] += new PointF(
                    (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                    (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                    );
    
                var dst = new Image<byte>(src.Width, src.Height, src.Channels);
    
                // 1. Use invert transform to interpolate pixel values in warpped image
                MLSD mlsd = new MLSD(2.0f, TransformType.Rigid);
                mlsd.InvTransformResampled(p, q, src, dst, src.Region);
    
                // 2. Sample codes for Multi-frame MLSD
                // In multi-frame MLSD, the P coefficients are computed only once and re-used
                // in future transforms where only Q coefficients are changed.
                var mlsdex = new MultiFrameMLSD(TransformType.Rigid, 2.0f);
                mlsdex.InvTransformResampled(p, q, src, dst, src.Region);
    
                // Reuse the pre-calculated P coefficients
                var mulmlsd = new MultiFrameMLSD(TransformType.Rigid, 2.0f, mlsdex.PCoefficient);
    
                // Transform the image for multiple times
                int WarpCount = 30;
                for (int i = 0; i < WarpCount; i++)
                {
                    q[MeshRows + 1] += new PointF(
                        (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                        (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                        );
                    q[MeshRows + 2] += new PointF(
                        (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                        (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                        );
                    q[MeshRows * 2 + 1] += new PointF(
                        (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                        (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                        );
                    q[MeshRows * 2 + 2] += new PointF(
                        (float)(src.Width * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10)),
                        (float)(src.Height * (rng.NextDouble() - 0.5) * 2 / rng.Next(6, 10))
                        );
    
                    mulmlsd.InvTransformResampled(q, src, dst, src.Region);
                }
            }
    Thanks.

    Tuesday, June 12, 2012 2:02 AM
    Owner
  • Hi ningx,

    Thanks for the reply.

    In the above given algorithm (TestMeshWrap) , It is randomly considering some portion of the image and wrap is applying for that. can we able to restrict the wrap, only to the portion where we tapped .

    Is there any way to achieve the twist (Tapped portion appear to be Circular)  effect on the Image.

    Wednesday, June 13, 2012 7:06 AM
  • Yes, you can. For example, to save computational cost, you can first crop out the region (the tapped area) you want to warp, then apply either mesh warp or MLSD, and copy the warped patch back to original image.

    In theory, twist effect can be achieved by mesh warp if the mesh is dense enough. However this is not very efficient. A better approach might be to fit/design a warp function F (e.g. p(u, v) = F(x, y) ) which transforms a point from (x, y) to (u, v). With this function, you know where the original pixel should be rendered on warpped image. In your case, F can be some kind of a circular curve.

    When you try the second approach, it is recommended to use invert transform ( F' = inv(F) ) to interpolate pixel values in target image by resampling pixels in source image. The details can be found on image processing books.


    Wednesday, June 13, 2012 8:37 AM
    Owner
  • Hi ,

    How to achieve the dense mesh.

    Friday, June 22, 2012 12:54 PM
  • My approach as follows --

    • identify the warp region (e.g. face area);
    • create a mesh by sampling the region (sample step in pixels depending on your scenario);
    • distort the mesh;
    • warp the region based on the two meshes;
    Monday, June 25, 2012 1:50 AM
    Owner