Gaze Fusion : Gaze Tracking and Facial Analysis for Marketing Analytics

Computer Vision #Machine Learning #Ads analysis #Marketing Analysis #Mediapipe

Before start, I would like to clarify a few points. I started this project purely for fun and educational purpose, dedicating my free time to its development. If you come across any errors or have suggestions for improvement, please feel free to comment. (Especially 3D space 🤣)

INTRODUCTION

The goal of this project is to analyze the user’s gaze on the screen and their emotional expressions. The intended purpose is to utilize these info for various applications — Ads, music videos or any other video content, whether for marketing or scientific experimentation.

The process begins with creating a Gaze Location Estimation, achieved by building a model using the user’s webcam data (user looking at predefined points on the screen). Subsequently, raw data is collected, consisting of both the user’s webcam feed and screen recordings(Ads/Video). This collected data is then used with the earlier model to predict the user’s gaze location. As for emotional expressions, I employ the DeepFace model directly.

Gaze Location Estimation

At first, I started with a mathematical method (for lack of a better term) to estimate location coordinates, which involved deriving location coordinates based on the distance between the eye pupil and the center of the eyeball.

Flipped right eye

First image ‘Assumption’ demonstrates mathematical method. Ex. small green circle indicates that when user’s eye moves towards the second quadrant (top left), it signifies that the user is looking at the top-left section of the screen.

However…. people do not stay still all the time. As the picture illustrates, when a user tilts their head upward (pitch up), this method becomes impractical due to the movement of the eye’s center. Also the white space above the eye pupil extends while the lower portion shortens. (depends on how the camera angle is set.)

To address this issue, I developed a model that compensates for head rotation. This model calculates the degree of angle and pixel distance changes in the user’s head position then corrects face landmark location. The correction process follows a sequence of adjustments — roll, pitch and yaw. (will explain this later)

For some reason, this method is not performing as expected(whoops 😂 ). I suspect that the issue may arise from extreme angles, particularly when the user yaws excessively. In such cases, Mediapipe may fail to detect correct eye landmarks, resulting in a reduced eye size. (Basically, it can’t 100% compensate for head rotation.)

After encountering the earlier issue, I decided to pursue a simpler solution Machine learning.

Collecting Training Data

Creating reference image

The first step involves creating a reference image, which serves as the image that I want user to focus on while I capture their image- used for feature extraction and training the model, which can be thought of as a calibration process. The reference image typically consists of a white background with 5 dots positioned on the screen. While it can be created using any application, I personally use Python for this.

import numpy as np
import cv2

# Define the screen resolution
screen_width = 1920
screen_height = 1080

# Create a blank white image
reference_image = np.ones((screen_height, screen_width, 3), dtype=np.uint8) * 255

#Draw markers for the screen corners (you can adjust the marker positions)
cv2.circle(reference_image, (200, 150), 10, (0, 0, 255), -1)  # Top-left corner
cv2.circle(reference_image, (screen_width - 200, 150), 10, (0, 0, 255), -1)  # Top-right corner
cv2.circle(reference_image, (200, screen_height - 150), 10, (0, 0, 255), -1)  # Bottom-left corner
cv2.circle(reference_image, (screen_width - 200, screen_height - 150), 10, (0, 0, 255), -1)  # Bottom-right corner

#Draw Center marker
center_x = screen_width // 2
center_y = screen_height // 2
cv2.circle(reference_image, (center_x, center_y), 10, (0, 0, 255), -1)  # Red circle at the center

# Save the reference image
cv2.imwrite('reference_image.png', reference_image)

I chose to position the reference point at x 200 and y 150 (not 150,150), as I believe that is too far off-screen, and people generally don’t tend to look in that area. Additionally, I decided to use only 5 dots. My reasoning is to avoid overtraining the model.

reference_image

I captured 20 (user) images for each dot with different poses, which typically took only 1–2 minutes per user. I want to simplify the process and avoid overwhelming them. While the number of dots used can vary depending on your judgment, my suggestion is to add 4 more dots to create a grid of 9 blocks, adhering to the photography theory known as the ‘rule of thirds’.

Feature Extraction

What to extract? As mentioned earlier, the data to be extracted includes normalized target eye landmark positions, roll, pitch, and yaw angles, as well as the size of the head. I will explain each of these aspects one by one.

Here is full-size face landmark image URL (Mediapipe). : https://storage.googleapis.com/mediapipe-assets/documentation/mediapipe_face_landmark_fullsize.png

eye landmarks

This image displays the eye landmarks that are used to define an area for the eyes.

Eye pupil - (landmark) no.473

Left - no.398 / Right - no.466, I haven’t used the rearmost eye landmarks (such as 463 and 263) due to extreme angles mentioned earlier (Mediapipe may fail to detect the correct positions).

Top - no.442 / Bottom - no.348, instead I opted for a flattened area of skin to ensure Mediapipe could collect consistent landmarks correctly.

#Ex
x466 = int(landmarks[466].x * image.shape[1])
y466 = int(landmarks[466].y * image.shape[0])

The data to collect includes the X and Y coordinates for each landmark. However, I do not use these values directly; instead, I normalize (compensate) from head rotation first.

#Normailize function
def normalize_rotation(point_x, point_y, center_x, center_y, angle_degrees):
    angle_radians = np.radians(angle_degrees)
    new_x = int(center_x + (point_x - center_x) * np.cos(angle_radians) - (point_y - center_y) * np.sin(angle_radians))
    new_y = int(center_y + (point_x - center_x) * np.sin(angle_radians) + (point_y - center_y) * np.cos(angle_radians))
    return new_x, new_y

This function use X/Y coordinates and head rotation angle (degree) as input. Return new X/Y.

Ref : https://www.researchgate.net/figure/Orientation-of-the-head-in-terms-of-pitch-roll-and-yaw-movements-describing-the-three_fig1_279291928

As the image illustrates head rotation . Yaw poses the most significant challenge because my method is limited to working with only one (right) eye. To compensate for head rotation, I calculate each rotation in the following order: Roll > Pitch > Yaw.

Head rotation — Roll 15 degree change

I utilize landmark no.151 and no.337 to calculate Roll change. These landmarks are chosen due to the prominent and flattened nature of the forehead area, ensuring consistent detection by Mediapipe.

This is a formula that I used.

Rolldeg = (math.degrees(math.atan2(y337 — y151, x337 — x151)))*-1

-1 is to correct direction.

Head rotation — Pitch change

For Pitch change, i use no.50 and no.4, same reason as Roll. It’s most linear and easy for model to detect.

You can change no.50 to no.280 if you wanna use the right cheek or write a condition to select which side to use.

Pitchdeg = (math.degrees(math.atan2(nor_y4 — nor_y50, nor_x4 — nor_x50)))*-1

*Don’t forget to normalized pitch landmarks before calculate Pitchdeg. I did for both X/Y.

Head rotation — Yaw change

Yaw change, use no.4 (same to pitch) and no.168

Yawdeg = (math.degrees(math.atan2(nor_y4 — nor_y168, nor_x4 — nor_x168)))-90

• Before calculating Yawdeg, I use Pitchdeg to normalize **only Y as it primarily reflects changes in the Y-axis.

#Ex
_,nor_y473 = normalize_rotation(x473,y473,nor_x50,nor_y50,Pitchdeg)

• Same for Yawdeg but this time only X-axis.

#Ex
nor_x473,_ = normalize_rotation(x473,y473,nor_x168,nor_y168,Yawdeg)

The next important feature is the distance between user and camera. When a user changes their position or gestures, this distance can also change. Normally, we could calculate the real distance by …

Ref : https://www.baeldung.com/cs/cv-compute-distance-from-object-video

As you can see, it involves a lot of variables. We’re not going to do engineer complex stuff like IR, remote or sonar. Instead, I came up with a simpler idea, considering that most of the variables are static. The only thing that changes over time is the distance so I use only the head size to indicate changes in distance.

head size reference

I use no.151 and no.9 to calculate a reference line representing the head size. This allows us to estimate changes in distance without needing to measure actual head size or use entire face. Here is formula I used.

Sizedist = int(math.sqrt((nor_x151 — nor_x9)2 + (nor_y151 — nor_y9)2))

Also, I stored no.151 as center of the head (coordinate), will use this in the future.

Lastly, the eyelid status is represented as a binary value that indicates whether the user’s eyes are closed or not. While it is not included in the training data, it is used as a filter to identify when a user is closing their eyes.

eyelid status

The condition is straightforward: open is as 1, close as 0. The value returns 0 when the Y-axis no.386 - no.374 ≤ 2

*2 this number may change depend on how you set camera and user (seat) distance.’

Here is an example image of what we are attempting to collect.

*This person image is generated by DALL·E AI.

The yellow line is a 90-degree line that I’ve added as a reference to gauge the degree of head roll.

I used an AI-generated image because I didn’t want to showcase my handsome face. 🤣🤣🤣🤣 Sorry for that!

Example of a training data dataframe.

*Region refers to point of interest that user is looking at within reference_image. This is predicted variable that we are aiming to obtain.

Collecting Live Data

One advantage of my method is that you don’t have to conduct training immediately after collecting training data (user looks at reference_image). Instead, you can store both training and live data then perform modeling and analysis later. This provides a benefit in real events when there are many volunteers (users).

The live data consists of two components:

• user’s webcam feed
• screen recorder

I recorded both at the same time (using cv2 and pyautogui) as images and then proceed to extract features from the webcam live data, following the same process as we did with the training data.

But there are a few extra steps to do.

• FPS — unlike training data, live data needs to be collected at a specific Frames Per Second (FPS) rate. This allows us to minimize missing data, especially during moments when the user blinks (eyelid status= closed). Therefore, each second generates multiple images then do a code to filter out the images during eye closure, selecting the closest to each second (I personally use 30 FPS).
• File naming, for easier matching of user webcam and screen recorder data, as well as for summary and visualization purposes, naming with timestamps would be highly beneficial.

#Create Folder
if not os.path.exists('src'):   
    os.makedirs('src')
if not os.path.exists('wc'):
    os.makedirs('wc')
#File naming
timestamp = datetime.datetime.now().strftime('%Y%m%d%H%M%S%f')
screen_filename = os.path.join('src', f'{timestamp}.jpg')
webcam_filename = os.path.join('wc', f'{timestamp}.jpg')

Modeling

Data Preparation

def normalize_eye_region(df):
    # Normalize Eye region matrix
    df['top_right_x'] = df['eye_right_x'] - df['eye_left_x']
    df['top_right_y'] = 0
    df['bt_left_x'] = 0
    df['bt_left_y'] = df['eye_bt_y'] - df['eye_top_y']
    df['bt_right_x'] = df['top_right_x']
    df['bt_right_y'] = df['bt_left_y']
    df['top_left_x'] = 0
    df['top_left_y'] = 0
    # Head position
    df['nm_head_center_x'] = df['head_center_x'] - mean_center_x
    df['nm_head_center_y'] = df['head_center_y'] - mean_center_y
    # Location normalize
    df['pupil_x'] = df['eye_center_x'] - df['eye_left_x']
    df['pupil_y'] = df['eye_center_y'] - df['eye_top_y']
    df['Width'] = df['top_right_x']
    df['Height'] = df['bt_right_y']
    return df

df_ref = pd.read_csv('ref_data.csv')
df = pd.read_csv('input_data.csv')

mean_center_x = round(df_ref.loc[df_ref['Region'] == 'Center', 'head_center_x'].mean())
mean_center_y = round(df_ref.loc[df_ref['Region'] == 'Center', 'head_center_y'].mean())

df_ref = normalize_eye_region(df_ref)
df = normalize_eye_region(df)
# Filter columns
df_ref = df_ref[['Region', 'roll_deg', 'pitch_deg', 'yaw_deg', 'head_size', 'nm_head_center_x', 'nm_head_center_y',
            'pupil_x', 'pupil_y', 'Width', 'Height']]

The normalize_eye_region function consists of three parts:

1. Normalizing the top-left eye coordinate to start at (0,0), ensuring all data share a similar scale (typical normalization).
2. Transforming head_center_X/Y into changes from the mean head_center (calculated from the training data, where users looked at center of the screen.
3. Add additional variables and normalize eye pupil X/Y coordinates.

Modeling and Evaluation

I chose CatBoost because the variable to predict is ‘Region,’ which is nominal and I want to treat all regions the same.

This is variable that I feed into model ‘Region’, ‘roll_deg’, ‘pitch_deg’, ‘yaw_deg’, ‘head_size’, ‘nm_head_center_x’, ‘nm_head_center_y’, ‘pupil_x’, ‘pupil_y’, ‘Width’, ‘Height’

I won’t be going into the details of modeling and fine-tuning, but I’ll mention something that I consider essential.

• Test 0.2 / Train 0.8
• StratifiedKFold — with shuffle = True, this approach ensures that all classes (region) receive equal treatment during (CV) the training process.
• micro-average F1 score as a evaluation metric. Same reason with StratifiedKFold.

After some fine-tuning and deal with overtraining (from 0.95+), I achieved a score of 0.9314, which is satisfactory for me.

Were able to achieve this high score because training data is quite easy to predict, with users looking at the very top corners of the screen.

Well, let’s just look what might interest…

Feature Importances Bar Chart

This chart illustrates that pupil_x has a strong effect with user behavior. It could imply that user tend to move their eyes sideways more than they yaw (move their entire head) to look at the screen. While Y-axis related variables do not show a significant. This outcome aligns with my expectations as I use a widescreen display, so I would naturally move horizontally (X-axis) more than vertically (Y-axis).

Enhancing Prediction

As mentioned earlier, the training data is relatively easy to predict because users tend to look at the corners, while real (live) case isn’t. Therefore, I came up with an idea to use the predicted probability of each region as a weight to calculate the X/Y coordinates where the user is looking.

Top-vs-Second Probability Difference Boxplot

As the boxplot indicates difference between top and second predicted classes is very close to each other. This observation is why I believe my idea might work.

second_highest_class = [class_labels[np.argsort(-probs)[:2]][1] for probs in y_input_pred_proba]
third_highest_class = [class_labels[np.argsort(-probs)[:3]][2] for probs in y_input_pred_proba]
p1 = [probs[np.argsort(-probs)[:1]] for probs in y_input_pred_proba]
p2 = [probs[np.argsort(-probs)[:2]][1] for probs in y_input_pred_proba]
p3 = [probs[np.argsort(-probs)[:3]][2] for probs in y_input_pred_proba]

df_output['Region2'] = second_highest_class
df_output['Region3'] = third_highest_class
df_output['probR1'] = p1
df_output['probR1'] = df_output['probR1'].apply(lambda x: x[0])
df_output['probR2'] = p2
df_output['probR3'] = p3
df_output['R2_diff'] = df_output['probR1']-df_output['probR2']
df_output['R3_diff'] = df_output['probR1']-df_output['probR3']

#Cal output region
q1_ref_diff = np.percentile(diff_ref, 25)
threshold_value = q1_ref_diff * 0.75
df_output['output_region'] = np.where(df_output['R3_diff'] <= threshold_value, 3,
    np.where(df_output['R2_diff'] <= threshold_value, 2, 1))

The first step is to obtain the top 3 predicted regions for each instance then calculate the differences from R1. Subsequently, determine which condition each instance falls into (as output_region). If either R2_diff or R3_diff falls within a threshold value (calculated from Quarter 1 of overall difference 0.75), output_region *will assume that value.

In my case threshold_value is 0.0095

df_output dataframe

#Convert Result to X/Y Coordinate
screen_width = 1920
screen_height = 1080
src_coor = pd.DataFrame([
    {"Region": "Top Left", "X": 200, "Y": 150},
    {"Region": "Top Right", "X": screen_width - 200, "Y": 150},
    {"Region": "Bottom Left", "X": 200, "Y": screen_height - 150},
    {"Region": "Bottom Right", "X": screen_width - 200, "Y": screen_height - 150},
    {"Region": "Center", "X": 960, "Y": 540}
    ])

#Add new cols + vals
df_output['out_x'] = 0
df_output['out_y'] = 0

#Masking
mask_region1 = df_output['output_region'] == 1
mask_region2 = df_output['output_region'] == 2
mask_region3 = df_output['output_region'] == 3

#output_region == 1
for index, row in df_output[mask_region1].iterrows():
    region = row['Region']
    x, y = src_coor.loc[src_coor['Region'] == region, ['X', 'Y']].values[0]
    df_output.at[index, 'out_x'] = x
    df_output.at[index, 'out_y'] = y

#output_region == 2
def two_region_weightXY(region1, region2, prob1, prob2, src_coor):
    x1, y1 = src_coor.loc[src_coor['Region'] == region1, ['X', 'Y']].values[0]
    x2, y2 = src_coor.loc[src_coor['Region'] == region2, ['X', 'Y']].values[0]
    x_intermediate = (prob1 * x1 + prob2 * x2) / (prob1 + prob2)
    y_intermediate = (prob1 * y1 + prob2 * y2) / (prob1 + prob2)
    return x_intermediate, y_intermediate

for index, row in df_output[mask_region2].iterrows():
    region1 = row['Region']
    region2 = row['Region2']
    prob1 = row['probR1']
    prob2 = row['probR2']
    #Apply fn
    x_intermediate, y_intermediate = two_region_weightXY(region1, region2, prob1, prob2, src_coor)
    df_output.at[index, 'out_x'] = int(x_intermediate)
    df_output.at[index, 'out_y'] = int(y_intermediate)

#output_region == 3
def three_region_weightXY(region1, region2, region3, prob1, prob2, prob3, src_coor):
    x1, y1 = src_coor.loc[src_coor['Region'] == region1, ['X', 'Y']].values[0]
    x2, y2 = src_coor.loc[src_coor['Region'] == region2, ['X', 'Y']].values[0]
    x3, y3 = src_coor.loc[src_coor['Region'] == region3, ['X', 'Y']].values[0]
    x_intermediate = (prob1 * x1 + prob2 * x2 + prob3 * x3) / (prob1 + prob2 + prob3)
    y_intermediate = (prob1 * y1 + prob2 * y2 + prob3 * y3) / (prob1 + prob2 + prob3)
    return x_intermediate, y_intermediate

for index, row in df_output[mask_region3].iterrows():
    region1 = row['Region']
    region2 = row['Region2']
    region3 = row['Region3']
    prob1 = row['probR1']
    prob2 = row['probR2']
    prob3 = row['probR3']
    x_intermediate, y_intermediate = three_region_weightXY(region1, region2, region3, prob1, prob2, prob3, src_coor)
    df_output.at[index, 'out_x'] = int(x_intermediate)
    df_output.at[index, 'out_y'] = int(y_intermediate)

Next, create a mask from output_region followed by the development of a function that corresponds to mask value. Each function takes predicted probability as weight and applies this weight to the Region X/Y values, returning user's gaze location as the result X/Y coordinates.

Ex. If Region(1) is Top left and Region2 Bottom left, have predicted probability R1 (probR1) = 0.5 and R2 (probR2) = 0.498 which under threshold_value then output_region will return 2, this falls under two_region_weightXY function. This function will apply weights from both regions and calculate the X/Y coordinates, which should fall within the Left-Center area.

Once we obtain final result (X/Y coordinates), the subsequent steps are relatively straightforward. Here’s a brief overview of what I did:

• Calculated a 9-grid map to facilitate easier interpretation.
• Split image names based on time (to enable easy matching between webcam images and screen recordings).
• Utilized OpenCV to draw a circle at the user’s gaze location.

final result dataframe

screen output

X/Y coordinates density Heatmap

As the heatmap illustrates, the majority of users’ gazes tend to fall in the middle of the screen. My suggestion is that this model may not be particularly useful for summarizing overall analysis, as most ads or videos aim to capture attention towards the center of the screen. However, it can still provide valuable insights when analyzing specific aspects, such as whether users are focusing on the intended target, like a promotional element.

Emotional Classification

I deployed the DeepFace model directly to user webcam images. I used the model from Serengil’s GitHub repository (please ensure the correct name).

URL : https://github.com/serengil/deepface

In this scenario, I employed only ‘emotion’ model.

df = pd.read_csv('output_data.csv')
folder_webcam = "wc/"
folder_output = "wc_output/"

if not os.path.exists(folder_output):
    os.makedirs(folder_output)

dominant_emotions = []

for index, row in df.iterrows():
    image_name = row['image_name']
    image_path = os.path.join(folder_webcam, image_name)
    image = cv2.imread(image_path)
    sentimental_result = DeepFace.analyze(image_path, actions=['emotion'], enforce_detection=False)
    dominant_emotion = sentimental_result[0]['dominant_emotion']
    dominant_emotions.append(dominant_emotion)
    if image is not None:
        cv2.putText(image, f'Emotion: {dominant_emotion}', (10, 60), cv2.FONT_HERSHEY_PLAIN, 4, (255,0,255), 3)
        output_path = os.path.join(folder_output, image_name)
        cv2.imwrite(output_path, image)
    else:
        print(f"Image {image_name} not found in {folder_webcam}")

df['sentiment'] = dominant_emotions

This code performs two tasks, first it conducts emotion classification, then it writes the results on both the image and dataframe.

Emotion Analysis pie chart

The results didn’t match my expectations; the model often misinterpreted neutral and questioning emotions as fear. To address this, I changed the keyword to ‘Question.’

Before using info model given, I recommend manually reviewing the results via the output webcam image. You can then adjust the classes to align more closely with the content of your video. For example, in my case I knew that my Ads video was comedic, so categorizing an emotion as ‘fear’ would be highly unlikely.

Video I used : https://www.youtube.com/watch?v=TcD4jZLGWjI

I’ve noticed that the model performs well in identifying happy faces (would call it Happy face Indicator 😂). Its performance with other emotions is ok. The issue might be related to my low-resolution webcam. Perhaps I’ll do it again with a higher resolution and better lighting settings.

I believe a useful visualization for this could be a bi-directional bar chart that normalizes certain emotions on a scale of [-1,1], where ‘Happy’ = 1, ‘Neutral’ = 0 and ‘Sad’ = -1. This chart could be valuable for studying how the sentimental tone changes over the course of a video.

We’ve reached the end of the article. Thank you for reading this far. 👏

If you have anything to share, please feel free to comment.

The more I know and learn, the more I realize how much I don’t know!

I will keep learning. Thank you in advance.