How Computer REALLY See: Unraveling the Code

Image created by Author using DALL-E 3

In the culmination of our four-part series, we delve into the heart of our digital odyssey: the Python script that brings our theoretical discussions into the tangible realm of interactive entertainment. Before we unravel the mysteries of this code, let’s do a recap, shall we? Till now, we’ve set a clear objective: making learning fun. We embarked on this journey by collecting a dataset, and after an enlightening exploration of Convolutional Neural Networks (CNNs), we created a model tailored for our purpose. This script is not merely lines of code; it’s a symphony of logic, an interplay of computational processes and human interaction, serving as a bridge between the abstract and the concrete.

You can find the full code over here.

[embed]Video created by author on Youtube.

The script begins by importing essential libraries: “*cv2” for computer vision, enabling our program to understand and manipulate images; “numpy ”for numerical operations on large, multi-dimensional arrays and matrices; and “keras.models*”, a part of the Keras deep learning library for loading our pre-trained model. These libraries are the building blocks of our game, allowing us to interpret and respond to the user’s physical movements.

In a blend of randomness and machine learning, “choice ”from the “random ” library and “load_model ”from keras.models come together to form the unpredictable heart of our game (we don’t want the model to be biased). The script uses “cvzone ”for hand detection and classification, turning a standard webcam into a portal for interaction, where human gestures become meaningful input.

The narrative of the script unfolds in a defined structure, set out by arrays and variables like “labels”(since we encoded the labels as rock, papers, scissors and restart earlier, the computer will produce a prediction based on the numbers, that are [0,1,2,3]. Hence the labels would decode those numbers back to tell us the sign in word format), mapping human gestures to game moves; “padding ”and “image_size”, defining the visual space (image_size for SqueezeNet model, padding see earlier article); and “classifier”, an instance of machine learning model ready to interpret these gestures.

Source

At this stage, you might wonder why there’s a “restart ”option in the game. Here’s a simple explanation: Our game keeps going on, and it keeps track of wins, losses, and ties for each round. If a student wants to stop playing and see their score, all they need to do is show a thumbs-up to the camera. This thumbs-up is like hitting a ‘‘restart’’ button. It shows all the game scores and then clears them for the next player to start fresh.

As we delve further, the script utilizes “cv2.VideoCapture” to capture live video, creating our digital game space. We employ “HandDetector ”and “Classifier” from cvzone to recognize the player’s hand movements, interpreting them as game moves like rock, paper, scissors, or restart.

Here are a few key variables:

1. seconds : Time allowed for the player to choose a move.
2. stat_screen : How long the statistics screen is shown.
3. winner_screen : Duration the winner’s screen is displayed after each round.
4. stats : Keeps track of wins.

We also introduce two critical functions:

1. draw_box : Draws the player’s hand box on the screen.
2. calculate_winner : Determines the round’s winner.

Image created by Author using DALL-E 3

The core gameplay loop continuously cycles through the webcam frames, flipping them for a more intuitive interaction and integrating a user interface directly into the live video. The script carefully evaluates each frame to identify and isolate the player’s hand from the background. Following the approach detailed in Article 2, we apply the Custom Hands script after obtaining hand landmarks from mediapipe, resize the image for consistency, and add white space to balance the composition.

To ensure the player is actively participating, the script uses the first function to draw a designated box on the screen. The game only progresses when the player places their hand within this box, preventing accidental inputs from being considered as active gameplay.

Once the hand is correctly positioned, the script gives the player a set amount of time (defined by the ‘seconds’ variable) to finalize their gesture. This allows for last-minute changes, ensuring the game captures and responds to the player’s final gesture just before the timer runs out.

After capturing the last frame, the script passes it to our trained model to predict the player’s move. Simultaneously, the computer randomly selects between rock, paper, and scissors. With both the player’s move and the computer’s move determined, we employ the calculate_winner function to decide the round’s outcome.

The winner is then displayed for a set duration (‘winner_screen’ seconds) before the game resets for another round. If the player shows a thumbs-up, the game will display the current statistics and restart.

This core logic embodies not just a game, but a condensed version of human experiences and decisions, rendered into code. It invites players to delve into programming, computer vision, and the spontaneous nature of human choice, offering a foundation for further exploration and creativity.

Image created by Author using DALL-E 3

So, that’s the end of our beautiful journey. We’ve successfully created a fun and educational learning experience for students. You should be proud of yourself for embracing new knowledge. I wrap up this series wishing you good luck in your future endeavors. I hope you enjoyed learning with me. Stay curious, keep exploring, and don’t hesitate to experiment with what you’ve learned. Your journey in tech and learning is just beginning. Stay tuned for my future articles, where we’ll uncover more fascinating worlds of knowledge together.