AI shuts my GF mouth

Youtube Channel — RsTaK

So, my GF’s (girlfriend) birthday was near and I, being a mediocre engineer, was confused what to plan for her birthday. She had a constant habit of breathing from her mouth. Even after pointing her about this, she repeats the whole process subconsciously.

Looking her birthday as a perfect opportunity, I, along with my best buddy “AI” decided to end this for the good. The video attached shows my journey from R&D stage to delivering an AI solution to solve this emerging problem of people breathing from their mouth subsequently. I thought it’s a good idea to share my findings in the form of a technical blog too. I’ll try to divide the blog in the following sections:

1. Defining the problem statement
2. Steps required for the solution

• Facial Landmark detection(Available models, pros and cons)
• Locating the Lip region
• Detect if mouth is open or not
• Robust solution and packaging

3. Conclusion

So, hey! There we go :)

1. Defining the Problem statement

https://www.okoa.org/articles/how-to-stop-mouth-breathing

For at least once, we all have breathed from our mouth. One reason can be because of the closed nostrils especially in winters. But sometimes it becomes a wrong habit and we start breathing from our mouth subconsciously even when our nostrils are open. Mouth breathing all the time isn’t good for our health. I would like to attach an article that talks more about mouth breathing, symptoms and complications.

Mouth Breathing

Coming from a computer science background, we spend lots of our time in front of our desktop or laptop and subconsciously lots of my friends along with my girlfriend started this habit of breathing from their mouth. I’m not a doctor or any medical expert who can medically help them in any regard. I’m just a casual AI enthusiast. So, i thought if there could be any app running on our workstation as soon as we start working and alerts us whenever we start breathing from our mouth unknowingly. This way one can easily track their habit and can work upon it in-order to improve it. Since, there’ll be a whole application running in the background analysing our lip region, it should be highly accurate and at the same time should consume less computation power.

And with this motivation, i started this project.

2. Steps required for the solution

In order to get to the solution, we need to address the following sub-problems:

2.1 Facial Landmark detection(Available models, pros and cons)

First step is detecting face and facial key-points in the video frame. Facial landmarks are used to localise and represent important regions of the face like mouth, eyes, eyebrows, nose, jawline etc.

Image by Author

After some google searches, i found some algorithms/libraries in that deals with the problem we are facing here.

• HOG + Linear SVM (DLIB)
• Haar Cascade (OpenCV)
• HRNET (Deep Learning solution)
• Partial Face Landmark Detector — PFLD (Deep Learning solution)
• Mediapipe Face Mesh

Based on some constraints, we’ll go with one of them like:

1. Accuracy: Our solution should be accurate and prone to false positive cases
2. Low Computation Power: Our solution will be running all the time in the background and hence CPU and Ram usage should be as low as possible

Considering the constraints, i tried all the algorithms one by one

2.1.1 HOG + Linear SVM (DLIB)

HOG + Linear SVM (DLIB)

This approach is accurate and fast but has some limitations like:

• Not invariant to changes in rotation and viewing angles
• Can’t detect small faces

2.1.2 Haar Cascade (OpenCV)

Haar Cascade (OpenCV)

It is an old algorithm and gives lots of false positive cases.

I would love to share another articles named “Face Detection Models: Which to Use and Why?”. It consists of very detailed comparisons between DLIB and openCV models

Face Detection Models: Which to Use and Why?

After that i was looking for some Deep Learning based solution and then i found these 2 model:

• HRNET (Deep Learning solution)
• Partial Face Landmark Detector — PFLD (Deep Learning solution)

2.1.3 HRNET (Deep Learning solution)

HRNET (Deep Learning solution)

Actually, i found out a thesis “FACIAL LANDMARK DETECTION ON MOBILE DEVICES” (http://jultika.oulu.fi/files/nbnfioulu-202012183424.pdf) where they did lots of benchmarking and found out that HR-NET is slow and won’t be good for real time applications.

HRNET for not realtime

2.1.4 Partial Face Landmark Detector — PFLD (Deep Learning solution)

This model was accurate though but CPU usage was pretty high, not fulfilling one of our constraints.

Partial Face Landmark Detector — PFLD (Deep Learning solution)

2.1.5 Mediapipe Face Mesh

MediaPipe Face Mesh estimates 468 3D face landmarks in real-time even on mobile devices. They have BlazeFace as their face detection model, that uses a lightweight feature extraction network inspired by, but distinct from MobileNetV1/V2, delivering better real-time performance.

2.2. Locating the Lip region

Since i was using MediaPipe Face Mesh, i used the canonical face mesh mapping in-order to get the lip region points.

Canonical face mesh mapping

In the MediaPipe source code, The FACEMESH_TESSELATION frozenset in the code defines the point-to-point connection pair between all points.

mediapipe/face_mesh_connections.py at master · google/mediapipe

I used the following tuples in order to get the ROI.

Lips upper inner: (78, 191), (191, 80), (80, 81), (81, 82), (82, 13), (13, 312), (312, 311), (311, 310), (310, 415), (415, 308), (308, 324),

Lips lower: (324, 318), (318, 402), (402, 317), (317, 14), (14, 87), (87, 178), (178, 88),(88, 95), (95, 78)

Lips ROI

2.3 Detect if mouth is open or not

Our problem is different that yawn detection since one won’t open his/her mouth a-lot when breathing from mouth. Here, i simply calculated the contour area inside the inner lip region. Let’s say if a threshold is there and the calculated contour area is more than the minimum threshold set, then it means mouth is open.

Mouth open logic

Now, you may ask that this hack may fail since when we speak, we open our mouth and according to this logic, our solution will falsely state speaking as breathing mouth.

Well, in order to solve this, earlier i was thinking to use VAD i.e., Voice activity detector with an intention that:

• VAD says voice is there and our solution say mouth open = Speaking
• VAD says no voice is there and our solution say mouth open = Breathing from mouth

But then i came up with a small trick. Again, if counter area is being more than the threshold constantly for some “X” number of frames then it means it’s a “breathing from mouth case” since while speaking, in order to pronounce any letter or word, one need to close their mouth or in short, contour area will change frequently after every frame. In case of subconsciously “breathing from mouth”, contour area won’t be altered much with a large variance.

Speaking Vs Breathing from Mouth logic

Then i added a small notification popup that will alert the user.

Popup

2.4 Robust solution and packaging

Now, the question is what should be the minimum threshold (min_area variable in above code). For this, i added a “slider” in the solution through which user can itself define the threshold as per their camera position and sitting angle. I then made a small Tkinter application for our solution, used PyInstaller to convert my files into executable format and NSIS to create an installer for windows.

Final Result

3. Conclusion

So, this is the my overall project. The video and the title is meant for the entertainment purpose only. With this i’m not trying to hurt anyone sentiments but even if i did subconsciously, My apologies.

I’m a ML enthusiast and i thought of sharing my journey and experiences with everyone via filming some devlogs (developer vlogs). If you like to be part of this journey too, feel free to checkout my YouTube channel. Always open for constructive criticism :)

YouTube Thumbnail

Youtube Channel — RsTaK