Fujitsu Develops AI-Video Recognition Technology to Promote Hand Washing Etiquette and Hygiene in the Workplace

TOKYO, May 26, 2020 - (JCN Newswire) - Fujitsu Laboratories Ltd. and Fujitsu Research and Development Center Co., Ltd. (FRDC) today announced the development of "Actlyzer" hand wash movement recognition technology, which leverages AI and machine learning techniques to identify complex hand washing movements from video data captured by camera.

Figure 1: 6 Steps for Correct Hand Washing

Figure 2: Conventional hand tracking tech (Left: Gesture recognition results; Right: Hand wash behavior)

Figure 3: Recognition of complex two-hand finger movements as a combination of the overall shape and movement patterns of both hands

Figure 4: Example of recognition display

Amidst the ongoing global COVID-19 pandemic, the importance of hand washing as a measure to protect people's health from bacteria, influenza, and other infectious diseases is gaining renewed attention worldwide. Under new regulations(1) planned to come into effect in June 2020 in Japan, food business operators will also be required to implement stronger measures to ensure hygiene in accordance with international HACCP(2) food safety standards, creating an urgent need for a non-invasive approach to quickly and accurately confirm that handwashing is carried out in a proper manner.

Anticipating this, Fujitsu has expanded the recognition function of its existing "Actlyzer behavioral analysis technology", which can recognize a variety of subtle and complex human movements without relying on large amounts of training data. Specifically, Fujitsu has refined recognition capabilities for hand movements to create technology to automatically recognize complicated hand movements performed during hand washing.

This technology makes it possible to easily determine whether someone is following each of the 6 steps for handwashing recommended by the Japanese Ministry of Health, Labour and Welfare, reducing the number of man-hours required for intrusive visual checks by inspectors for on-site sanitation management.

Moving forward, Fujitsu envisions this technology being used in a variety of other contexts, including in medical facilities, schools, hotels, and venues for large events, and plans to conduct field trials and additional research and development into it as a potential solution for its AI portfolio in the future.

Developmental Background

The Japanese Ministry of Health, Labour and Welfare recommends that people follow 6 steps to ensure their hands are properly washed and prevent food poisoning as well as the spread of infectious diseases (Figure 1).

At present, food service industry workers are required to perform these six steps when washing their hands and confirm that during each step the different parts of their hands have been rubbed more than a certain number of times. To this end, food service providers fill out a check sheet and make a self-report, and workers on-site undergo visual checks by supervisors. Because verification is still conducted manually, however, costs associated with human error, securing resources for inspectors, etc. remain a persistent challenge. As many manufacturers, including in the food industry, continue the march towards process automation for inspections by using machine learning technologies, hand washing checks represent a prime candidate for process optimization through automation.

Challenges

Gesture recognition using deep learning is a common technique for identifying hand and finger movements. This conventional technique can detect multiple feature points, such as joints and fingertips, from an image of the hand, and determine the hand gesture based on the positional relationship of the feature points (Figure 2 Left). However, one issue with the existing technology was that when people wash their hands correctly, both hands overlap and are lathered with soap, which obscures the detection points on the fingers and prevents accurate gesture recognition (Figure 2 Right).

Newly Developed Technology

To resolve this challenge, Fujitsu Laboratories Ltd. and FRDC have developed a new AI technology that automatically and accurately recognizes hand movements under the conditions described above, expanding the recognition function of the original Actlyzer behavioral analysis technology.

With the new technology, the complex hand movements of handwashing are captured as a combination of hand shape and repetitive rubbing motions, detected by two deep learning engines: Hand Shape Recognition and Motion Recognition (Figure 3).

The two-hand shape recognition engine uses a learned model of a basic shape of two hands, which is a typical form of hand movement in which hands are placed on top of each other, to determine the handshape for each frame of the image. Focusing on the overall shape solves the problem of when fingertips and joint feature points cannot be correctly detected due to hand overlap or foam. In addition, Fujitsu's unique AI technology "High Durability Learning (High Durability Learning)", which can track data changes, is applied to ensure that the basic shape of both hands is recognized with high accuracy even when the camera position or lighting changes during operation.

The motion recognition engine uses a learned model that detects periodically changing motion from successive frames and counts the number of iterations as the number of rubs from the iteration pattern and its period. In addition, the results of these two recognition engines are fed back to each other to improve recognition accuracy. The motion recognition engine sets a threshold value for the magnitude of the motion to be judged in accordance with the steps recognized by the two-hand shape recognition engine to prevent detection of erroneous periods, such as hand tremors not related to foam motion or rubbing. The two-hand shape recognition engine improves detection accuracy by filtering the judgment result using the repetition pattern period detected by the motion recognition engine.

Effects

A hand-wash video data set with approximately 2000 variations; including people, camera positions, and soap types, was independently filmed and collected for learning and evaluation. It was confirmed that the accuracy of 6 steps of correct hand washing is average of 95% or more, and that the accuracy of the number of hands rubbing movements is 90% or more. When operating the system on-site, the omission of action could be prevented because the person washing their hands always knows how long to scrub their hands and which of the six stages they've completed because the system determines it's displayed on the screen until it's completed. In addition to this, the system automatically records data with the starting timestamp and length of action of completing each six steps, and the number of times of rubbing. (Figure 4).

This technology automates on-site handwashing checks for workplaces that require strict hygiene control, eliminating the number of man-hours needed for visual confirmation and manual recording. Additionally, because the system doesn't recognize incorrect or incomplete hand washing, Fujitsu expects the solution will help educate users and ensure standardization for the right way of handwashing.

(1) New regulations Act for Partial Revision of the Food Sanitation Act
(2) HACCP Hazard Analysis and Critical Control Point. A sanitation management method that ensures safety by controlling processes that are particularly important for eliminating or reducing hazardous factors in the entire process from the receipt of raw materials to the shipment of products, based on the business operator's understanding of hazards such as contamination by food-borne bacteria. This is becoming a mandatory method mainly in developed countries.

About Fujitsu

Fujitsu is the leading Japanese information and communication technology (ICT) company offering a full range of technology products, solutions and services. Approximately 130,000 Fujitsu people support customers in more than 100 countries. We use our experience and the power of ICT to shape the future of society with our customers. Fujitsu Limited (TSE:6702) reported consolidated revenues of 3.9 trillion yen (USD35 billion) for the fiscal year ended March 31, 2020. For more information, please see www.fujitsu.com.
About Fujitsu Laboratories

Founded in 1968 as a wholly owned subsidiary of Fujitsu Limited, Fujitsu Laboratories Ltd. is one of the premier research centers in the world. With a global network of laboratories in Japan, China, the United States and Europe, the organization conducts a wide range of basic and applied research in the areas of Next-generation Services, Computer Servers, Networks, Electronic Devices and Advanced Materials. For more information, please see: http://www.fujitsu.com/jp/group/labs/en/.

About Fujitsu R&D Center

Fujitsu R&D Center Co., Ltd. is the first wholly-owned corporate research and development institution with independent legal status established in China by Fujitsu group in Japan with an investment of USD4.4 million. The research area of FRDC covers all the business fields of Fujitsu, namely information processing, communications, semiconductors and software services. Founded in February 1998, the company's business scope covers the research, development and technical services of system hardware and software of information and communication technology, electronic equipment, network technology, communication technology, information service technology, material technology, ecological protection and environmental governance technology, etc. For more information, please see: http://www.fujitsu.com/cn/frdc/

Figure 1: 6 Steps for Correct Hand Washing

Figure 2: Conventional hand tracking tech (Left: Gesture recognition results; Right: Hand wash behavior)

Figure 3: Recognition of complex two-hand finger movements as a combination of the overall shape and movement patterns of both hands