[STORY] A Scientific Analysis of Your Skin Condition in Just 30 Minutes: The Growing Potential of AI Dermatology plus AI: a DX through Co creation

Toshiba and… skincare? While it may seem unexpected for a company that offers solutions in energy, infrastructure, and B2B, Toshiba's collaboration with FANCL represents part of the company's push towards business model DX, digital transformation.FANCL a Japanese company known for its additive-free skincare products, wanted to improve its customer consultation capabilities. Toshiba's role in the project to do this was, of course, technological-specifically, the implementation of AI technology in FANCL's skin analytics system. How did Toshiba become involved in this project, and what kind of value has it created? We spoke to two engineers who were involved in the development and implementation of the technology to find out.

Exploring Ways to Speed Up Skin Analytics Using AI

A FANCL skin analytics session goes as follows. A special tape is applied to your skin, then gently removed. Cells from the stratum corneum-the outermost layer of the epidermis-that adhere to the tape are then photographed with a microscope. While the AI system analyses the image, you answer a series of questions about your skin. When the analysis is complete, the system tells you about the condition of your stratum corneum, and the potential for various skin troubles, and shows you a radar chart of your stats for "seven skin strengths." Staff members-beauty experts-then offer you skincare counseling based on this data.

This skin analytics service, "AI Skin Beauty Instinct Analysis," is available at about 180 FANCL company-owned retail stores. It takes about 30 minutes, and is offered free of charge. The service initially began as a way to give people, both men and women, a deeper understanding of their skin needs, at a time when more and more people are suffering from skin problems, such as rough skin and acne caused by masks.

The "AI Skin Beauty Instinct Analysis" identifies seven skin strengths and five major skin troubles. The system does this through a mathematical analysis that involves multiplying the numerical value assigned to the form of the stratum corneum cells, and to the presence of proteins involved in skin beauty and aging. Toshiba's role it this was in developing an AI system that could use the image from the microscope to estimate and assign numerical values to the form of the stratum corneum cells and the presence of proteins. To learn more, we spoke with Masaya Eto, who was responsible for developing and verifying the AI system, about the background to the project.

Masaya Eto, Expert, Software & AI Technology Center, Toshiba Digital Solutions Corporation

"FANCL had identified a protein associated with skin beauty, and named it the Stratum Corneum Biomarker." They used this knowledge to offer a paid service, "Stratum Corneum Biomarker Analysis," online and in some of their company-owned stores, but ran into certain challenges.

"What made things difficult was that the stratum corneum cells they collected for these analyses had to be put through expensive analytical equipment and analyzed by researchers. It was only in the flagship store that they were able to give users their results on the spot, and even that took two hours. In other stores, the skin cells had to be sent by post to a lab for analysis. This took considerable time and effort, about two weeks for the entire process, and also required users to visit the store again to get their results. FANCL wanted a solution that would let them see the analysis results on the spot in all of their company-owned stores, so staff could offer users skincare counseling right away. And that's where Toshiba's SATLYS AI Platform came in," said Eto.

Differences in service before and after the implementation of Toshiba's AI system

What FANCL wanted was a system that could use AI to analyze an image, estimate the value of stratum corneum biomarkers, and provide a mathematical analysis of skin condition that staff members in-store could use to offer skincare counseling. As to why Toshiba was chosen out of the many, many companies that develop AI systems, that came down to Toshiba's all-in-one, comprehensive approach.

"We were different in that we were able to provide comprehensive support, all the way from the initial stages-identifying issues and setting KPIs-to proposing various AI solutions, developing the AI system, and even building, operating, and maintaining the customer infrastructure. They thought highly of this support system, and the number of talented people we have working in AI, and that's what led to this collaboration," said Eto.

Breaking the Process Down Into Phases, Going for Base Hits Instead of Home Runs

The most important thing when it comes developing an AI system is the initial task analysis and issue identification process. People at Toshiba refer to this phase as 'digital consulting,' and spent considerable time in discussions to define the requirements for the AI system. Eto says, "I think part of what made this project successful is how clear the customer's issues were from the start-the analysis was too time-consuming, the cost was prohibitive, and it took too much time to get the results."

SATLYS AI development/implementation flow for solving customer issues

Even then, Eto says, they were unsure at first whether they could actually develop such a system. "Just on an intuitive level, I felt it would be difficult for an AI system to judge these stratum corneum biomarkers. But then again, it was such meaningful work. I asked if I could take on the challenge, and so the project began."

Toshiba's initial goal for the AI was a system that could estimate the values for seven biomarkers from the image of the stratum corneum, such as those associated with the skin barrier, antioxidants, and inflammation. However, Eto's intuition, honed from years and years of working in AI development, turned out to be right.

"We developed a prototype AI, fed it 93 stratum corneum images, got it to estimate the biomarker values from these images, and compared these estimates to the actual values. The result was a correlation coefficient of 0.22 to 0.54 between the estimated and actual values, which is… not great. With an analysis like that, we couldn't say that the system had really analyzed the stratum corneum. Even though we did not have high hopes, we were disappointed." said Eto.

How did the team get past this? The answer lay in a rather ingenious twist, something only Toshiba with its wealth of experience in AI co-creation could have come up with. From the beginning, the Toshiba team had divided the AI development process into phases, with development proceeding according to these phases. In the end, it was going through all three phases, and many, many discussions with the team at FANCL, that resulted in an AI system usable in the skin analytics service.

The three phases of AI development as implemented by FANCL and Toshiba

"Establishing a large scope for issue analysis from the start, and then working towards a perfect solution, will make a project drag on, and up the risk of failure," said Eto. "Our approach from the start was to break down what needed to be done, and to go through these steps while discussing everything with FANCL, so we could get results quickly. If I were to use a baseball analogy, our strategy was to win the game by trying to get as many hits as possible, instead of going for home runs.

"In Phase 1, we used stratum corneum images taken with our research cameras to see if the AI could estimate biomarker values, but the results were disappointing. So we moved onto Phase 2."

In proceeding to Phase 2, FANCL and Toshiba reassessed and reestablished their overall goal. That was not to offer estimates of biomarker values, but to offer skincare counseling that suited the customer, based on estimates of their skin condition. In that case, they thought, couldn't they estimate skin condition at the required level by combining these biomarker values with some other element? This, then, became the focus of their efforts.

The AI system that was developed on this basis was one that could estimate the form of stratum corneum cells. Research conducted by FANCL had already shown that the form of a person's stratum corneum cells is associated with skin condition. Using this knowledge, the Toshiba team developed an AI that could estimate the form of individual stratum corneum cells by enhancing their contours and surfaces, and assign numerical values to the estimates. This AI system returned a more than satisfactory recognition rate of 77%.

Toshiba AI system automatically recognizes stratum corneum cells

A System that Reeducates and Improves Itself

Here's how it works. Toshiba's AI system estimates the form of the stratum corneum cells and the biomarker values, after which FANCL's mathematical analysis kicks in calculate values for various indicators, including skin moisture content, barrier function, resilience, wrinkles, and inflammation. The results of this analysis are then delivered from the cloud to the store and the customer, and staff can provide skincare counseling. This is truly a feat of co-creation, with Toshiba streamlining the analysis process using its image analysis AI, and FANCL using its expertise in skincare research to offer estimates of user skin condition.

With success in this phase, it was on to Phase 3-final confirmation using images from the actual cameras that would be used to provide the service, in preparation for its launch in stores. But even with all this done, the journey was still far from over. Toshiba still had to build, operate, and maintain the system for the official service, so that it could be used on devices in the approximately 180 FANCL stores throughout Japan.

This was where AI implementation specialist Kazuyuki Tanaka joined the project.

"We made it so the 'AI Skin Beauty Instinct Analysis' service could be used in-store by allowing devices there to access the Toshiba AI system in the cloud, and FANCL's mathematic analysis," said Tanaka.

Kazuyuki Tanaka, Specialist, Data Management Services Development Group, AI Operation & Services Dept.,
Digital Engineering Center, Toshiba Digital Solutions Corporation

And that's not all. The system even came equipped with a new feature, one that perhaps symbolizes Toshiba's capabilities. "We included a feature that would allow the system to keep learning even after the start of the service, instead of just operating on initial estimations. What's important with AI systems is what happens after they're installed," said Tanaka. But what does that mean?

As Tanaka explains, "Every time the 'AI Skin Beauty Instinct Analysis' service is used it collects the data. We can use it to reinforce its capabilities; maintain the accuracy of its estimates, and perhaps even improve it. So what we did was we implement a feature that would allow the system to learn and update itself-not just in terms of the AI, but in terms of FANCL's mathematical analysis as well.

Co-Creating with Customers, Even in Entirely New Fields

And so the "AI Skin Beauty Instinct Analysis" service was launched. And the results were very clear.

"It's been a huge hit with customers, to the point where people have to make reservations. A lot of people are very happy to have been able to gain this understanding of their skin condition. The staff have been able to recommend products based on solid reasoning, which makes customers more confident about the products they're buying. And the researchers who previously analyzed stratum corneum samples visually now have more time that they can spend on new research," said Eto.

Toshiba's efforts this time around were in an area a bit far removed from the company's specialties of energy and infrastructure. Eto and Tanaka had this to say about the project.

"Of course, we at Toshiba didn't have any knowledge of dermatology. What we did have, though, was the ability and expertise to solve the customer's issues with AI. I think that's why this co-creation effort with FANCL went so well.

It doesn't matter if it's a field that's entirely new to us. As long as the customer has the concepts and the data, Toshiba will be able to work closely with and co-create with them. Our experiences in this project have given us a newfound confidence, and it would be great if customers from different industries and business categories would give us a ring to see what we can accomplish together," said Eto.

"The customer's business environment is constantly changing. That means, of course, that the technologies they use, including AI, must constantly evolve as well. At Toshiba, we're always working to stay prepared, so that we can offer the best solutions using optimal technologies," said Tanaka.

The "AI Skin Beauty Instinct Analysis" service takes FANCL's real-world data, analyses it using AI in cyberspace, then feeds it back into the real world. In that sense, it is a textbook example of cyber physical systems^* giving way to DX. We are in an era where the necessity of DX is being shouted from the rooftops, and this particular effort, which incorporated AI into business in a clever fashion, changed the very structure of the service itself. It is, in other words, an example of DX that transformed the experience of in-store skincare counseling and the sale of skincare products. Both Eto and Tanaka agree: "It's a model case of DX through co-creation."

*Systems in which data obtained from the real world is analyzed in cyberspace, then fed back into the real world as useful information

The "AI Skin Beauty Instinct Analysis" is available to anybody who visits a direct-sales FANCL store in Japan, regardless of gender or age. So if you are in Japan, why not try it out? Seeing is believing, after all. While you may not believe outright that a sample of your skin on some specialized tape will tell you all about your skin condition, actually walking into the store and seeing this melding of FANCL and Toshiba technologies might just change your mind.

Note: "Skin Beauty Instinct Analysis" is a registered trademark or trademark of FANCL Corporation.

Public link

Please use the above public link if you want to share this noodl on another website.