More barrier-free software access with AI

NewIn: Chunyang Chen

More barrier-free software access with AI

In this new issue of NewIn we learn more about Chunyang Chen. He conducts research at the TUM Campus in Heilbronn on how software development can be simplified and automated by artificial intelligence (AI), especially Large Language Models. In this interview he explains how he aims to make software more accessible for people with visual impairments or other disabilities.

Display external content

At this point content of an external provider (source: www.xyz.de) is integrated. When displaying, data may be transferred to third parties or cookies may be stored, therefore your consent is required.

You can find more information and the possibility to revoke your consent at www.tum.de/datenschutz.

What got you interested in this research topic?

When I started doing research on the subject of apps and developing my own apps, I came across recommendations from Google and other companies on how to make apps more accessible for people with disabilities. I have to admit that I had never given it any thought. I then tried to use a mobile phone blindfolded and soon realized that it was hugely difficult even with special tools.

Another big realization came a few years ago when I attended a scientific conference where a scientist gave his talk via video link. The speaker was a young man, lying in bed, who could only move his head and one finger. This scientist was doing research on barrier-free access to websites and apps. That's when I understood how important this research is, given the number of people like him who can access online content only with great difficulty.

What technical resources are available to help people with difficulties find their way around the online world?

As a rule, they use screen readers, which read the contents of web pages from left to right and top to bottom. This works quite well for texts. But images or buttons for users to click are a bigger challenge. In this case, descriptive texts must be placed that can be read aloud by the screen readers. However, these texts are often not there or good enough to be read by blind users or those with other disabilities.

Can you help these teams with your research?

I hope so. We have developed algorithms to scan the apps before they are released to users. They point out areas where improvements are needed. At the same time, our algorithms suggest various solutions. We pass these regulations on to developers so that they can update their software.

In technical terms, we use different methods such as natural language processing, computer vision, large language models e.g., GPT-4 and adapt them to our needs by proposing proper "prompt engineering" methods. That means that we use in-context learning and feedback to drive the meta-data generation to make the mobile app accessible. Some data comes from Google recommendations or examples from other companies that have already created barrier-free access to their content. So we optimize the interaction with these models for our purposes.

What do you want to achieve with your research?

I moved to TUM in the spring to further develop my research career and apply that into real-world applications. In Germany, I hope that I'll be able to build a network with various industry partners and that my approach will be implemented in many companies.

• After completing his studies in China and his doctoral thesis in Singapore, Prof. Chen spent six years in Australia, working as a faculty. In the spring of this year, he arrived at the TUM Campus in Heilbronn and is conducting research there at the point where software engineering meets artificial intelligence.
• At TUM Campus Heilbronn, scientists from the fields of informatics and management conduct teaching and research on the digital transformation.
• All episodes of the NewIn series

News about the topic