AI at banks and in­sur­ers: au­to­mat­i­cal­ly fair

Erscheinung:16.08.2024 | Topic Banken, VersicherungenAI at banks and insurers: automatically fair?

(BaFinJournal) The automation of the financial industry is advancing at a rapid pace. Artificial intelligence and machine learning promise great economic opportunities. But their use also raises some critical questions.

By Lydia Albers, Dr Matthias Fahrenwaldt, Ulrike Kuhn-Stojic, Dr Martina Schneider, BaFin Banking Supervision and Insurance Supervision

In the financial industry, as in many other sectors, the use of artificial intelligence (AI) and machine learning (ML) is on the rise. These technologies can be used to accelerate processes and analyse large quantities of data quickly and effectively. But when machines make decisions, problems can arise. This is because their logic only appears neutral at first glance. Highly automated decision-making processes with little human monitoring can amplify existing discrimination risks.
Financial services providers (see info box) and state supervisory authorities must therefore take measures to prevent unjustified discrimination. But what exactly is discrimination? In the financial industry, as in many other sectors, the use of artificial intelligence (AI) and machine learning (ML) is on the rise. These technologies can be used to accelerate processes and analyse large quantities of data quickly and effectively. But when machines make decisions, problems can arise. This is because their logic only appears neutral at first glance. Highly automated decision-making processes with little human monitoring can amplify existing discrimination risks.

Financial services providers (see info box) and state supervisory authorities must therefore take measures to prevent unjustified discrimination. But what exactly is discrimination?

EU law distinguishes between direct and indirect discrimination. In the case of direct discrimination, a person is treated less favourably on the basis of protected grounds (see info box "protected grounds"). Indirect discrimination occurs when an apparently neutral rule disadvantages a person or a group sharing the same characteristics (see info box "Various forms of discrimination"). Indirect discrimination does not relate to discriminatory treatment itself, but the impacts of seemingly neutral provisions, criteria and practices. In some cases it can be difficult to determine whether a situation is to be regarded as direct or indirect discrimination.

Fairness is a multifaceted challenge

In the context of AI and ML, questions of discrimination are often discussed under the umbrella term of "fairness". This term encompasses three important aspects.

Firstly, algorithmic fairness: here the design of algorithms should ensure that persons and groups of persons are treated uniformly. This is often achieved by means of quantitative approaches. In its simplest form, this may involve comparing the share of positive loan decisions for women with the share of positive decisions for men. However, such a statistical approach based on characteristics of groups of people is not suitable for identifying individual discrimination. For this reason, further-reaching measures may be required depending on the individual situation.

Secondly, the term fairness relates to discrimination in the legal sense. This varies according to national jurisdiction. In Germany, for example, the General Equal Treatment Act sets out the definition of (legally) impermissible unequal treatment (sections 19 and 20 of the AGG). Protected grounds are also defined in this Act (see info box "Protected grounds").

Finally, the term fairness also relates to "bias". In the 2021 publication "Big data and artificial intelligence: Principles for the use of algorithms in decision-making processes", BaFin and the Deutsche Bundesbankdescribed bias as the systematic distortion of results. The term bias has many dimensions. In the simplest cases, bias relates to the datasets used to train algorithms. In practice, datasets might exclude certain customer groups, such as single women. Such datasets would then not be representative of a company's customer base, which could result in unintentional discrimination since the algorithm would be incapable of adequately accounting for this group of persons and would therefore make assessments without any corresponding empirical basis.

The relevant literature contains numerous approaches for ensuring fairness in AI/ML-modelling. Key indicators are often used to measure fairness in algorithmic decisions. However, each indicator stands for a specific definition of fairness, and the various indicators are not necessarily compatible. A different approach uses further algorithms to explain AI/ML algorithms. However, these algorithms also have their shortcomings: it is easy to create examples in which algorithms make highly unfair decisions while typical explainability tools fail to flag these as discrimination (see info box "Fairness metrics and explainable AI"). In these examples, explainable artificial intelligence tools fail to recognise a protected characteristic as a significant factor even though the algorithm proposes a decision based on precisely that variable.

Generative artificial intelligence, for example large language models, can exacerbate the problem of a lack of fairness. Where such technologies are acquired from third-party providers (as is normally the case), users do not have any detailed information on the dataset used to train the models or on how they function. Such a lack of transparency is not found in conventional mathematical models, which clearly model causal relationships between various input factors and the model result, making them easier to understand.

BaFin addresses possible cases of discrimination

As part of its supervisory activities, BaFin takes account of possible cases of discrimination that may arise due to the automation of the financial industry.

In general, this issue is relevant to requirements for proper business organisation (in accordance with section 25a (1) sentence 1 of the German Banking Act and section 23 (1) of the German Insurance Supervision Act). In other words, these general governance requirements also apply to AI/ML processes. Firms must therefore adapt or expand their governance processes to take account of AI/ML. Current supervisory law already allows BaFin to address unjustified discrimination in the context of AI/ML and to demand firms comply with the relevant requirements. In its supervision of compliance with rules of conduct, BaFin expects institutions and companies to clearly set out responsibilities. This also applies to the use of AI/ML. To mitigate risks, it is also key to raise awareness and provide appropriate training for staff members entrusted with the development and use of AI/ML. Furthermore, the General Data Protection Regulation sets out the extent to which persons may be subject to decisions based on automated processing or profiling (Article 22 of the DSGVO).

The European Artificial Intelligence Act (see info box "Artificial Intelligence Act"), which entered into force on 1 August, is an important step in the legal treatment of bias risks in AI systems. In order to prevent discrimination and promote fairness and transparency, the Regulation sets out comprehensive rules for AI systems, in particular high-risk AI systems, which include models used to evaluate credit scores or creditworthiness alongside several applications in life and health insurance. This includes risk management and quality management, in addition to detailed documentation and high data quality standards. Article 10 of the AI Act also contains specific rules for minimising bias.

Financial services providers must prevent unjustified discrimination

For BaFin, one thing is clear: financial services providers must prevent unjustified discrimination against their customers caused by the use of AI/ML. Companies must establish review processes in order to identify possible sources of discrimination and take measures to mitigate them. As part of this, they must consider the limits of existing procedures for preventing unintended discrimination. Reliable and transparent governance and data management are crucial in order to ensure the fair and non-discriminatory treatment of consumers.

In some cases, human supervision may be essential to guarantee responsible operation, compensate for any technical deficiencies, and close any data gaps. Furthermore, in their choice of AI model, companies can make a considerable contribution to promoting transparency. Where results are comparable, simple models (e.g. in the simplest case logistic regression, which is commonly used today) are to be favoured over more complex black box methods. Such methods are not only easier to understand and interpret, they also allow bias to be better identified.

If supervised companies' use of AI and ML methods results in illegal discrimination, BaFin will take decisive action and implement suitable measures, for example in the context of supervising violations of statutory provisions.

The AI Act provides a sound regulatory framework for this. It lays down EU-wide legal requirements for companies regarding explainability, transparency and fairness. And there is a chance that it will also set EU-wide standards - for an effective supervisory approach to unjustified discrimination by AI/ML in the financial industry.

Did you find this article helpful?