Are our financial lives controlled by biased algorithms?

Jamie Heinemeier Hansson had a better credit score than her husband, tech entrepreneur David. They have equal shares in their property and file joint tax returns.

Yet David was given permission to borrow 20 times the amount on his Apple Card than his wife was granted.

The situation was far from unique. Even Apple’s co-founder Steve Wozniak tweeted that the same thing happened to him and his wife despite having no separate bank accounts or separate assets.

The case has caused a stink in the US. Regulators are investigating. Politicians have criticised Goldman Sachs, which runs the Apple Card, for its response.

What the saga has highlighted is concern over the role of machine-learning and algorithms – the rules of computer calculations – in making decisions that are clearly sexist, racist or discriminatory in other ways.

Society tends to assume – wrongly – that computers are impartial machines that do not discriminate because they cannot think like humans.

The reality is that the historic data they process, and perhaps the programmers who feed or create them, are themselves biased, often unintentionally. Equally, machines can draw conclusions without asking explicit questions (such as discriminating between men and women despite not asking for gender information).

How are our lives affected?

A whole range of issues in our daily lives have been changed, and undoubtedly improved, by the use of computer algorithms – from transport and technology to shopping and sport.

Arguably, the clearest and most direct impact is on our financial lives. Banks and other lenders use machine-learning technology to assess loan applications, including mortgages. The insurance industry is dominated by machines’ conclusions of levels of risk.

For the consumer, the algorithm is central in deciding how much they have to pay for something, or whether they are even allowed to have that product at all.

Image copyright Getty Images

Take insurance: the so-called “postcode lottery” comes from the fact that an algorithm will decide why two people with identical properties and with an identical security system will pay different amounts for their home insurance.

The algorithm uses postcodes to look up the crime rates in those areas, and subsequently makes a judgement on the likelihood of a property being burgled and sets the premium accordingly.

With credit scores, any machine’s conclusion on how reliable you are at repaying can affect anything from access to a mobile phone contract to where you can rent a home.

In the Apple Card case, we do not know how how the algorithm makes its decisions or which data it uses, but this could include historic data on which sorts of people are considered more financially risky, or who have traditionally made applications for credit.

So are these algorithms biased?

Goldman Sachs, which operates the Apple Card, says it does not even ask applicants their gender, race, age and so on – it would be illegal to do so. Decisions were, therefore, not based on whether they were a man or a woman.

However, this ignores what Rachel Thomas, director of the USF Center for Applied Data Ethics in San Francisco, calls “latent variables”.

“Even if race and gender are not inputs to your algorithm, it can still be biased on these factors,” she wrote in a thread on Twitter.

For example, an algorithm will not know someone’s gender, but it may know you are a primary school teacher – a female-dominated industry. Historic data, most controversially in crime and justice, may be drawn from a time when human decisions by police or judges were affected by somebody’s race.

The machine learns and replicates conclusions from the past that may be biased.

Image copyright Science Photo Library

Image caption Data has become extremely valuable for companies

It will also be worse at processing data it has not seen before. Someone who is not white or has a strong regional accent may be not be so well recognised by automated facial or voice recognition software which has mostly been “trained” on data taken from white people with no regional accents – a source of anger for some ringing a call centre.

What can be done about this issue?

The impartiality, or otherwise, of algorithms has been a hotly-debated issue for some time, with relatively little consensus.

One option is for businesses to be completely open about how those algorithms are set. However, these products are valuable commercial property, developed over years by highly-skilled, well-paid individuals. They will not want to just give their secrets away.

Most retailers, for example, would be delighted to be handed Amazon’s algorithms for free.

Another option is algorithmic transparency – telling a customer why a decision has been made, and which elements of their data were the most significant. Yet, there is little agreement on the best way to set out such information.

One answer could be more algorithms based on less specific information.

Jason Sahota, chief executive of Charles Taylor InsureTech, which provides software for the insurance industry, says there is an increasing use of pooled policies. An insurer may offer group health cover via an employer for a certain group of workers. Those insured do not need to fill out individual forms, as the insurer assesses their risk as a whole.

He says that consumers’ demand for fewer clicks and quicker payouts was happening as the insurance underwriting process was being simplified.

Stripping out too much data would make it difficult to differentiate applicants and policies, which would lead to homogenised products that would cost more.

Instead, Mr Sahota argues that people should be told why information is asked for and how it is used.

If something is found to be unintentionally biased, then – rather than simply blaming the data – he says it is important to do something to find a way to overcome the problem.