Concerns about individual privacy are not the only risks organisations face when leveraging big data and machine learning to drive growth. Another is bias inherent in algorithms used to make decisions about individuals or groups of consumers. These biases often result from two factors: first, the fact that machine learning models are trained using historical data; second, human involvement in creating algorithms that may reflect personal prejudices. As a result, they may inadvertently discriminate against certain groups – such as women or minorities – or reinforce existing disparities based on income level, race, gender identity, geographic location, age group, religion, or sexual orientation.

A study by researchers at MIT, for instance, found that facial recognition technology sold by IBM was more accurate in identifying white men than darker-skinned women. In another case, Alphabet’s Google Street View cars were more likely photograph neighbourhoods populated mainly by people of colour than those mostly inhabited by white residents – an oversight resulting from the absence of detailed demographic information needed train computer vision system identify which areas map.