Mitigating malicious intent with MISUSE threat modelling

The MISUSE thread model framework can help technologists build safe applications and devices and identify how perpetrators may manipulate technology to harm vulnerable individuals.

Technology is intertwined with nearly all aspects of society, with the influence being two-way. While people and society drive technological change, changing technologies can in turn shape society and the individual, and those changes can be incredibly fast paced.

Feeding this rapid pace of change is an increased focus across the industry on accelerating technological discovery. This has led to technologists producing exciting new applications and devices that literally transform the way we work and live. Yet sometimes, the drive towards the new and the emphasis on the positive can – at first – mean that any downsides of an invention are not recognised.

While it is true that technologies are neither inherently good nor inherently bad, their use can lead to good and bad outcomes. At its best, technology supports initiatives of all kinds. At its worst, there are unanticipated consequences or even malevolent uses.

One area where this contrasting nature of technology is particularly evident is coercive control – a pattern of dominating behaviour aimed at instilling fear and compliance. Technology can be key in supporting vulnerable individuals – enabling them to record evidence, find helpful information and access support. Yet even the most well-meaning of technologies can also be leveraged by perpetrators to facilitate malicious aims, such as control, harassment, and stalking.

Recognising this as a growing issue, in May 2020 the IBM Policy lab published our Five Technology Design Principles to Combat Domestic Abuse which both raised awareness of the issue of technology-facilitated abuse and proposed a way of resisting it through design. However, while many technologists have a key desire to build safe applications and devices, identifying how perpetrators may manipulate technology to harm vulnerable individuals, and devising measures to lessen those manipulations, is no easy task.

One type of methodology often used to uncover and minimise security vulnerabilities is threat modelling, which is a practical framework for understanding, identifying, prioritising and mitigating risks. Yet many traditional threat modelling methods have a point of view that is inwards focused, considering threats against company assets. Due to this viewpoint, it can be difficult to apply these frameworks when thinking of threats to individuals, as the aims of a perpetrator of coercive control will differ from those of hackers.

To shift thinking towards an outward-facing focus that considers risk towards the individual, an IBM team created the MISUSE threat model framework. This framework introduces a different threat modelling perspective, helping technologists recognise the full range of harms their technologies could pose to individuals.

MISUSE is an acronym used to identify possible malevolent intents of a perpetrator of technology-facilitated abuse. It highlights six threat dimensions, which encapsulate potential aims for maliciously leveraging technology against a vulnerable person.

MANIPULATE – Steering, controlling, or influencing vulnerable individuals.

ISOLATE – Controlling contact to cut vulnerable individuals off from their support system.

SPY – Monitoring and tracking activities, conversations, and whereabouts.

UNDERMINE – Wearing down a vulnerable individual’s self-esteem or lessening how they are perceived by others.

SCARE – Unnerving, worrying or frightening vulnerable individuals.

EMBARRASS – Causing a vulnerable individual to feel self-conscious, anxious, or ashamed.

Having these six threat dimensions at the heart of MISUSE threat modelling enables technologists to gain insight into how their creations could be re-purposed for harm. With this understanding they can work towards mitigating those malicious intents by advancing the security, privacy, and usability of their technologies.

Tempering any optimism bias and recognising that technology can be – and is being – manipulated for harm doesn’t mean that we shouldn’t continue to be excited and hopeful about the potential of new technologies. In fact, by embracing the MISUSE framework to think more broadly about how to build safety into design, the benefits of technology will become more evident. Technologists will not only improve the lives of some of society’s most vulnerable people but enhance digital technologies for all.