Developing a Typology of Adversarial Attacks on ML Systems

Our analyses of n1=62 adversarial attack examples and n2=30 academic articles revealed five unique types of adversarial attacks: (i) poisoning, (ii) evasion, (iii) sponge, (iv) intellectual property theft, and (v) deep fake attacks. The ENISA framework had covered only the first two: poisoning and evasion attacks (ENISA 2021). Sponge attacks were introduced by Shumailov and colleagues. Many adversarial attacks targeted intellectual property (IP) inside the ML system. By just providing inputs to ML system and observing outputs, the adversary was able to identify and steal the model inside the system such as a classification boundary. Surprisingly, such IP thefts were not included as an adversarial attack type in previous frameworks. We decided to include it as IP theft can directly affect the bottom line of an organization and it is a risk that needs to be mitigated. Deep fake attacks that fool ML systems or humans are also becoming pervasive, but not covered in previous frameworks. In our sample, deep fake attacks mostly targeted ML systems. For example, a deep fake algorithm was used to create a fake video of the U.S President. The algorithm targeted the fake video to social media platforms. Content moderation algorithms failed to identify it as fake content; recommender algorithms recommended the content to users; users liked or disliked the video; and the fake video was disseminated at a large scale. Deep fake algorithms also target humans directly. For instance, in 2019, adversaries used ML to impersonate a chief executive’s voice, targeted the fake voice to employees of the company to request a fraudulent transfer of $243,000. An employee thought the voice was that of his boss and made the transfer. As our focus is on adversarial attacks on ML systems, deep fake attacks targeting humans are out of scope and reserved for future research.