Detail CWE-1039

The product uses an automated mechanism such as machine learning to recognize complex data inputs (e.g. image or audio) as a particular concept or category, but it does not properly detect or handle inputs that have been modified or constructed in a way that causes the mechanism to detect a different, incorrect concept.

Mode Pengenalan

Architecture and Design : This issue can be introduced into the automated algorithm itself due to inadequate or inappropriate training data, as well as lack of validation, verification, testing, and evaluation of the algorithm. These factors can affect the overall robustness of the algorithm when introduced into operational settings.
Implementation : The developer might not apply external validation of inputs into the algorithm.

Platform yang Berlaku

Bahasa

Kelas: Not Language-Specific (Undetermined)

Teknologi

Nama: AI/ML (Undetermined)

Konsekuensi Umum

Contoh yang Diamati

Mitigasi yang Memungkinkan

Phases : Architecture and Design
Algorithmic modifications such as model pruning or compression can help mitigate this weakness. Model pruning ensures that only weights that are most relevant to the task are used in the inference of incoming data and has shown resilience to adversarial perturbed data.
Phases : Architecture and Design
Consider implementing adversarial training, a method that introduces adversarial examples into the training data to promote robustness of algorithm at inference time.
Phases : Architecture and Design
Consider implementing model hardening to fortify the internal structure of the algorithm, including techniques such as regularization and optimization to desensitize algorithms to minor input perturbations and/or changes.
Phases : Implementation
Consider implementing multiple models or using model ensembling techniques to improve robustness of individual model weaknesses against adversarial input perturbations.
Phases : Implementation
Incorporate uncertainty estimations into the algorithm that trigger human intervention or secondary/fallback software when reached. This could be when inference predictions and confidence scores are abnormally high/low comparative to expected model performance.
Phases : Integration
Reactive defenses such as input sanitization, defensive distillation, and input transformations can all be implemented before input data reaches the algorithm for inference.
Phases : Integration
Consider reducing the output granularity of the inference/prediction such that attackers cannot gain additional information due to leakage in order to craft adversarially perturbed data.

Metode Deteksi

Dynamic Analysis with Manual Results Interpretation

Use indicators from model performance deviations such as sudden drops in accuracy or unexpected outputs to verify the model.

Dynamic Analysis with Manual Results Interpretation

Use indicators from input data collection mechanisms to verify that inputs are statistically within the distribution of the training and test data.

Architecture or Design Review

Use multiple models or model ensembling techniques to check for consistency of predictions/inferences.

Catatan Pemetaan Kerentanan

Justifikasi : This CWE entry is a Class, but it does not have Base-level children.
Komentar : This entry is classified in a part of CWE's hierarchy that does not have sufficiently low-level coverage, which might reflect a lack of classification-oriented weakness research in the software security community. Conduct careful root cause analysis to determine the original mistake that led to this weakness. If closer analysis reveals that this weakness is appropriate, then this might be the best available CWE to use for mapping. If no other option is available, then it is acceptable to map to this CWE.

Catatan

Further investigation is needed to determine if better relationships exist or if additional organizational entries need to be created. For example, this issue might be better related to "recognition of input as an incorrect type," which might place it as a sibling of CWE-704 (incorrect type conversion).

Referensi

REF-16

Intriguing properties of neural networks
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus.
https://arxiv.org/abs/1312.6199

REF-17

Attacking Machine Learning with Adversarial Examples
OpenAI.
https://openai.com/index/attacking-machine-learning-with-adversarial-examples/

REF-15

Magic AI: These are the Optical Illusions that Trick, Fool, and Flummox Computers
James Vincent.
https://www.theverge.com/2017/4/12/15271874/ai-adversarial-images-fooling-attacks-artificial-intelligence

REF-13

CommanderSong: A Systematic Approach for Practical Adversarial Voice Recognition
Xuejing Yuan, Yuxuan Chen, Yue Zhao, Yunhui Long, Xiaokang Liu, Kai Chen, Shengzhi Zhang, Heqing Huang, Xiaofeng Wang, Carl A. Gunter.
https://arxiv.org/pdf/1801.08535

REF-14

Audio Adversarial Examples: Targeted Attacks on Speech-to-Text
Nicholas Carlini, David Wagner.
https://arxiv.org/abs/1801.01944

REF-1547

Practical Black-Box Attacks against Machine Learning
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, Ananthram Swami.
https://arxiv.org/pdf/1602.02697

REF-1548

LLM STINGER: Jailbreaking LLMs using RL fine-tuned LLMs
Piyush Jha, Arnav Arora, Vijay Ganesh.
https://arxiv.org/pdf/2411.08862

REF-1549

Universal and Transferable Adversarial Attacks on Aligned Language Models
Andy Zou1, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, Matt Fredrikson.
https://arxiv.org/abs/2307.15043

Pengajuan

Modifikasi