Chi tiết CWE-1434

The product has a component that relies on a generative AI/ML model configured with inference parameters that produce an unacceptably high rate of erroneous or unexpected outputs.

Các cách xuất hiện

Build and Compilation : During model training, hyperparameters may be set without adequate validation or understanding of their impact.
Installation : During deployment, model parameters may be adjusted to optimize performance without comprehensive testing.
Patching and Maintenance : Updates or modifications may be made to the model that alter its behavior without thorough re-evaluation.

Nền tảng áp dụng

Ngôn ngữ

Lớp: Not Language-Specific (Undetermined)

Kiến trúc

Lớp: Not Architecture-Specific (Undetermined)

Công nghệ

Tên: AI/ML (Undetermined)
Lớp: Not Technology-Specific (Undetermined)

Hậu quả phổ biến

Các biện pháp giảm thiểu tiềm năng

Phases : Implementation // System Configuration // Operation
Develop and adhere to robust parameter tuning processes that include extensive testing and validation.
Phases : Implementation // System Configuration // Operation
Implement feedback mechanisms to continuously assess and adjust model performance.
Phases : Documentation
Provide comprehensive documentation and guidelines for parameter settings to ensure consistent and accurate model behavior.

Phương pháp phát hiện

Automated Dynamic Analysis

Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.
Hiệu quả : Moderate

Manual Dynamic Analysis

Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.
Hiệu quả : Moderate

Ghi chú ánh xạ lỗ hổng

Lý do : This CWE entry is at the Base level of abstraction, which is a preferred level of abstraction for mapping to the root causes of vulnerabilities.
Bình luận : Carefully read both the name and description to ensure that this mapping is an appropriate fit. Do not try to 'force' a mapping to a lower-level Base/Variant simply to comply with this preferred level of abstraction.

Ghi chú

This weakness might be under-reported as of CWE 4.18, since there are no clear observed examples in CVE. However, inference parameters may be the root cause for various vulnerabilities - or important factors - but the vulnerability reports may concentrate more on the negative impact (e.g. code execution) or the weaknesses that the insecure settings contribute to. Alternately, dynamic techniques might not reveal the root cause if the researcher does not have access to the underlying source code and environment.

Tài liệu tham khảo

REF-1487

We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs
Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala.
https://arxiv.org/abs/2406.10279

Gửi

Sửa đổi