X.org libFS 1.0.4 and earlier allows X servers to trigger allocation of insufficient memory and a buffer overflow via vectors related to an unexpected sign extension in the FSOpenServer function.
Improper Restriction of Operations within the Bounds of a Memory Buffer The product performs operations on a memory buffer, but it reads from or writes to a memory location outside the buffer's intended boundary.