Bug 691590 - Segfault in tile_by_steps_trans
Summary: Segfault in tile_by_steps_trans
Status: RESOLVED FIXED
Alias: None
Product: Ghostscript
Classification: Unclassified
Component: Graphics Library (show other bugs)
Version: master
Hardware: PC Linux
: P4 normal
Assignee: Ray Johnston
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-09-02 15:34 UTC by Tim Waugh
Modified: 2011-07-11 06:14 UTC (History)
3 users (show)

See Also:
Customer:
Word Size: ---


Attachments
ghostscript-transbytes-null.patch (733 bytes, application/octet-stream)
2010-09-02 15:34 UTC, Tim Waugh
Details
Ausgabe.pdf (68.76 KB, application/pdf)
2010-10-21 13:42 UTC, Tim Waugh
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Waugh 2010-09-02 15:34:47 UTC
Created attachment 6698 [details]
ghostscript-transbytes-null.patch

I've been sent a PDF (it's private, unfortunately) which crashes ghostscript-8.70 and 8.71 in tile_by_steps_trans.

The reason for the segfault is that ptile->ttrans->transbytes is NULL the third time this function is called.

By placing a check for NULL before we call down into pat_trans_fill I've managed to avoid the segfault, but the file now gives a PostScript stack trace instead, so I think this is more of a work-around than a fix.

Is there any legitimate reason for ptile->ttrans->transbytes to be NULL at this point, or should I be looking further up the call trace?

Original bug report:
  https://bugzilla.redhat.com/show_bug.cgi?id=589569

#0  memcpy () at ../sysdeps/x86_64/memcpy.S:161
#1  0x00000031f892cef5 in tile_rect_trans_simple (xmin=<value optimized out>, 
    ymin=<value optimized out>, xmax=461, ymax=274, px=<value optimized out>, 
    py=<value optimized out>, ptile=0x651478, fill_trans_buffer=0xef7dc0)
   from /usr/lib64/libgs.so.8
#2  0x00000031f892d539 in tile_by_steps_trans (xmin=0, ymin=0, xmax=461, 
    ymax=274, ptile=0x651478, fill_trans_buffer=0xef7dc0, 
    phase=<value optimized out>) at base/gxp1fill.c:551
#3  gx_trans_pattern_fill_rect (xmin=0, ymin=0, xmax=461, ymax=274, 
    ptile=0x651478, fill_trans_buffer=0xef7dc0, phase=<value optimized out>)
    at base/gxp1fill.c:802
#4  0x00000031f89bdcfc in pdf14_tile_pattern_fill (dev=0x8339f8, 
    pis=<value optimized out>, ppath=0x646ce0, params=0x7fffffffc750, 
    pdcolor=0x80d2d0, pcpath=<value optimized out>) at base/gdevp14.c:1687
#5  pdf14_fill_path (dev=0x8339f8, pis=<value optimized out>, ppath=0x646ce0, 
    params=0x7fffffffc750, pdcolor=0x80d2d0, pcpath=<value optimized out>)
    at base/gdevp14.c:1478
#6  0x00000031f8ba1076 in gx_fill_path (ppath=0x646ce0, pdevc=0x80d2d0, 
    pgs=0x629828, rule=-1, adjust_x=76, adjust_y=<value optimized out>)
    at base/gxpaint.c:48
#7  0x00000031f8b6b051 in fill_with_rule (pgs=0x629828, rule=-1)
    at base/gspaint.c:310
#8  0x00000031f893b7ae in interp (pi_ctx_p=0x604368, 
    pref=<value optimized out>, perror_object=0x7fffffffd660)
    at psi/interp.c:1275
Comment 1 Ray Johnston 2010-09-03 17:13:52 UTC
Tim, please test with the 9.00 release candidate. There have been many
fixes made in this area.
Comment 2 Ray Johnston 2010-09-09 18:18:47 UTC
Reassign to submitter for testing with 9.00 rc3 (with his private file).
Comment 3 Tim Waugh 2010-09-24 15:56:54 UTC
I can't reproduce the problem with the 9.00 release.
Comment 4 Tim Waugh 2010-10-01 15:07:32 UTC
Apologies -- I re-tested the wrong input file.

I *can* still reproduce this problem in 9.00.
Comment 5 Tim Waugh 2010-10-21 13:42:44 UTC
Created attachment 6813 [details]
Ausgabe.pdf

Here is the input file, shared with permission.
Comment 6 Ray Johnston 2011-04-12 23:55:41 UTC
Command line I used was:

 gswin32c -dMaxBitmap=100000000 -sDEVICE=png16m -o nul: Ausgabe.pdf

I can reproduce this on HEAD (rev 12391). It may be that some 'optimization'
is biting us. The pattern is very small. At resolutions higher than 91 dpi.
the file runs to completion.

The -dPDFDEBUG log (with -Zv set on a DEBUG build) shows:

/p13 scn
<<
/Length 76 0 R
/Filter /FlateDecode /Type /XObject /Subtype /Image /Width 12 /Height 20 /ColorSpace /DeviceGray /BitsPerComponent 8 >>
stream
13 endobj
%FilePosition: 5615
endobj
%Pattern: << /PatternType 1 /Matrix [2.58333302 0 0 0.05 0 1410] /Resources << /XObject << /x77 {77 0 resolveR} >> >> /T
ilingType 1 /BBox [0 0 12 20] /File ( q 12 0 0 20 0 0 cm /x77 Do Q \n) /PaintType 1 /XStep 29798 /PaintProc {<< /XObject
 << /x77 {77 0 resolveR} >> >> .pdfpaintproc} /Length {79 0 resolveR} /.pattern_uses_transparency true /YStep 29798 >>
/a0 gs
0 0 1152 1466 re
f
[v](0x1e5e050)shape.alpha = 1
[v](0x1e5e050)opacity.alpha = 1
gx_pattern_load: pushing the pdf14 compositor device into this graphics state
[v]gs_pdf14_device_push
[v]pdf14_open: width = 13, height = 1
[v]base buf: 13 x 1, 4 color channels, 4 planes
[v]set_marking_params, opacity = 1, shape = 1, bm = 0
   pushing the pdf14 compositor device into this graphics state
[v]gs_pdf14_device_push
[v]pdf14_open: width = 13, height = 1
[v]base buf: 13 x 1, 4 color channels, 4 planes
[v]set_marking_params, opacity = 1, shape = 1, bm = 0
%Begin PaintProc<< /XObject << /x77 {77 0 resolveR} >> >>
[v]gs_push_transparency_state NOT sending
q
[v]gs_push_transparency_state NOT sending
12 0 0 20 0 0 cm
/x77 Do
[v](0x1e5e050)shape.alpha = 1
[v](0x1e5e050)opacity.alpha = 1
[v]pushing soft mask color sending
[v](0x1e5e050)gs_begin_transparency_mask [0 0 1 1]
      subtype = 1  Background_components = 0 no TR
[v](0x1e5e050)gx_begin_transparency_mask [0 0 1 1]
      subtype = 1  Background_components = 0  Num_grp_clr_comp = 1 no TR
pdf14_begin_transparency_mask, bg_alpha = 255
[v]pdf14_update_device_color_procs
[v]pdf14_update_device_color_procs,num_components_old = 3 num_components_new = 1
[v]pdf14_push_transparency_mask, idle=0, replacing=0
[v](0x1e5e050)opacity.alpha = 1
[v](0x1e5e050)shape.alpha = 1
[v](0x1e5e050)blend_mode = Compatible
[v](0x1e5e050)shape.alpha = 1
[v](0x1e5e050)opacity.alpha = 1
[v]set_marking_params, opacity = 1, shape = 1, bm = 0
[v]set_marking_params, opacity = 1, shape = 1, bm = 0
[v]xstate_changed set true, gstate level is 20
[v](0x1e5e050)gs_end_transparency_mask(0)
[v]popping soft mask color sending
[v](0x1e5e050)gx_end_transparency_mask(0)
pdf14_end_transparency_mask
[v]pdf14_pop_transparency_mask, idle=0
[v](0x1e5e050)begin_transparency_group [0 0 1 1] Num_grp_clr_comp = 0
     (no CS)  Isolated = 1  Knockout = 0
[v](0x1e5e050)gx_begin_transparency_group [0 0 1 1] Num_grp_clr_comp = 0
     (no CS)  Isolated = 1  Knockout = 0
[v]pdf14_begin_transparency_group, I = 1, K = 0, alpha = 1, bm = 0
[v]pdf14_update_device_color_procs
[v]pdf14_update_device_color_procs,num_components_old = 3 num_components_new = 3
[v]Transparency group color space update
[v]pdf14_push_transparency_group, idle = 0
[v]base buf: 13 x 1, 4 color channels, 4 planes
[v](0x1e5e050)shape.alpha = 1
[v](0x1e5e050)opacity.alpha = 1
[v]set_marking_params, opacity = 1, shape = 1, bm = 0
[v]set_marking_params, opacity = 1, shape = 1, bm = 0
[v]set_marking_params, opacity = 1, shape = 1, bm = 0
[v]gs_end_transparency_group
[v]gx_end_transparency_group
[v]pdf14_end_transparency_group
pdf14_pop_transparency_group y0 = 0, y1 = 1, w = 13, alpha = 255, shape = 255, tag =  bm = 0
[v]pop buf, idle=0
Q
[v]gs_pop_transparency_state sending
[v]gx_pop_transparency_state
pdf14_pop_transparency_state
%%EOF
[v]gs_pop_transparency_state NOT sending
%End PaintProc2476
[v]pdf14_push_transparency_group, idle = 0
[v]base buf: 461 x 274, 1 color channels, 1 planes
[v]pdf14_tile_pattern_fill, (0, 0), 461 x 274 pat_id 1691

Continuing to breakpoint to look at the pattern cache when it is written.
Comment 7 Ray Johnston 2011-04-13 01:58:31 UTC
Michael, what's the idea trying to poach this bug ;-)

I've continued looking at this and have isolated it into 
    pdf14_get_buffer_information
where the buf->bbox is inverted (p=[13,1], q=[0,0]) so that the width and
height get calculated as negative values. Then the 'return' at line 1350
exits BEFORE the transbuff is set up.

I'll continue to look into why the buf->bbox is backwards.

I have determined that doing abs() around the calculation for width and
height prevents the crash, but need more study to find out why the bbox
is inverted and why it works when the resolution is > 91.
Comment 8 Ray Johnston 2011-07-11 06:14:00 UTC
Pretty sure that the final fixes for patterns with transparency have fixed
this. It no longer fails for me on HEAD.