Bug 689493

Summary: Core dump with DEVICE=tiffsep with transparency
Product: Ghostscript Reporter: Marcos H. Woehrmann <marcos.woehrmann>
Component: Graphics LibraryAssignee: Ray Johnston <ray.johnston>
Status: NOTIFIED FIXED    
Severity: critical    
Priority: P2    
Version: master   
Hardware: PC   
OS: Linux   
Customer: 190 Word Size: ---

Description Marcos H. Woehrmann 2007-10-03 10:30:45 UTC 
The customer reports and I've verified that gs8.60 and gshead (r8268) both core dump with the attached 
file when using -sDEVICE=tiffsep.

The command line I used:

  bin/gs -sDEVICE=tiffsep -sOutputFile=test ./axa.pdf
Comment 1 Marcos H. Woehrmann 2007-10-03 10:31:05 UTC 
Created attachment 3433 [details]
axa.pdf
Comment 2 Marcos H. Woehrmann 2007-10-03 10:32:02 UTC 
The customer suggested a fix:

Update: in fact  the problem has nothing to do with spot colors: tiffsep
(and possibly other devices supporting spots) crashes immediately, with
all files having transparency.

The cause is this test  (in gdev_prn_allocate):

if (new_height < max_ulong/(ESTIMATED_PDF14_ROW_SPACE(new_width) >> 3))

which, at least on Linux, causes a floating point exception when
new_width=0, i.e. as soon as gdev_prn_open() is called.
This bugs does not affect other devices, because the test is only
executed when the "flag page_uses_transparency" is set, and for some
reasons it is set with tiffsep, but not with non-spot devices (at the
time gdev_prn_open() is called).

Now, I have tried 2 fixes:

 if (new_width>0 && new_height <
max_ulong/(ESTIMATED_PDF14_ROW_SPACE(new_width) >> 3))

Or

 if (new_width==0 || new_height <
max_ulong/(ESTIMATED_PDF14_ROW_SPACE(new_width) >> 3))

Both seems to work, but please tell me which one is better.
Comment 3 Ray Johnston 2007-10-12 13:31:59 UTC 
fixed as of rev. 8289

The problem was that when the pdf14 device, several of the color procs were
substituted including get_color_comp_index. When the Separation name is unknown
we need to call the target (tiffsep) get_color_comp_index.