Created attachment 4577 [details]
Pantone.pdf

The core dump does not occur under linux on my amd64 box and valgrind reports no difference 
between gs8.63 and head (r9198).  Here's the gdb output from Mac OS X:

.
.
.
%%SeparationName: PANTONE 143 U
%%SeparationName: PANTONE 144 U
%%SeparationName: PANTONE 145 U

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00000914
0x0016627e in pdf14_mark_fill_rectangle (dev=0x187ec34, x=0, y=0, w=1984, h=74, 
color=18374686481802330112) at gdevp14.c:1771
1771	    src_alpha = src[num_comp] = (byte)floor (255 * pdev->alpha + 0.5);
(gdb) where
#0  0x0016627e in pdf14_mark_fill_rectangle (dev=0x187ec34, x=0, y=0, w=1984, h=74, 
color=18374686481802330112) at gdevp14.c:1771
#1  0x00165924 in pdf14_fill_rectangle (dev=0x187ec34, x=0, y=0, w=1984, h=74, 
color=18374686481802330112) at gdevp14.c:1608
#2  0x0043601c in gx_dc_pure_fill_rectangle (pdevc=0xbfff99e4, x=0, y=0, w=1985, h=2835, 
dev=0x187ec34, lop=16636, source=0x0) at gxdcolor.c:414
#3  0x00437d00 in gx_general_fill_path (pdev=0x187ec34, pis=0xbfff862c, ppath=0xbfffa800, 
params=0xbfff99a4, pdevc=0xbfff99e4, pcpath=0x0) at gxfill.c:459
#4  0x00438cba in gx_default_fill_path (pdev=0x187ec34, pis=0xbfff862c, ppath=0xbfffa800, 
params=0xbfff99a4, pdevc=0xbfff99e4, pcpath=0x0) at gxfill.c:687
#5  0x0016496b in pdf14_fill_path (dev=0x187ec34, pis=0xbfff9cac, ppath=0xbfffa800, 
params=0xbfff99a4, pdcolor=0xbfff99e4, pcpath=0x0) at gdevp14.c:1100
#6  0x001fee26 in clist_playback_band (playback_action=playback_action_render, cdev=0x1810234, 
s=0xbfffcc1c, target=0x187ec34, x0=0, y0=0, mem=0x1809634) at gxclrast.c:1784
#7  0x0020642d in clist_playback_file_bands (action=playback_action_render, crdev=0x1810234, 
page_info=0xbfffd74c, target=0x1722138, band_first=0, band_last=0, x0=0, y0=0) at gxclread.c:696
#8  0x00206008 in clist_render_rectangle (cldev=0x1810234, prect=0xbfffdc60, bdev=0x1722138, 
render_plane=0xbfffdd9c, clear=1) at gxclread.c:625
#9  0x00205cc5 in clist_rasterize_lines (dev=0x1810234, y=0, line_count=1, bdev=0x1722138, 
render_plane=0xbfffdd9c, pmy=0xbfffdd98) at gxclread.c:539
#10 0x0020578e in clist_get_bits_rectangle (dev=0x1810234, prect=0xbfffe028, params=0xbfffdf98, 
unread=0x0) at gxclread.c:430
#11 0x0021cca0 in clist_get_bits_rect_mt (dev=0x1810234, prect=0xbfffe028, params=0xbfffdf98, 
unread=0x0) at gxclthrd.c:472
#12 0x0047e55c in gx_default_get_bits (dev=0x1810234, y=0, data=0x1717034 "", 
actual_data=0xbfffe0cc) at gdevdgbr.c:51
#13 0x001f4b6d in gdev_prn_get_bits (pdev=0x1810234, y=0, str=0x1717034 "", 
actual_data=0xbfffe0cc) at gdevprn.c:1227
#14 0x00315d71 in tiffsep_print_page (pdev=0x1810234, file=0xa04d8ee8) at gdevtsep.c:1050
#15 0x001f3dd0 in gx_default_print_page_copies (pdev=0x1810234, prn_stream=0xa04d8ee8, 
num_copies=1) at gdevprn.c:834
#16 0x001f3b0a in gdev_prn_output_page (pdev=0x1810234, num_copies=1, flush=1) at 
gdevprn.c:769
#17 0x00481bf8 in gx_forward_output_page (dev=0x181f234, num_copies=1, flush=1) at 
gdevnfwd.c:164
#18 0x003f0a57 in gs_output_page (pgs=0x1218034, num_copies=1, flush=1) at gsdevice.c:134
#19 0x000f744f in zoutputpage (i_ctx_p=0x1229184) at zdevice.c:354
#20 0x000ae5fe in call_operator (op_proc=0xf7351 <zoutputpage>, i_ctx_p=0x1229184) at 
interp.c:111
#21 0x000b221c in interp (pi_ctx_p=0x11018cc, pref=0xbfffec54, perror_object=0xbfffee10) at 
interp.c:1534
#22 0x000aece6 in gs_call_interp (pi_ctx_p=0x11018cc, pref=0xbfffed58, user_errors=1, 
pexit_code=0xbfffee18, perror_object=0xbfffee10) at interp.c:496
#23 0x000aeb43 in gs_interpret (pi_ctx_p=0x11018cc, pref=0xbfffed58, user_errors=1, 
pexit_code=0xbfffee18, perror_object=0xbfffee10) at interp.c:454
#24 0x000a2637 in gs_main_interpret (minst=0x1101878, pref=0xbfffed58, user_errors=1, 
pexit_code=0xbfffee18, perror_object=0xbfffee10) at imain.c:214
#25 0x000a318b in gs_main_run_string_end (minst=0x1101878, user_errors=1, 
pexit_code=0xbfffee18, perror_object=0xbfffee10) at imain.c:526
#26 0x000a3053 in gs_main_run_string_with_length (minst=0x1101878, str=0x110a308 
"<2e2f50616e746f6e652e706466>.runfile", length=36, user_errors=1, pexit_code=0xbfffee18, 
perror_object=0xbfffee10) at imain.c:484
#27 0x000a2fbb in gs_main_run_string (minst=0x1101878, str=0x110a308 
"<2e2f50616e746f6e652e706466>.runfile", user_errors=1, pexit_code=0xbfffee18, 
perror_object=0xbfffee10) at imain.c:466
#28 0x000a6078 in run_string (minst=0x1101878, str=0x110a308 
"<2e2f50616e746f6e652e706466>.runfile", options=3) at imainarg.c:798
#29 0x000a6028 in runarg (minst=0x1101878, pre=0x49e324 "", arg=0x1102248 "./Pantone.pdf", 
post=0x4a2725 ".runfile", options=3) at imainarg.c:788
#30 0x000a5d0b in argproc (minst=0x1101878, arg=0xbffff983 "./Pantone.pdf") at imainarg.c:723
#31 0x000a44ae in gs_main_init_with_args (minst=0x1101878, argc=5, argv=0xbffff868) at 
imainarg.c:207
#32 0x00001d9d in main (argc=5, argv=0xbffff868) at gs.c:77
(gdb) 

This is _probably_ a clist problem, but if it is a transparency issue, please
reassign to Michael

A quick view shows that pdf14_mark_fill_rectangle allocates arrays for 64 
color components and then tries to process 69 color compomnents (at the crash 
point pdf14_fill_rectangle::buf->n_chan == 69, pdf14_mark_fill_rectangle uses 
it to initialize num_comp that is used as the index limit for unpacking colors 
to the local arrays). So the crash mechanizm is C stack damage. Thus :

1. We definitely observe a bug in gdevp14.c which doesn't check for the real 
number of color components against the size of internal arrays.

2. The check first must happen somewhere before gdevp14.c, because other 
modules may have same problem. IMO the check must happen when adding spot 
colors to the color space, and report error about running over the hardcoded 
internal limit.

3. Both gdevp14 and color spaces are owned by Michael, so it should go to him.

4. I guess the problem first happens in r9071 (which is marked 
as "syntactically equivalent") because it actually changes the C stack layout 
with adding new functions. I guess older revisions also damage C stack, but 
the function return address is not overwritten so they occasionally work well.

5. IMO the key problem here is Ray's limitation for the number of color 
components. First we need to obtain a right answer for the key question, 
because other problems are mionor and easy fixing. Therefore I pass it to Ray. 

(Well in the past many times I suggested to replace color index with pointers, 
but it was declined, so probably now it worth to remind that once again).

Further information from the customer:

I changed my mind concerning the way GS should deal with this problem: "it should emit a warning 
'maximum number  of components reached', or something like that, and exit with an error. And not ignore 
the additional components, as initially suggested."

Also, the customer is concerned that I obsfucated the original issue in my comment #0; the problem he 
originally reported was that "all pixels of the yellow separation are at 100%" (I did mention that the colors 
were incorrect, but spent more time describing the seg fault).

This file now renders properly in the softmask branch.  The file has over 100 
sep colors, with overprinting, a soft mask and transparency group.  The colors 
that are mapped to the alternate color space (CMYK) will not overprint 
correctly -- i.e. the previously painted colors will be blown away.  This is 
also true with AR (see it with overprint preview on), as it is not possible to 
handle overprinting correctly once you have mapped to the alternate color 
space -- hence it is ignored. For completeness, I will attach a version of the 
file that does not have overprinting.

Created attachment 4767 [details]
Pantone_no_overprint.pdf

Comment on attachment 4767 [details]
Pantone_no_overprint.pdf

Attached file is the same file, but the spot colors are not overprinted the 50%
black

This renders correctly with the softmask branch.

This is fixed with the soft mask merge.  Note that Adobe Acrobrat Viewer 
renders the file incorrectly also due to the overprint issue described in the 
other comments.

Changing customer bugs that have been resolved more than a year ago to closed.