Created attachment 1589 [details]
File showing problem

Simple example continaing the gentoo logo.

Core dump obtained by running gs -sDEVICE=bitcmyk -r4000 -sOutputFile=x test.ps


larger image would result in core dump at lower resolution, but is harder to
upload

Comment on attachment 1589 [details]
File showing problem

When downloaded & checked bug wasn't reproduced, will upload more suitable
example!

Created attachment 1590 [details]
File actually showing problem

the command
gs -sDEVICE=bitcmyk -r1800 -sOutputFile=x stm07024.pdf

really does show the problem :)

Created attachment 1594 [details]
patch

The immediate cause of the problem is clear. On many 64-bit platforms 
  sizeof(int) == 4
  sizeof(unsigned) == 4
  sizeof(void *) == 8

In our case "ptc->raster" is unsigned int, "by * ptc->raster" is a 4-byte
unsigned value. When "by" is negative the product is a large number, which
makes an invalid pointer when added with "ptc->tdata".

I don't know whether, the negative "by" value is valid. After the fix GS
finishes without errors, but generates a PS file that is somewhat different
from the file generated on x86 box.

That looks like it's fixed the problem thanks Alex.

How many more places do you think this could be an issue, as I guess it's a
general problem with unsigned & negative int multiplications on this platform?
And is there anyway we can help review the code?

Alex,

Where is the seg fault occuring?  Is it in init_tile_cursor or at some other 
routine?  We have had a report of an IBM compiler running under AIX that did 
not like bad pointer values even if the pointer not used.

I am bothered by your statement that you get a different output (with your 
patch) for the AMD64 verus x86 machines.  Do you have a way of comparing bit 
files?  If so then what are the apparent differences between the files? 
I tried different combinations of casting on m x86 and I do not see any 
differences.  (I do not have an AMD64.)  Do you see different output for other 
resolutions that do not seg fault?

I did do some checking and it is true that the value of by is negative (i.e. 
it is -1) on some ocassions.  This only seems to occur with rectangles that 
are out of the band to be imaged.  I add the followint check into
gx_dc_ht_colored_fill_rectangle and it prevents by from being negative (at 
least with this file at 1800 dpi).

diff -u -r1.16 gxcht.c
--- src/gxcht.c 19 Jun 2005 20:45:57 -0000      1.16
+++ src/gxcht.c 4 Aug 2005 22:03:38 -0000
@@ -610,6 +610,8 @@

     if (w <= 0 || h <= 0)
        return 0;
+    if (x + w <= 0 || y + h <= 0)
+       return 0;
     if ((w | h) >= 16) {
        /* It's worth taking the trouble to check the clipping box. */
        gs_fixed_rect cbox;

It appears that gx_dc_ht_colored_fill_rectangle only verifies that 'large' 
rectangles (> 16 pixels) may not be imaged.  Judging from this problem, it 
would seem that at least some sort of check to prevent references outside of 
the image buffer, etc. may be needed.  I think that 'small' rectangles will be 
tossed later in the pipeline (i.e. in the device's copy_mono or copy_color 
routines.)  Thus I would not expect any differences in output.

Your patch is safe and I do not seen any reason not to commit it.  I would 
like at least a partial explanation about the differences between x86 and 
AMD64 output.

The code produces different results on 32bit x84 to x86_64 I think due to the 
pointer wrapping around, or the compiler deciding it must be in a particular 
range (ie int not unsigned int).

The code below is a trivial example demonstrating the problem.  

#include <stdio.h>
int main (void) {
        char text[]             = "abcdefghijklmnopqrstuvwxyz";
        int x                   = -1;
        unsigned int y  = 8;

        printf("char is: '%c'\n", *(text + 25 + (x*(int)y))); // works on both
        printf("char is: '%c'\n", *(text + 25 + (x*y)));      // 32bit only
        return 0;
}

On the 32bit platform you're either adding -8 & going back 8 bytes, or adding 
2^32-8, wrapping around & therefore effectively going back 8 bytes.
On the 64bit platform, I think we attempt to move forward 2^32-8 bytes (which 
is valid in 64bit platforms so long as the position you're at is <2^64-2^32) 
and as this is well beyond the allocated space, (for text in the example or ptc-
>tdata in gxcht.c), we segmentation fault.

I understand the issues about signed versus unsigned and 32 versus 64 bit 
pointers.  What I do not understand why there is an effect in this particular 
case in Ghostscript.  I think that the situations in which the value of 'by' 
is negative ghostscript will later clip the values since the rectangle being 
imaged is outside of the band.

Does this same problem appear with another 4 bit cmyk device like pkmraw.  If 
so, do the output file differ between an x86 machine versus the AMD64?

COmment #4 from Alex implies that there is still a problem with the output on 
the AMD64 device.  Is this true?  If I do not get some sort of feedback, I am 
going to close this bug report since I am unable to do any testing.

Since I am not getting any feed back about if there was a remaining problem 
with this test file.  I am closing this report.  I did commit Alex's patch.

http://ghostscript.com/pipermail/gs-cvs/2005-August/005686.html

*** Bug 688486 has been marked as a duplicate of this bug. ***

*** Bug 688642 has been marked as a duplicate of this bug. ***