Bug 688252 - Signal 11 in set_color_ht_le_4 on AMD64
Summary: Signal 11 in set_color_ht_le_4 on AMD64
Alias: None
Product: Ghostscript
Classification: Unclassified
Component: General (show other bugs)
Version: 8.51
Hardware: PC Linux
: P3 normal
Assignee: Dan Coby
QA Contact: Bug traffic
: 688486 688642 (view as bug list)
Depends on:
Reported: 2005-08-01 07:13 UTC by Alistair Braidwood
Modified: 2008-12-19 08:31 UTC (History)
3 users (show)

See Also:
Customer: 632
Word Size: ---

File showing problem (6.28 MB, application/postscript)
2005-08-01 07:26 UTC, Alistair Braidwood
patch (652 bytes, patch)
2005-08-03 07:22 UTC, Alex Cherepanov
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alistair Braidwood 2005-08-01 07:13:34 UTC
compiled with gcc 3.4.4 on AMD64 with -m64 flag, the bitcmyk device core dumps.

The failure is dependent on resolution and size of image in the input postscript/pdf

I think this is NOT related to http://bugs.ghostscript.com/show_bug.cgi?id=688247
Comment 1 Alistair Braidwood 2005-08-01 07:26:33 UTC
Created attachment 1589 [details]
File showing problem

Simple example continaing the gentoo logo.

Core dump obtained by running gs -sDEVICE=bitcmyk -r4000 -sOutputFile=x test.ps

larger image would result in core dump at lower resolution, but is harder to
Comment 2 Alistair Braidwood 2005-08-01 07:34:35 UTC
Comment on attachment 1589 [details]
File showing problem

When downloaded & checked bug wasn't reproduced, will upload more suitable
Comment 3 Alistair Braidwood 2005-08-01 07:38:14 UTC
Created attachment 1590 [details]
File actually showing problem

the command
gs -sDEVICE=bitcmyk -r1800 -sOutputFile=x stm07024.pdf

really does show the problem :)
Comment 4 Alex Cherepanov 2005-08-03 07:22:12 UTC
Created attachment 1594 [details]

The immediate cause of the problem is clear. On many 64-bit platforms 
  sizeof(int) == 4
  sizeof(unsigned) == 4
  sizeof(void *) == 8

In our case "ptc->raster" is unsigned int, "by * ptc->raster" is a 4-byte
unsigned value. When "by" is negative the product is a large number, which
makes an invalid pointer when added with "ptc->tdata".

I don't know whether, the negative "by" value is valid. After the fix GS
finishes without errors, but generates a PS file that is somewhat different
from the file generated on x86 box.
Comment 5 Alistair Braidwood 2005-08-03 10:00:21 UTC
That looks like it's fixed the problem thanks Alex.

How many more places do you think this could be an issue, as I guess it's a
general problem with unsigned & negative int multiplications on this platform?
And is there anyway we can help review the code?
Comment 6 Dan Coby 2005-08-04 15:26:13 UTC

Where is the seg fault occuring?  Is it in init_tile_cursor or at some other 
routine?  We have had a report of an IBM compiler running under AIX that did 
not like bad pointer values even if the pointer not used.

I am bothered by your statement that you get a different output (with your 
patch) for the AMD64 verus x86 machines.  Do you have a way of comparing bit 
files?  If so then what are the apparent differences between the files? 
I tried different combinations of casting on m x86 and I do not see any 
differences.  (I do not have an AMD64.)  Do you see different output for other 
resolutions that do not seg fault?

I did do some checking and it is true that the value of by is negative (i.e. 
it is -1) on some ocassions.  This only seems to occur with rectangles that 
are out of the band to be imaged.  I add the followint check into
gx_dc_ht_colored_fill_rectangle and it prevents by from being negative (at 
least with this file at 1800 dpi).

diff -u -r1.16 gxcht.c
--- src/gxcht.c 19 Jun 2005 20:45:57 -0000      1.16
+++ src/gxcht.c 4 Aug 2005 22:03:38 -0000
@@ -610,6 +610,8 @@

     if (w <= 0 || h <= 0)
        return 0;
+    if (x + w <= 0 || y + h <= 0)
+       return 0;
     if ((w | h) >= 16) {
        /* It's worth taking the trouble to check the clipping box. */
        gs_fixed_rect cbox;

It appears that gx_dc_ht_colored_fill_rectangle only verifies that 'large' 
rectangles (> 16 pixels) may not be imaged.  Judging from this problem, it 
would seem that at least some sort of check to prevent references outside of 
the image buffer, etc. may be needed.  I think that 'small' rectangles will be 
tossed later in the pipeline (i.e. in the device's copy_mono or copy_color 
routines.)  Thus I would not expect any differences in output.

Your patch is safe and I do not seen any reason not to commit it.  I would 
like at least a partial explanation about the differences between x86 and 
AMD64 output.
Comment 7 Alistair Braidwood 2005-08-05 01:26:25 UTC
The code produces different results on 32bit x84 to x86_64 I think due to the 
pointer wrapping around, or the compiler deciding it must be in a particular 
range (ie int not unsigned int).

The code below is a trivial example demonstrating the problem.  

#include <stdio.h>
int main (void) {
        char text[]             = "abcdefghijklmnopqrstuvwxyz";
        int x                   = -1;
        unsigned int y  = 8;

        printf("char is: '%c'\n", *(text + 25 + (x*(int)y))); // works on both
        printf("char is: '%c'\n", *(text + 25 + (x*y)));      // 32bit only
        return 0;

On the 32bit platform you're either adding -8 & going back 8 bytes, or adding 
2^32-8, wrapping around & therefore effectively going back 8 bytes.
On the 64bit platform, I think we attempt to move forward 2^32-8 bytes (which 
is valid in 64bit platforms so long as the position you're at is <2^64-2^32) 
and as this is well beyond the allocated space, (for text in the example or ptc-
>tdata in gxcht.c), we segmentation fault.
Comment 8 Dan Coby 2005-08-05 09:35:23 UTC
I understand the issues about signed versus unsigned and 32 versus 64 bit 
pointers.  What I do not understand why there is an effect in this particular 
case in Ghostscript.  I think that the situations in which the value of 'by' 
is negative ghostscript will later clip the values since the rectangle being 
imaged is outside of the band.
Comment 9 Dan Coby 2005-08-09 16:14:13 UTC
Does this same problem appear with another 4 bit cmyk device like pkmraw.  If 
so, do the output file differ between an x86 machine versus the AMD64?
Comment 10 Dan Coby 2005-08-18 09:50:57 UTC
COmment #4 from Alex implies that there is still a problem with the output on 
the AMD64 device.  Is this true?  If I do not get some sort of feedback, I am 
going to close this bug report since I am unable to do any testing.
Comment 11 Dan Coby 2005-08-30 20:46:05 UTC
Since I am not getting any feed back about if there was a remaining problem 
with this test file.  I am closing this report.  I did commit Alex's patch.

Comment 12 Alex Cherepanov 2006-01-22 22:13:28 UTC
*** Bug 688486 has been marked as a duplicate of this bug. ***
Comment 13 Konstantin Belyanski 2006-04-11 03:05:13 UTC
*** Bug 688642 has been marked as a duplicate of this bug. ***