Bug 695235 - Careless casting when dealing with doubles causes off-by-one error
Summary: Careless casting when dealing with doubles causes off-by-one error
Status: RESOLVED FIXED
Alias: None
Product: Ghostscript
Classification: Unclassified
Component: Regression (show other bugs)
Version: 9.07
Hardware: PC Linux
: P4 normal
Assignee: Chris Liddell (chrisl)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-05-15 03:17 UTC by Tim Waugh
Modified: 2014-09-02 01:46 UTC (History)
0 users

See Also:
Customer:
Word Size: ---


Attachments
ghostscript-trio-g.patch (964 bytes, patch)
2014-05-15 03:17 UTC, Tim Waugh
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Waugh 2014-05-15 03:17:06 UTC
Created attachment 10908 [details]
ghostscript-trio-g.patch

Noticed when diagnosing a pdfwrite issue on a ppc64 system.

In TrioWriteDouble(), the number of digits for the integer part of the output is calculated as follows:

3162:  integerNumber = trio_floor(number);
[...]
3174:      integerDigits += (int)TrioLogarithm(integerNumber, base);

TrioLogarithm returns a long double:

TRIO_PRIVATE trio_long_double_t
TrioLogarithm
TRIO_ARGS2((number, base),
           trio_long_double_t number,
           int base)
{
[...]
          result = trio_log10(number);
[...]
}

(gdb) whatis result
type = trio_long_double_t
(gdb) whatis trio_long_double_t
type = long double

The problem can show up when 1000.0 < number < 1001.0. In this case, integerNumber is 1000, and TrioLogarithm returns (a floating point representation of) 3.0.

However, simply casting that to int can lead to problems, depending on exactly how the floating point representation works. In this instance, casting to int gave the expression the value 2, not 3.

The result of all this is that gs_sprintf("%s", 1000.184) can result in "000.184", not "1000.18".

I've had success using a work-around that detects this off-by-one error and corrects it: see the attached patch. I'm not sure if this is the best approach.
Comment 1 Marcos H. Woehrmann 2014-05-29 16:54:18 UTC
Assigning to Chris since he added the trio library.
Comment 2 Chris Liddell (chrisl) 2014-09-02 01:46:40 UTC
I have committed the patch - thank you.

http://git.ghostscript.com/?p=ghostpdl.git;a=commitdiff;h=884b9e45


Once again, I feel inclined to express my fury at glibc for including the locales stuff in such a way that calls like sprintf() change functionality "under our feet" - it's just wrong........