| Summary: | Careless casting when dealing with doubles causes off-by-one error | ||
|---|---|---|---|
| Product: | Ghostscript | Reporter: | Tim Waugh <twaugh> |
| Component: | Regression | Assignee: | Chris Liddell (chrisl) <chris.liddell> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | ||
| Priority: | P4 | ||
| Version: | 9.07 | ||
| Hardware: | PC | ||
| OS: | Linux | ||
| Customer: | Word Size: | --- | |
| Attachments: | ghostscript-trio-g.patch | ||
Assigning to Chris since he added the trio library. I have committed the patch - thank you. http://git.ghostscript.com/?p=ghostpdl.git;a=commitdiff;h=884b9e45 Once again, I feel inclined to express my fury at glibc for including the locales stuff in such a way that calls like sprintf() change functionality "under our feet" - it's just wrong........ |
Created attachment 10908 [details] ghostscript-trio-g.patch Noticed when diagnosing a pdfwrite issue on a ppc64 system. In TrioWriteDouble(), the number of digits for the integer part of the output is calculated as follows: 3162: integerNumber = trio_floor(number); [...] 3174: integerDigits += (int)TrioLogarithm(integerNumber, base); TrioLogarithm returns a long double: TRIO_PRIVATE trio_long_double_t TrioLogarithm TRIO_ARGS2((number, base), trio_long_double_t number, int base) { [...] result = trio_log10(number); [...] } (gdb) whatis result type = trio_long_double_t (gdb) whatis trio_long_double_t type = long double The problem can show up when 1000.0 < number < 1001.0. In this case, integerNumber is 1000, and TrioLogarithm returns (a floating point representation of) 3.0. However, simply casting that to int can lead to problems, depending on exactly how the floating point representation works. In this instance, casting to int gave the expression the value 2, not 3. The result of all this is that gs_sprintf("%s", 1000.184) can result in "000.184", not "1000.18". I've had success using a work-around that detects this off-by-one error and corrects it: see the attached patch. I'm not sure if this is the best approach.