Bug 694546 - devices/gdevpbm.c: likely undefined operation
Summary: devices/gdevpbm.c: likely undefined operation
Status: RESOLVED FIXED
Alias: None
Product: Ghostscript
Classification: Unclassified
Component: Other Driver (show other bugs)
Version: master
Hardware: PC Linux
: P4 minor
Assignee: Robin Watts
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-08-29 04:02 UTC by jsmeix
Modified: 2013-08-30 05:16 UTC (History)
1 user (show)

See Also:
Customer:
Word Size: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description jsmeix 2013-08-29 04:02:51 UTC
When compiling ghostscript-9.10rc1
in our openSUSE build system build fails with this message:
--------------------------------------------------------------------------
gcc  -DHAVE_MKSTEMP -DHAVE_FILE64 -DHAVE_FSEEKO -DHAVE_MKSTEMP64   -DHAVE_SETLOCALE   -DHAVE_BSWAP32 -DHAVE_BYTESWAP_
H -DHAVE_STRERROR -fPIC  -O2 -Wall -Wstrict-prototypes -Wundef -Wmissing-declarations -Wmissing-prototypes -Wwrite-strings -Wn
o-strict-aliasing -Wdeclaration-after-statement -fno-builtin -fno-common -DHAVE_STDINT_H=1 -DHAVE_DIRENT_H=1 -DHAVE_SYS_DIR_H=
1 -DHAVE_SYS_TIME_H=1 -DHAVE_SYS_TIMES_H=1 -DHAVE_INTTYPES_H=1 -DGX_COLOR_INDEX_TYPE="unsigned long long" -march=i586 -mtune=i
686 -fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -DGS_DEVS_
SHARED -DGS_DEVS_SHARED_DIR=\"/usr/lib/ghostscript/9.10\"  -I./obj -I./base -I./obj -I./devices  -I./devices/vector  -o ./obj/
gdevpbm.o -c ./devices/gdevpbm.c
./devices/gdevpbm.c: In function 'pnmcmyk_print_page':
./devices/gdevpbm.c:1097: warning: operation on 'pcmyk' may be undefined
./devices/gdevpbm.c:1097: warning: operation on 'pcmyk' may be undefined

[...]

I: Program causes undefined operation
   (likely same variable used twice and post/pre incremented
   in the same expression).
   e.g. x = x++; Split it in two operations.
E: ghostscript sequence-point ./devices/gdevpbm.c:1097
--------------------------------------------------------------------------
Our openSUSE build system has special checks for certain
compiler warnings that we regard as severe to let the build fail.

In devices/gdevpbm.c:1097 I have (from "cat -n" output):
--------------------------------------------------------------------------
  1093                  cmy = ( ((255 - *pcmyk++) * lum_red_weight) +
  1094                          ((255 - *pcmyk++) * lum_green_weight) +
  1095                          ((255 - *pcmyk++) * lum_blue_weight) +
  1096                          (lum_all_weights / 2))
  1097                          / lum_all_weights;
  1098  
  1099                  k = *pcmyk++;
--------------------------------------------------------------------------

I do not have the knowledge to decide if this code is really correct
but on a first glance the many "*pcmyk++" therein look somehow strange
at least to me.

Therefore - to be on the safe side - I would like if you could have
a look and verify if it is really correct.
Comment 1 Ken Sharp 2013-08-29 05:00:48 UTC
(In reply to comment #0)

> I do not have the knowledge to decide if this code is really correct
> but on a first glance the many "*pcmyk++" therein look somehow strange
> at least to me.
> 
> Therefore - to be on the safe side - I would like if you could have
> a look and verify if it is really correct.

Yes its correct.
Comment 2 jsmeix 2013-08-29 05:11:00 UTC
This means to make the compiler happy I can change it to:
------------------------------------------------------------------------
                cmy = ( ((255 - *pcmyk) * lum_red_weight) +
                        ((255 - *pcmyk) * lum_green_weight) +
                        ((255 - *pcmyk) * lum_blue_weight) +
                        (lum_all_weights / 2))
                        / lum_all_weights;
                pcmyk++;
                pcmyk++;
                pcmyk++;
------------------------------------------------------------------------

Or is actually this meant:
------------------------------------------------------------------------
                cmy = ((255 - *pcmyk++) * lum_red_weight);
                cmy += ((255 - *pcmyk++) * lum_green_weight);
                cmy += ((255 - *pcmyk++) * lum_blue_weight);
                cmy += (lum_all_weights / 2);
                cmy /= lum_all_weights;
------------------------------------------------------------------------
Comment 3 Robin Watts 2013-08-29 05:16:07 UTC
Ken is wrong - the code is definitely wrong.

(In reply to comment #2)
> This means to make the compiler happy I can change it to:
> ------------------------------------------------------------------------
>                 cmy = ( ((255 - *pcmyk) * lum_red_weight) +
>                         ((255 - *pcmyk) * lum_green_weight) +
>                         ((255 - *pcmyk) * lum_blue_weight) +
>                         (lum_all_weights / 2))
>                         / lum_all_weights;
>                 pcmyk++;
>                 pcmyk++;
>                 pcmyk++;
> ------------------------------------------------------------------------

Your code would be even worse. A correct version would be:

>                 cmy  = ((255 - *pcmyk++) * lum_red_weight);
>                 cmy += ((255 - *pcmyk++) * lum_green_weight);
>                 cmy += ((255 - *pcmyk++) * lum_blue_weight);
>                 cmy  = (cmy + (lum_all_weights / 2)) / lum_all_weights;

(subject to cmy being the right type).

I will get this fixed.
Comment 4 Robin Watts 2013-08-29 09:50:48 UTC
Fixed in:

commit 621b010d1a6e12f7754508fee8a860345b4f84df
Author: Robin Watts <robin.watts@artifex.com>
Date:   Thu Aug 29 16:47:29 2013 +0100

    Bug 694546: Fix undefined behaviour in gdevpbm.

    Thanks to jsmeix for reporting this.

If you know of any other such warnings, please let us know! Thanks.
Comment 5 jsmeix 2013-08-30 02:26:10 UTC
I addition to our autometed checks in our openSUSE build system
I searched for "undefined" in all our build log files for all
our various build targets (openSUSE_Factory openSUSE_12.3 openSUSE_12.2
openSUSE_12.1 SLE_11_SP3 SLE_11_SP2 SLE_11_SP1 SLE_11 where each
is built both for 32 bit i586 and 64 bit x86_64 architecture.
In our various build targets we use several GCC versions:
gcc-4.8 (openSUSE_Factory)
gcc-4.7 (openSUSE_12.3 openSUSE_12.2)
gcc-4.6 (openSUSE_12.1)
gcc-4.3 (SLE_11_SP3 SLE_11_SP2 SLE_11_SP1 SLE_11)

I found nothing.

This means with your commit 621b010d1a6e12f7754508fee8a860345b4f84df
our GCC versions do no longer report something as undefined.


Robin Watts,

I have a question regarding my comment#2
and your commit 621b010d1a6e12f7754508fee8a860345b4f84df
versus your comment#3:

Your commit 621b010d1a6e12f7754508fee8a860345b4f84df
is what I suggested as "is actually this meant" in comment#2
but in your comment#3 your "correct version" is different
any there you wrote "Your code would be even worse" and
"subject to cmy being the right type".

Now I am puzzled.

I am not an expert for those subtle details in C
but I think that my "make the compiler happy" code
is not worse but explicit code for what actually happens.
As far as I understand it "cmy = ... *pcmyk++ ... ;" results
that pcmyk gets incremented at the next sequence point and
as far as I see the next sequence point here is ';'.

Furthermore I think that my "is actually this meant" code
(and your commit 621b010d1a6e12f7754508fee8a860345b4f84df)
and your "correct version" result the same value in cmy.

I would appreciate a short note whether I am right or wrong.
Many thanks in advance!
Comment 6 Robin Watts 2013-08-30 04:43:41 UTC
(In reply to comment #5)
> This means with your commit 621b010d1a6e12f7754508fee8a860345b4f84df
> our GCC versions do no longer report something as undefined.

Thanks. I checked the warnings here from our build systems, and that was the only example I could find flagged up too.

> I am not an expert for those subtle details in C
> but I think that my "make the compiler happy" code
> is not worse but explicit code for what actually happens.

OK, so the original code was:

  cmy = ( ((255 - *pcmyk++) * lum_red_weight) +
          ((255 - *pcmyk++) * lum_green_weight) +
          ((255 - *pcmyk++) * lum_blue_weight) +
          (lum_all_weights / 2))
        / lum_all_weights;

Clearly the author expected the code to be evaluated from left to right, where each *pcmyk++ would be executed before the next. This would result in the R value being read, then the G value, then the B value, and the pointer being left pointing to the next B value. So the author hoped that the code would actually do:

  int R = *pcmyk++;
  int G = *pcmyk++;
  int B = *pcmyk++;
  cmy = ( ((255 - R) * lum_red_weight) +
          ((255 - G) * lum_green_weight) +
          ((255 - B) * lum_blue_weight) +
          (lum_all_weights / 2))
        / lum_all_weights;

You suggested that we could use:

                cmy = ( ((255 - *pcmyk) * lum_red_weight) +
                        ((255 - *pcmyk) * lum_green_weight) +
                        ((255 - *pcmyk) * lum_blue_weight) +
                        (lum_all_weights / 2))
                        / lum_all_weights;
                pcmyk++;
                pcmyk++;
                pcmyk++;

which would actually have been the same as doing:

  int R = *pcmyk;
  cmy = ( ((255 - R) * lum_red_weight) +
          ((255 - R) * lum_green_weight) +
          ((255 - R) * lum_blue_weight) +
          (lum_all_weights / 2))
        / lum_all_weights;
  pcmyk += 3;

Clearly that's not what the original author intended as the G and B values are completely ignored.

> I would appreciate a short note whether I am right or wrong.
> Many thanks in advance!

Hopefully this makes it clear. Thanks again for bringing this to our attention.
Comment 7 jsmeix 2013-08-30 05:16:17 UTC
We (i.e. Robin Watts and I) have the same understanding of the code.

The only thing that is perhaps not yet fully clear is
how my comment#2 was meant.

My "make the compiler happy" code was in response to
Ken Sharp's statement that the original code is correct.

If the original code was correct,
then I could change it to my "make the compiler happy" code.

I wrote the "make the compiler happy" code to make it
explicitly clear what the original code actually does.

Because I assumed that the original code was not correct
I added the "is actually this meant" code in comment#2

My "is actually this meant" code was meant as suggestion
what you may use (if the original code is not correct) and
in your commit 621b010d1a6e12f7754508fee8a860345b4f84df
you use it so finally everything is now clear and resolved.

Many thanks for your time to explain it to me.