The customer reports and I've confirmed that Ghostscript uses a large (~2 gigs) amount of memory converting the attached file at 1600 DPI. This happens both with gs8.60 and gshead (r8268). The command line I used for testing: bin/gs -sDEVICE=ppmraw -sOutputFile=test.ppm ./M2M_CP_WAN_VPN.pdf
Created attachment 3432 [details] M2M_CP_WAN_VPN.pdf
The problem is that the 'maskbuf' is an element of the 'pdf14_ctx' structure, and is not "pushed/popped" by the transparency group begin/end. The condition can be detected by setting a conditional breakpoint in 'pdf14_pop_transparency_mask' at "ctx->maskbuf = tos;" (line 930 currently) when "ctx->maskbuf != NULL". Overwriting a non-NULL ctx->maskbuf means that the pointer to this (previous) maskbuf will be lost.
Changing assignment because I'm working on similar bugs.
The problem here is that clist interpreter skips a transparency group for some band, so that a mask immediately follows another mask. It doesn't happen in the PS interpreter. This info may be obtained when running with -ZvL! . All the old code, Tor's code, and my last patch assume that such sequence can't happen. Rather I think this case has to be allowed, I'd like to understand first why it happens with the current code. IMO it should not, and the reason lies definitely in the clist writer. Changes to gdevp14.c don't fix the problem but hide it.
The comment #4 is partially incorrect. The words "clist interpreter skips a transparency group, so that" to be removed. As mentioned in Comment #4, old revisions assume that each mask must be followed by a group. Likely this was a base assumption for the original design for transparency code. This assumption is not true in following cases : 1. A PDF command stream executes "/d1 gs /d2 gs" where d1 and d2 both define an untrivial SMask (not important whether they are same mask or different masks). In this case the second mask replaces the first one, and the first one to be released. 2. A transparent group include a soft mask image. This case happens in the test document for this bug. According to PLRM, the image's soft mask must *temporary* replace the groups soft mask. In this case the first mask must not be released. To repair this design hole we need 2 mask fields in pdf14_ctx_s : one for the group mask, and the other for the soft image's mask. They may be not NULL simultaneousely and must behave as a 2-element stack. Fortunately the ps interpreter distinguishes .begintransparencymaskgroupand .begintransparencymaskimage, so we don't need to patch the PDF interpreter. But this information (the distinguishing) is later missed in the PS interpreter and it does not come to the device interface. Due to that it can't be accounted by the clist interpreter and by the pdf14 device. Thus any local patch, which is restricted with gdevp14.c, can't fix the bug. A right fix will include changes to gstrans.h, gstrans.c, gdevp14.h, gdevp14.c, ztrans.c . Besides all that, the document uses overprint with a transparency. I'm not clear how it was intended to work in the original design, because each of them wants to install its own compositor devices. This needs a further analyzis.
My last comment doesn't account .discardtransparencymask and .discardtransparencygroup . I think they need additional chamges about soft mask images (the case (2) in comment #2). I recommend to make a separate patch for them.
A correction to Comment #5 : The statement : "we need 2 mask fields in pdf14_ctx_s : one for the group mask, and the other for the soft image's mask" to be removed. One field is enough. .begintransparencymaskgroup and .begintransparencymaskimage must pass an additional bit to infrom whether the topmost pdf14_ctx_s:stack element to be replaced with the new mask, or the new mask to be added as a new stack element.
Patch to HEAD : http://ghostscript.com/pipermail/gs-cvs/2008-January/008088.html It fixes the memory leak, but the document demonstrates another problem : a shadow of a text is not rendered. Need a further investigation, so don't close the bug now.
More patches to HEAD : http://ghostscript.com/pipermail/gs-cvs/2008-January/008089.html http://ghostscript.com/pipermail/gs-cvs/2008-January/008090.html See also a patch above. Now closing.
Patch for 8.60 needed by customer is attached to bug 689532 that also fixes this problem in 8.60. Adapted from Igor's patches, but without all of the unrelated changes, and including the patch found for a leak with the file for bug 589534 that was committed to trunk as rev 8704.