Ghostscript cannot set a /SMask in the graphic state using an image with /SMask attribute. instead, it uses the image and ignores /SMask attribute. The above description and the attached test is courtesy of Alex, from Bug 691157. Since Bug 691157 exposed and was diagnosed as being a separate issue, which is now fixed, I've opted to create a new bug to track this problem. I'm assigning to Michael to investigate - if it looks like this is another interpreter issue, then you can assign back to me.
Created attachment 6966 [details] test job for smask problem
Setting customer priority (190 is P1).
This is a strange one. There are multiple softmasks being created where I would think there should only be one. This is in the non-clist case.
So I am counting a total of 4 soft masks created for this thing during rendering. The mask is getting mixed up as it ends up divided between a hard mask and an alpha plane at some point. I will dig into the file tomorrow to understand what weird thing is going on in it.
Testing a fix for this. The issue was that the soft mask contains a soft mask. In this case, when the soft mask group is popped, it really needs to blend in the alpha portion of the current transparency stack group to create the mask buffer.
Fixed with rev 11929