Created attachment 11933 [details] Original document When freetype font renderer is used (the default) some color issues occur for certain documents. In the attached example the original ps document (printed powerpoint) is converted to pcl6 using pxlcolor device, then back to pdf - the resulting colors of some texts do not match. It is also observed when printing a PCL document on some printer models. With the -dDisableFAPI=true option this issue does not happen.
Created attachment 11934 [details] Mismatched output
(In reply to Dmitry Pankratov from comment #0) > When freetype font renderer is used (the default) some color issues occur > for certain documents. Since FreeType isn't responsible for the colour which is used to fill the outline this seems unlikely, however we will look at it, but see below. > In the attached example the original ps document (printed powerpoint) is > converted to pcl6 using pxlcolor device, And the command line used was ? > then back to pdf - And again, what was the command line which was used ? Why bother converting to PDF and adding another conversion step ? If you want to view the PXL, use GhostPCL. > the resulting > colors of some texts do not match. Specifically *which* text ? This is a 12 page document, you need to state where the problem is in the document. [later] This looks to me like a problem with the pxlcolor device, the text involved is large in every case and I suspect that the difference is rather more to do with whether the text is drawn as text, or as a filled path. I can believe that is different with the two font systems. It looks to me like the pxlcolor device is not applying the current colour to the 'text' and instead defaults to black. So not Font API at all.
> And the command line used was ? gs -sDEVICE=pxlcolor -dBATCH -dNOPAUSE -dNOINTERPOLATE -dCompressMode=2 -sOutputFile=output.pcl orig.ps (tried with other compress modes as well) > And again, what was the command line which was used ? Why bother converting to > PDF and adding another conversion step ? If you want to view the PXL, use GhostPCL. GhostPCL was used to convert it to pdf (just to view and attach it; not strictly necessary indeed): gpcl6 -sDEVICE=pdfwrite -dNBATCH -dNOPAUSE -sOutputFile=output.pdf output.pcl > Specifically *which* text ? This is a 12 page document, you need to state where the problem is in the document. Sorry, should have been more specific. Page 1 - all text is rendered black instead of blue and red. Page 2 - one letter is black (all must be red), Page 11 - all text is black (should be colorized), Page 12 - one letter is black instead of red). I have attached an original document and the produced output. Actually it can be seen already on the first page.
Can you attach the two versions of PCLXL output also. It is possible that it is one of the known rasterop problems.
(In reply to Hin-Tak Leung from comment #4) > Can you attach the two versions of PCLXL output also. > > It is possible that it is one of the known rasterop problems. Since its starting from PostScript, and creating the PXL output using the pxlcolor device, it can't possibly contain rasterops. And even if it did, the PXL interpreter would render the PXL 'correctly' which it does not do.
Created attachment 11936 [details] Bad PCL output a PCL output with bad colors
Created attachment 11937 [details] Good PCL output A PCL output produced with -dDisableFAPI=true
Viewing both attachments with pcl6 (9.15) on X11, both are correct. This seems to be back in the pdfwriter territory.
The strange thing is that when PCL is sent to some real printers the paper output is ok on Xerox but is printed wrongly on Ricoh devices.
Yes, just running pcl6 -sDEVICE=pdfwrite -sOutputFile=a.pdf ... on attachment 11936 [details] and 11937, 11936 generates a pdf with the wrong color on the first page. You can just compare how the first page looks on X11 /screen vs how the first page of the output pdf looks. It is back to pdfwriter and Ken, I think.
(In reply to Dmitry Pankratov from comment #9) > The strange thing is that when PCL is sent to some real printers the paper > output is ok on Xerox but is printed wrongly on Ricoh devices. The latter is probably a firmware bug with some ricoh printers - you might actually want to mention the actual model because George of Ricoh may be interested.
(In reply to Ken Sharp from comment #5) > (In reply to Hin-Tak Leung from comment #4) > > Can you attach the two versions of PCLXL output also. > > > > It is possible that it is one of the known rasterop problems. > > Since its starting from PostScript, and creating the PXL output using the > pxlcolor device, it can't possibly contain rasterops. And even if it did, > the PXL interpreter would render the PXL 'correctly' which it does not do. I mean PCL/PCLXL containing rasterops coonverts to raster outputs fine, but do not convert well to pdf - that's a known problem for years. Indeed, the PXL interpreter do render 11936 correctly to screen, just not to pdf.
(In reply to Hin-Tak Leung from comment #11) > (In reply to Dmitry Pankratov from comment #9) > > The strange thing is that when PCL is sent to some real printers the paper > > output is ok on Xerox but is printed wrongly on Ricoh devices. > > The latter is probably a firmware bug with some ricoh printers - you might > actually want to mention the actual model because George of Ricoh may be > interested. Most of the models we tried on Ricoh devices do have this issue (for example MPC3002); it is very similar (bot not always 100% identical) to the problem with pdfwrite.
Both PCLXL files contains ROP3's. 11936 uses ROP3 136 in addition to the 3 ( 204, 240, 252) which 11937 already uses.
(In reply to Hin-Tak Leung from comment #8) > Viewing both attachments with pcl6 (9.15) on X11, both are correct. This > seems to be back in the pdfwriter territory. This is not what I see. Running the current PXL code with the supplied files and going to the display device I see that the two files render differently. The 'bad' file produces the same incorrect output as is reported against pdfwrite. (In reply to Hin-Tak Leung from comment #10) > Yes, just running > > pcl6 -sDEVICE=pdfwrite -sOutputFile=a.pdf ... > > on attachment 11936 [details] and 11937, 11936 generates a pdf with the > wrong color on the first page. You can just compare how the first page looks > on X11 /screen vs how the first page of the output pdf looks. > > It is back to pdfwriter and Ken, I think. No. Running the two PXL files to the display device (on windows) instead of sending to pdfwrite, I see *exactly* the same problems. However, if I run to tiff24nc, the output is correct, so perhaps the problem is in the PXL interpreter which is operating differently depending on the colour space of the device. Weirdly if I run the output to tiff32nc the output is quite broken. So it seems there are ROPS in the file. Why this should be so starting from PostScript I cannot fathom, but it puts the problem squarely in the pxlcolor device again I would say.
The pxlcolor driver can use ROP3's for pattern fill's, among other usages; I don't think PS vs pdf input upstream is relevant for that usage, at least. Just for the chance that it is a recent regression (as I was using pcl6 9.15 to view), viewing 11936 with latest-ish HEAD gpcl6 on X11 gives correct color too. So It seems that both the windows display driver and pdfwriter driver are broken, if it doesn't display correctly on windows; or the PXL interpeter is not behaving for more than 1 drivers.
(In reply to Hin-Tak Leung from comment #16) > The pxlcolor driver can use ROP3's for pattern fill's, among other usages; I > don't think PS vs pdf input upstream is relevant for that usage, at least. While the device might reasonably choose to use a language feature to represent objects (eg patterns) it clearly isn't essential, because some of the text on page 2 doesn't do so. Note that on the second page just one character is incorrect, all the others are correct. If we use the old font engine instead of FreeType then all the characters are correct. Obviously there is some difference in the treatment of these characters (possibly text vs path), and IMO its unreasonable that the pxlcolor device is treating the two situations differently. > So It seems that both the windows display driver and pdfwriter driver are > broken, I disagree most strongly with this statement. ROPs cannot be represented in PDF (or PostScript), so we choose to ignore them. That's not 'broken' its a design decision. The alternative would be to render the content to a bitmap and 'wrap' that as a PDF, something we have considered on several occasions. Of course, users often complain about that, because the resulting file size is large. You can't have it both ways though. I have no idea why the Windows display device doesn't deal with these ROPs, if in fact this is the problem. Note that the tiff32nc output was also (wildly) incorrect, Given that I've only tried 4 devices and 3 of them gave incorrect output, that's not a great track record. > if it doesn't display correctly on windows; or the PXL interpeter is > not behaving for more than 1 drivers. Apparently it doesn't work correctly on some printers either (see comment #9). While you might say those Ricoh printers are 'broken', when we encounter similar problems in the PostScript world we modify the output of ps2write in order to work around the problems. Since whatever is being done by pxlcolor is clearly not essential to its operation (because if we don't use FreeType it doesn't happen, and doesn't happen with all the text anyway) it would obviously be an improvement if it didn't do it at all. The problem needs to be addressed in the pxlcolor device. And with that I'm bowing out of this conversation and leaving it to Henry.
while the pxl standard has the concept of font and text, the pxldriver does not use those concepts - everything is graphics. Howeve, pixels vs paths and fills are treated differently, as they should be, since that's what vector devices do. fills are not blocks of pixels. It is also known that non-HP printers don't necessarily implement every part of an HP spec, nor correctly all the time. It is only of worry if an HP printer does not print the file as intended. The rest is 3rd party printer firmware bug. it is worth noting that the two files differ by nearly 20% in size. And quite obviously with/without freetype uses the driver(s) differently.
As for why rop3 affect different part of texts different, I can think of one reason: interaction with bandmode rendering of the input. The pxl driver does not have knowledge of graphic states outside the current band. So you can probably get a wholepage to be all correcr or all wrong, by setting a large MaxBitmap to render whole page to pxl, instead of bands. As a guess.
The bmax_LARGE constant from base/gsfont.c somehow affects the output - smaller values making it better (but not perfect) and larger values render the output even worse (more characters are black). I was bisecting the sources and found out this relation, might be helpful for troubleshooting.
It seems to me that it's fairly easy to see where the blame lies. Do rendering devices (TIFF, png, pnm etc) render the Postscript correctly? Does pdfwrite convert the Postscript to PDF correctly? If the answer to both of those questions is "yes", which it seems to be that it is, then the problem is with the PXL writer device, and that needs to be fixed. Not least because, the FAPI code does nothing with the rest of the graphics pipeline that cannot be done from an input file. Thus even if I could tweak the FAPI behaviour to not trigger the problem with the PXL output, there'd be nothing to stop an input file triggering the same issue.
Chris, I think you are not understanding the issues fully. The bottom line is, pxl to x11 and some printers work as intended. The same pxl to some other non-HP printers does not, to pdf does not, and according to Ken, to windows display, does not. So even for raster outputs ( x11 and windows display) the verdict is split. How that pxl is generated, isn't really relevant, as long as it is valid pxl and print correctly on HP printers. Of course it is possible to 'dumb down' the pxl driver not to write things that specific conversion (i.e pdfwrite) cannot cope, but then, so can the pdfwriter conversion to convert to raster then wrap. Neither are relevant, as in the end of the day, gpcl6 must be able to cope with arbitrary and expressive valid pxl, wherever it comes from. I don't think this is a FAPI on/off problem - it is just the newer code allows more high-level translation and preservation of graphic objects without rastering, and that exposes known problem of pcl/pxl rasterops don't translate well to the final pdf.
Perhaps another question is: why is the reporter converting twice (ps -> pxl -> pdf), instead of just once (ps -> pdf)?
(In reply to Hin-Tak Leung from comment #23) > Perhaps another question is: why is the reporter converting twice (ps -> pxl > -> pdf), instead of just once (ps -> pdf)? The document has to be printed on non-PS printer - therefore pxl conversion. Back conversion to PDF is to demonstrate the wrong output, it's not needed per se.
(In reply to Dmitry Pankratov from comment #24) ... > The document has to be printed on non-PS printer - therefore pxl conversion. > Back conversion to PDF is to demonstrate the wrong output, it's not needed > per se. Two things: - just as many printers claiming to be pdf capable but choke on pdf's which are more feature-rich (and crashes by gs 9's pdf output, when they upgrade from gs 8 to gs 9), there are printers which claim to be PCL/PCL XL capable but cannot cope with some PCL/PCL XL features. PCL/PCL XL is a HP standard and only HP printers may be expected to support PCL/PCL XL fully. Many non-HP printers don't. - there are known problems with PCL/PCL XL converting to pdfs - if you want to demonstrate a problem with PCL/PCL XL being wrong, you should use one of the bitmap formats, such as bmp/tiff/png. I just did: pcl6 -sDEVICE=png16m -o a%02d.png 11936 (to convert to a01.png ... a10.png, one png file per page) and same with 11937, and there is no problem.
Hm, to me it looks like the vendor's firmware issue (if it exists) is unusually similar to the pdfwrite (and some other devices btw) output - meaning that the same issue somehow persists in the GS as well. I've found at least two factors that affect the output - the DisableFAPI switch and bmax_LARGE constant (comment #20). Not sure it is really relevant as I am not a gs developer. If this is strictly a vendor issue still to my opinion it makes sense to introduce a workaround if possible of course (because nobody will fix a firmware on millions of devices).
I just downloaded the win32 binary of ghostspcl 9.16 from www.ghostscript.com, and ran it under wine. pcl6-9.16-win32.exe -sDEVICE=display ... is indeed broken, but pcl6-9.16-win32.exe -sDEVICE=png16m -o a%2d.png ... is correct. (and png16m also work correctly on native linux) So we have X11 and png16m as correct, display and pdf being wrong; Xerox printers print correctly and Ricoh print wrongly.
(In reply to Dmitry Pankratov from comment #26) > Hm, to me it looks like the vendor's firmware issue (if it exists) is > unusually similar to the pdfwrite (and some other devices btw) output - > meaning that the same issue somehow persists in the GS as well. I've found > at least two factors that affect the output - the DisableFAPI switch and > bmax_LARGE constant (comment #20). Not sure it is really relevant as I am > not a gs developer. > > If this is strictly a vendor issue still to my opinion it makes sense to > introduce a workaround if possible of course (because nobody will fix a > firmware on millions of devices). Like I said, only HP printers are expected to support all PCL/PCL XL features fully; many non-HP printers also do well, other non-HP printers don't. As a "workaround", you could try using the cljet5c device instead. That's a more primitive PCL 5 driver.
Sorry I only look at non customer bugs once and a while. Here's what I see, the conversion from PS -> PXL is correct and the conversion from PXL -> PDF is wrong. Verified on an HP color laserjet printer (out target of emulation). I didn't verify but I'm sure the fapi business is the difference of producing an outline vs. raster for the text. The incorrect output black text instead of blue is all produced by images with an indexed color space. I suspect pdfwrite is tripping over that. We've had other troubles with that in PCL (not XL). I'll produce a simplified sample and attach it to the bug shortly.
Actually a simplified test file of the gs PXL output seemed unnecessary, putting a breakpoint on the first "image" will catch a character with a bad color (black not blue). But if we need one let me know.
Does anybody know which specific Ricoh model does not print this XL correctly?
(In reply to Henry Stiles from comment #31) > Does anybody know which specific Ricoh model does not print this XL > correctly? Please see comment #13.
(In reply to Dmitry Pankratov from comment #32) > (In reply to Henry Stiles from comment #31) > > Does anybody know which specific Ricoh model does not print this XL > > correctly? > > Please see comment #13. Thank you, missed that, lots of comments to go through ;-)
Created attachment 12352 [details] pxl image color space gpcl6 -sDEVICE=ppmraw -o out.%d.ppm badimage.pxl is correct. gpcl6 -sDEVICE=tiff24nc -o out.%d.tiff badimage.pxl The first page and first image shows the issue so the file is not reduced.
(In reply to Henry Stiles from comment #34) > Created attachment 12352 [details] > pxl image color space > > gpcl6 -sDEVICE=ppmraw -o out.%d.ppm badimage.pxl > > is correct. > > gpcl6 -sDEVICE=tiff24nc -o out.%d.tiff badimage.pxl > > The first page and first image shows the issue so the file is not reduced. The bad image should be reproduced with pdfwrite as the output device. We believe CMYK output devices are okay. It's been reported the windows viewer fails as well.
The characters on the first page are represented with an image, 1 bit per pixel and indexed color space indexing a 2 entry palette, see gdevpx.c:pclxl_fill_mask(). I didn't see a non trivial rop when I first examined this file but now I see it isn't a logical operation exactly, but the transparency of the source (image) is set to true. Which causes the image code to interpret this as a non trivial raster operation and use the rop device to render the images. I imagine we could fix the image code to recognize this case as a mask and not go through the ROP code. However, with the exception of the the ghostscript device that generated this PXL I doubt we'd ever see anything like this in real world files. This does raise the question if text should be handled with vectors and not bitmaps in the gdevpx.c device. Of course, it would be great to do real fonts, but the benefits of using bitmaps over vectors is questionable. Further the logical operations and transparency needed for the bitmap implementation are not implemented correctly on many devices. This is not the first example (comment #13) where we've had reports of ROPS or transparency not working on PCL clones. If we want to handle the widest range of PCLXL printers ROPS and transparency should be replaced with simpler operations in the ghostscript PCLXL device. Feedback welcome.
(In reply to Henry Stiles from comment #36) ... If we want to handle the widest > range of PCLXL printers ROPS and transparency should be replaced with > simpler operations in the ghostscript PCLXL device. > > Feedback welcome. Granted rasterizing everything always works, the disadvantage of that approach is fairly well-understood : output file size will increase, sometimes very substantially, and there is also the potential of poorer print quality when the printer/downstream might internally rasterize again such input. The worst case file size is about 120MB per color A4 page, at 600 dpi, I think. The code should just rasterize if the rops method does not exist, so it is probably a relatively simple matter to run stock code and mod'ed code , with that routine removed, once and compare size difference, for a range of input files (say, the cluster tests), though this will take some computing resource. And everything downstream (pxl to pdf, etc) will have substantially larger size and possibly worse graphic quality too. There is also the issue of whether to cater for non-HP printers which do not implement the whole pxl spec fully nor correctly, and to what extent one want to work around printer firmware bugs...
Created attachment 12358 [details] patch to handle text as vector Picking up on Henry's comment #36 (!) The attached patch disables the glyph cache, causing all text to be emitted as paths instead of bitmaps. For the supplied input file the resulting PXL renders correctly on pdfwrite and all the other problematic devices I have tried (tiff32nc, windows display device). I don't have a Ricoh PXL printer, but I very strongly suspect it will work correctly there too. The size of the resulting PXL file is also smaller: pre-FreeType - 2,010,525 FreeType - 2,377,758 patched - 2,010,691 This hints at what the likely change was, and I very strongly suspect its not the use of FreeType as such, I suspect that previously the cache was smaller. When a glyph is too big to fit in the cache it is rendered as vector paths, if the cache was smaller previously (ie pre-FreeType) the glyph bitmaps would have been less likely to fit and more likely to be emitted as vector, not bitmaps. Ideally we would capture the outlines or preserve/convert the original fonts, which would give us the benefits of small size *and* scalability, however that would be considerable work, and capturing the outlines looks problematic as the font would have to be injected into the PXL stream before it was used. There is a balance to be struck, potentially, between file size and reliability (and consistency!). As Henry noted on IRC PXL doesn't have an 'imagemask' in the sense that PostScript does, the ROP is the equivalent operation, which is why it is used for text bitmaps. For text in small point sizes using bitmpas probably results in smaller file sizes, and PXL isn't intended to be scalable anyway, as I understand it PCL and PXL are only supposed to work at 300 or 600 dpi. On the other hand, using vectors does, for at least some cases, result in output which is more correct, even on some clone PCL printers. Additionally, large point size text will always exceed the glyph cache size, and be emitted as vector data, which can potentially lead to inconsistent results on these devices I suspect this is the source of the problem on the Windows display device, the glyph which is black is the smallest glyph on the page ('I') and probably the only one rendered as an image. I don't think we test the PXL devices on regular commits, this isn't my area of responsibility, and Henry may want to make this behaviour contingent on a command line control, so I'm leaving it up to him what to do with this. The patch is exactly the same as we use in the pdfwrite device to 'flatten' fonts on demand.
> > Ideally we would capture the outlines or preserve/convert the original > fonts, which would give us the benefits of small size *and* scalability, > however that would be considerable work, and capturing the outlines looks > problematic as the font would have to be injected into the PXL stream before > it was used. > Thanks for looking at this Ken, I was thinking of something even simpler, bring any mask back to rectangles as follows: diff --git a/devices/vector/gdevpx.c b/devices/vector/gdevpx.c index 47e9cd4..a07a8de 100644 --- a/devices/vector/gdevpx.c +++ b/devices/vector/gdevpx.c @@ -152,7 +152,7 @@ static dev_proc_strip_copy_rop(pclxl_strip_copy_rop); map_color_rgb, /* differs */\ gdev_vector_fill_rectangle,\ NULL, /* tile_rectangle */\ - pclxl_copy_mono,\ + NULL, /* pclxl_copy_mono, */ \ pclxl_copy_color,\ NULL, /* draw_line */\ NULL, /* get_bits */\ @@ -169,7 +169,7 @@ static dev_proc_strip_copy_rop(pclxl_strip_copy_rop); NULL, /* copy_rop */\ gdev_vector_fill_path,\ gdev_vector_stroke_path,\ - pclxl_fill_mask,\ + NULL,\ gdev_vector_fill_trapezoid,\ gdev_vector_fill_parallelogram,\ gdev_vector_fill_triangle,\ That weighs in at 2.5M for this particular file. The fonts are rendered with scanline height rectangles. The change is "ham fisted", I don't know if copy_mono() and fill_path() is frequently used beyond fonts. If the high level image code ends up calling pclxl_copy_mono() for 1 bit images we'd likely have bloat problems. We need to get some runs from Marcos to find out.
Marcos, can you test each of the patches from comment #38 and #39 looking for changes in output size? Or if there is a nice way cluster users can do this themselves, let us know.
Current versions of pxlcolor and pdfwrite still have the same problems. The tiff32nc device SEGVs now. I don't have access to the cluster to test the patch.
This commit 496f84da7abe3dbea5657d5af836f9425fba2a4f is (an updated version of) my original patch. Testing my proposed solution and Henry's showed that Henry's resulted in considerably larger output files when the input was text heavy.