This is a reminder to implement the memory limitation again (-K option on the commandline, this is at the moment uncommnented with a #ifdef OLD_ALLOCATOR). We need (system requirement) to limit the amount of memory that the GhostPDL is allowed to allocated.
I have also an xps file that causes memory-exhaustion (keeps claiming memory). the xps-file (40MB) is uploaded to peeves (@ $MYHOME/20090910_xps_memory/*.xps) What I see is that after each page is printed the allocated memory grows. It looks like the resources used for the page are not released. At the end of the job memory is released again (i.e. the ctx->first_part list). However at our current resolution (600dpi, with transparency). It is impossible for us to print the complete job. I.e. the job uses far more than 0.5 GB and we get a hangup due to swapping? Is it possible to release page-resources at the end of a page?
Please assign this back to me when you are done with Comment #1.
Freeing resources at the end of the page is easier than it sounds. The XPS spec has support for DiscardControl parts that can be used to let the interpreter know when a resource is safe to free. Unfortunately, the test file you sent does not have these parts (nor have I seen any other files that use them). To compound matters, the only other way to know which resources are used is by looking at the ".rels" parts, and they're usually tacked on at the end of the file. GhostXPS uses a streaming architecture and so cannot look ahead or go back and reload parts. Fixing this issue will require that the file uses .rels and DiscardControl parts as they were intended (not going to happen, microsoft's XPS producers don't do it) or requiring that GhostXPS has random file access to the XPS document. We can make some adjustments and release the parsed representation of resource parts between each page though. That should help alleviate some of the memory pressure. It means that we have to free fonts, image data and parsed XML representations and recreate them from the raw data on each page. Unfortunately we cannot get away with discarding the raw data unless DiscardControl parts are present or we have random file access.
After some more investigation, we already do free the parsed representation of resource parts when we can. We cannot free font resources because pdfwrite needs them to be alive for the duration of the job. So it's not likely that much can be done without random file access to the XPS file.
Tor a question: What would be needed/changed if not the *.xps file (PK-zip container) is processed as a stream, but when the directory-path is given to the parser (i.e. the unzipped *.xps). Is only xpszip.c to be replaced with a xpsfilesystem.c ? Then you would have a 'random' access, and a part->data is then only allocated when used (filled from a file) and released again. This should keep the memoryconsumption more in control. Also, it should then be possible to 'jump' to a certain page to be rasterized (i.e. kind of pagerange printing with startpage > 1).
I am working on this very thing, a separate mode for parsing XPS files from disk where it reads just the part data it needs by seeking in the file (using FILE*). It doesn't need to unzip the XPS to a directory, it just needs random access within the XPS file. I just need to add handling of interleaved parts and freeing the memory between pages, and make sure I cover all the cases so that both the streamable and seekable modes work well before I have a commit ready.
I've implemented a random file access mode in the XPS parser, which can be experimentally enabled by setting an environment variable. In a soon-to-come patch to plmain the mode is going to be automatically enabled if the input is a seekable file, so the environment variable is just for testing.
I tested it and it works, but the memory consumption still grows with each page. The amount of memory used is the same as before, minus the size of the xps-job (which is not stored in memory anymore). The growing memory is most probably due to the resources (images) that are used to create a page (scaled to destination resolution) are kept in cache. Perhaps this cache could also be flushed )LRU mechanism?) as it can now more easily be re-created when an element is re-used on another page?
Assign back to me when the xps issues are resolved and I'll add the -K option.
Memory use on PRI019_WinHEC06.xps still grows by about 10-20 megabytes on every page.
However, with the transparency device disabled memory use stays at a constant total of 15 mb throughout the job. Perhaps this is related to the memory issues in 690951 which are also only present when transparency is enabled.
Created attachment 5910 [details] leaks.awk Apply the following patch and then run gxps with -ZA and -l flags and pipe the output through "awk -f leaks.awk" to get a summary of all leaks. Index: gsalloc.c =================================================================== --- gsalloc.c (revision 10411) +++ gsalloc.c (working copy) @@ -727,6 +727,7 @@ gs_ref_memory_t * const imem = (gs_ref_memory_t *)mem; obj_header_t *pp; gs_memory_type_ptr_t pstype; + gs_memory_struct_type_t saved_stype; struct_proc_finalize((*finalize)); uint size, rounded_size; @@ -777,10 +778,20 @@ rounded_size = obj_align_round(size); finalize = pstype->finalize; if (finalize != 0) { + + /* unfortunately device finalize procedures will clobber the + stype which is used for later debugging with "A" debug + tracing, so we save stype it in a local. */ + if (gs_debug['a']) + saved_stype = *pstype; + if_debug3('u', "[u]finalizing %s 0x%lx (%s)\n", struct_type_name_string(pstype), (ulong) ptr, client_name_string(cname)); (*finalize) (ptr); + + if (gs_debug['a']) + pstype = &saved_stype; } if ((byte *) ptr + rounded_size == imem->cc.cbot) { alloc_trace(":-o ", imem, cname, pstype, size, ptr);
These is the big pdf14 transparency leak. It occurs every time an opacity mask/softmask is used: leak [a1:+b.]pdf14_buf_new (pdf14_pop_transparency_mask) (191520) = 0x1008bd058 There is another smaller leak here: leak [a1:+b ]pdf14_begin_transparency_mask (256) = 0x1010ed398 And some small non-transparency related leaks: leak [a1:+b.]scan_font_resource(ps_name) -bytes-*(16=16*1) = 0x101002e78 leak [a1:+b.]gs_type42_font_init(GSUB) -bytes-*(3662=3662*1) = 0x101043858 leak [a1:+bf]mem_open (128) = 0x101044c50 leak [a1:+<f]gx_path_add_line line(32) = 0x1010e1070 This leak has its own bug: leak [a1:+b.]gs_type42_font_init(GSUB) -bytes-*(3662=3662*1) = 0x101043858
I found the fix for this. There is a ref counting issue with respect to the soft masks. I need to do a regression test to make sure all is well.
So my fix has some issues with the some of the files that test the soft mask with the extended graphic state. This will take me a little longer than I had hoped.
Fixed with rev 10644
Changing customer bugs that have been resolved more than a year ago to closed.