I compiled the trunk (revision 11362) for pcl (with ufst). in main: nmake -f pcl6_msvc.mak DEBUG=1 TDEBUG=1 MSVC_VERSION=9 PL_SCALER=ufst when run at default resolution (72dpi). devenv debugobj/pcl6.exe with project-properties: -sDEVICE=jpeg -sOutputFile=/dev/null -dNOPAUSE J341.pcl J341.pcl J341.pcl and breakpoint set at plmain.c:line343: if (!filename) where it gets the name of the next job on the commandline. set a watch at: (gs_malloc_memory_s *)(pjl_mem->non_gc_memory) and look for used/max_used fields. you will notice that used/max_used will not grow after the 1the job has been processed. Do the same with resolution 150dpi. project-properties: -r150 -sDEVICE=jpeg -sOutputFile=/dev/null -dNOPAUSE J341.pcl J341.pcl J341.pcl you will see that used/max_used will grow at each break (start of next job) with about 5-6MB per job. There seems to be a memory-leak related to non-standard resolution (! 72dpi). The testfile J341.pcl is from a very dumb application (downloading 2 fonts over and over again (using different softfontids), so the job as such uses a lot of memory. What was surprising is that memory consumption keeps growing with each time the job is processed (also to be seen in the taskmgr of windows). I'm trying to put the testfile (58MB of data) on peeves.ghostscript.com but I seem to have the wrong key?
*** Bug 691373 has been marked as a duplicate of this bug. ***
uploaded testfile to peeves.ghostscript.com:${MYHOME}/data4_691374
(In reply to comment #0) > I compiled the trunk (revision 11362) for pcl (with ufst). > in main: nmake -f pcl6_msvc.mak DEBUG=1 TDEBUG=1 MSVC_VERSION=9 PL_SCALER=ufst > > when run at default resolution (72dpi). > > devenv debugobj/pcl6.exe > with project-properties: > -sDEVICE=jpeg -sOutputFile=/dev/null -dNOPAUSE J341.pcl J341.pcl J341.pcl > and breakpoint set at plmain.c:line343: if (!filename) > where it gets the name of the next job on the commandline. > > set a watch at: (gs_malloc_memory_s *)(pjl_mem->non_gc_memory) and look for > used/max_used fields. > > you will notice that used/max_used will not grow after the 1the job has been > processed. > > Do the same with resolution 150dpi. > project-properties: > -r150 -sDEVICE=jpeg -sOutputFile=/dev/null -dNOPAUSE J341.pcl J341.pcl > J341.pcl > Detecting memory leaks with the chunk allocator can be tricky. Revision 11375 has a definition HEAP_ALLOCATOR_ONLY which will cause all memory operations to go out to the system malloc, free, and realloc. Then third party tools like valgrind or intruments.app (mac os x) can be used to see what is left over after the job is finished. I am still investigating this but the left over allocations appear very similar to 691378, objects (macros and downloaded fonts) allocated by the pcl file but not cleaned up. Here is some of the valgrind output (I put an exit() call in after 500 pages): ==20706== ==20706== 2,900,843 bytes in 1,967 blocks are possibly lost in loss record 25 of 27 ==20706== at 0xB3C516: malloc (vg_replace_malloc.c:193) ==20706== by 0x27B7E1: gs_heap_alloc_bytes (gsmalloc.c:182) ==20706== by 0x3A97E: gx_ttfMemory__alloc_bytes (gxttfb.c:320) ==20706== by 0x382EE: Load_TrueType_Programs (ttload.c:218) ==20706== by 0x39FE8: Face_Create (ttobjs.c:883) ==20706== by 0x2D5FD: ttfFont__Open (ttfmain.c:309) ==20706== by 0x3B03B: ttfFont__Open_aux (gxttfb.c:440) ==20706== by 0x2903B5: gx_attach_tt_interpreter (gxccman.c:241) ==20706== by 0x290745: gx_add_fm_pair (gxccman.c:347) ==20706== by 0x28E6C4: gx_lookup_fm_pair (gxccache.c:137) ==20706== by 0x2952A8: show_proceed (gxchar.c:1038) ==20706== by 0x2947C9: continue_show (gxchar.c:762) ==20706== ==20706== ==20706== 4,085,852 bytes in 155 blocks are possibly lost in loss record 26 of 27 ==20706== at 0xB3E2B8: realloc (vg_replace_malloc.c:472) ==20706== by 0x27BAD4: gs_heap_resize_object (gsmalloc.c:273) ==20706== by 0x33FF09: append_macro (pcparse.c:221) ==20706== by 0x340BF4: pcl_process (pcparse.c:535) ==20706== by 0x2BCD: pcl_impl_process (pctop.c:461) ==20706== by 0x322E1D: pl_process (pltop.c:162) ==20706== by 0x372FCD: pl_main (plmain.c:436) ==20706== by 0x37503E: main (plmain.c:1327) ==20706== ==20706== ==20706== 29,393,226 bytes in 108,363 blocks are possibly lost in loss record 27 of 27 ==20706== at 0xB3C516: malloc (vg_replace_malloc.c:193) ==20706== by 0x27B7E1: gs_heap_alloc_bytes (gsmalloc.c:182) ==20706== by 0x338160: pcl_character_data (pcsfont.c:628) ==20706== by 0x340C89: pcl_process (pcparse.c:547) ==20706== by 0x2BCD: pcl_impl_process (pctop.c:461) ==20706== by 0x322E1D: pl_process (pltop.c:162) ==20706== by 0x372FCD: pl_main (plmain.c:436) ==20706== by 0x37503E: main (plmain.c:1327) ==20706==
I uploaded a file dl_J341.pcl.gz (containing only the download objects (fonts/macros). If sending this job multiple times (and having the debugger attached), set a breakpoint at approx. line 479 in plmain.c (at the pl_main_cursor_close()). Then look at: (gs_malloc_memory_s*)(pjl_mem->non_gc_memory) (used/max_used). Now only download data is processed, no pages ripped. Everytime the job is processed the amount of memory used is enlarged by 2MB. This seems strange to me. I used the pcl6_msvc.mak build (with gdevbit.c compiled in). But the pcl6_gcc.mak, or any other device should show the same. run : -r600 -sDEVICE=bit -sOutputFile=NUL -dNOPAUSE dl_J341.pcl dl_J341.pcl dl_J341.pcl
(In reply to comment #4) > I uploaded a file dl_J341.pcl.gz (containing only the download objects at peeves.ghostscript.com:${MYHOME}/data4_691374 > (fonts/macros). If sending this job multiple times (and having the debugger > attached), set a breakpoint at approx. line 479 in plmain.c (at the > pl_main_cursor_close()). Then look at: > (gs_malloc_memory_s*)(pjl_mem->non_gc_memory) (used/max_used). Now only > download data is processed, no pages ripped. Everytime the job is processed the > amount of memory used is enlarged by 2MB. This seems strange to me. > > I used the pcl6_msvc.mak build (with gdevbit.c compiled in). But the > pcl6_gcc.mak, or any other device should show the same. > > run : -r600 -sDEVICE=bit -sOutputFile=NUL -dNOPAUSE dl_J341.pcl dl_J341.pcl > dl_J341.pcl
(In reply to comment #5) > (In reply to comment #4) > > I uploaded a file dl_J341.pcl.gz (containing only the download objects > at peeves.ghostscript.com:${MYHOME}/data4_691374 > > (fonts/macros). If sending this job multiple times (and having the debugger > > attached), set a breakpoint at approx. line 479 in plmain.c (at the > > pl_main_cursor_close()). Then look at: > > (gs_malloc_memory_s*)(pjl_mem->non_gc_memory) (used/max_used). Now only > > download data is processed, no pages ripped. Everytime the job is processed the > > amount of memory used is enlarged by 2MB. This seems strange to me. > > > > I used the pcl6_msvc.mak build (with gdevbit.c compiled in). But the > > pcl6_gcc.mak, or any other device should show the same. > > > > run : -r600 -sDEVICE=bit -sOutputFile=NUL -dNOPAUSE dl_J341.pcl dl_J341.pcl > > dl_J341.pcl Yes I did notice there is a leak with macros across jobs and I'll fix that, but this is completely different than the other memory exhaustion example 691378, as far as I can tell, every memory object in that file needs to be kept in memory. I am stuck on that problem, let me know if you find anything that indicates my analysis is incorrect.
in data4_691374 on peeves you find: dlfonts.pcl (only the download fonts) dlmacros.pcl (only the macro definitions) remain.pcl (only the job, without the above contents). When printing the dlmacros.pcl multiple times I did not see a significant growth in memory, but when printing dlfonts.pcl multiple times I see a growth of 2MB per job. So Macros seems to be alright, but Fonts seem to loose memory.
(In reply to comment #7) > in data4_691374 on peeves you find: > dlfonts.pcl (only the download fonts) > dlmacros.pcl (only the macro definitions) > remain.pcl (only the job, without the above contents). > > When printing the dlmacros.pcl multiple times I did not see a significant > growth in memory, but when printing dlfonts.pcl multiple times I see a growth > of 2MB per job. > > So Macros seems to be alright, but Fonts seem to loose memory. Yes I analyzed this last night the small allocations associated with each glyph from the 400 or so fonts downloaded with each file lead to pathological fragmentation in the allocator almost all the overhead (2MB) per job are extra chunks produced by the allocator. The easiest way to see this is compile with HEAP_ALLOCATOR_ONLY which uses the system's malloc and free, and observe there is no growth. For a more detailed look one can use -ZA and the script gs/toolbin/leaks.tcl to get an exact report of allocations left over. In normal usage where fonts are deleted as they are used this is not an issue. I am hesitant to slow down the allocator with compaction to satisfy these pathological test cases.
(In reply to comment #4) > I uploaded a file dl_J341.pcl.gz (containing only the download objects > (fonts/macros). If sending this job multiple times (and having the debugger > attached), set a breakpoint at approx. line 479 in plmain.c (at the > pl_main_cursor_close()). Then look at: > (gs_malloc_memory_s*)(pjl_mem->non_gc_memory) (used/max_used). Now only > download data is processed, no pages ripped. Everytime the job is processed the > amount of memory used is enlarged by 2MB. This seems strange to me. > > I used the pcl6_msvc.mak build (with gdevbit.c compiled in). But the > pcl6_gcc.mak, or any other device should show the same. > > run : -r600 -sDEVICE=bit -sOutputFile=NUL -dNOPAUSE dl_J341.pcl dl_J341.pcl > dl_J341.pcl Note the EOF does not delimit a job in the default configuration - see pcl_impl_process_eof(). We assume some users may want to customize this function. There should be no difference in memory consumption sending these files separately or concatenated as one unless you have changed pcl_impl_process_eof().
Henry, I don't know which allocator you are using (nogc?), but the chunk allocator is very efficient with annealing adjacent free space which can help a lot with fragmentation issues compared to the nogc method. Just thought I'd mention it in case it helps.
(In reply to comment #10) > Henry, I don't know which allocator you are using (nogc?), but the chunk > allocator is very efficient with annealing adjacent free space which can > help a lot with fragmentation issues compared to the nogc method. > > Just thought I'd mention it in case it helps. Yes the chunk allocator does work better with these jobs. Last I checked though the performance of the chunk allocator was quite a bit slower than nogc for normal jobs, I thought possibly related to not supporting freelists, but I'll have a look at it again since I'm working in this area anyway.
Ray's chunk allocator does look promising I will integrate it with pcl, xps, and friends this evening. That should be sufficient time before the release (August) to see if there are any problems, if so it is trivial to switch back.
With the new gsmchunk.c memory-manager it looks ok. After 2 jobs (J341.pcl) there is no more memorygrowth. So this bug can be closed.
Changing customer bugs that have been resolved more than a year ago to closed.