Hi, We observed that GS8.63 performance is degraded compared to GS8.54 version. We are in the process of getting the exact test cases, but just wanted to give a headsup, as this is a very critical issue for us. Regards Subrahmanyan.
We observed that the performance is degraded with all the transformers : Ps->pcl Ps->pdf Ps->ps Pdf->ps We submitted about 50000 jobs and we found that performance is degraded significantly. Since this is a major concern for us, could you please let us know if this issue is already known and fixed already, at the earliest? Thanks Subbu
We cannot address general (non-specific) issues. Please attach sample files (or email to support) and the exact command line or invocation procedures used. Also specify the version of Ghostscipt that you found to exhibit the problem. Note that there are some memory (and garbage collection) issues that affected previous versions of Ghostscript. Some output devices (such as pdfwrite) were more prone to performance degradation with memory issues. Please test with the 8.64 release, and also please test with a more relaxed demand on the garbage collector using the option: -c " 80000000 setvmthreashold " -f immediately preceding the input file (or sent via 'run_string' api calls) prior to processing the input.
Created attachment 4861 [details] input.pdf PDF file used in the test.
Created attachment 4862 [details] input.ps Input postscript file used in the test.
We compared with GS 8.54 and GS8.63 version And with GS8.63 we found the performance degradation. Following are the commands we used: PS->PDF /hp/om/bin/gs -sDEVICE=pdfwrite -I/hp/om/lib/FONTS/Unicode/Resource - I/hp/om/lib/FONTS/Unicode -I/hp/om/lib/FONTS/Soft_Horizons -I/hp/om/lib/PS -q - dBATCH -dNOPAUSE -dSAFER -dSHORTERRORS -dWRITESYSTEMDICT -dGHOSTSCRIPT - dDEVICEWIDTHPOINTS=612 -dDEVICEHEIGHTPOINTS=792 -sOutputFile=out.pdf input.ps PDF->PS /hp/om/bin/gs -sDEVICE=psmono -q -dNOPAUSE -dBATCH -dSAFER -dSHORTERRORS - dWRITESYSTEMDICT -I/hp/om/lib/FONTS/Unicode/Resource - I/hp/om/lib/FONTS/Unicode -I/hp/om/lib/FONTS/Soft_Horizons -I/hp/om/lib/PS - sOutputFile=out.ps input.pdf PS->PS /hp/om/bin/gs -sDEVICE=pswrite -I/hp/om/lib/FONTS/Unicode/Resource - I/hp/om/lib/FONTS/Unicode -I/hp/om/lib/FONTS/Soft_Horizons -I/hp/om/lib/PS -q - dBATCH -dNOPAUSE -dSAFER -dSHORTERRORS -dWRITESYSTEMDICT -dGHOSTSCRIPT - dDEVICEWIDTHPOINTS=612 -dDEVICEHEIGHTPOINTS=792 -sOutputFile=out.ps in.ps Please help us in resoloving the issue. Please let us know if you need any more details. Regards Subrahmanyan.
How do I use this option: -c " 80000000 setvmthreashold " I got the following error when I used with gs: %%[ Error: undefined; OffendingCommand: setvmthreashold ]%% Artifex Ghostscript 8.63: Unrecoverable error, exit code 1 ----------------------------- /hp/om/bin/gs -I/hp/om/lib/FONTS/Soft_horizons -I/hp/om/lib/PS - sDEVICE=pdfwrite -q -dBATCH -dNOPAUSE -dSAFER -dSHORTERRORS -dWRITESYSTEMDICT - dGHOSTSCRIPT -dDEVICEWIDTHPOINTS=612 -dDEVICEHEIGHTPOINTS=792 - sOutputFile=out.pdf -c " 80000000 setvmthreashold " -f input.ps %%[ Error: undefined; OffendingCommand: setvmthreashold ]%% Artifex Ghostscript 8.63: Unrecoverable error, exit code 1
Try 'setvmthreshold' instead of 'setvmthreashold'
Yes, it worked with 'setvmthreshold'. but what does that number "80000000" signify? Thanks Subrahmanyan.
Standard PostScript operators are documented in the PostScript Language Reference Manual. From the 3rd Edition, p688: setvmthreshold int setvmthreshold – sets the allocation threshold used to trigger garbage collection. If int is less than the implementation-dependent minimum value, the threshold is set to that minimum value. If int is greater than the implementation-dependent maximum value, the threshold is set to that maximum value. If int is -1, the threshold is set to the implementation-dependent default value. All other negative values of int result in a rangecheck error. Modifications to the allocation threshold parameter are subject to save and restore. In an interpreter that supports multiple contexts, this parameter is maintained separately for each context. The parameter specified by setvmthreshold is the same as the VMThreshold userparameter set by setuserparams (see Appendix C).
I'm working on reproducing this; if successful I'll look for the applicable revision(s).
The input.ps file you attached processes very quickly (less than 1 second), so it's hard to use it for performance testing. I built a longer file by concatenating it to itself 1000 times. It's not easy to do this for input.pdf, so I haven't done any PDF->PS testing (I could convert the 1000 page PS file to PDF, but that would obviously be a different file than input.pdf). For PS->PDF using pdfwrite I see about a 3% increase in cpu time from 8.54 to 8.63; from 1:14 to 1:16. Is this the same issue you are seeing or something more significant The PS->PS time increase is more significant, from 0:33 to 0:51. I'm looking into this now.
Hi Marcos, Thanks for your update. yes we are seeing the similar issues. Basically in our performance test we submit the same job continously for an hour and compare the number of jobs processed in an hour. We observed significant degradation. Thanks Subrahmanyan.
Hi, I actually tried using GS8.64 and with that also I see similar degradation when compared GS8.54. I actually tried on Linux platform. Also with the option: -c " 80000000 setvmthreshold " -f I don't see much difference. Thanks Subbu
Hi, I saw a mail from Ray, which said to run the test by leaving the Ghostscript open and sending it jobs to process and write the output to different files each time, COuld you please let us know the steps to run the test by leaving the ghostscript open, for multiple jobs with different input and output document formats?Is there a way to provide us a solution, that does not have a startup overhead and works for all transformations? Would it be possible to provide client/server model for ghostscript? Thanks Subbu
Ghostscript can work as a server in most cases. Ghostscript can read and write pipes, which may connect to other program that listens to a port, enumerates a directory, or fetches the data from the database. The program needs to pass to Ghostscript the input steam itself including EOF flag, the output file name, and other parameters. Ghostscript needs to split the input stream into jobs and re-install an output device for every job. The exact implementation will depend on the host operating system, the choice of communication method, the choice of error reporting, etc. This method currently doesn't work for pdfwrite device. This is a known bug that deserves a separate entry in the tracker.
Hi, Is there any documentation on how Ghostscript can work as a server. Also would it work fine when we have multiple jobs submitted concurrently. Thanks Subbu
You can have as many 'server' processes as you wish. Each process would be a single instance of Ghostscript. A simple example using python: import os F=os.popen3("/Artifex/ghostpdl/gs/debugobj/gswin32c.exe -q -sDEVICE=bbox -dNOPAUSE -dNOPROMPT -dNOOUTERSAVE") F[0].write("save (pswrite) selectdevice << /OutputFile (x.ps) >> setpagedevice (examples/tiger.eps) run restore (done) = flush\n") result = F[1].readline() # The 'result' string should be 'done\n', otherwise an error. F[0].write("save (tiffg4) selectdevice << /OutputFile (x.tif) >> setpagedevice (examples/tiger.eps) run restore (done) = flush\n") result = F[1].readline() # The 'result' string should be 'done\n', otherwise an error. The 'save ... restore' insures that the device will be closed. This is needed for pswrite so that it finishes the output file. The '<< /Outputfile (...) >> setpagedevice' determines where the file will be written. This simple framework shows how a python 'client' can use a Ghostscript process as a 'server' to convert a PS or PDF input file to some other format. As Alex notes, this will not currently work with pdfwrite. The python 'client' can be extended to have multiple 'server' processes by issuing multiple 'popen3' calls (saving the file handles in different variables of course).
Created attachment 4888 [details] client_server.py Python example showing multiple 'gs' server instances. On my multi-core processor I get 100% CPU utilization with NUM_SERVERS == 2 or more. With NUM_SERVERS=2 it completes 1000 jobs (50% to pcl, 50% to ps) in 101 seconds. With NUM_SERVERS=1 it takes 159 sec. From this example, it is clear that a client-server model can help if there are multiple server CPU's. I suggest that the customer implement such a model to improve the performance.
Hi, When I tried this, it just hung with this below mesg: Total print jobs = 1000 Starting server: 0 Also in our case the job submission happens from a different process altogether, not sure how we can use this in our case. Thanks Subbu
To implement a server to dispatch to multiple gs processes, the 'master' process would receive the requests (from a pipe or a hot folder, for example) and then would dispatch each job to an already open gs process. As far as the 'hang' starting the first server process you report, I really can't diagnose this without access to the system you are testing on. I recommend looking at your system's process monitor tools (Windows "Task Manager" or unix "top") to see if the process is starting and whether or not it is stalled. I assume that you have examined the example code and modified the hard coded paths in INFILE and popen3() call.
Hi, I am actually running on a Linux box and had changed the paths according to my setup in the python script. And I can see the gs process running when I do "ps -aef". Thanks Subbu
We have more customers reporting this issue now. THanks Subbu
While I have found causes for some performance degradation in newer versions when processing files with lots of pages, this issue of single invocation of Ghostscript per job is not generally of concern since the proposed solution of leaving Ghostscript 'open' and processing the files that way avoids the 'startup' overhead altogether. This is the way PostScript printers work and the mode in which performance is worth addressing -- not the single job mode. Closing this as WONTFIX.
All of our code versions (back to version 6.00) are on our subversion based server. With an svn client program on a unix or linux platform there is a script we use to track down which changes caused bugs in toolbin/search-svn-revs. This tool is what Marcos, myself and a few others have used in the past and the documentation (such as it is) in in the script. Part of the challenge you will face is writing a 'test' script to let the script decide on 'pass'/'fail' since there will be timing differences in versions that are minor, so you need set benchmark 'pass' times and possibly run it several times to see which patches contribute to what performance changes. I suppose you could also check out, compile and test EVERY version since 8.64 and record the test times. Just for your information, the svn revisions corresponding to release versions are: release rev 8.54 6789 8.63 8909 8.64 9434 As you can see from the www.ghostscript.com website, the svn repository can be accessed with: svn co http://svn.ghostscript.com/ghostscript/trunk/gs or, for a specific revision, e.g. 6789: svn co -r6789 http://svn.ghostscript.com/ghostscript/trunk/gs
Created attachment 5222 [details] time-log.txt These timings show the progression of timings for execution of the bash: time ( x=1; while [ $x -lt 200 ] ; do bin/gs -q -sDEVICE=pdfwrite -o x.pdf Bug_690352_input.ps > $O 2>&1; x=$(( $x + 1 )); done ) after a "make clean" on each revision. I ran the above for every 5 revisions from 6794 to 9859 (sure is nice having a system that can do a make -j5 in under 30 sec). As can be seen, there are a few points that show jumps in the user time.
A little follow up testing shows that the biggest single change, between 12.7 seconds user time to 25.7 seconds is due to the rev 8084 that effectively enables 'fontconfig' by default on platforms where it is available. Running configure (or autogen.sh) with --disable-fontconfig (which results in the CAPOPT= line in Makefile without -DHAVE_FONTCONFIG) reverts to the 8083 timing. If platform fonts are enabled, and not needed for the conversion, then disabling this hook will dramatically reduce the 'overhead' of starting a gs process. Note that since all of the times above we for a loop running 200 simple jobs, the added time per job is only 65 milliseconds, but since the entire job, including performing the PS to PDF conversions and writing the file only takes 63 milliseconds, this is a relatively large penalty. Unfortunately, testing HEAD with --disable-fontconfig shows no time difference for this test, but HEAD has improved to 22 seconds (compared to 25.7 for rev 8084).
Hi, I just tried the test with and without --disable-fontconfig option. But I didn't see any difference with and without the change. did I make any mistake? Thanks Subbu
Hi, I was trying the client_server.py python script on Unix. It seems that the GS process just waits and does not take the input, whatever is written to the stdin. It works fine on windows. Is there any setting to be done on Unix, so that GS takes the values written to stdin by the process? Thanks Subbu
Time In Seconds To process 50000 jobs Transformation GS8.63 GS8.54 % Slowdown pdf to ps 2434 1747 39 % ps to pcl 1924 1401 37 % ps to pdf 2748 1423 93 % ps to ps 2677 1433 87 % Most if not all of the slowdown reflects changing startup times, of course the customer should be running ghostscript as a server and not restarting each time.
Created attachment 5479 [details] rev-times.log The attachment is 'user' times running 200 jobs converting input.ps to PDF. The 'autogen.sh' options used were: --disable-contrib --disable-fontconfig The timing was collected using: time ( x=1; while [ $x -lt 200 ] ; do bin/gs -I../fonts -q -sDEVICE=pdfwrite -o x.pdf input.ps >> /dev/null 2>&1; x=$(( $x + 1 )); done ) The initial revision was rev 8.54 (rev 6790). As expected there was some variation in the timings, but the significant changes in the averages are annotated. The last two lines in the log are the timing improvements that are possible with very simple changes: reducing the ht cache size and COMPILE_INITS=0. The more significant degradation that resulted from the massive ESP Ghostscript merge will take much more analysis. The 'prof' results for this rev will be attached next with some discussion.
Created attachment 5481 [details] rev-times.png Plot of data from Comment #30
Created attachment 5482 [details] performance.png I performed a similar analysis as Ray using the command lines given in Comment #5 and the input.pdf and input.ps test files but tested ps to pdf, ps to ps, and pdf to ps. I did not test with --disable-fontconfig.
The axes on your performance graph aren't labelled. Can you please describe what you've plotted?
The X axis is revision and the Y axis is time in seconds.
Thanks.
When we use "(pswrite) selectdevice" in GS> prompt on Linux, we get this error. It works on windows. GS>(pswrite) selectdevice GPL Ghostscript 8.70: **** Could not open the file . Error: /invalidfileaccess in --setdevice-- Operand stack: --nostringval-- Execution stack: %interp_exit .runexec2 --nostringval-- --nostringval-- -- nostringval-- 2 %stopped_push --nostringval-- --nostringval-- % loop_continue 1753 2 3 %oparray_pop --nostringval-- --nostringval-- false 1 %stopped_push .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- 1741 1 3 %oparray_pop --nostringval-- Dictionary stack: --dict:1147/1684(ro)(G)-- --dict:0/20(G)-- --dict:70/200(L)-- Current allocation mode is local Current file position is 23 GS<1>
Hi, Could someone please respond to my Comment #36 . Thanks Subbu
Comment #36 is not relevant to the performance issue, so really does not belong here. Also there is not enough information in the comment. Please send an email to support with the command line used to invoke Ghostscript. Most likely, the default OutputFile is invalid so that when the pswrite attempts to open the file for writing, the error occurs. On most installations, the default output is the printer (PRN: on Windows, /dev/lp0 on linux). How a particular OS does or does not allow writes to this (even if there is no printer) is not controlled by GS.
Hi, As mentioned in Comment #18, we just tried to use the instructions provided in the python script. First we execute the following command on Linux: /hp/om/bin/gs -I/hp/om/FONTS/Soft_Horizons -I/hp/om/lib -q -dNOPAUSE - dNOPROMPT -sDEVICE=bbox -sOutputFile=/dev/null -dNOOUTERSAVE Then in the GS prompt we provide the below commands: currentglobal true setglobal globaldict /OuterSave save put (pswrite) selectdevice with this we are getting the below error: --------------------------- GPL Ghostscript 8.70: **** Could not open the file . Error: /invalidfileaccess in --setdevice-- Operand stack: false --nostringval-- Execution stack: %interp_exit .runexec2 --nostringval-- --nostringval-- -- nostringval-- 2 %stopped_push --nostringval-- --nostringval-- % loop_continue 1753 3 3 %oparray_pop --nostringval-- --nostringval-- false 1 %stopped_push .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- 1741 2 3 %oparray_pop --nostringval-- Dictionary stack: --dict:1148/1684(ro)(G)-- --dict:1/20(G)-- --dict:70/200(L)-- Current allocation mode is global Last OS error: 2 Current file position is 23 --------------------------- Is there anything wrong in this? The same steps works fine on windows. We tried GS version 8.70 In some comment it is mentioned that pdfwrite is not supported in the GS server mode. So I get the same error for pdfwrite device also. Our understanding was, only pdfwrite is not supported in GS server mode, but on Unix, we are seeing the issue with pswrite, ps2write device also, it works fine with 'ljet4' and 'psmono' device. Hope this is clear now, could you please help us here. Thanks Subbu
Since the start-up time dominates, we performed profiling and tracing of Ghostscript simply starting up and then quitting. This led to reducing the number of different paths that were searched for the PostScript init files and Resource files (including fonts) as well as turning off the IdiomRecognition that slows down all ‘bind’ operations. Also, there are garbage collection actions of the PostScript ‘VM” that are executed during the initialization and start of jobs that are optional. Instead the ‘vmthreshold’ operator can be used to allow garbage collection to only occur when it is required by the memory allocation and ignore the ‘vmreclaim’ operator. The changes were selected because these are readily applied to the 8.70 and 8.71 PRE-RELEASE versions (8.71 will be released in early February 2010) and thus do not require a major effort for HP to apply to these and subsequent releases. The methods to achieve these performance improvements are detailed below. All times for our measurements are ‘real’ time seconds (also called wall clock times) provided by the linux shell ‘time’ function. The ‘user’ time collected during the runs correlates to the ‘real’ time reported by the shell. The timings given by HP were: ps->pdf ps->pcl ps->ps pdf->ps HP 8.54 1423 1401 1433 1747 HP 8.63 2748 1924 2677 2434 HP 8.63 / 8.54 93% 37% 87% 39% For these particular versions, the timings for 50,000 jobs (files “input.ps” and “input.pdf” from the bug 690352) on our server were: ps->pdf ps->pcl ps->ps pdf->ps gs 8.54 4480 4145 3815 5515 gs 8.63 5885 5510 5370 6995 gs 8.63/gs 8.54 31% 33% 41% 27% As mentioned above, the observed percentage performance degradation from 8.54 to 8.63 did not match those reported by HP for ps->pdf and ps->ps but were similar for ps->pcl and pdf->ps. The timings we observed for 8.53 vs. 8.70 (both “out of the box” builds from the releases) were: ps->pdf ps->pcl ps->ps pdf->ps gs 8.53 4295 4170 3575 5315 gs 8.70 6260 5615 4945 7010 8.70 original /8.53 46% 35% 38% 32% After the optimization improvements to 8.70, and similar changes to 8.71 PRE-RELEASE the timings and relative performance are: ps->pdf ps->pcl ps->ps pdf->ps gs 8.53 4295 4170 3575 5315 Optimized gs 8.70 4020 3545 3425 5700 optimized 8.70 / 8.53 -6% -15% -4% 7% Optimized gs 8.71 3775 3475 3105 5475 optimized 8.71 / 8.53 -12% -17% -13% 3% The improvements were substantial and exceeded the goal specified in the SOW, and the changes to the code base since 8.70 that are in the 8.71 PRE-RELEASE provide even greater performance gains for the this usage of Ghostscript. Detailed description of the optimization. 1. The ‘autogen.sh’ was used to generate a ‘Makefile’ for the system with the parameters: ./autogen.sh --disable-contrib --disable-fontconfig –disable-cups In particular, the ‘fontconfig’ was determined to be a major contributor to the slowdown of releases. Support for fontconfig was added after the 8.53 release. 2. The resulting ‘Makefile’ was modified to set: SEARCH_HERE_FIRST=0 GS_LIB_DEFAULT= COMPILE_INITS=0 Note that these changes could be performed as command line macros on the make invocation, but it making the changes directly to the Makefile allowed for easy tracking of the differences. 3. The following command line options were added to the invocation of bin/gs: -I.:Resource/Init –dNOGC -sGenericResourceDir=Resource/ -c "\ << /IdiomRecognition false >> setuserparams" –f The ‘\’ in the above line is not needed but is shown for to indicate that the line is to be joined. Since the GS_LIB_DEFAULT paths were eliminated by the changes that were made in (2), explicit search paths are established by the -I.:Resource/Init and the Resource file locations are specified by -sGenericResourceDir=Resource/ IdiomRecognition is disabled using the setuserparams operator so that the extra processing of the bind operator can be avoided. Testing Details (scripts). The scripts make use of ‘shell arithmetic’ to increment the ‘x’ count variable. Timings of the loop structure, without the bin/gs invocation within the loop were performed to confirm that the shell loop overhead was less than 1%. The script used to perform the timing on the 8.53 release, built with the default build settings (which did not include COMPILE_INITS=1) was: echo Revision: 8.53 >> time.log COUNT=50000 O=time.log date >> time.log echo "================= PS to PDF ==========================" >> time.log time ( x=1; while [ $x -lt $COUNT ] ; do bin/gs -I../fonts -q -sDEVICE=pdfwrite -sOutputFile=x.pdf -dBATCH -dNOPAUSE ../bugs/dazel/Bug_690352_input.ps >> $O 2>&1 x=$(( $x + 1 )) done ) >> time.log 2>&1 date >> time.log echo "================= PS to PCL (ljet4) ===================" >> time.log time ( x=1; while [ $x -lt $COUNT ] ; do bin/gs -I../fonts -q -sDEVICE=ljet4 -sOutputFile=x.pcl -dBATCH -dNOPAUSE ../bugs/dazel/Bug_690352_input.ps >> $O 2>&1 x=$(( $x + 1 )) done ) >> time.log 2>&1 date >> time.log echo "================= PS to PS (pswrite) ==================" >> time.log time ( x=1; while [ $x -lt $COUNT ] ; do bin/gs -I../fonts -q -sDEVICE=pswrite -sOutputFile=x.ps -dBATCH -dNOPAUSE ../bugs/dazel/Bug_690352_input.ps >> $O 2>&1 x=$(( $x + 1 )) done ) >> time.log 2>&1 date >> time.log echo "================= PDF to PS (pswrite) =================" >> time.log time ( x=1; while [ $x -lt $COUNT ] ; do bin/gs -I../fonts -q -sDEVICE=pswrite -sOutputFile=x.ps -dBATCH -dNOPAUSE ../bugs/dazel/Bug_690352_input.pdf >> $O 2>&1 x=$(( $x + 1 )) done ) >> time.log 2>&1 date >> time.log The script used for 8.70 and 8.71 test (which show the additional command line options was: echo Revision: 8.70 optimized > time.log COUNT=50000 O=time.log date >> time.log echo "================= PS to PDF ==========================" >> time.log time ( x=1; while [ $x -lt $COUNT ] ; do bin/gs -sGenericResourceDir=Resource/ -q -I.:Resource/Init -sDEVICE=pdfwrite -sOutputFile=x.pdf -dBATCH -dNOPAUSE -dNOGC -c "<< /IdiomRecognition false >> setuserparams" -f ../bugs/dazel/Bug_690352_input.ps >> $O 2>&1 x=$(( $x + 1 )) done ) >> time.log 2>&1 date >> time.log echo "================= PS to PCL (ljet4) ===================" >> time.log time ( x=1; while [ $x -lt $COUNT ] ; do bin/gs -sGenericResourceDir=Resource/ -q -I.:Resource/Init -sDEVICE=ljet4 -sOutputFile=x.pcl -dBATCH -dNOPAUSE -dNOGC -c "<< /IdiomRecognition false >> setuserparams" -f ../bugs/dazel/Bug_690352_input.ps >> $O 2>&1 x=$(( $x + 1 )) done ) >> time.log 2>&1 date >> time.log echo "================= PS to PS (pswrite) ==================" >> time.log time ( x=1; while [ $x -lt $COUNT ] ; do bin/gs -sGenericResourceDir=Resource/ -q -I.:Resource/Init -sDEVICE=pswrite -sOutputFile=x.ps -dBATCH -dNOPAUSE -dNOGC -c "<< /IdiomRecognition false >> setuserparams" -f ../bugs/dazel/Bug_690352_input.ps >> $O 2>&1 x=$(( $x + 1 )) done ) >> time.log 2>&1 date >> time.log echo "================= PDF to PS (pswrite) =================" >> time.log time ( x=1; while [ $x -lt $COUNT ] ; do bin/gs -sGenericResourceDir=Resource/ -q -I.:Resource/Init -sDEVICE=pswrite -sOutputFile=x.ps -dBATCH -dNOPAUSE -dNOGC -c "<< /IdiomRecognition false >> setuserparams" -f ../bugs/dazel/Bug_690352_input.pdf >> $O 2>&1 x=$(( $x + 1 )) done ) >> time.log 2>&1 date >> time.log
Changing customer bugs that have been resolved more than a year ago to closed.