Bugzilla – Bug 693121
/tmp file not deleted
Last modified: 2015-01-03 11:27:24 PST
Temporary files are not being deleted properly during the cluster runs. It would be good to understand why gs is leaving the /tmp files around (i.e. this bug is not a suggestion to add an "rm" command to the regression procedure)
I don't understand why removing the /tmp/gs_* files is not a good idea -- the
clist and pdfwrite temp files are really of no use that I know of.
Reporting how many files were deleted, or after running every test spitting
out a message to the cluster log indicating which job left junk laying around,
then removing the /tmp files seems like a better approach.
Is this possibly while 'miles' and 'kilomters' are dead ?
BTW, the underlying issue is that Ghostscript doesn't keep track of tempfiles
and make sure and delete them in gs_abort (or a SEGV). In order to clean up,
we'd need to do both.
(In reply to comment #2)
> BTW, the underlying issue is that Ghostscript doesn't keep track of tempfiles
> and make sure and delete them in gs_abort (or a SEGV). In order to clean up,
> we'd need to do both.
Oh for some reason I assumed we were using tmpfile() which would delete the files automatically. This seems like a bug we should fix. I didn't look carefully but doing a system call trace I see unlink calls on the /tmp files in normal operation. So I assume these files we see being left behind only happen with errors. tmpfile() is the easiest solution but I am not sure how portable that is.
(In reply to comment #3)
> Oh for some reason I assumed we were using tmpfile() which would delete the
> files automatically. This seems like a bug we should fix. I didn't look
> carefully but doing a system call trace I see unlink calls on the /tmp files in
> normal operation. So I assume these files we see being left behind only happen
> with errors. tmpfile() is the easiest solution but I am not sure how portable
> that is.
From K&R2: "tmpfile creates a temporary file... that will be automatically removed when closed or when the program terminates normally."
So tmpfile should be available in any ANSI C implementation, but it may still leak - certainly in the SEGV case, and maybe in the gs_abort() case.
> From K&R2: "tmpfile creates a temporary file... that will be automatically
> removed when closed or when the program terminates normally."
> So tmpfile should be available in any ANSI C implementation, but it may still
> leak - certainly in the SEGV case, and maybe in the gs_abort() case.
The GNU stuff lib doc says:
This function creates a temporary binary file for update mode, as if by calling fopen with mode "wb+". The file is deleted automatically when it is closed or when the program terminates. (On some other ISO C systems the file may fail to be deleted if the program terminates abnormally).
but I guess we should consider doing a portable complete implementation if we do anything at all about this.
Temp files are left around during cluster runs because the cluster software terminates ghostscript jobs that timeout by calling kill. This causes any temp files opened by that job to be orphaned. However, based on customer comments that they occasionally have ghostscript temp files showing up as well, it seems likely a seg fault can also leave temp files behind.
The only solution I've been able to come up with is to unlink the temp file immediately after it is opened. Testing shows this results in no temp files remaining, even if ghostscript is killed, however, there are two problems with this approach: I don't think it is portable to Windows and it causes three of the regression files to fail:
In each case the error message is of the form:
Copyright (C) 2012 Artifex Software, Inc. All rights reserved.
This software comes with NO WARRANTY: see the file PUBLIC for details.
% Start time = 0.227705, memory allocated = 3419480, used = 3214730
Error: /undefinedfilename in --run--
(1 ) (/tmp/gs_e7cgeO) (/tmp/gs_e7cgeO) (r)
%interp_exit .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- --nostringval-- false 1 %stopped_push 1900 1 3 %oparray
--dict:1169/1684(ro)(G)-- --dict:1/20(G)-- --dict:78/200(L)--
Current allocation mode is local
Last OS error: No such file or directory
GPL Ghostscript 9.07: Unrecoverable error, exit code 1
% Final time = 0.31234, memory allocated = 3421368, used = 3135866
% Exiting instance 0x0x1d4d350
Command exited with non-zero status 1
These three files are PDF Portfolio files and reading them causes Ghostscript to open attempt to the temp file after it is closed, which doesn't work since it disappears when closed is called. Presumably there could be a parameter added to gp_open_scratch_file_generic() to indicate that the temp file being opened needs to be kept around, but I'm sure if this is a reasonable solution.
Re-assigning to Henry to decide how to proceed.
1. There's a way to create files on Windows that disappear when they are closed.
2. Processing of portable collections can be modified to use file objects instead of the file names.
I thought there was a problem doing this with clist files because the clist code closes and reopens its temp files. Thus they can't just disappear when they're closed.
The clist MT code closes the temp files and re-opens them. This is because
Windows does not seem to allow shared 'read' access to a file that is still
opened for writing (even with w+ mode). On linux, opening the files using
fdopen(fileno(tempfile), "rb") works, iirc, but we'd need to verify that
since once I encountered the problem with Windows, I changed the code.
If there is a method for Windows to open a file that is still open for "w",
that should allow the Windows code to be changed to use the 'delete on close'
(In reply to comment #7)
> 1. There's a way to create files on Windows that disappear when they are
Does this include if Ghostscript seg faults or is killed?
tmp files are left behind on the cluster because ghostscript crashes or is killed because it times out, otherwise the would be deleted when Ghostscript exits, even if they weren't closed. Searching the internets shows this is a common issue; the tmp file deletion handler don't get called when abnormal termination occurs.
I think we agreed Chris was going to research this bug and report back to Ray, so changing assignment and CC: list.
On IRC we discussed putting our temp files in a directory, and clearing
that directory when we start (or possibly even upon normal exit).
To prevent "windows" where files are closed, but still in use (as the
clist MT rendering does) we discussed using a 'marker' file to disable the
deletion while the file is closed. This is needed if more than one gs
instance is running.
This needs some feasibility testing.
I believe Ray and I have discussed this, and exhausted my knowledge/ideas on the subject.
If you re-work how Ghostscript deals with temporary files,
I like to bring up a related issue:
There is a general move away in various Linux distributions
from /tmp/ being on harddisk to /tmp/ as a tmpfs (i.e. basically
a special kind of RAM disk that can be swapped out if needed).
If temporary files in Ghostscript can be huge, it could be a problem
to have them on a tmpfs (may eat up RAM and/or become suddenly
very slow when swapping happens).
As far as I know having /tmp/ on tmpfs is related to systemd
and as far as I know the systemd developers demand that /var/tmp/
has to be used nowadays for bigger temporary files.
But as far as I understand it, using /var/tmp/ in general would be
not in good compliance with the Filesystem Hierarchy Standard.
But when Ghostscript itself carefully cleans up its temporary files,
I think it can use /var/tmp/ by default as well.
Perhaps a configure option "--with-tmpdir" to specify the directory
where Ghostscript will store its temporary files would be best?
At openSUSE we have this Wiki page
I am afraid, I could not help if you have further questions.
I am not at all an expert here - I act only as messenger.
Update with new info regarding clist temp files.
I made some changes as I implemented the BGPrint and am now able to open the
clist temp files for threads WITHOUT closing the main thread's files. This
means that we can now switch to having temp files "delete on close" (which
is platform dependent, of course).
I will try this, but I think we are OK.
With --saved-pages, we need to be able to close the clist files and not have
them go away. At the very least, this will require an internal change so that
the clist can open clist files in a 'retain' mode, and if we do that and crash,
the clist files will not be deleted.
This may require us to hook uncaught exceptions and make sure and delete a
list of files (or at least try to).
Taking this back to review. Thanks, Paul.
I'll post comments here, but I am a bit concerned about the use of unbuffered
IO, so I might try some tests on an old slow laptop I have (kept for just such
The nightly regression run shows extensive slowdown which I'm able to reproduce
on Windows on my laptop.
gswin32c -q -Z: -sDEVICE=pbmraw -o nul: -r300 -dMaxBitmap=0 $* /tests/pdf/Bug6901014_Fourth_crash_signature_testcase.pdf
takes 35 seconds but less than 5 seconds with -sBandListStorage=memory
I have a patch in progress that adds caching that reduces the time back to
less than 5 seconds.
Performance improved by addition of a cache, commit 64f9cafc