s/Generice/Generic/ in above TO REPRODUCE instructions.

(In reply to comment #0)

> In this form, the workaround is successful. However, it is undesirable
> because the user's environment setup (.profile for example) must reflect
> explicit knowledge of where ghostscript's shared Resource directory is,
> and the version of ghostscript currently installed.

Or you could use -I to add a path to the search.

This seems to have been broke for some time, it would be more helpful to quote a version number where this actually works.

Using -I doesn't work for resources of course, unless you add a complete resource tree.

Do you get the warning about GenericResourceDir not pointing to a valid resource directory ? (I don't see how you can avoid this).

> This seems to have been broke for some time, it would be more helpful
> to quote a version number where this actually works.

Sorry - I had just done a Fink upgrade without noting what the
pre-upgrade version was. Fortunately, Fink seems to keep the old .deb
around: ghostscript_8.61-5_darwin-i386.deb and judging by the timestamp
I had installed it in August 2011. It might not have been the current
version even then, but it's what was packaged in Fink.

> Do you get the warning about GenericResourceDir not pointing to a valid
> resource directory ?

If I set GS_LIB with ghostscript's own Resource directory ahead of the
user's, no. In any other case, yes. In any case I was going to file a
separate bug report about this warning ... not that the warning's a bad
idea, but the test used to generate it doesn't seem to be the right test.

Note that putting ghostscript's Resource directory ahead of the user's
in GS_LIB is not a completely adequate workaround for this bug, because
user-dir-ahead-of-system-dir is needed for the use case described in
http://anastigmatix.net/postscript/Packager.html#ofn

> In any case I was going to file a separate bug report about this warning
> ... not that the warning's a bad idea, but the test used to generate it
> doesn't seem to be the right test. (bug 694512)

This is terribly hard to investigate given that the change seems to have occurred over 7 years ago. I no longer have the relevant version of the development environment for Windows installed, and even the Linux environment seems to have changed somewhat.

A Git bisect (run for me by Chris) identifies the culprit commit as a2d599de but all that does is set COMPILE_INITS=1 as the default. There are 4 recently prior (within 6 months) commits referencing COMPILE_INITS which were committed by Ray. I'm going to pass this one over to him.

My own personal preference would be to remove the documentation about relative paths in GenericResourceDir, since it clearly doesn't work. Given that nobody has noticed this in the past 7 years, this can't be a hugely important feature. Copying the Ghostscript standard resources to a local folder, adding extra resources there and then specifying an absolute GenericResourceDir, allows for the use of custom resources with only a very small extra use of disk space.

This would also resolve the related bug #694512 since the warning would be correct. Note that since the warning is emitted by gs_res.ps, which is located in Resource/Init its not at all clear to me how you could trigger this warning, except of course by attempting to use a relative path for GenericResourceDir.

If it seems that bleak at the moment, I hope you'll consider
leaving this bug open for a bit before you pull the trigger
and close it with a doc "fix". I'll see if I can make time to
look further into it.

I think the judgment above ("can't be a hugely important feature"
because "nobody has noticed this in the past 7 years") might not
reflect some of the unique or unusual properties of ghostscript
as a product:

* It has several different audiences (people who only see it as
a viewer for .ps files, OEMs embedding it in devices, people who
use it as an environment for programming in PostScript). These
audiences will differ in what they notice, and will also differ in
how frequently they update to the latest gs versions.

* Because ghostscript is an implementation of a language whose
last significant spec revision was something like 16 years ago,
those audiences who use it as either a viewer for .ps files or
as a PostScript programming environment are not subject to a press
of new features encouraging them to update. This is true not only
at the level of individual users but for syadmins and package
repository maintainers also. You can already see from my own experience
in this report that the reason I didn't notice the bug until August 2013
is because it wasn't present in the package-repository version that
I installed in 2011. If the bug's been present for seven years then that
package repository two years ago was at least five years behind ... and
many people don't notice or care about /that/ either, as long it works
as an implementation of the language. Between repository maintainers,
sysadmins, and end-users, there can be three layers of delay that make
it dangerous to draw conclusions from the timing of a bug's discovery.

* What's worse, it may *contribute* to that delay in updating at all
three levels (packagers, admins, and end users) if their experience
tends to be that updating the product may expose them to random breakage
or disappearance of fully documented features.

* Leaving aside the time-to-discovery issue, there's another point
that weighs against concluding that this "can't be a hugely important
feature": look at other commonly used implementations for programming
or scripting languages. Can you think of any that lack a comparable
feature? Consider what you'd have to do if Python or Perl, say, didn't
understand a path of contributed, site-specific, and user-specific modules,
and forced you to do what you suggest would cost only a "small extra use
of disk space":

- If you were a sysadmin relying on a package repository system, you'd
have to catch every time a Python or Perl update got installed (maybe
by writing a yum plugin or whatever) and respond by regenerating your
directory tree of site additions with copies of the new distributed files.

- If you were a user on that system, you would have to catch every time
that happened plus every time the admins update or change site-specific
additions, and re-copy all of that stuff into your local directory.

There's a good reason so many languages have a module path feature, and
a good reason ghostscript used to be one of them.

Also, the use case I linked above (http://anastigmatix.net/postscript/Packager.html#ofn)
is an example of where a "make a big copy of everything" workaround
doesn't solve the problem.

It seems as if the biggest problem here is that a regression didn't
get caught promptly because a regression test was lacking for the
documented behavior. In my opinion, the far better resolution would
be to restore the documented behavior, and I'll try to look a bit
further for the problem and do what I can to help. If it can be fixed,
and you show me where the regression tests go in your build system (I
looked around some in the git repo but didn't catch on to the plan),
I'll also be happy to write one for this feature so it doesn't
get clobbered again.

(In reply to comment #7)
> If it seems that bleak at the moment, I hope you'll consider
> leaving this bug open for a bit before you pull the trigger
> and close it with a doc "fix". I'll see if I can make time to
> look further into it.

I'm sure Ray will consider your comments before deciding, however....


> * It has several different audiences (people who only see it as
> a viewer for .ps files, OEMs embedding it in devices, people who
> use it as an environment for programming in PostScript). These
> audiences will differ in what they notice, and will also differ in
> how frequently they update to the latest gs versions.

We are well aware of this.

 
> * Because ghostscript is an implementation of a language whose
> last significant spec revision was something like 16 years ago,

Given that Ghostscript itself also processes PDF files, this is not actually true, its the case only for PostScript files.


> as an implementation of the language. Between repository maintainers,
> sysadmins, and end-users, there can be three layers of delay that make
> it dangerous to draw conclusions from the timing of a bug's discovery.

Given the sheer number of users Ghostscript has, if it isn't discovered in a reasonable period of time, its not a widely used feature.


> * Leaving aside the time-to-discovery issue, there's another point
> that weighs against concluding that this "can't be a hugely important
> feature": look at other commonly used implementations for programming
> or scripting languages. Can you think of any that lack a comparable
> feature? 

The feature you are referring to is not a feature of the language, but of the implementation. The language does not offer this feature at all so the language actually does lack the feature. It is also, as you pointed out, a rather elderly language now, not having been revised for 14 years.


> There's a good reason so many languages have a module path feature, and
> a good reason ghostscript used to be one of them.

There is still such a feature, and even for the case of resources, there is. What there isn't is a 'search list' of directories. You can still have multiple installations and switch between them easily enough.

 
> Also, the use case I linked above
> (http://anastigmatix.net/postscript/Packager.html#ofn)
> is an example of where a "make a big copy of everything" workaround
> doesn't solve the problem.

I didn't read this in enough detail to understand why this is a problem (in fact I'm not sure there is enough detail here to understand why this is a problem).

However, you seem to be arguing somewhat circularly here, given that this is one of your own applications. Obviously we are aware that you use this feature, but it is not clear to me from that link why simply having multiple full copies of the resources doesn't work.

 
> It seems as if the biggest problem here is that a regression didn't
> get caught promptly because a regression test was lacking for the
> documented behavior. In my opinion, the far better resolution would
> be to restore the documented behavior,

No, the problem here is that because a regresison didn't get caught for a VERY long time I *can't* 'restore the documented behaviour' because so much has changed in the last 7 years. In fact, as I stated before, so much has changed that I cannot build the old version on Windows, and going back any earlier than we have already gone (because even that commit is not the *source* of the problem) caused the build to fail on Linux, we think because the system is 64-bit and part of the build system doesn't want to work on it.

Trying to return to such an elderly build is now too much effort for a free user, which means we can't examine the code as it worked before.

So really what you are asking for is not to 'restore the behaviour' its to 'recode the behaviour from scratch', in effect its a request for a new feature.


> further for the problem and do what I can to help. If it can be fixed,
> and you show me where the regression tests go in your build system (I
> looked around some in the git repo but didn't catch on to the plan),

You can't run our regression tests from outside our system I'm afraid. Also the way our regression tests run I don't think it would be feasible to test this. While ideally we would like to regression test everything, Ghostscript has so many features, a large number of which interact, that its already impossible to test every feature and combination. 

I don't have the exact numbers to hand but I believe currently we run some 60,000 tests which takes around 30 minutes on a cluster of 12 machines. Those are the tests run on every commit to the repository, there are additional nightly and weekly tests which test still more features or combinations and those I think are already at saturation point (ie adding more tests would mean that the next test would be due before the previous one had finished)

I appreciate that this is a feature which you find useful, and you argue eloquently for its inclusion, nevertheless so much time has passed since this feature was broken that it is likely to take significant effort for us to resurrect it. In my opinion it is more effort than can be justified for the utility of the feature, but its up to Ray now.

(In reply to comment #8)
> > * Because ghostscript is an implementation of a language whose
> > last significant spec revision was something like 16 years ago,
> 
> Given that Ghostscript itself also processes PDF files, this is not actually
> true, its the case only for PostScript files.

I did leave myself open for that, didn't I? :) But again, this highlights
the differences in what different audiences see. The PDF support is of
interest to the "just give me a viewer" crowd and the "I'm a printer OEM"
crowd, but because one doesn't write programs in PDF, it's not of equal
interest to the programming crowd (other than as a convenient way to distill
the output of a program into a static document). As far as the /programming
language/ gs implements is concerned, the spec is pretty mature.

> > * Leaving aside the time-to-discovery issue, there's another point
> > that weighs against concluding that this "can't be a hugely important
> > feature": look at other commonly used implementations for programming
> > or scripting languages. Can you think of any that lack a comparable
> > feature? 
> 
> The feature you are referring to is not a feature of the language, but of
> the implementation. The language does not offer this feature at all so the
> language actually does lack the feature.

I don't think I left myself as wide open in this case, because I did say
"implementations for programming and scripting languages." I'm pretty
sure there are other examples (not trying to enumerate them right now)
where a programming language spec itself didn't address the need for
segregating distribution/conrib/site/user/etc. search areas, but developers
of an implementation pretty much as a matter of course understood what's
wrong with not having that, and addressed it as an implementation feature.
Seeing that exactly that happened years ago with ghostscript itself (and
before my involvement, so at least one other person saw why it would be good),
I'll stick with my story here.

> There is still such a feature, and even for the case of resources, there is.
> What there isn't is a 'search list' of directories. You can still have
> multiple installations and switch between them easily enough.

You've responded directly to several points in my earlier comment, but
not directly to the thought experiment of what admins' and users' jobs
would look like at the time of every update if, say, Python or Perl worked
that way. I think if you were to begin a response to that point, you
would convince yourself that "have multiple installations and switch
between them" doesn't really address it. (Or, you'd convince me that it
does and I simply haven't seen how. One of those things would happen.)

> > (http://anastigmatix.net/postscript/Packager.html#ofn)
> > is an example of where a "make a big copy of everything" workaround
> > doesn't solve the problem.
> 
> I didn't read this in enough detail to understand why this is a problem (in
> fact I'm not sure there is enough detail here to understand why this is a
> problem).

That particular use case involves setting up a build environment for
resources (some of which require bootstrapping, i.e., working prior
versions available) where the bootstrap versions can be found when the
new ones haven't been written yet, and the new ones are naturally written
into an earlier-searched directory without replacing the bootstrap versions
until after testing. Ghostscript's old behavior really shone for this
purpose; there was virtually no effort needed to make it work that way.
What coding would be necessary to hack up any kind of similarly useful
behavior without a path mechanism, I haven't begun to estimate yet. :(
It would probably end up resembling the current build process wrapped in
some ugly shell script that shovels things around into temp directories,
take as long to get working as a resource-path patch for gs, and be less
satisfying and generally useful.

> However, you seem to be arguing somewhat circularly here, given that this is
> one of your own applications.

I'm not sure if I'm caught in a catch-22 here ... I wanted to include
an example from my own use so I would not seem to be contriving general
arguments without disclosing a personal stake ... but then providing
that example may give an appearance of circularity. The thing is, I've
also offered general arguments that aren't just contrivances; the example
use case of my own was to be one among them.

> I don't have the exact numbers to hand but I believe currently we run some
> 60,000 tests which takes around 30 minutes on a cluster of 12 machines.
> Those are the tests run on every commit to the repository, there are
> additional nightly and weekly tests which test still more features or
> combinations and those I think are already at saturation point (ie adding
> more tests would mean that the next test would be due before the previous
> one had finished)

It's reassuring to hear that there are so many regression tests already.
But it's still not reassuring that something clearly documented with its
own second-level heading in Use.htm just got accidentally clobbered at
some unknown time in the past. If there were one or a few regression tests
in your suite for each first- and second-level heading in Use.htm, it seems
that would be no more than 0.1% of the tests you already run. I'm sure many
of the tests that are currently in the suite test much more subtle things.
That's good, but it's good to test the obvious stuff too.

I'm throwing stones from a glass house; I've written a lot of code without
good test coverage, but I can sometimes be talked into adding a test if I
see that I've broken something without it.

Clearly the whole question of testing may be moot depending on whether
Ray decides to abandon the feature. Again, I'm willing to at least try
my hand at a patch if you don't close with a doc fix right away. But
I'm pretty sure a test for this feature would add very few cycles to
the testing.

> I appreciate that this is a feature which you find useful, and you argue
> eloquently for its inclusion, nevertheless so much time has passed since
> this feature was broken that it is likely to take significant effort for us
> to resurrect it. In my opinion it is more effort than can be justified for
> the utility of the feature, but its up to Ray now.

Thanks for listening and taking the time for thoughtful replies.

Hi, I stumbled on this bug report after spending considerable time failing to get Ghostscript to load an external resource file in the manner (fwiw, I'm using the version of Ghostscript in Debian wheezy, GPL Ghostscript 9.05 (2012-02-08)).

Being able to load external named resources from a search path that includes user-specific resources, site-specific resources and common system resources strikes me as a very useful feature to have. Practically every programming language has such a feature for locating modules/library files etc.

I suspect the fact that this feature was unnoticed for so long indicates that the Ghostscript mechanism for doing this was poorly documented, error-prone and overcomplicated (for example, it's hardly intuitive that GenericResourceDir is interpreted as the only resource dir if it's an absolute path, and an entry in a list of search paths if it's relative...), such that users who tried to load external resources eventually gave up and included the resource definition in the documents they were printing instead. It's something of a chicken and egg situation - if the method for loading external resources is complicated and error prone, few people will use external resources; if few people use external resources, then few will notice when said mechanism finally breaks...

Therefore, I would like to suggest that you consider reimplementing the ability to search multiple locations for named resources without trying to exactly replicate the previous mechanism which is documented and broken - after all, if no-one was using it there aren't any backwards-compatibility issues to consider. And reimplementing this functionality from scratch is possibly easier than tracking down the regression.

I don't think that the suggested work-around - having each user create a complete copy of the Ghostscript library hierarchy - is viable. Users would have to maintain such directories, recreating or updating them each time a new version of ghostscript was installed or new site-specific resources were installed. Of course this task could be automated, but automating the task of compiling a single resource directory from multiple source directories is more complicated than simply having Ghostscript search those directories at resource load time in the first place.

I'd like to fix this bug, similarly to the way described by the immediately previous comment and the advice given to me by rayjj in IRC. Specifically, I want to make three changes:

 1. Add a `LocalResourceDirList` system parameter in the file "gs_res.ps" which, by default, is an empty array.

 2. Since this parameter is a list, and not an integer or string, the current options for specifying system parameters aren't sufficient. I can either add a new option for specifying arrays (it looks like neither "-l" for list nor "-a" for array are currently in use), add a special case to the "-s" option to handle this parameter, or set the parameter in string form and then convert to a list as-needed.

I prefer the first choice since it's most consistent with the intent of already-existing behavior (that is, having an option for each type of parameter); however, there is the issue of syntax. Postscript's syntax for lists and strings could work, but it would be rather complicated to implement without allowing free-form input (unless the scanner is flexible enough in that regard). Additionally, depending on the directory name, it could require many awkward escape sequences. Another option is to use whatever path-separated-list format the operating system uses, but it could be confusing and awkward if in the future additional array parameters are added that have nothing to do with directories. A third option is to find some unique list format that takes all platforms into consideration, but I don't know the syntax of non-Unix/non-Windows directories, so I don't know if it's workable, and it would be rather complicated. To avoid all that, a dedicated option for this particular parameter, or an option for "directory list" parameters could be added, and then the platform-specific path-separated-list format could be used. That has the downside of confusing the user about whether the option actually refers to a system parameter, as it's either not being represented as a generic system parameter (but as a dedicated option) or it's being represented as a type that doesn't exist in Postscript (a "directory list"). I think that this is a minor downside, and I suggest representing it as a "directory list type" system parameter, perhaps using the flag "e" as "d" is already taken.

The second choice, adding a special case to the "-s" option, has the benefits of allowing the user to use the path-separated-list syntax of their operating system, but it has the downside of adding complexity to the parsing of that option as well as being inconsistent with the pattern already established of having an option for each type of parameter.

The third option, converting the parameter as-needed from a string to a list, is doable, but imo will unnecessarily complicate the `lib_file_open_search_with_combine` function as well as trigger redundant processing of the parameter every time that function is called.

 3. Finally, in `lib_file_open_search_with_combine`, check each directory in `LocalResourceDirList` for the resource provided it's not an empty list. I see a variety of ways to access this parameter and I'm not entirely sure which is preferred. It can be accessed directly with `zgetsystemparam` or `currentparam1`; or it can be accessed by obtaining and going through the list returned by `zcurrentsystemparams`, `currentparams`, and `current_param_list`. My guess is that `zgetsystemparam` is the way to go, but I figured it was best to ask.

I forgot one thing: should this list be searched before or after LIBDIR? Before would allow the user to override the existing resources, but it has the downside of increasing the search time in cases where the default resource is desired, which I would guess is the most common case. I personally think it should be done after because the ability to override existing resources doesn't seem to be a valuable enough feature to justify the increased search time.

s/LIBDIR/LIBPATH

I haven't added info to this bug in a while, but I have been looking at it.

For starters, I'm now using a workaround that accomplishes (most of) what
the old behavior was giving me.  From exactly what is and isn't working, one
can learn more about what the code is actually doing.

Although the documented behavior of trying GenericResourceDir appended to
elements of GS_LIB doesn't happen any more, if I add a resource directory, e.g.,
/usr/share/site-ghostscript/Resource at the front of GS_LIB, ghostscript will
find resources there; no fussing with GenericResourceDir needed. This will seem
encouraging in simple tests, but will turn out to have broken font loading in
a confusing way, where it first reports finding the proper font file, then
reports being unable to load it and substituting another, then finally conks
out on /invalidaccess.  What it turns out to be doing is looking in every place
but the right one for encoding vectors, and to fix that, the installation's
original Resource directory needs to also be mentioned in GS_LIB:

GS_LIB=/usr/share/site-ghostscript/Resource:\
/usr/share/ghostscript/<ver>/Resource

Then it works happily. It still gives the totally spurious warning reported
as bug 694512 but works fine. (That warning's a good example of how LBYL can
introduce bugs that EAFP wouldn't.)

It's not perfect to have to explicitly specify the distribution's own
Resource directory, as that path contains a version, requiring the site's
environment settings to be changed every time gs is updated. A worthwhile,
simple change would be for gs to supply that element of the path automatically,
so the site environment variable only has to mention the site directories.
(And lose that bug 694512 warning.) But as a workaround, this is mostly
doing what I need.

There is, however, still one piece of documented behavior that doesn't happen.
In section 10.2 under resource-related parameters, USe.htm says:

  Due to the extended search method, Ghostscript uses GenericResourceDir only as
  a default directory for resources being not installed. Therefore
  GenericResourceDir may be considered as a place where new resources to be
  installed. The default implementation of the function ResourceFileName uses
  GenericResourceDir when (1) it is an absolute path, or (2) the resource file
  is absent. The extended search method does not call ResourceFileName .

That is, given that design, you can use ResourceFileName to find the directory
in which a newly to-be-installed resource should be put; by setting
GenericResourceDir to an absolute path, one can make ResourceFileName return
this intended (e.g. writable) location even when there is already a version
of the same resource later on the path. The trouble is, the test (1) was lost
in this change:

http://git.ghostscript.com/?p=ghostpdl.git;a=commit;f=gs/Resource/Init/gs_res.ps;h=bb40d003b2d5578874600559c5c2511546c0c46f
http://bugs.ghostscript.com/show_bug.cgi?id=691408#c37
... leaving only the test (2); you only get the intended installation location
if no version of the resource is already installed. The comment explaining the
change where test (1) was lost did not seem to show familiarity with the
reasons test (1) was there, which are in this commit message:

http://git.ghostscript.com/?p=ghostpdl.git;a=commit;f=gs/doc/Use.htm;h=c5253751c494869b9bd8bb7752be077204e19958

But it shouldn't be hard
to restore that lost test, and be back to working according to the behavior
described in 10.2. The different, 8.1 described behavior of tacking
GenericResourceDir onto each GS_LIB element could then be removed as
documentation rot, and the actual behavior could be documented better
in its place.

Interestingly, this actual behavior that still mostly works (and should be
easily restorable) seems to predate the 8.1-described, now rotted behavior,
which seems to have been born here:

http://git.ghostscript.com/?p=ghostpdl.git;a=commit;h=1ddf1f5b73c6eeccf5539a7555416fe36f6cff2f

It seems as if the road that led to this point involved at least two different
ideas of what the behavior should be, neither of which had regression tests,
which could have alerted the different people quasi-independently working on
those parts of the code to the possible conflicts in what they were doing.

... Nothing I just said means I'd be against implementing another all-new, all-different way to get the desired behavior, as Jordy seems to be describing,
though possibly that's overkill if one of the earlier behaviors can be easily resurrected.

But if an all-new approach is to be added, it will definitely help a lot to carefully revise both places in Use.htm where the older approaches were described, to remove the traces of those and make sure what's documented is only what's intended to work. And ideally any new approach will be able to accomplish the same things that were documented for the old approach(es) when it(they) worked, such as being able to use ResourceFileName to find a writable location to supersede an existing resource, as in the commit comments here:

http://git.ghostscript.com/?p=ghostpdl.git;a=commit;f=gs/doc/Use.htm;h=c5253751c494869b9bd8bb7752be077204e19958

With respect to the question of whether local dirs go in front or behind, I don't think I'm persuaded by a search-time argument for making it impossible to supersede system resources. That kind of sounds like premature optimization (at a real functionality cost), at least unless there's some real data showing resource search time as a big fraction of gs run time.

(In reply to Chapman Flack from comment #15)
...
> With respect to the question of whether local dirs go in front or behind ... ... sounds like premature
> optimization ...

I think you have completely missed the point. There were a lot of discussions over many years - and heated ones, and strong opinions expressed - about whether local resources should go in front/behind, and whether relative paths should be allowed, and how easy it should be to override its default behavior, and they are not speed-related. The main considerations are security- and ease-of-maintainence related: gs are often run as part of a system's printing workflow at elevated privileges, and also that subtle or obvious problems happen if one overrides part of its internals accidentally and/or indiscriminantly.

I see you are filing it under Mac OS X - that has another interesting and specific set of problems from the fact that file names are only case-preserving but not case-sensitive. e.g. you would find that you can name files "example", "EXAMPLE", "examPLE" but not in the same directory.

I don't think I want to be drawn into this again so I should stop. In any case, most of those thoughts should go to the #ghostscript IRC channel, rather than a bug report.

(In reply to Hin-Tak Leung from comment #16)

Just quickly:

> I think you have completely missed the point. There were a lot of
> discussions over many years - and heated ones, and strong opinions expressed
> - about whether local resources should go in front/behind, and whether
> relative paths should be allowed, and how easy it should be to override its
> default behavior, and they are not speed-related.

I was responding to comment #12, which presented a speed-related argument.

> The main considerations
> are security- and ease-of-maintainence related: gs are often run as part of
> a system's printing workflow at elevated privileges, and also that subtle or
> obvious problems happen if one overrides part of its internals accidentally
> and/or indiscriminantly.

Understood. In that context it does make a difference whether alterations to the search path are being made from inside a (possibly sneaky) document, or in the environment inherited by gs (presumably put there deliberately and purposefully by the person responsible for making that printing workflow work), which is the same sort of distinction already supported by the SAFER mechanisms.

> I see you are filing it under Mac OS X

I did file it under Mac OS X, because that was the machine I was on at the time, but the behavior this report is about can be seen on other platforms, and the largely platform-independent gs_res.ps is where most of the action is.

(In reply to Jordy Dickinson from comment #11)

I've been ignoring this report mostly, as I'm not familiar with the full history, and much too busy to look into it properly, however I feel this needs some comment.


>  1. Add a `LocalResourceDirList` system parameter in the file "gs_res.ps"
> which, by default, is an empty array.
> 
>  2. Since this parameter is a list, and not an integer or string, the
> current options for specifying system parameters aren't sufficient. I can
> either add a new option for specifying arrays (it looks like neither "-l"
> for list nor "-a" for array are currently in use), add a special case to the
> "-s" option to handle this parameter, or set the parameter in string form
> and then convert to a list as-needed.

In my opinion this is not the way to proceed. Existing 'lists' are specified using a string, delimited with ';' for each entry. If that isn't sufficient then you should use PostScript to create the array.

I'm very much against adding new switch types.


> The second choice, adding a special case to the "-s" option, has the
> benefits of allowing the user to use the path-separated-list syntax of their
> operating system, but it has the downside of adding complexity to the
> parsing of that option as well as being inconsistent with the pattern
> already established of having an option for each type of parameter.

We already use a ; delimited string in the -I switch, OS-specific delimiters are a *really* bad idea as it will be inconsistent across platforms. 


>  3. Finally, in `lib_file_open_search_with_combine`, check each directory in
> `LocalResourceDirList` for the resource provided it's not an empty list. I
> see a variety of ways to access this parameter and I'm not entirely sure
> which is preferred. It can be accessed directly with `zgetsystemparam` or
> `currentparam1`; or it can be accessed by obtaining and going through the
> list returned by `zcurrentsystemparams`, `currentparams`, and
> `current_param_list`. My guess is that `zgetsystemparam` is the way to go,
> but I figured it was best to ask.

I haven't looked into the specifics, but since you are apparently in the C world here, you don't want to mess with PostScript. There is no good reason to run the PostScript operator and have (many) parameters created in PostScript memory and returned on the stack.

Stick with the C world, if you want a system parameter, look at how the existing code retrieves it. Pulling parameter lists is *slow*, avoid it where possible.

Could I just ask for a little clarification, please?

Am I right in asserting that the contentious change in behaviour here is that, at some point, the value of a relative GenericResourceDir setting (that is, one that does not start with either "/" or "\" depending on platform) stopped being enumerated relative to the value(s) of the GS_LIB variable?

(In reply to Chris Liddell from comment #19)
> Am I right in asserting that the contentious change in behaviour here is
> that, at some point, the value of a relative GenericResourceDir setting
> (that is, one that does not start with either "/" or "\" depending on
> platform) stopped being enumerated relative to the value(s) of the GS_LIB
> variable?

It's a little more complicated: at different times in history there have
been two different behaviors either of which would be usable to achieve
the effect of resource directories on a path.

The one I mentioned in the initial filing of this bug (and only one I knew
of at the time) was actually the younger one historically: it was first
documented here:
http://git.ghostscript.com/?p=ghostpdl.git;a=commit;h=1ddf1f5b73c6eeccf5539a7555416fe36f6cff2f
That was the method that involved tacking a relative GenericResourceDir
onto elements of GS_LIB in turn, and it has rotted in the way you describe.

However, there was also an older behavior, stemming from here:
http://git.ghostscript.com/?p=ghostpdl.git;a=commit;f=gs/doc/Use.htm;h=c5253751c494869b9bd8bb7752be077204e19958

that was also usable to achieve a path of resource directories, and it
still (mostly) works. Instead of relying on GenericResourceDir being
appended to elements of GS_LIB, you just include resource directories
themselves among the elements of GS_LIB. (The original commit comments
explained, "There is no file name conflicts because lib does not contain
subdirectories, but Resource always store files in subdirectories.")

This method, as I say, still mostly works; it is not just gone like the
append-GenericResourceDir method.  It does, however, suffer from three
mild symptoms of code rot:

1) You get a spurious warning described in bug #694512.
2) You have to explicitly add the gs installation's own Resource directory
   if you add any other, or gs fails to find encoding vectors, and gives
   unexpected /invalidaccess loading fonts.
3) The behavior in that commit comment's numbered paragraph (2) is broken;
   it was intended that ResourceFileName return a name based on
   GenericResourceDir either if GenericResourceDir is absolute OR if there
   is no resource already installed by that name, and the first branch of
   that OR is no longer tested. That was broken in
http://git.ghostscript.com/?p=ghostpdl.git;a=commit;f=gs/Resource/Init/gs_res.ps;h=bb40d003b2d5578874600559c5c2511546c0c46f
http://bugs.ghostscript.com/show_bug.cgi?id=691408#c37

   which means now you can't use ResourceFileName to reliably find
   a directory intended for new resource installation in the case where
   a version of that resource is already present further down the path.

So, although I initially filed about the loss of the younger-and-more-
completely-gone behavior, I can see now that the same need could also be
met simply by fixing the three minor code rot problems with the older-
and-still-mostly-working behavior.

(In reply to Ken Sharp from comment #18)
> (In reply to Jordy Dickinson from comment #11)
>
> > [...]
>
> In my opinion this is not the way to proceed. Existing 'lists' are specified
> using a string, delimited with ';' for each entry. If that isn't sufficient
> then you should use PostScript to create the array.
>
> I'm very much against adding new switch types.
> 
> > [...]
> 
> We already use a ; delimited string in the -I switch, OS-specific delimiters
> are a *really* bad idea as it will be inconsistent across platforms. 
> 

I'm not familiar with all of the system parameters but none of those documented in PRLM or any that I've seen in the code are "list" types. I know that the "-I" option uses that approach but as far as I can tell from the code, it eventually adds the directories to a list using `file_path_add` which references `gp_file_name_list_separator` which, unless I'm simply reading things wrong, varies depending on platform. Thus, it uses OS-specific delimiters. I don't think this is a bad idea because 1) platforms vary in their directory syntax and conventions and the option most familiar to users should be chosen and 2) directory lists generally specify absolute paths with OS-specific path separators, neither of which are portable or consistent anyway.

I understand not wanting to add another option, and in order to avoid that I could drop the system parameter approach and accept it as an environment variable, in much the same way that `GS_LIB` is done. Perhaps `GS_LOCAL_LIB`. But unless you wanted me to break away from the method that's used to parse "-I" and (presumably) `GS_LIBS`, I would still have to use an OS-specific directory-list format. Actually, this approach seems far simpler and in fact more consistent with already-existing behavior.

> >  [...]
> 
> I haven't looked into the specifics, but since you are apparently in the C
> world here, you don't want to mess with PostScript. There is no good reason
> to run the PostScript operator and have (many) parameters created in
> PostScript memory and returned on the stack.
> 
> Stick with the C world, if you want a system parameter, look at how the
> existing code retrieves it. Pulling parameter lists is *slow*, avoid it
> where possible.

Fair enough. The environment variable approach would make this a moot point, but it's still good to know in general.

(In reply to Jordy Dickinson from comment #21)

> > We already use a ; delimited string in the -I switch, OS-specific delimiters
> > are a *really* bad idea as it will be inconsistent across platforms. 
> > 
> 
> I'm not familiar with all of the system parameters but none of those
> documented in PRLM or any that I've seen in the code are "list" types.

There are many lists in PostScript, these use an array or dictionary. But we are discussing Ghostscript's command line, which hasn't a lot to do with the PLRM. Also, this isn't really anything to do with PostScript system parameters, its already been noted that the PLRM doesn't allow for a search list for Resources, so what you are trying to achieve here is technically incorrect (at best an extension) according to the specification.


> I
> know that the "-I" option uses that approach but as far as I can tell from
> the code, it eventually adds the directories to a list using `file_path_add`
> which references `gp_file_name_list_separator` which, unless I'm simply
> reading things wrong, varies depending on platform.

The subpath delimiters are OS-specific, the delimiters between search terms are ';'. Clearly the subpath delimiters should remain OS-specific, I'm talking about the delimiters between search items.

I'm also not happy about an environment variable, stuff that happens magically and invisibly is always difficult to explain to users. I already have enough trouble with PHP 'programmers' who have no clue what an environment variable is or how to set it. If we adopt anything it should be a command line parameter.

I believe this is fixed with:
http://git.ghostscript.com/?p=ghostpdl.git;a=commitdiff;h=27ce88cc

(In reply to Ken Sharp from comment #22)
> (In reply to Jordy Dickinson from comment #21)
> 
> > > We already use a ; delimited string in the -I switch, OS-specific delimiters
> > > are a *really* bad idea as it will be inconsistent across platforms. 
> > > 
> > 
> > I'm not familiar with all of the system parameters but none of those
> > documented in PRLM or any that I've seen in the code are "list" types.
> 
> There are many lists in PostScript, these use an array or dictionary.

> But we are discussing Ghostscript's command line, which hasn't a lot to do
> with the PLRM. Also, this isn't really anything to do with PostScript system
> parameters, its already been noted that the PLRM doesn't allow for a search
> list for Resources, so what you are trying to achieve here is technically
> incorrect (at best an extension) according to the specification.

Since it doesn't break backwards-compatibility I like to consider it an extension.

> > I know that the "-I" option uses that approach but as far as I can tell from
> > the code, it eventually adds the directories to a list using `file_path_add`
> > which references `gp_file_name_list_separator` which, unless I'm simply
> > reading things wrong, varies depending on platform.
> 
> The subpath delimiters are OS-specific, the delimiters between search terms
> are ';'. Clearly the subpath delimiters should remain OS-specific, I'm
> talking about the delimiters between search items.

I'm not sure what you mean by "subpath delimiters" or "search items", but all I'm suggesting is that directory lists be formatted according to the operating systems pre-existing conventions, as is already done with "-I" and "GS_LIB". So a directory list in Unix might look like "/foo/bar:/baz" and in Windows might look like "c:\foo\bar;c:\baz".

> I'm also not happy about an environment variable, stuff that happens
> magically and invisibly is always difficult to explain to users. I already
> have enough trouble with PHP 'programmers' who have no clue what an
> environment variable is or how to set it. If we adopt anything it should be
> a command line parameter.

I'm not sure then exactly how you think it should be represented on the command-line. If we can't add another option, then we're left only with "-s" to represent a system parameter as a string, and then we'd have to either do some processing later to convert it to an array or otherwise convert it every time we search for directories. I suppose converting it at some point in the initialization process is plausible and something I could look into. Alternatively we could represent it both as an environment variable and an option, like how we have both `GS_LIB` and the "-I" option, but that would involve adding another option which you've stated you don't want to do.

(In reply to Chris Liddell from comment #23)
> I believe this is fixed with:
> http://git.ghostscript.com/?p=ghostpdl.git;a=commitdiff;h=27ce88cc

I can still reproduce the bug following the instructions given by the reporter.

(In reply to Jordy Dickinson from comment #25)
> (In reply to Chris Liddell from comment #23)
> > I believe this is fixed with:
> > http://git.ghostscript.com/?p=ghostpdl.git;a=commitdiff;h=27ce88cc
> 
> I can still reproduce the bug following the instructions given by the
> reporter.

OKay, partially fixed - it works if you're not using init files in the romfs.

(In reply to Jordy Dickinson from comment #25)
> (In reply to Chris Liddell from comment #23)
> > I believe this is fixed with:
> > http://git.ghostscript.com/?p=ghostpdl.git;a=commitdiff;h=27ce88cc
> 
> I can still reproduce the bug following the instructions given by the
> reporter.

For clarity's sake, at the moment there are three different approaches
in play for accomplishing the same effect:

#1 the append-GenericResourceDir-to-GS_LIB approach, dating to
   commit 1ddf1f5b of 26 October 2004, currently not usable because broken.

#2 the include-resource-dirs-in-GS_LIB approach, dating to
   commit c5253751 of 11 April 2003, currently mostly usable but
   slightly broken.

#3 the LocalResourceDirList approach, proposed by Jordy on 10 December 2014,
   currently not usable because design/implementation needed.

As a gs user, I don't really care which of those 3 is ultimately left
standing, as long as (a) it's documented, (b) stray references in the
docs to other methods no longer standing are removed, and (c) it gets
a regression test so we don't end up right back here again.

But this does sort of mean that in any comments of the form "believed
fixed" or "still reproducible" it's probably a good idea to say which
of the three approaches is believed fixed, or which one was tried and
still didn't work. :)

(In reply to Chapman Flack from comment #27)
> (In reply to Jordy Dickinson from comment #25)
> > (In reply to Chris Liddell from comment #23)
> > > I believe this is fixed with:
> > > http://git.ghostscript.com/?p=ghostpdl.git;a=commitdiff;h=27ce88cc
> > 
> > I can still reproduce the bug following the instructions given by the
> > reporter.
> 
> For clarity's sake, at the moment there are three different approaches
> in play for accomplishing the same effect:
> 
> #1 the append-GenericResourceDir-to-GS_LIB approach, dating to
>    commit 1ddf1f5b of 26 October 2004, currently not usable because broken.
> 
> #2 the include-resource-dirs-in-GS_LIB approach, dating to
>    commit c5253751 of 11 April 2003, currently mostly usable but
>    slightly broken.
> 
> #3 the LocalResourceDirList approach, proposed by Jordy on 10 December 2014,
>    currently not usable because design/implementation needed.
> 
> As a gs user, I don't really care which of those 3 is ultimately left
> standing, as long as (a) it's documented, (b) stray references in the
> docs to other methods no longer standing are removed, and (c) it gets
> a regression test so we don't end up right back here again.
> 
> But this does sort of mean that in any comments of the form "believed
> fixed" or "still reproducible" it's probably a good idea to say which
> of the three approaches is believed fixed, or which one was tried and
> still didn't work. :)

None of the above. What I did was fix a mistake, as mentioned in the commit message.

It now works if you don't read init files from the romfs (COMPILE_INITS=0). There is something "special" happening when we use the romfs which I don't fully understand.

I don't believe any of the above listed approaches is appropriate, nor necessary.

Actually, no, this now works for me. I consider this to be resolved with the current code (including the above linked patch).