This can be quite confusing when it actually happens - it doesn't strike me as particularly intuitive. What's wrong with keeping ref->u.r.obj nil or indicating failure somehow differently?
The problem with keeping it nil is that then it'll try to load the object every time, and fail every time.
OTOH, it fails while the document is encrypted and won't, when it's not. If you've however already tried once, you won't succeed after decrypting - which seems counter-intuitive to me. How expensive is dereferencing an indirect reference when its object isn't available? Too expensive for the odd case where it actually fails (which AFAICT we wouldn't expect for sane documents, would we)?
The problem is a bug introduced by the code that loads the pagetree before the decryption has been installed, which it shouldn't be doing! (See pdf_open.c where pdf_loadxref calls pdf_getpagecount). If we fix that we should never read objects that are encrypted without decrypting them first, and won't be running into this issue. Since I've fixed that pagetree loading bug I am closing this as invalid, unless you still think this is an issue?
Since you're explicitly catching the error, I guess I'll just have to watch the debug output more closely. I'd however appreciate a comment (along your comment #1) in fz_resolveindirect as to why you resolve the indirect to itself, as this isn't really obvious from reading the code.