Bug 698830

Summary: denial of service in seek_file
Product: MuPDF Reporter: Ruikai Liu <lrk700>
Component: mupdfAssignee: Sebastian Rasmussen <sebastian.rasmussen>
Status: RESOLVED FIXED    
Severity: blocker CC: sebastian.rasmussen
Priority: P1    
Version: unspecified   
Hardware: PC   
OS: All   
Customer: Word Size: ---
Attachments: POC pdf
ASAN backtrace
GDB backtrace

Description Ruikai Liu 2017-12-21 23:51:59 UTC
Created attachment 14549 [details]
POC pdf

Hi,

The fuzzer found another crash when running `mutool info` with the attached PDF file. The source code we use is at commit 7307a7f232b27b22176b8f68364e43e09ce393cc.

By debugging it, it shows that the FILE pointer is NULL when calling `fseeko` in `seek_file`:

(gdb) bt
#0  0x00007ffff66f9b80 in fseeko64 () from /usr/lib/libc.so.6
#1  0x000000000048d30b in seek_file (ctx=0x2756080, stm=0x2766bb0, offset=24641, whence=0) at source/fitz/stream-open.c:112
#2  0x000000000048de3d in fz_seek (ctx=0x2756080, stm=0x2766bb0, offset=24641, whence=0) at source/fitz/stream-read.c:157
#3  0x0000000000524b0c in next_null (ctx=0x2756080, stm=0x27cf460, max=1) at source/fitz/filter-basic.c:31
#4  0x00000000004c2508 in fz_read_byte (ctx=0x2756080, stm=0x27cf460) at include/mupdf/fitz/stream.h:374
#5  0x00000000004c3536 in pdf_lex (ctx=0x2756080, f=0x27cf460, buf=0x2766d90) at source/pdf/pdf-lex.c:507
#6  0x00000000004e6d9f in pdf_load_obj_stm (ctx=0x2756080, doc=0x2766c10, num=2, buf=0x2766d90, target=1) at source/pdf/pdf-xref.c:1593
#7  0x00000000004e7d44 in pdf_cache_object (ctx=0x2756080, doc=0x2766c10, num=1) at source/pdf/pdf-xref.c:1975
#8  0x00000000004e7f9d in pdf_resolve_indirect (ctx=0x2756080, ref=0x277d090) at source/pdf/pdf-xref.c:2025
#9  0x00000000004e8068 in pdf_resolve_indirect_chain (ctx=0x2756080, ref=0x277d090) at source/pdf/pdf-xref.c:2051
...

Though I'm not sure why it failed to open file(maybe because resource is exhausted since there're many warnings on exception stack overflow).
Comment 1 Sebastian Rasmussen 2017-12-22 08:55:09 UTC
I can confirm that the attached file produces an issue on git HEAD when built with ASAN. Due to the holidays this may take some extra time to resolve.
Comment 2 Sebastian Rasmussen 2017-12-22 09:33:35 UTC
Created attachment 14550 [details]
ASAN backtrace
Comment 3 Sebastian Rasmussen 2017-12-22 09:33:54 UTC
Created attachment 14551 [details]
GDB backtrace
Comment 4 Sebastian Rasmussen 2017-12-22 09:34:44 UTC
The backtraces seems to suggest that MuPDF enters an endless loop, which is eventually terminated with a FILE * being set to zero. I need to investigate this further.
Comment 5 Sebastian Rasmussen 2018-01-28 17:20:03 UTC
I have been able to isolate one problem related to this bug. There is a tentative patch for this issue in 68f4f5e070848f8ffee7ba3796100d7ca9ada875.

This is not the complete fix however, because now I wind up in what looks like an endless loop of errors instead. I have determined that if I set the error stack depth to 32 positions mutool draw -s t poc.pdf finishes in 2 seconds. When I increase the error stack depth to 36 positions running the same command finishes in ~20 seconds. Increasing the error stack depth further to 39 positions causes the execution to run for ~90 seconds. The normal error stack depth is 256 positions.

So at the moment I conclude that the is some O(n^2) behavior in the error handling. I believe, but have not yet proved, that there is some kind of circular dependency between a pdf object and and a pdf object stream and this dependency leads to the extremely deep callstack which eventually causes mupdf to run out of error stack. Hopefully it is possible to break this circular dependency somewhere, but I don't know where yet.
Comment 6 Sebastian Rasmussen 2018-01-29 15:38:45 UTC
*** Bug 698965 has been marked as a duplicate of this bug. ***
Comment 7 Sebastian Rasmussen 2018-02-01 09:18:12 UTC
This issue was fixed in these two commits.

commit b03def134988da8c800adac1a38a41a1f09a1d89
Author: Sebastian Rasmussen <sebras@gmail.com>
Date:   Thu Feb 1 16:36:14 2018 +0100

    Bug 698830: Avoid recursion when loading object streams objects.
    
    If there were indirect references in the object stream dictionary and
    one of those indirect references referred to an object inside the object
    stream itself, mupdf would previously enter recursion only bounded by the
    exception stack. After this commit the object stream is checked if it is
    marked immediately after being loaded. If it is marked then we terminate
    the recursion at this point, if it is not marked then mark it and
    attempt to load the desired object within. We also take care to unmark
    the stream object when done or upon exception.

commit 26527eef77b3e51c2258c8e40845bfbc015e405d
Author: Sebastian Rasmussen <sebras@gmail.com>
Date:   Mon Jan 29 02:00:48 2018 +0100

    Bug 698830: Don't drop unkept stream if running out of error stack.
    
    Under normal conditions where fz_keep_stream() is called inside
    fz_try() we may call fz_drop_stream() in fz_catch() upon exceptions.
    The issue comes when fz_keep_stream() has not yet been called but is
    dropped in fz_catch(). This happens in the PDF from the bug when
    fz_try() runs out of exception stack, and next the code in fz_catch()
    runs, dropping the caller's reference to the filter chain stream!
    
    The simplest way of fixing this it to always keep the filter chain
    stream before fz_try() is called. That way fz_catch() may drop the
    stream whether an exception has occurred or if the fz_try() ran out of
    exception stack.
Comment 8 Sebastian Rasmussen 2018-09-13 17:24:28 UTC
*** Bug 698833 has been marked as a duplicate of this bug. ***