I noticed a potential integer overflow in the gauss5x5 function that may lead to a heap overflow. The vulnerable line is: ``` buf = fz_malloc(ctx, w * 3 * 5 * sizeof(uint16_t)); ``` If `w` is large enough, the multiplication can overflow. Since the result is passed as a `size_t` to `fz_malloc`, this won't typically cause problems on 64-bit systems due to the larger address space. However, on 32-bit systems where `size_t` is only 4 bytes, the multiplication can wrap around and result in a much smaller allocation than expected — potentially causing a heap overflow when the buffer is used. Additionally, as noted in the comments, fz_malloc may return NULL when the requested size is zero, so it's important to check the return value before using the buffer.
Just following up with a concrete example to illustrate the impact. You can find the relevant code here: https://github.com/ArtifexSoftware/mupdf/blob/master/source/fitz/warp.c#L651-L665 For instance, if `src->w` is set to a large value like 0x08888888 (which is not negative number), the computed allocation size becomes 0x000000c2 on a 32-bit system due to integer overflow during the multiplication.
Please don't play with the priorities.
Do you have a file that exhibits this problem?
(In reply to Sebastian Rasmussen from comment #3) > Do you have a file that exhibits this problem? Hey Sebastian, I don't have a file that triggers it right now. When I first reported the issue, it was more of a theoretical overflow. I chatted with Robin at the time in discord — he said it's a valid issue but since the code isn't actually called, it's low priority. So I didn't go further with making a sample.
Fixed with: commit ef7122162c801f1c7a48fcfb535e748fc6224999 Author: Robin Watts <Robin.Watts@artifex.com> Date: Mon Aug 4 15:17:09 2025 +0100 Bug 708511: Harden fz_warp_pixmap against overflows. Thanks to Amir Mohammad Jahangirzad for the report that lead to this. With sufficiently large pixmaps, we could cause overflows in some of the calculations here. The code is designed to work with pixmaps that are small enough that this should never be a problem, but it doesn't hurt to protect the code a bit. We rearrange some expressions so that they won't overflow if int is smaller than size_t; often this is done by putting a size_t as the first thing to multiply. Thanks for the report.