• Yang Wang's avatar
    dsputil_mmx: fix incorrect assembly code · 845e92fd
    Yang Wang authored
    In ff_put_pixels_clamped_mmx(), there are two assembly code blocks.
    In the first block (in the unrolled loop), the instructions
    "movq 8%3, %%mm1 \n\t", and so forth, have problems.
    From above instruction, it is clear what the programmer wants: a load from
    p + 8. But this assembly code doesn’t guarantee that. It only works if the
    compiler puts p in a register to produce an instruction like this:
    "movq 8(%edi), %mm1". During compiler optimization, it is possible that the
    compiler will be able to constant propagate into p. Suppose p = &x[10000].
    Then operand 3 can become 10000(%edi), where %edi holds &x. And the instruction
    becomes "movq 810000(%edx)". That is, it will stride by 810000 instead of 8.
    This will cause a segmentation fault.
    This error was fixed in the second block of the assembly code, but not in
    the unrolled loop.
    How to reproduce:
        This error is exposed when we build using Intel C++ Compiler, with
        IPO+PGO optimization enabled. Crashed when decoding an MJPEG video.
    Signed-off-by: default avatarMichael Niedermayer <michaelni@gmx.at>
    Signed-off-by: default avatarDerek Buitenhuis <derek.buitenhuis@gmail.com>
dsputil_mmx.c 154 KB