• Alex Converse's avatar
    Change highbd variance rounding to prevent negative variance. · 022c848b
    Alex Converse authored
    Always round sum error and sum square error toward zero in variance
    calculations. This prevents variance from becoming negative.
    Avoiding rounding variance at all might be better but would be far
    more invasive.
    
    Change-Id: Icf24e0e75ff94952fc026ba6a4d26adf8d373f1c
    022c848b
variance.c 21.4 KB