1. 22 Jun, 2016 1 commit
    • Yaowu Xu's avatar
      Prevent negative variance · ef665996
      Yaowu Xu authored
      Due to rounding used computation, HDB variance computation may produce
      slightly negative values. This commit adds clamping to make sure
      output variance values for 10 and 12 to be non-negative.
      
      Change-Id: Id679aa55a4c201958c4c7d28cd8733b9246a71c8
      ef665996
  2. 16 Jun, 2016 1 commit
    • Yaowu Xu's avatar
      vpx_dsp/variance.c: change to use correct type · e5e998a6
      Yaowu Xu authored
      This commit change to use int64_t to represent the sum of pixel
      differences, which can be negative.
      
      This fixes a number of ubsan warnings.
      
      BUG=webm:1219
      
      Change-Id: I885f245ae895ab92ca5f3b9848d37024b07aac98
      e5e998a6
  3. 13 Jan, 2016 1 commit
  4. 25 Nov, 2015 1 commit
    • Alex Converse's avatar
      Change highbd variance rounding to prevent negative variance. · 022c848b
      Alex Converse authored
      Always round sum error and sum square error toward zero in variance
      calculations. This prevents variance from becoming negative.
      Avoiding rounding variance at all might be better but would be far
      more invasive.
      
      Change-Id: Icf24e0e75ff94952fc026ba6a4d26adf8d373f1c
      022c848b
  5. 07 Jul, 2015 1 commit
  6. 26 May, 2015 1 commit