1. 22 Jun, 2016 1 commit
    • Yaowu Xu's avatar
      Prevent negative variance · ef665996
      Yaowu Xu authored
      Due to rounding used computation, HDB variance computation may produce
      slightly negative values. This commit adds clamping to make sure
      output variance values for 10 and 12 to be non-negative.
      Change-Id: Id679aa55a4c201958c4c7d28cd8733b9246a71c8
  2. 16 Jun, 2016 1 commit
    • Yaowu Xu's avatar
      vpx_dsp/variance.c: change to use correct type · e5e998a6
      Yaowu Xu authored
      This commit change to use int64_t to represent the sum of pixel
      differences, which can be negative.
      This fixes a number of ubsan warnings.
      Change-Id: I885f245ae895ab92ca5f3b9848d37024b07aac98
  3. 13 Jan, 2016 1 commit
  4. 25 Nov, 2015 1 commit
    • Alex Converse's avatar
      Change highbd variance rounding to prevent negative variance. · 022c848b
      Alex Converse authored
      Always round sum error and sum square error toward zero in variance
      calculations. This prevents variance from becoming negative.
      Avoiding rounding variance at all might be better but would be far
      more invasive.
      Change-Id: Icf24e0e75ff94952fc026ba6a4d26adf8d373f1c
  5. 07 Jul, 2015 1 commit
  6. 26 May, 2015 1 commit