In model_rd_for_sb function, the spatial domain SSE and variance
are checked to see if transform coefficients are quantized to 0.
Besides that, this patch adds another set of thresholds that are
much more strict. These thresholds are used to conduct a partition
block level check to measure if all its TX blocks are skippable
for YUV planes. If it is true, x->skip is set for this partition
block, and thus its mode search is terminated.

This speeds up the encoding at very low prediction error case,
such as screen sharing application. This patch covers what
rd_encode_breakout_test() does, so that function is removed.

Borg test at speed 3 shows:
For stdhd set, psnr: +0.008%, ssim: +0.014%;
For derf set, psnr: +0.018%, ssim: +0.025%.
No noticeable speed change.

Change-Id: I4e5f15cf10016a282a68e35175ff854b28195944

Change-Id: Ie35a965a6b8de536ccaf61ff61498620d22db205

The concept:

There's too much noise in source pixels for variance and at low bitrate
the reconstructed looks nothing like the source so we have problems
getting good partitionings with either.   This skirts the issue by using
a box blur scaled down version for variance calculations.  To compare
against source_var_ moved keyframe to be rd based like source_var.

Change-Id: Ie3babdbfadae324b7b5a76bea192893af27f0624

Change-Id: Ie69a164bc166b6a8819777038d65a7d9f9c3361f

This commit breaks the overly broad header files into more
targeted and smaller ones, to help better structure the system
layout.

Change-Id: I7b24559d3ea6e582cf5d9bbe8f71459f9824d71b

Change-Id: I3a7f83ab1dbfcedc8a82fe798c2fa30dd9c7d696

Change-Id: I6f2865bb8ba9295f5c45a4cad065aecbe1e63c32

The basic data defs should be above block operation level.

Change-Id: I7dd9836d01120ab75e0c472baac9f15495ed0db5

Fixed an encoder crash. Set skip_txfm to 0 for cases that skip_txfm
isn't calculated. Put memcpy of skip_txfm at right place.

Change-Id: Ib3b6afc1b251a85b2a853c8138fb3393f48cfef6

Change-Id: If0ea98aa139d14d40cd924114e18396aff36b5a5

The functions b_width_log2 and b_height_log2 only do direct
table fetch. This commit unifies such use cases by using the
table directly and removes these functions.

Change-Id: I3103fc6ba959c1182886a2799d21b8b77c8a7b6b

Add comments on the use case of these definitions. Further reduce
the scope of header file in vp9_context_tree.h.

Change-Id: Ic4a7638e838d0ac441b64abfc56e57354c059d75

The coefficient range checking is enabled when configured with
--enable-debug --enable-coefficient-range-checking
for vpxdec to detect ill-formed input stream. This addresses the
problem raised by issue #792.

Change-Id: I3f9ea541de4dc742dd64389d6c5f543fb1c4f052

Also fixes a case of distortion becoming negative and messing
up the RDCOST computation.

Change-Id: Id345af9e8dfff31ade622be5756e51f2cdface53

Change-Id: I264ee35044a5973c7725daba7af870968353a3c1

Change-Id: Ia212390cf8d36db5436bb0f0e1b696f70066341a

This commit fixes a buffer pointer mis-use in store_coding_context.
The compression performance for stdhd set of speed 3 is improved by
0.097%. It fixes issue 869.

Change-Id: Idc59e22035eaf39f7133ca04174894374d647ff7

This SSE2 is based on VP8 denoiser's SSE2 code. In VP8, there are
only 16x16 blocks in denoiser, while in VP9, there are 13 different
block sizes.

By adding this SSE2 code, the improvement of encoder speed is around
20%(using C code vs using SSE2 code), vary for different clips.

The unit test for VP9 denoiser is to confirm that the SSE2 code is
bit-exact with the C code. The unit test covers all block size.

Change-Id: Ic8d8ac26db4ea40a5f146b5678a065af07eaaa3d

Bit-stream clarification related to Issue 868.

Change-Id: I92a7bc5b7782c9ea5c3f6cceec761742183c9514

Adjustments to the GF interval choice and minimum boost.
Adjustment to the calculation of 2 pass worst q.
Compared to 09/29 head there is metrics hit on derf of
(-0.123%,-0.191%)

Compared to the September 29 head and a baseline on
September 18 baseline the accuracy of the VBR rate control
measured on the derf set is as follows:-

Mean error %  / Mean abs(error %)
Sept 18 baseline (-7.0% / 14.76%)
Sept 29 head (-15.7%, 19.8%)
This check in (-1.5% / 14.4%)

The mean undershoot is reduced slightly but the
worst case overshoot on e.g. harbour/highway is
increased. This will be addressed in a later patch.

Change-Id: Iffd9b0ab7432a131c98fbaaa82d1e5b40be72b58

It is possible that the GOLDEN reference frame is not avaiable, in
which setting the predicted mv will be associated with a residual
value of INT_MAX. This commit checks this condition before
left shift and comparison with that of ALTREF frame, to avoid
overflow issue.

Change-Id: Ib98c3149dbdd016f2fe5beaafb13f67d469dd07c

Resolves a visual studio warning, and includes some cleanups.

Change-Id: I6a7576ef323c475b7d1c659800cd82c6cb1fd18d

This commit adds proper initialization of segment id for variance AQ
mode in non-rd coding path. It fixes the enc/dec mismatch issue of
rt=7 with --aq-mode=1, as reported in issue #816

Change-Id: I02fa41b96345bf2e66077d5ea553f85ba800f7bb

Change-Id: Icbcfe7b56d88474f4398b4c5b52f6719d551ab4a

This commit enables the encoder to skip split partition search if
the bigger block size has all non-zero quantized coefficients in low
frequency area and the total rate cost is below a certain threshold.
It logarithmatically scales the rate threshold according to the
current block size. For speed 3, the compression performance loss:
derf  -0.093%
stdhd -0.066%

Local experiments show 4% - 20% encoding speed-up for speed 3.
blue_sky_1080p, 1500 kbps
51051 b/f, 35.891 dB, 67236 ms ->
50554 b/f, 35.857 dB, 59270 ms (12% speed-up)

old_town_cross_720p, 1500 kbps
14431 b/f, 36.249 dB, 57687 ms ->
14108 b/f, 36.172 dB, 46586 ms (19% speed-up)

pedestrian_area_1080p, 1500 kbps
50812 b/f, 40.124 dB, 100439 ms ->
50755 b/f, 40.118 dB,  96549 ms (4% speed-up)

mobile_calendar_720p, 1000 kbps
10352 b/f, 35.055 dB, 51837 ms ->
10172 b/f, 35.003 dB, 44076 ms (15% speed-up)

Change-Id: I412e34db49060775b3b89ba1738522317c3239c8

Incorporates the WRAPLOW macro into the non-highbitdepth transforms
to aid hardware verification between a software C model and an
intended hardware implementation though the use of the configure
options: --enable-experimental --enable-emulate-hardware.
Note that to avoid further discrepancies between the sse/sse2
implementations of the transforms and the C implementation, when the
emulate hardware option is invoked, we also disable sse/sse2/etc.

Also incudes some minor cleanups/renaming etc.

Change-Id: Ib864d8493313927d429cce402982f1c8e45b3287

Adds proper scaling for highbitdepth in a rdopt cost.

Change-Id: I066694799a7f491b830945ef1c66eb202071c355

To fix a VS warning.

Change-Id: I4c530c0afe8d06acdb8cc78b7995aba57a25373d

This commit changes the tables to be read only, which fixes
issue #866

Change-Id: I85bbe03f9d344f50570f8c1c61699bdc5cee248f

Change-Id: I0e262ede7eb4a4ae0c86181922d744e542e93350

Change-Id: I034b73bb27e8023733aae63e6f8565f0a64c7267

Change-Id: I5a566d6ade720f212a60c0ad5d6f1ee1d1d37f2e

Fixes an intermittent assert failure for highbitdepth.

Change-Id: If8cad0209a94f1184b69c7b3f1d587934f857d9b

Adds enums and corresponding vpx level code.

Change-Id: Ia402d47490a4466988d7edc6b7e3e5163f20a381

Change-Id: Id92544762e7b96d3c729dfc8e04ecff91cbcc7f9

Change-Id: Icfc3fb62cc0b05e435814035bfe1f2e2870442b4

Miscellaneous bug-fixes for high bitdepth functionality.
With this patch, high bit-depth profiles become mostly functional,
except for an intermittent assert failure issue that is being
tracked.

Change-Id: I6a7fcbdcf1e5b09842e88535f8442d2e1230748c

This commit removes unused header file vp9_onyxc_int.h and repeatedly
included file vpx_ports/mem.h from vp9_block.h

Change-Id: I400b210bd1da48f1880bd50a8f4a6e2c690e15a1

Block transform skipping was implemented based on DCT's energy
conservation property. Modified the thresholds using zero bin
parameters. AC and DC coefficients were checked separately to
allow better identifying of skippable blocks.

Borg test at speed 3 showed:
stdhd set: psnr gain: 0.153%, ssim gain: 0.051%;
derf set: psnr gain: 0.023%, ssim gain: 0.036%

For most test clips, the encoding speedup is 1% - 2%.
parkrun(720p): 7.5% speedup, park_joy(1080p): 3.5% speedup.

Change-Id: If28eb81113a077414f5ca7b021c14f9069b373bb

The file vpx_integer.h has been included and used in the parent
file vp9_common.h.

Change-Id: I9c65f08353576f9ef1e5ea17244fc5ca964ec002