Rework unit test for 8x8 transformation

This commit reworks the unit test for 8x8 forward/inverse
transformation. It adds extreme input value test to detect overflow
issues in the intermediate steps.

It temporarily disables unit test for the SSSE3 version, which
showed overflow failure in the new test conditions.

Change-Id: I7caf10bba4b6db031add65d8c0eb99426b38aa42