Allow key frame more flexibility in mode search

This commit allows the key frame to search through more prediction
modes and more flexible block sizes. No speed change observed. The
coding performance for rtc set is improved by 1.7% for speed -5 and
3.0% for speed -6.

Change-Id: Ifd1bc28558017851b210b4004f2d80838938bcc5