Fix potential invalid partition size use

For blocks at frame boundary, the selected block size sometimes needs
to be smaller than that was first given. This commit forces such block
size change only between square blocks, so as to avoid the potential
use case containing 32x16 + 16x8 + 16x8, for 1080p sequences.

Local test suggested no visible coding speed difference. Borg test
reveals no difference in terms of compression performance.

Change-Id: Ie8de87f3c6febc3acf11b4cbfdf2077f9f6def52