fix potential use of uninitialized rate_y

This issue likely doesn't appear in the unmodified encoder, but
sufficient hacking on the mode selection loop can expose it.

Change-Id: I8a35831e8f08b549806d0c2c6900d42af883f78f