Make filter RD code and encode breakout variance size-independent.

Static threshold results slightly up (+0.1% on derf), probably b/c
we now take the filter (sharp/lowpass) into account for the breakout
decision.

Change-Id: I9f597601da434205142afd05f32690e7ba8fd690