vp9_loopfilter.c: cosmetics

- fix indent, spelling
- drop some whitespace in some comments
- add an assert in vp9_setup_mask, it shouldn't be called on decode
  error

Change-Id: Ic312a815e977a6f9cb81ceb7b039eeada76c5aa0