Fix checking the Qt::BypassWindowManagerHint flag

Change-Id: I24ca4854aa6756f4cea4f4f5a471ffd9bf33c620
Reviewed-by: Johan Helsing <johan.helsing@qt.io>
Reviewed-by: Erik Larsson <erik@ortogonal.com>
Reviewed-by: Laszlo Agocs <laszlo.agocs@theqtcompany.com>