The Exit code of external programs were not stopping the CI script, thus displaying a green tick...
This is now done manually.
For Ninja, we generate the error code if there is a failed command in the build.
For MSVC, we use the error code from $LastExitCode
For both, we throw an exception. This is not pretty, but this works