Fix destroy listener signal connection

Remove it from the list in the callback, not when a new buffer
is created later on.

Change-Id: I2328edec9728752d18efaecede19eb4527d0f578
Reviewed-by: Louai Al-Khanji <louai.al-khanji@theqtcompany.com>