In small numbers, the sponges are not a problem.
However, in some cases they compete for real estate with the zoas.
In your case, I would recommend removing the sponges. You can use tweezers or just use a stick to rub the sponges out.
If the sponge frags attach somewhere else and survive, then...