If you don't wan't to snatch it, send him a reminder or a PM. Or just wait.
There is no way to cancel it.
But there is nothing wrong with claiming victory (as long as your opponent is not on holiday). You both agreed upon the time constraints. It is just another chess rule. If the waiting bothers you, claim the victory, otherwise just wait.
Edit: Also many people claim victory when the vacation flag is up. Also partly because of the strange way that it is implemented here. It is basically all up to you.