It used to be that kids were warned that they should be "good" or else Sinterklaas would take them in a bag to Spain....... not sure why that would be punishment, but alright....
Anyway, for some reason these days the idea of an old holy man in a dress abducting children to a far away country is not so popular anymore and I don't think they still say it to children (not sure though).
As far as I know and remember from being a child Zwarte Piet has never been considered "sinister". Children often find them funny because they act "crazy" and in fact rather childlike.
With discussions such as these people often forget that this is a tradition aimed at children. And children don't give a flying fudge about Piet's colour.