I agree with kpoos and you; kids are priority no exceptions.
He has to know you are serious with the toys.
I would also add that the children become a part of obedience with him as well. If they request he sit, down, stay, come - he must learn to do this so he views them as leaders. (my 20 year old even needed some coaching on this recently on getting the dogs to listen to her)
Have you tried exercises where you sit and play with the toy and take it? Then request he sit or another command before receiving?
If not I would try it and then have the children with you, beside you doing the same thing. Another thing would be to place the toy up away from him but where he can see it, and when he wants it, the children get the toy and give it to him once he sits and waits calmly. Always correcting the growl during the exercise.