Ball to hand isn't deliberate handball and shouldn't be judged as such.
However, to avoid controversy refs nowadays tend to give them all.
Same with fouls. They are given for just about any contact, deliberate or not.