In directional verbs with path movement, the place determining the start of the path is the one indicating the grammatical subject. In other directional verbs in which no movement takes place, the fingers are facing the object.
"When I was a kid my parents looked after me. Now, however, it's me who looks after them."
"My friend gave me money and next week I'll give it back to him."