There have been discussions about this before and majority agreed that 'knowing who downvotes a post/comment will breed retaliation-acts', and that is not something desirable here. So, getting to know whoever downvotes will do more harm to Relevant than good.
Regarding upvotes, from my observation, you get notified when someone whose upvote matters (higher Reputation members) upvote your post/comment but not when lower reputation members do.