It would return to your hand.
I don't have access to the MTG site to be able to cite the rulebook, but it's all down to the stack.
The stack is played out in order, so new items go to the top of the stack and then the stack resolves from the top down. In this instance Hand of Death was cast and went on the stack, then you activated the Merfolk's ability which went above Hand of Death on the stack.
When the spells and abilities on the stack resolve:
- Darting Merfolk's ability resolves, returning it to your hand.
- Hand of Death tries to resolve, but the card it targeted is no longer on the battlefield, and does not exist to be targeted.
- Hand of Death is countered and has no effect. (It goes to your opponent's graveyard.)
I recommend reading up on the Stack, understanding it is vital to getting the most out of Magic: The Gathering: Guide to the Stack