TL;DR : They are technically breaking the rules but you should allow it as it doesn't affect the game.
Looking at the rules here it says :-
Draw a card from either the Draw Pile or the Discard Pile, and add it
to your hand. And then, discard any card that you want onto the
There is also nothing in the rules about 'still touching' anywhere that I can see. So technically the answer to your question is no that is not allowed.
However, there is also an answer about just being nice person when playing games. If a player picks up a publicly visible discard and decides that is not what they want to do, and the next player has not taken a turn, I see no good reason why they shouldn't put it be back and be able to draw a card. Nothing has changed that can not be undone. No new information was available.
Of course if a player drew a card from the deck and saw it then that would have to stick and shouldn't be returned as they have new information that was not available at the start of the turn.