Board & Card Games Stack Exchange is a question and answer site for people who like playing board games, designing board games or modifying the rules of existing board games. It only takes a minute to sign up.
Yes, the Rite of Replication would fail if you destroy the creature it's targeting. It plays out like this:
Your opponent casts Rite of Replication targeting their creature.
You cast Flesh to Dust in response to Rite of Replication.
Your Flesh to Dust resolves, and their creature dies.
Your opponent's Rite of Replication tries to resolve, but since the creature it targeted is no longer on the battlefield, it doesn't resolve because of rule 608.2b, which says
If the spell or ability specifies targets, it checks whether the targets are still legal. A target that's no longer in the zone it was in when it was targeted is illegal [...] If all its targets, for every instance of the word “target,” are now illegal, the spell or ability doesn’t resolve. It’s removed from the stack and, if it’s a spell, put into its owner’s graveyard.