When the creature dies, both abilities go on the stack. They are placed on the stack in "Active Player, Non-active Player" (APNAP) order. This means that the player whose turn it is places all of their abilities that triggered on the stack, and then the other player will place all of their abilities that triggered on the stack, above the active player's abilities. If both abilities belong to the same player, that player chooses the order. Then the top ability on the stack resolves, moving the creature. When a card changes zones it becomes an entirely new object, and the only ability that can track it to the new zone is the ability that caused it to move. So when the second ability tries to resolve it won't be able to do anything because it can't find the creature it is trying to target.
This means that the exact result depends on who controls the effects, and whose turn it is.
If you control Endless Cockroaches that you enchanted with False Demise, then when the cockroaches die you choose which order the effects go on the stack. If you order them so the cockroaches' ability is on top it will return to you hand. If you order them so the False Demise ability is on top, it will return to the battlefield.
If you control Endless Cockroaches, and your opponent enchanted it with False Demise and the Cockroaches die on your turn then the Cockroaches ability will be placed on the stack first, and then False Demise will be placed on top of it. False Demise will resolve first, and your opponent will get your Cockroaches.
If instead it was your opponent's turn, then the False Demise trigger will be placed first and the Cockroach trigger will be placed on top, meaning that the Cockroach trigger will resolve first and the roaches will be returned to you hand.
From the description, it sounds like the enchantment in question is Grave Betrayal. If that is the case, then the roaches will always go back to their owners hand. That's because Grave Betrayal reads
Whenever a creature you don't control dies, return it to the battlefield under your control with an additional +1/+1 counter on it at the beginning of the next end step. That creature is a black Zombie in addition to its other colors and types.
Grave Betrayal's ability triggers and goes on the stack the same way I described above. But even if it resolves first, it doesn't try and move the creature until the beginning of the end step. That means that when the Cockroaches' ability resolves, it will still be able to find the Cockroaches and will always move the roaches to their owner's hand, and when the Grave Betrayal trigger tries to move the roaches at the beginning of the end step they won't be able to find them.