You want to go through each of your dice from lowest to highest and, if your chance of success is at least as good if you re-roll that die than if you do not re-roll that die, add that die to your list of re-rolls. As soon as you find a die that worsens your current chance of succeeding, stop, and re-roll all the previous dice in this procedure. If you run out of dice, re-roll everything.
For example, if you roll [4, 5, 5] and are trying to beat a 17, here is what it would look like:
Base case, re-roll nothing: chance of success = 0.
Consider re-rolling : chance of success = 0. This is the same, so keep going
Consider re-rolling [4, 5]: chance of success = 1 / 36. This is better than the previous proposition, so keep going.
Consider re-rolling [4, 5, 5]: chance of success 4 / 216. This is worse than the previous proposition, so stop and re-roll the previous proposition, namely [4, 5].
Optimal result: re-roll [4, 5].
The reason that this works is that if re-rolling a given die is not helpful, you won't want to re-roll any die with a higher result as it will be even less beneficial. Similarly, if re-rolling a given die is helpful, you will want to re-roll any die lower than it at least as much, because re-rolling that will be even more beneficial. This assumes all of your dice are the same size (ex. D6s). The analysis is much more complicated with dice of different sizes.
Here's a more formal (i.e. Python) statement of the algorithm:
def find_rerolls(dice, target):
if sum(dice) >= target:
# If you are already over the target, don't re-roll anything
rerolls =  # This is the list of dice to be re-rolled, which starts empty
p = 0 # This is the probability that your current rerolls will be greater than the target value
for die in sorted(dice): # for each die, starting with the lowest
new_rerolls = rerolls + [die]
# This is the total we would need to beat given the dice we are re-rolling
reroll_target = target - sum(dice) + sum(new_rerolls)
new_p = probability_greater_than_target(len(new_rerolls), reroll_target)
if new_p >= p:
# re-rolling this die improves your chance of success (or is at least break-even), so do it
p = new_p
rerolls = new_rerolls
# Re-rolling that die hurts. Given all other dice are at least as high, stop trying
# Re-roll everything
The only thing missing from this is the implementation of
probability_greater_than_target(number_of_dice, target) (i.e. what is the probability that a set of dice will be higher than a target without re-rolls), which is then a well-formed question for math stack exchange.