For example, if I draw a 4 (move 4 spaces backward) and the only piece that could make the move is in my "safe zone", do I have to make the move?
Yes. From a PDF version of the rules, you are required to move if a legal move is available:
... if at any time you can move, you must move, even if it's to your disadvantage.
and from the description of the Safety Zone, moving backward out of the zone is legal:
... however, a pawn may move backward out of its SAFETY ZONE and on subsequent turns move back into the ZONE as cards permit.
Note that trading places on an 11 is an exception to this - you may decline to switch places with another pawn if you like - but yes, in your situation, you must move a pawn out of your safety zone.
Confusion arose when there was a disagreement about whether you move within the safety zone. One says you must move, while another says once in the zone, you cannot move unless you draw the exact number To arrive in your home. Since the rules specify having to move out on a four, it stands to reason that you must move if possible, but you cannot enter home on a number greater than required to arrive there.