This is a good question, and the answer basically depends on how one should interpret the phrase "as many cards as you have remaining to play" in the card text.
One interpretation is that this refers strictly to the number of cards the rules tell you to play (with the following parenthetical remark making a special exception for "Play All", which one might otherwise interpret as allowing you to draw the whole deck!). Under this interpretation, you would indeed have the option of drawing 4 cards (or 3 cards after playing one — unless your read the FAQ literally, in which case you might not be able to do anything after playing your last card!), regardless of how many cards you actually have in hand.
The other interpretation is that this phrase refers to the number of cards you could actually play at this point (assuming that none of the effects of those cards granted you extra plays or altered the rules!). Under this interpretation, the parenthetical remark about "Play All" merely serves to clarify the "general principle" that you cannot play (and Swap Plays For Draws therefore will not let you draw) more cards than you have in your hand. Thus, with only one card in hand, you would only be able to use Swap Plays For Draws to draw one extra card (or no cards at all, if you already played your one card).
Alas, both of those interpretations seem plausible on the face of it, and the Fluxx Rules don't seem to provide any explicit guidance on the matter. The only even marginally relevant bits in the rules that I could find are the following FAQ entries:
Q: What if the Rules require me to Play 4 but I only have 2 cards?
A: Play as many as you can. Your turn ends when you have no cards left in your hand, even if you are supposed to play more.
and possibly:
Q: How do I interpret the Play All card? If, by playing my cards, I receive more cards in my hand, do those new cards have to be played as well, so that I will always end up with zero cards in my hand at the end of my turn as long as Play All is on the table?
A: Yes.
A strictly literal reading of the former entry ("Your turn ends when you have no cards left in your hand") suggests that, regardless of the interpretation chosen, you cannot actually activate Swap Plays For Draws after playing the last card in your hand, since your turn will already have ended!
On the other hand, it doesn't specifically say that the number of cards you "have remaining to play" is limited by the number of cards you have left in your hand — it just says that your turn ends when your hand runs out, even if, by the current rules, you're supposed to play more cards. Thus, while this reading is arguably more consistent with interpretation #2 above, it doesn't actually contradict either of them. And it's IMO not even clear whether this FAQ entry is really meant to be read that literally.
The only reason I have for quoting the second FAQ entry is to emphasize that the special rule that Swap Plays For Draws has, when combined with Play All, is, while sensible enough (at least when compared to the alternative of drawing the whole deck), not actually an accurate match for how Play All normally works.
Normally, under Play All, you'd be able to (and would have to) keep playing until you literally cannot play any more cards, and literally combining this with the basic Swap Plays For Draws rule, without the special exception, would effectively have you draw the whole deck (and the discards too!) until there are no more cards to draw. Instead, the special rule for Swap Plays For Draws + Play All in the parentheses just has you count your cards to determine the draw/play amount — which is exactly how the FAQ entry says Play All does not normally work. Thus, whereas the first FAQ entry above might be taken as an argument in favor of the second interpretation, by similar logic this one would seem to be an argument against it.
Thus, pending a clarification from the designers, I'd have to say that this is something your table may simply have to make a ruling on, and stick to it. FWIW, I don't think either choice will actually break the game, although they do each offer a different set of strategic options.
