Andrew Looney (Fluxx, Icehouse) has written several great articles about game design on his blog, which I'd encourage you to check out (even if you don't like Fluxx). For the purposes of your question, I think this graphic is very helpful (and funny).
He breaks playtesting down into three phases: inner circle of friends, outer circle of friends and random strangers. I have conducted playtesting on one game, and I agree wholeheartedly with these divisions. I don't know how far along you are, so I'll write this assuming you've only done the "mental" playtesting necessary to build a prototype and are looking for a first test run. There might be more detail in here than you need, but I think it's important to define the phases in order to help you find the appropriate people to fill them.
For initial testing, it's important to have people who are both friends and gamers. For me, this was my game group. To your concern about them not liking it, Looney's instruction about this phase is to "repeat until fun", and I think that's a great distillation of the point: until the huge bugs are ironed out, you don't want to "waste" the interest level of people you don't know until your product is fairly polished. The friends part is important because they'll be patient with you, and the gamers part is important so that they'll have some frame of reference for any familiar mechanics, be able to make good suggestions from experience, and roll with things if you need to change them midstream. I'll also add that you shouldn't bother with a rulebook at this phase, at least not for others; you may end up changing things mid-game, let alone between games. Think of this as alpha or maybe even pre-alpha testing. If you don't have any gamer friends (and don't want to pay people to playtest), you really should make some if you want to have success designing. Many areas have game groups that you can find via BoardGameGeek or Meetup; if not, try starting a Meetup group yourself. (You may have success pulling pre-alpha testers off the street, but then I think your concern about them not liking the game and never coming back is valid.)
Once you've gotten things to the point where everyone is having fun (for me, my metric was when my friends started requesting the game), you can move on to the outer circle. For me, this was people like my father-in-law; he'll play games if asked, but won't seek them out on his own. It's helpful to get fresh eyes on it at this stage because the people who've seen the game since its inception have preconceived notions about it and, often, you'll run into situations where a mechanic or situation is a lot better than it was, but still not good enough. These folks are still friendly enough with you that they'll put up with a game with some kinks in it, but often not patient enough to sit though the broken sessions of your pre-alpha period. Family members and non-gamer friends (unless they're militant non-gamers) should do nicely here. This is probably more like alpha testing; you still should be the one explaining the game and possibly playing, but things should be less volatile at this point; changes should be more along the lines of tweaks than overhauls, and you probably shouldn't make any significant changes mid-session "to see what happens".
By the time you get to random stranger testing, you should have a product that someone could pick up off of a shelf and teach themselves how to play. The artwork doesn't have to be professional (or existent), but rules and player aids should exist, and everything should be clear. It's better at this point if you can be there in person, but don't say anything unless the group has answered a rules question incorrectly. This is the fly-on-the-wall point, like dropping your baby off at their first day of kindergarten and letting them go off into the world. Finding players for this stage is fairly easy, because you can find plenty on BoardGameGeek if you can't find anyone locally. I did my testing at a convention, which is a convenient place if you've got one you can reasonably get to. The key part here is to know your audience. If you're making a children's game, Origins or Gen Con is probably not the best place to approach people. Likewise, you wouldn't try a church youth group if your game is a 4-hour brain burner with a provocative theme. Try to find random strangers who are in your target audience; typically, this will be gamers, but not always.