I was pretty reluctant to get into magic because I knew that:
So, though I know it is possible to get started at low cost, I want to know how to participate in the richness of deck building without going over $100.
Is there a way to do this?
Yes. The key is to get common and uncommon cards, avoiding rare and mythic rare cards (Note: A year after answering this question, I wrote a lengthy comprehensive guide on how to keep cost low while still having fun with MtG. I still stand behind the following answer, but I did find out other ways to keep costs low).
Here's how:
If you don't already know the game, your buying will be much more efficient and effective if you learn it first. You can learn the rules and play some sample games on an iOS or Android device with the official app (at the time of this writing, it's Duels of the Planeswalkers). You don't have to spend any money to play a few sample duels but if you want to play all of the different duels you'll have to pay to unlock additional duels ($15 on iOS). You should begin to get a feel for different possible deck strategies.
Alternatively, if you have a friend with several (or more) decks, have your friend teach you the game using his or her decks. Be sure to try out several different styles of decks.
Read about deck types.
Armed with knowledge of the basic deck types, you're ready to go to a local store that processes a lot of cards. A good resource for finding a store near you is the official store locator.
Stores will vary in their policies so you'll have to find one that sells bulk common cards at very low cost. The store I go to lets me sift through many thousands of unsorted common and uncommon cards and pile them into a box that fits about 500 cards, at a cost of $5. It's time consuming, but I get to choose the cards.
This low card cost is possible because some stores generate so many common and uncommon cards as byproducts of events or buying collections that they can't possibly sell them all. They pick out the rares and unusually valuable uncommons/commons. The remaining commons/uncommons will eventually get recycled, so some store owners are happy to sell them for something.
The first time you do this, you will make mistakes, and you'll quickly realize that you don't have enough of certain kinds of cards and too many junky cards. That's okay. You've only spent $5. So you go back to the store a few more times. By your 4th trip you should be a lot better at knowing which cards are interesting and which to avoid. It helps to bring a list of what kinds of cards you want, including specific cards.
By the time you've accumulated around 2500-3000 cards (at least a few hundred of these will be lands and dual lands), you'll find that it's possible to construct a wide variety of decks and pursue a wide range of deck strategies. You have the potential to create hundreds or maybe even thousands of different, effective decks. However, you'll only be able to have 12-15 decks going at one time due to limited number of lands, and different decks competing for certain cards that tend to get used in many decks.
All this can be done for around $25-$30.
It is also possible to buy "fat packs" on sale for $30 or so, and then you can sell off the rares and mythic rares for $15-$45, depending on the market value of what you were lucky (or unlucky) enough to get. But doing it this way subjects you to a lot of randomness and you may get stuck with lots of commons you don't want.
If you decide to participate in tournaments (which you can do with the pauper format or the peasant format), you will likely want to supplement with a few commons you lack. So you may end up spending another $5 or $10 to supplement your several thousand card collection.
All told, it is possible to get quite a variety for less than $50, which will provide plenty of fun and variety in casual play. For a few more dollars, you can fine tune some of your decks to compete well in pauper or peasant formats.
EDIT: About half a year after writing this, I started to get serious about competing with Pauper. While still much less expensive than other formats, there is little chance of staying under $50 if you want many competitive pauper decks. Depending on how many competitive decks you want, your total cost could range from $300 to $1200 to have a wide variety of top pauper decks all built and ready to play.
Expenses for competitive pauper are higher than casual play because there is high demand for many "staple" commons that cost $2 or more, such as Lightning Bolt, Rancor, Gitaxian Probe, etc. that you are very unlikely to find in a penny commons bin. If you want to stay under $500 total you'll have to avoid cards costing well over $5 such as Sinkhole, Three Visits, Chain Lightning, and Oubliette. Most competitive Pauper decks do not need individual cards costing over $5.
In addition to the good ideas in Joe's answer, another thing you can do is find budget deck lists that you like and only buy the specific cards you need for those decks.
To find a deck list you can find a fourm online that has deck lists like MTG Salvation, there they have different forums for each format, and most of them include Budget sub-forums. They also have forums for the Pauper (all commons) and Peasant (a few uncommons, but mostly commons) formats which due to their nature tend to be fairly budget friendly.
Once you have a deck list, either taken from a list online or created yourself, the next thing to do, before you go out and buy the cards, is test the deck and make sure that you like it and it plays well. There are a couple different ways that you can test your deck without buying the cards, the first is by creating proxies of the cards to use only while testing. The simplest way of doing that is to just write everything about a card on a piece of paper and put it in a sleeve with another card behind it for support.
The other way of testing a deck without buying the cards is with a system like TappedOut, this site lets you input a deck list and then lets you play test it against a 'goldfish' (an opponent that has 20 life, but never plays cards or responds to what you do in any way).
This testing phase is important because it allows to try a deck out to make sure you like it, and it allows you to make sure things like the mana curve is right and the color balance between lands and spells is right and other things like that.
Once you are done testing you can finally go and but your cards, if you have a local gaming store that sells bulk commons/uncommons and you can get everything you want that would be the best place to get your cards because with getting mostly inexpensive cards shipping online will probably end up being fairly expensive.
Your question focuses more on casual play than tournament play. But for a long time, there was a nice column that dedicated itself to creating tournament level decks on the cheap. That column is no longer being maintained, but was called "Magic on a Budget" and I've found the thinking there extremely helpful.
In case your local store doesn't have a bargain bin for commons, Wizards specifically releases a product that can help with this, the "Deck Builder's Toolkit"
It contains over 150 cards, which allows for roughly 5 decks at a time, but obviously those decks can be broken apart and re-configured. Critically it also contains a suite of Basic Lands, which can actually be a pain to get a hold of, as stores need to keep hold of some of their opened land stock for Limited events.
The cards will also be only from one Standard environment, so you won't get into a situation where you're playing cards with wildly different themes or over-powered combinations. Everything will "fit together" nicely.
Finally, this is a giftable product, where you can give someone the box as a whole product and it has the necessary instructions for them to get started. And don't underestimate the value of having a box to hold Magic cards in.
