So I am blanking on the word for this, but the goal is for the character to get stronger over time (power curve), where the character gains levels/items. What I am looking for is different ways a player can add new spells to their kit (imagine every one is a wizard, and they need to create a spell book and add to it through the game). I don't want to have a level up system. for example, things like every few turns, they draw a new spell card and add it to their inventory, killing an enemy and it drops loot. What other way can the player add to their kit to handle more difficult enemies?
I'm commenting your examples first and then adding a couple more:
- Drawing from a deck:
It's a way of randomizing how the character evolves, but it needs to be balanced as players expect to learn more powerful spells when they advance in their adventure. So it could be done this way if you arranged the deck previously or separated the cards in more decks. In addition, forcing players to learn the spell they draw could make them feel they're not free when choosing their way to play, so you could make them draw two or more cards and choose between them.
Being another random system of learning spells, can be felt odd as a monster could carry equipment, gold and other treasures with it but not a spell which is not something physical. It could be solved by making all enemies wizards who carry spellbooks with them, but I'm not sure that's the game your looking for. So I'm not sure about how this system would work. Anyway, it would be too random again, and needs some kind of arrangement like the previous one.
Essentially a experience system, where instead of earning experience you earn gold (or other sort of money) and you spend it on spells. The main difference with the experience system is that you can save the money to buy spellbooks more powerful than other, or choosing to split it between less powerful spells which give you a more dynami way of playing.
- "Copying" or learning from the enemy
Make spells only available for a player after they've seen it used by another character or enemy. Not every spell they see, of course, but make them remember the spells they've seen and after a few turns making them stop and meditate/concentrate/etc to replicate one of the spells they've seen.
- Some kind of "enlightment"
The wizard performs a very difficult move and they are successfull, then he discovers new ways of magic. Players would need to accomplish an achievement in order to unlock a more powerful or different spell. Achievements could be hidden or shown, and in case of being hidden even keep hidding them after someone gets the spell, so other players need to figure out how the player did it. But again, I'm not sure which kind of game you are trying to create, so maybe the achievements won't fit in it.
Note: there are many details about the pacing of the game that would really clear things up. For instance, when/how do you encounter "more difficult enemies"? Do you just work through a deck of increasingly difficult challenges (enemies)? If so, you can regularly schedule upgrades (say, every 4 or 5 encounters).
Now when you say that you want to avoid a "level[ing] up system", that's also unclear. Are you opposed to tracking XP? To having multiple stats that depend on the character's level?
As for specific mechanisms, the first one that comes to mind is the way that technology works in Twilight Imperium 4ed. If you look at the bottom right card in this image, you can see two blue icons on that card. That is a prerequisite for researching that technology (you need to have two technologies that provide that blue icon before researching "CARRIER II").
In your game, you would have several weak/simple spells with no prerequisites. Then tier 2 spells would have 1 or 2 prerequisites (maybe as flexible as "any two spells"). You can make the better and better spells require more icons, so a super Fireball would require 3 fire icons, and a Firestorm would require one each of fire, water, and wind.