Yeah, IMHO the concept of magic in fantasy can really be a make-or-break element. As a reader, I'm willing to forego a lot of science for the good of the story. As a writer though, I find it hard to create a world that strays too far; I try to make everything explainable, which can be tough and, bluntly, defeats the whole point of magic. But you're right, there has to be some sort of balancing of magic, e.g. it makes the user weak, or has unintended consequences, etc.
I can say this, I am more a fan of sword vs. sorcery. I loathe magic users in books that use spells, incantations, sacrifices, etc. I prefer a lighter brand of fantasy than all that, but that's just me and my tastes.
If you ever get stuck, just throw a red priestess in your book somewhere, everyone else does.
