Also, I'm snurching this next bit from brokensymmetry, because I liked it and think it should be read by more people, and I figure it's fair, since he said I played a role in the formation of these thoughts:
The idea stated bluntly is this: the right mathematical language with which to describe magick, assuming you want to describe magick mathematically, would seem to be category theory.
Of course this may be because category theory is the right mathematical language with which to describe everything. It's becoming increasingly important to pure math, physics (including string theory), computer science, and so on.
But I can't explain the idea without explaining category theory, which I have neither the time nor skill to do, and there don't seem to be any good online introductions. And it doesn't help that my understanding of the subject is still very rudimentary.
What I can say is that a lot of magick seems to me to be about defining some sort of equivalences between things (sephiroth, colors, elements, major arcana, deities, etc) and the relationships and processes between things (what is done to the voodoo doll will happen to the person). In particular these equivalences are not equalities but some sort of "equality with regards to a particular attribute or structure." Category theory is all about modeling mappings between things, processes and equivalences in an abstract and general way.
For fans of "Full Metal Alchemist," the law of equivalent exchange could be rephrased as "the only transformations allowed by alchemy are isomorphisms".
My own magickal inquiries have been curtailed by my workload and the glumness that winter brings, but this is headed, essentially, in the direction I was headed, only I'll never be a good enough mathematician to be a very good magickian. Also, this last bit from brokensymmetry, which says it all:
[I should also add by way of clarification that I'm quite prepared to believe magick works, but only for suitable definitions of magick best exemplified by a bit from one of Terry Pratchett's novels where Magrat has fallen off her broom and performs a spell which changes her from a panicked, terrified woman plummeting towards the ground into a calm, rational woman plummeting towards the ground.]