I had a post here a while back (before I lost my PW and had to create a new account) talking about how and why certain materials and certain lubes are incompatible. It all comes down to what plastics are used to make each kind of toy...there's a lot of polymer science behind it, and it sounds complicated but it's not that bad:
-Silicone toys are incompatible with silicone-based lubricants.
-Latex toys are incompatible with oil-based lubricants (including coconut oil, vegetable shortening, lard, vegetable/olive oil, butter, etc.).
If you start to dissolve the toy with lube, it is conceivable that some of the other things in the toy - plasticizers, possibly filler materials - can leach out into your body. The concentrations won't be high and unless your body is particularly sensitive you should be fine, but the old rule applies: if you're not willing to put it in your mouth, would you really be willing to put it in any other hole? Some people are very particular and refuse to use, say, a toy made out of the "jelly" material because those tend to leak more plasticizers than other materials even if you don't expose the toy to the wrong kind of lube. I myself tend to stick with silicone or vinyl rubber and use water-based lube, although I have a jelly double dildo for when I want to do depth play...
Of course, the bigger practical concern with using the wrong lube is that you've melted your toy...which is fun for nobody. That is also why you shouldn't just worry about mixing the wrong lube with the wrong toy, but also be cognizant of how you store your toys (such as what happened to Mr. bliindsided). Sometimes the leakage of plasticizers from one will get on another and cause it to dissolve...that's how I lost a couple of my older plugs, because I was storing them in a smallish box together and didn't give them space (or use something as a barrier). Toys can get expensive, and if you clean them and take care of them they should last much longer than 3-4 weeks - especially if you bought a quality product in the first place.