Because you're
coming at an issue from a certain metaphorical angle. So the background that leads you to your opinion is where you're
coming from. I could say something like, "I think spanking children is bad, but then again this might
come from my strict upbringing." My perspective and opinion
come from somewhere. So I guess you could literally say, "I know where your opinion comes from", but that's just not how the idiom goes and it makes perfect sense to me how it is. Maybe it's just really hard to understand if you didn't grow up around it, and that's fair.
