My theory is mutants.

What if horses were having unicorns? They would then be seen as mutant horse babies and because of their non-pure bred nature they were shunned, given as food to orphanages in the dark ages, whatever you can think of whether it be light or dark. I mean come on. Doesn’t mutant horse baby sound like something you pull away from and shout AHHH at?

However, I’m sure there would be sympathizers with the unicorn kind. When a lion and a tiger have a child with one another then their offspring are known as ligers or tiglons (depending on the sexes) they are primarily thought of as sterile or extremely hard to mate. Maybe unicorns or mutant horse babies are similar. Horses mated with another creature and their unicorn offspring we’re extremely hard to mate afterward and thus led to the eventual extinction of the unicorn kind.

So maybe unicorns hard two hardships against them. They were disfigured revolting mutant babies and had a hard time having the babies. After all its not like the human race has had a stellar past with things being different than the norm.