There is much that mortal men do not know about the Sidhe and the other dark fey who inhabit the realms beyond the human lands. There is even more that mortals think that they know about the fey, but are mistaken. Much of this is because they rarely, if ever, actually encounter any of the fey folk.
It is for this reason that of all the various branches of the fey, the ones that humanity does have at least some practical knowledge of are the changelings.
Changelings target the newborn, usually within the first six weeks of their birth. It is customary for a newborn to always be watched over, and perhaps have a small piece of iron lain on or above their body, to protect them from being taken away by a changeling.
Should the proper precautions not be taken, it is possible that one of the Sidhe (or one of the other lesser branches of the fey) will show up with a changeling, and quickly make a switch, leaving the changeling behind to be raised unknowingly by the humans, while the fey steal away with the human newborn. The more beautiful the child, the more likely it is to be targeted.
Should the changeling's true nature not be detected, it will grow up among the humans, doing its best not to give its true nature away. Whatever the original human child might have grown up to be, the changeling will be thin, graceful, and beautiful as it grows up. They are intelligent and perceptive, but their insight is invariably drawn toward deception and mischief. A few might develop a genuine fondness for its adoptive family, but most secretly take a cruel delight at their continued deception.
Unlike most of the fey, changelings can abide the touch of iron for short periods of time, although they do not like it, and as they grow older they will make excuses not to handle such items (rare is the changeling that attempts to infiltrate the household of a blacksmith). Injuries caused by items made of iron are especially harmful to changelings, although not as much so as it is to the other fey.
Should the changeling fall prey to mortal accident or violence, in its death its body will become withered, hideous, and charred. Depending on the changeling's personal temperament, most will eventually flee back toward the faerie lands, usually during childhood or early adolescence (certainly before they enter adulthood, although there may be rare exceptions to this).
And what of the mortal children stolen away? If the human parents eventually discover the deception that has been inflicted upon them, most will assume that their true child is forever lost to them.
They are correct, but not entirely in the manner they believe.
Once they have been taken into the faerie lands, the human child is aged quickly (the flow of time can differ in the faerie lands, in some areas moving faster than normal, in others slower - of course, the fey know where these temporal anomalies are, but the unwary human traveler can easily fall prey to such traps), and when they are old enough, they are forced to do the menial labor that the Sidhe and other fey would never demean themselves to perform. As they grow, what natural beauty they might have had is drawn away from them, and leeched into the lands and its fey inhabitants; in this the fey help to maintain their unnatural beauty.
Some of these once-humans eventually escape from their captors (this is easier than one might suspect, given that they fey easily grow bored and distracted over time). Some of them have made their way back into human lands, where they find themselves shunned by humanity for their terrible appearance. These once-humans can breed true among themselves, and so they live among the dark forests and hidden ruins, just outside the lands of man.
And so it is that the once-humans, who now call themselves goblins, hate both fey and man... and not without reason.