Draco Malfoy bullies you because of the way you speak. You defend yourself but he just ends up laughing in your face even more. You keep to yourself, preferring to enjoy your own company. People do not bother approaching you either way, it will only result in pure confusion on their part, but who could blame them? Your thick Scottish brogue was basically a whole foreign language to them.