I would argue that the way a symbol is used affects and changes its meaning. Yes, it's historically been used as a symbol of good fortune, balance, and all sorts of other good things. It has ALSO been historically used as a symbol of hatred, fascism, and genocide.
Symbols are not unchangeable things handed down to humanity from a greater power. Symbols are created by humans and hold only the meanings given to them. Given that the swastika has been given both meanings, it holds both meanings.
If you're using it in its original context, with respect to the cultures it came from, and practicing within those cultures, that's fine. But it's ridiculous and closed-minded to call anyone who sees it as a symbol of hate "brainwashed." Your point of view requires you to completely ignore a large and important part of human history, and as dark and evil as that part of history is, it's still important to acknowledge it and its effects.
Furthermore, the use of this symbol by the Nazis is not exclusively the domain of history. It happens in the modern day too. When there are groups actively using this symbol to represent themselves and preaching hate, bigotry, fascism, and genocide under its banner, it's completely irresponsible to ignore that meaning. Modern neo-Nazi groups still use this symbol.