EA is probably too broad of an umbrella. I think there's a distinction to be made between trying to find the best per dollar donation (givewell etc) and people who use it as justification to amass wealth. That said, most of the people I know who declare themselves effective altruists also openly state that the world would be better off if all the wealth and decision making power were concentrated in a small group of hundreds of people who are "smarter" than everyone else. They tend to think that they would belong that group. Anecdotal, but it does make me skeptical whenever I meet someone who uses EA to describe themselves.