In extreme terms (which is applicable for most of us), if someone isn't building you up, then they're bringing you down. There is no neutrality.

Surround yourself with those that will build you up, strengthen, encourage, and challenge you. You will have seasons that these things may not seem to be happening, but never under estimate the value of perseverance. As you surround yourself with those that are building you up, do not forget the call to service, the call on you to selflessness. Assure yourself that you are one that builds others, and others are sure to come along that will in turn, directly or indirectly, build you up.