No. Social Security counts only the wages you earn from a job, or your net profit if you're self-employed. Non-work income such as pensions, annuities, investment income, interest, capital gains and other government benefits are not counted and will not affect your Social Security benefits. For more information, Social Security suggests that you read the following publications which are available online: Retirement Benefits (Publication No. 05-10035) and How Work Affects Your Benefits (Publication No.05-10069)."