It is true that if you withdraw your contributions from the non-covered pension system and give up all rights to the pension, you can avoid GPO.   However, losing a lifetime pension (reflecting both your contributions and your employer's contributions), is an important decision and needs careful consideration.  If you choose to withdraw your contributions from the non-covered pension system in order to avoid the GPO, you may do so at any point before you begin receiving the pension.

NOTE:  You can also be exempt from the windfall elimination provision (WEP) if you withdraw your contributions to the non-covered pension system, but the rules for the WEP are different.  To be exempt from the WEP, you need to withdraw your contributions before you become eligible for the pension from non-covered employment.