SHP2, encoded by PTPN11, is a non-receptor protein tyrosine phosphatase (PTP) containing two tandem Src homology-2 (SH2) domains. It is expressed ubiquitously and plays critical roles in growth factor mediated processes, primarily by promoting the activation of the RAS/ERK signaling pathway. Genetic and biochemical studies have identified SHP2 as the first bona fide oncoprotein in the PTP superfamily, and a promising target for anti-cancer and anti-leukemia therapy. Here, we report a structure-based approach to identify SHP2 inhibitors with a novel scaffold. Through sequential virtual screenings and in vitro inhibition assays, a reversible competitive SHP2 inhibitor (C21) was identified. C21 is structurally distinct from all known SHP2 inhibitors. Combining molecular dynamics simulation and binding free energy calculation, a most likely binding mode of C21 with SHP2 is proposed, and further validated by site-directed mutagenesis and structure–activity relationship studies. This binding mode is consistent with the observed potency and specificity of C21, and reveals the molecular determinants for further optimization based on the new scaffold.