After completing his graduation, John Pippy was assigned to the 1st Cavalry Division at Fort Hood, Texas where he served on active duty from 1992 to 1995.
Later, he left active duty and joined the Pennsylvania National Guard, one of the largest National Guards of United States, where he still holds the rank of Major.
In 1996, he was elected as a Republican Party member of the Pennsylvania House of Representatives from the 44th district, defeating his Democrat Party opponent, Andrew McGraw.
In 1998, prior to general elections, the Democratic Party members challenged his candidacy for re-election based on a clause in the Pennsylvania Constitution requiring candidates for the General Assembly to swear that they have been residents of the state for at least four years.
After much debate, the challenge was eventually dismissed when the Assembly passed a bill, deleting the residency provisions from the candidate affidavit. Pippy emerged as the clear winner of 1998 re-election and ran unopposed in the general elections of 2000 and 2002.
In 2003, when the Pennsylvania State Senator resigned to take a seat in the U.S. House of Representatives, John Pippy announced his candidacy for the seat. Though shortly after announcing his nomination, his unit was called to active duty and he returned to serve in the Iraq War.
Subsequently, he received an initial ruling from the Army Reserve headquarters which barred his candidacy but after intervention from higher authorities, a waiver was issued allowing him to remain in the elections.
John Pippy won the seat with 67% of the vote but at the time, he was serving on active duty in Maryland with his unit. Somehow, he managed to return to Harrisburg on a one-day pass and took his oath of office, but returned to active duty immediately afterwards.
In January 2004, he returned home from Iraq and resumed his service as the Member of the Pennsylvania Senate from the 37th district. The same year, he ran for re-election against the Democrat Gianni Floro, winning with 67% of the vote.
In 2008, he once again won the general election for the Pennsylvania Senate with 64.8% of votes, against the Democratic Party candidate, Amy Jude Schmotzer.
In the 2009–2010 legislative session, he served as a member of several committees such as ‘Appropriations’, ‘Banking and Insurance’, ‘Law & Justice’, ‘Transportation’, and ‘Game and Fisheries’ committee.
Later, he served as the Chairman of the Senate Urban Affairs and Housing Committee, and also chaired the Legislative Budget and Finance Committee.
In June 2012, shortly after the Pennsylvania legislature approved the state budget for the following fiscal year, Pippy resigned from his state senate seat prior to the end of his term, to become head of the PA Coal Association.
John Pippy also became a member of numerous non-legislative committees such as Autism Caucus, Base Development Caucus, Brain Injury Caucus, Coal Caucus, Early Childhood Education Caucus, Environmental Caucus, Hunger Caucus, Life Sciences Caucus, and Pro-Life Caucus.
He holds several other important posts, serving on the Advisory Board for Penn State Beaver, the Board of Pittsburgh Ballet Theater, and the Board of Pittsburgh Botanical Gardens. He is also a member of University of Pittsburgh Cancer Institute and the Sewickley Valley Chamber of Commerce.