Hilfiger was born and raised in Elmira, New York. The second of eight children, he grew up in an Irish-American family; he claims direct descent from Robert Burns.^[1] His parents originally intended for him to be an engineer. He attended Elmira Free Academy for high school. Rather than furthering his education, he started to work in retail at the age of 18. Hilfiger would go to New York City to buy jeans and bell-bottom pants, which he customized and resold at a local downtown Elmira store, Brown's.

He later opened his own store, named The People's Place, around the block in downtown Elmira. Although the store was a hot spot for teens with frequent contests and live DJ appearances, there were often more people hanging out than shopping. Over the years, a number of stores closed in downtown Elmira as shopping traffic shifted to the new Arnot Mall in Horseheads, New York. It wasn't long before The People's Place became another casualty. After seven years, The People's Place went bankrupt, when Hilfiger was 25. The site of the original store has since been demolished to make room for First Arena, home of the Elmira Jackals Hockey team.