Much of the Gardens is located in a valley that the planners of Duke University in the early 1920s hoped to turn into a lake with elegant fountains, but funds were short and the project was abandoned. Instead, the idea of a garden arose in the early 1930s, due to the vision and enthusiasm of Dr. Frederic M. Hanes, an early member of the original faculty of the Duke Medical School.

Dr. Hanes possessed a special love for gardening and was determined to convert the debris-filled ravine, by which he walked daily, into a garden of his favorite flower, the iris. He persuaded his friend, Sarah P. Duke, widow of one of the University's founders, Benjamin N. Duke, to give $20,000 to finance a garden that would bear her name.
In 1935, more than 100 flower beds (in the area which would become today's South Lawn) were in glorious bloom with 40,000 irises, 25,000 daffodils, 10,000 small bulbs, and assorted annuals, all of which were washed away in heavy summer rains and the flooding stream. By the time of Sarah P. Duke's death in 1936, the original gardens were destroyed. Dr. Hanes convinced her daughter, Mary Duke Biddle, to construct a new garden on higher ground, as a fitting memorial to her mother. Ellen Shipman (1869-1950), a pioneer in American landscape design, was selected to do the plans for both the construction and the plantings for the new gardens.

Duke Gardens is considered Shipman's greatest work and a national architectural treasure, most of the some 650 other gardens she designed having long since disappeared.

The Sarah P. Duke Gardens today consists of four major parts: the original Terraces and their immediate surroundings, the H.L. Blomquist Garden of Native Plants (a representation of the flora of the southeastern United States), the William L. Culberson Asiatic Arboretum (devoted to plants of eastern Asia), and the Doris Duke Center Gardens. There are five miles of allées, walks, and pathways throughout the gardens.