This article radically reframes Henry James's The Portrait of a Lady in relation to the Underground Railroad, the transatlantic slave trade, US slavery, and racial housing segregation. Focusing on the house in Albany, New York, where Isabel Archer stays in the 1850s, it asserts that Isabel's pursuit of freedom is grounded in her 1850s childhood when the Underground Railroad was particularly active in Albany. It examines the Albany home within the historical context of the 1870s and 1880s, when, respectively, Isabel returns to Albany and The Portrait of a Lady was first published. Taking into account the Supreme Court's 1883 Civil Rights Cases, which facilitated housing discrimination against Black Americans, this article argues that the Albany home, whose door remains bolted to the “vulgar street,” protects James's American lady from a vulgarity associated then with African Americans. By examining a critically overlooked function of the Albany house—its spatial representation of the novel's plot—this article shows how narratives of Black people escaping slavery along with late nineteenth‐century definitions of vulgarity centrally define Isabel's pursuit of freedom. Analyzing the plot's architecture, the article reveals how James strategically alludes to two novels about racism—overtly to George Eliot's Daniel Deronda and covertly to Harriet Beecher Stowe's Uncle Tom's Cabin—to construct his highly influential narrative about a white woman's transatlantic journey toward freedom.