Walk past Fifth Avenue and 34th Street in Manhattan today, and you will see the Empire State Building and the steady stream of tourists waiting for the observation deck. What you will not see is any trace of the fact that for 72 years before the skyscraper was built, that corner held a brownstone where a woman named Caroline Astor decided who in the United States counted as a real person. Her list contained 400 names, kept on a slip of paper in her writing desk on the second floor. If you were on it, you were invited to her ballroom.
If you were not, you were nobody. That is only one chapter of a much longer story. The Astor family arrived in America in 1784. By the time the Empire State Building replaced their hotel, they had been the wealthiest family in New York for almost a century.
Over 242 years, the family went from a German immigrant landing in Baltimore Harbor with about $25 to a great-great-great-grandson who lost his seat in the British House of Lords in April 2026, after the family had held it for 110 years. Along the way, the Astors lost a passenger on the Titanic, produced the most famous elder abuse trial in modern American history, and built a swimming pool in Buckinghamshire that helped bring down a British government in 1963. Most American dynasties either keep their money or implode publicly. The Astors did neither.
They faded. The fortune held, but the addresses that defined them vanished. The Astors built more vanished real estate than any other American family. Almost every famous address they put their name on no longer exists or no longer belongs to them.
350 Fifth Avenue was demolished in 1895. 338 Fifth Avenue came down in 1893. The Waldorf-Astoria Hotel built on top of those two lots was demolished in 1929. 840 Fifth Avenue was demolished in 1926.
Beechwood in Newport was sold in 1940, and Cliveden in Buckinghamshire was given to the National Trust in 1942. Hever Castle in Kent was sold in 1983. The pattern is most concentrated on a single block in Midtown Manhattan. In the early 1860s, the Astor brothers, John Jacob III and William Backhouse Jr.
, built two adjacent brownstones on Fifth Avenue between 33rd and 34th streets. Their wives lived in those houses and did not speak to each other for almost three decades. By the 1890s, both brothers were dead, and the heirs decided independently to tear the brownstones down. In 1893, William Waldorf Astor demolished his father’s house and built the original Waldorf Hotel on the lot, specifically to overshadow the brownstone of his aunt next door.
Two years later, Caroline Astor and her son tore down their own house to build the Astoria Hotel on the lot directly next door, to neutralize the first one. By 1897, the two hotels had merged into the Waldorf-Astoria. That building was demolished in 1929, and the Empire State Building was raised on the lot. Today you can stand in the lobby of one of the most photographed buildings on Earth and not realize you are standing on top of three different vanished Astor structures: two brownstones from before the Civil War, the two hotels that replaced them, and the combined hotel those two became.
Sixty-four years of Astor real estate on the same lot, and none of it survives. The Newport story tells the same thing more slowly. Beechwood, the family summer house, was sold in 1940. It changed hands several times.
Larry Ellison, the founder of Oracle, bought it in 2010 and has spent over a hundred million dollars restoring it to look the way it did in 1881. The Astors are not part of the restoration. They are the ghosts the restoration is answering. Cliveden in Buckinghamshire was given to the National Trust in 1942 and is now a luxury hotel.
Hever Castle in Kent, where Anne Boleyn was raised before she married Henry VIII, was bought by an Astor in 1903 and sold by the family 80 years later. The Astor identity for two and a half centuries was tied to their addresses. By the late 20th century, the addresses were no longer the family’s. The fortune kept doing fine.
The fortune is not the part of the story that gets lost. The buildings are. The man who started it all arrived in Baltimore Harbor in March of 1784 with about $25 and a stock of German flutes. He bought the land before there were addresses to put on it.
In 1807, Aaron Burr needed to leave the country. The former vice president had been acquitted at his treason trial that summer, but his political career was finished after killing Alexander Hamilton in a duel three years earlier. To fund his trip to Europe, he had to liquidate Richmond Hill, his Manhattan estate, roughly where the West Village is now. The buyer he found was an immigrant fur trader named John Jacob Astor.
Astor bought Richmond Hill at a discount, subdivided the land, and began leasing the parcels. The estate that had been one man’s country house became dozens of building lots, each rented under terms Astor controlled. Other people saw Manhattan as places where you lived. He saw it as inventory.
Born in 1763 in Waldorf, the youngest son of a German butcher, Astor left for London at 16 to work for an older brother who made musical instruments. He sailed for America in November of 1783. The ship was caught in ice in Chesapeake Bay for two months. While they waited, Astor met another passenger, a returning fur trapper.
That conversation became the rest of his life. He arrived in Baltimore in March of the following year and spent the next 30 years building an export empire in pelts. The American Fur Company, founded in 1808 with the explicit blessing of President Thomas Jefferson, eventually monopolized the fur trade across the Great Lakes and the Missouri River Basin. Underneath that surface business, Astor was running ships across the Pacific, carrying furs to Canton and tea and silk back to New York.
By 1816, he was carrying something else. That year, the American Fur Company shipped ten tons of Turkish opium to Canton aboard a packet ship called the Macedonian. China had banned opium imports 17 years earlier. Astor was the first American known to have run the route.
The chartered enterprise the Jefferson administration helped create ran narcotics into a country that had banned them. Astor did it for three years before shifting his opium operation to Britain, where it was legal. Whether the China money paid for the Manhattan land is something historians have argued about for 200 years. Probably yes.
The same ships also came home with tea and silk, so the trail is hard to follow exactly. What is certain is that by the 1820s, no other private buyer in New York could match his rate of acquisition. Farms, leases on farms, ground rents, Old Trinity Church land grants. Most of it sat north of Houston Street in what was then countryside.
He almost never built. He left the buildings to other people. He wanted the dirt underneath them and the right to collect rent on it forever. In 1834, at 71, he sold the fur business outright and put the proceeds into Manhattan real estate.
He spent his last 14 years buying property in New York. On March 29, 1848, he died at home, 84 years old. The fortune was almost entirely real estate, and the real estate was almost entirely on a single island. The son who inherited it was the second son.
The first, John Jacob Astor II, had been institutionalized for mental illness in his 20s and would die in confinement 21 years later. The man who would run the empire was a younger brother, William Backhouse Astor, whom his father had pulled out of a German university to come home and learn the family business. William had wanted to be a classics professor, lecturing on Homer and Virgil and Tacitus in a small town in Germany. What he became was the landlord of New York.
He was 25, studying classical philology at the University of Göttingen, when his father called him home in 1817. He ran the family’s Manhattan property for the next 58 years from a townhouse on Lafayette Place. He kept his own ledgers. He never enjoyed the work and never pretended to.
The biographical record describes a curmudgeon who avoided social events, did not host dinner parties, and spent his evenings reading classical literature in the original languages. The work, however, was being done. Under his management, the Astor position in Manhattan roughly doubled. He perfected a structure his father had used.
The Astors did not build tenements. They owned the dirt. They leased the lots on long ground leases to operators who put up the buildings and managed the tenants. The operators handled what happened inside.
The Astors collected the ground rent regardless. In 1845, a public health study found that 25 percent of children born in lower Manhattan tenements died before their first birthday. One in three did not see their fifth. The conditions described in the report would have been recognizable to anyone living on the densest blocks of Astor land.
William’s name appeared nowhere in the report. The buildings were not registered to him. The tenements that had killed the children had been built and managed by operators on land William had leased to them. When city reformers tried to pass laws requiring better light, ventilation, or fire safety, William fought them.
He funded lawyers in the New York legislature and supported Tammany Hall, the political machine that kept the building codes loose. The compromises tended to look the same: a tenement footprint reduced by five percent, the rent raised by ten percent to compensate. By 1867, a magazine published a list of the largest private fortunes in New York. William’s was $16 million.
The man in second place was $8 million behind him. By his death in 1875 at 83, the family fortune had moved from $20 million to $50 million. William’s sons were very different. The older son, John Jacob III, was bookish.
The younger son, William Backhouse Jr. , was a yachtsman who avoided his own marriage. Their wives lived next door to each other on Fifth Avenue and did not speak. The wife of the younger son was Caroline Schermerhorn.
The next 30 years of New York high society would be hers. Every social claim a person could make in 19th century New York had to pass through a small calling card. The card had your name on it. The butler placed it on a silver tray and carried it upstairs.
A woman somewhere in the house looked at the card. If she returned the call, you were part of her circle. If she did not, you were nothing. The woman in charge of this system between the late 1860s and the early 1900s was Caroline Schermerhorn Astor.
She had been born a Schermerhorn, descendant of New York’s original Dutch land-owning families. Her 1853 marriage to William Backhouse Jr. combined three centuries of Manhattan pedigree with unlimited Astor money. Through the 1860s, she was just another wealthy woman on Fifth Avenue.
Her life was the brownstone at 350 Fifth Avenue and a ballroom upstairs that held 400 people. In 1872, a Southern man named Ward McAllister attached himself to her. McAllister had grown up in Savannah and decided New York needed a system for separating real society from the new industrial money flooding in from the Midwest. He proposed organizing the city’s old families into a private cartel that met at exclusive winter balls.
The Patriarch Balls. Twenty-five gentlemen, each empowered to bring five women and four men of their own selection. Caroline approved. For the next two decades, the Patriarch Balls were the spine of the city’s social order.
McAllister managed the guest lists. Caroline funded and approved. The most famous test was the Vanderbilts. Alva Vanderbilt had been trying for years to get into Caroline’s circle and was systematically excluded.
In 1883, she announced she would host a costume ball at her new Fifth Avenue mansion and let it be known that Caroline’s daughter, Carrie, was rehearsing a quadrille for the event. Caroline asked for an invitation. The answer was that without a calling card from Mrs. Astor, no Vanderbilt invitation could come.
Caroline sent the card. The Vanderbilts were in. McAllister, by the late 1880s, had become a problem. He had started talking to newspapers.
He published a memoir in 1890 called Society as I Have Found It, which made him sound like a pompous clown. The press started calling him Mr. Make-a-Lister. In February of 1892, he gave the New York Times what he called the official list of the 400: 319 names from 169 families.
The number came from his earlier remark to a reporter that there were only about 400 people in fashionable society. The same number conveniently fit Caroline’s ballroom. The Times printed it. The country read it.
McAllister died alone in 1895, dining at his club. Caroline did not attend his funeral. She kept hosting through the late 1890s from a larger chateau on Fifth and 65th designed to hold 1,200. After her last great ball in 1905, she fell on her staircase and broke her hip.
She died three years later suffering from advanced dementia. By then, the system was unraveling. The next generation of hostesses cared less about policing who counted and more about using their houses for political access. The 400 as a working institution did not survive her by 10 years.
What survived was the idea that in any country with new money, a small group of people decides who else is real. She had one son. His name was John Jacob Astor IV. The hotel war that split the family, the marriage that scandalized it, the night on the Atlantic in 1912—all of those stories begin with her.
In the summer of 1892, William Waldorf Astor had his servants tell American reporters that he had died of pneumonia. He waited to read his own obituaries. When the ruse was exposed, he was mocked publicly across the United States. This is the man who hated his aunt enough to build a hotel to spite her.
The aunt was Caroline Astor. The two of them had been at war for nearly a decade over a single question: who got to be called Mrs. Astor. Until 1887, there had been two Mrs.
Astors. When William Waldorf’s mother Charlotte died that year, Caroline shortened her own card from Mrs. William Astor to simply Mrs. Astor.
William Waldorf inherited the head of family role and tried to assert the title for his wife, Mary. Caroline disagreed. What followed was less an argument than an architectural war. By the early 1890s, William Waldorf had inherited his father’s brownstone.
Caroline still lived next door in hers. In 1893, William Waldorf had his father’s brownstone torn down and erected the original Waldorf Hotel on the lot. Thirteen stories, designed in the form of a German Renaissance chateau. It loomed over Caroline’s residence next door.
Inside were public dining rooms and bars, which polite society at the time did not patronize. Caroline waited two years, then tore down her own brownstone and built the Astoria Hotel on the lot. Sixteen stories, designed by the same architect who had built the Waldorf, deliberately so. The two structures abutted each other.
In 1897, they were connected by a corridor and rebranded as the Waldorf-Astoria. The rivalry was buried under a single business arrangement. William Waldorf had already given up by then. In 1890, he had begun renting Lansdowne House in London.
Three years later, he bought Cliveden, the country estate of the Duke of Westminster. By 1899, he had taken British citizenship. He was made a baron in 1916 and a viscount the year after. The title would stay in his line for over a century.
What he had done by leaving was split the family into two branches that would not meaningfully reconnect. The British branch survived. The American branch had its own next problem coming. That problem was on a ship in the North Atlantic.
He was 47. She was 18. The wedding was in Newport because no Episcopal church in New York would perform it. The bride’s name was Madeleine Talmadge Force.
The groom was John Jacob Astor IV, only son of Caroline Astor, head of the American branch of the family. By any standard of 1911, the marriage was a scandal. The age gap alone was disqualifying. The groom had divorced his first wife two years earlier.
The bride’s family was respectable but minor. The Episcopal Church refused to perform the ceremony. The groom had been born in 1864, raised at 350 Fifth Avenue, sent to Harvard. By his 30s, his net worth was approaching $87 million.
He let his managers handle the buildings. What occupied his time was side projects: a bicycle brake patent, a vibratory disintegrator, a turbine engine design, a yacht, and a science fiction novel about life on Saturn. His one consequential real estate project was the St. Regis Hotel, built between 1904 and 1909 at Fifth Avenue and 55th.
He intended it as a private hotel for old money, contrasting deliberately with the Waldorf-Astoria his cousin had built downtown. The St. Regis still stands, the only major Astor building of his generation that was not eventually demolished. He divorced his first wife, Ava Lowle Willing of Philadelphia, in 1909.
Two years later, he met Madeleine Force at Bar Harbor in the summer of 1910. She was 17. They were engaged the following summer. The wedding was in September of 1911.
The newlyweds went to Egypt to wait out the gossip. By February of 1912, Madeleine was pregnant. They arranged to come home on a new British ship making its maiden voyage from Southampton to New York. They boarded at Cherbourg on April 10, 1912, with a private suite, a maid, a nurse, two manservants, and an Airedale terrier named Kitty.
Four nights later, the Titanic struck an iceberg. Around 12:40 a. m. , as lifeboat number four was being loaded, Astor helped Madeleine into the boat.
He asked the officer in charge whether, given that his wife was pregnant, he might join her. Second Officer Charles Lightoller, who had been told that the rule was women and children only, said no. Astor accepted the answer and stepped back from the boat. The Titanic broke apart and went down at 2:20 a.
m. Astor was last seen at the deck rail watching the boats go. The recovery ship Mackay Bennett pulled his body from the North Atlantic on April 20. The wallet identified him.
The gold pocket watch was still in his vest. Madeleine survived. She was 18, pregnant, and held a stipend that ended on remarriage. She remarried twice.
The bulk of the fortune went to Vincent, John Jacob Astor IV’s 20-year-old son from his first marriage, who ran the family’s American real estate position for the next 40 years. The unborn child was born in August of 1912 and was named John Jacob Astor. Three thousand miles away, his cousin’s branch was settling into Cliveden, the country estate that would within 20 years host the dinner parties that pulled British foreign policy toward appeasement. Cliveden sits on 150 acres of Buckinghamshire chalk above the Thames.
The Astors took it over in 1893, when William Waldorf bought it from the Duke of Westminster. He gave it to his son as a wedding present in 1906. The son’s wife was a Virginian named Nancy Langhorne. Within 15 years, she became the first woman ever to take a seat in the British House of Commons.
Through the 1920s, Nancy gathered around her a circle of British political figures, journalists, and intellectuals. By the early 1930s, the circle had a public profile and a working position on what should be done about the rising power of Germany. The position was that England should not fight Hitler. Lord Lothian argued Germany had legitimate grievances about Versailles.
Geoffrey Dawson, editor of The Times, suppressed reporting of German atrocities. Lord Halifax met Hitler in 1937 and came back impressed. Nancy and her husband Waldorf supported the line and hosted the meetings. In 1937, a left-wing journalist named Claud Cockburn coined the phrase Cliveden Set in his newsletter The Week.
The phrase stuck. Within a year, the Cliveden Set was being blamed for British foreign policy. The accusation was overstated. The group did not run British policy.
It provided a respectable upper-class sounding board for a position the government already held. Munich was Chamberlain’s deal, not Cliveden’s. The perception of influence, once it had taken hold, did not lift. By the time war was declared in 1939, Nancy was being treated in the left-wing British press as an honorary collaborator.
She kept her Plymouth seat, but her political career was effectively over. Waldorf and Nancy gave Cliveden to the National Trust in 1942. The terms let them keep living there as tenants. Waldorf died in 1952.
Nancy retired from Parliament in 1945 and died in 1964. But that was not the end of the British Astor story. By the early 1960s, parts of the estate were leased out to tenants. One of them was an osteopath named Stephen Ward, who occupied a cottage on the grounds called Spring Cottage.
On a July night in 1961, at the swimming pool, Ward introduced a 19-year-old woman to the British Secretary of State for War. That introduction would, within two years, bring down the British government. Christine Keeler was 19 years old when Stephen Ward introduced her to John Profumo at the Cliveden pool on the night of July 8, 1961. Keeler was a London showgirl.
Ward was the osteopath who had befriended her, brought her to weekend parties at Cliveden, and used her as social currency among his wealthy male clients. Profumo was the British Secretary of State for War, 46, married to the actress Valerie Hobson. Profumo’s wife was at the party. So was the Astor host of the weekend, the third Viscount Astor, who lived at Cliveden as the National Trust tenant and was friendly with Ward.
What Profumo did after the encounter was begin a brief affair with Keeler. It ended when British intelligence informed Profumo that Keeler was simultaneously sleeping with Yevgeny Ivanov, a Soviet naval attaché and known intelligence officer, also in Ward’s social circle. That is the part that turned a private affair into a national security crisis. In March of 1963, Profumo was asked about Keeler in the House of Commons.
He denied any impropriety. The denial was made on the floor of the house under privilege. Two months later, evidence surfaced that the denial had been a lie. He resigned in June.
The fall took the rest of the government with it. Prime Minister Harold Macmillan, already weakened, never recovered politically and resigned in October. The Conservative Party lost the next general election. Stephen Ward was prosecuted on charges of living off immoral earnings, found guilty, and committed suicide before sentencing.
Christine Keeler did jail time on a perjury charge. William Waldorf Astor III, the third Viscount, was treated by the British press as collaterally implicated for the rest of his life. His name had not been on the indictment of anyone. The pool had simply been on his estate.
He died in 1966, three years after the resignation. The scandal had three lasting effects. British intelligence and security culture tightened around political vetting in ways that lasted into the 21st century. Tabloid journalism discovered that the private lives of cabinet ministers were now legitimately newsworthy and never went back.
And Cliveden kept operating as a National Trust property, but the Astor family connection became in popular British memory primarily about the pool. In 2006, journalist Phillip Knightley argued that the establishment had used Stephen Ward as a sacrifice. Nobody on the establishment side of the affair, including the Astor whose pool it began at, faced consequences anywhere close to what Ward got. By the early 1970s, the surviving Astor children had largely left British public life.
The British branch retreated to Hever Castle in Kent and to private business interests. The American branch was now running on Vincent Astor’s foundation in the hands of his widow. That was not the last problem the family produced. What does it look like when an American dynasty ends?
It looks like an apartment on Park Avenue where a Childe Hassam painting of Fifth Avenue with flags hung on a library wall, worth $10 million, until the day someone took it down to sell. The apartment belonged to Brooke Astor, the widow of Vincent Astor. By 2002, she was 100 years old, in early-stage Alzheimer’s, and increasingly in the care of a son who was systematically removing things from her life that she had stopped noticing. The painting was one of those things.
Brooke had married into the family. Vincent, her third husband, had inherited the bulk of the fortune from John Jacob Astor IV in 1912. He died in 1959 and left her his foundation. Brooke ran it for 38 years, gave away approximately $200 million, and closed it in 1997.
She walked the city, visiting the New York Public Library, the Bronx Zoo, the Metropolitan Museum. She showed up at neighborhood centers in Harlem and the Lower East Side with envelopes of cash. Then her son moved in. His name was Anthony Marshall, Brooke’s only child.
By the early 2000s, when she could no longer follow most conversations, he began consolidating control of her affairs. Will amendments increased his share by tens of millions. He gave himself a salary of nearly $2 million a year. He sold the Hassam painting and kept the proceeds.
In July of 2006, Brooke’s grandson Philip filed a guardianship petition in Manhattan court alleging elder abuse. The petition described an apartment allowed to deteriorate, a mother in soiled clothes on a sofa, and a son looting the estate. The trial of Anthony Marshall took place in 2009. It ran for five months.
He was convicted on 14 counts including grand larceny. Brooke had died on August 13, 2007, before the trial began. She was 105. The British branch had a title.
The fourth Viscount served in the House of Lords for over 40 years. In April of 2026, after Lords reform removed the remaining hereditary peers, he lost his seat. The Astor parliamentary presence begun in 1916 was over. What is left?
A foundation that closed itself in 1997. A British title that no longer votes. A handful of Manhattan buildings that no longer carry the name. A grave at Trinity Cemetery in Washington Heights.
A German butcher’s son arrives in Baltimore in 1784 with $25. Two hundred forty-two years later, the British branch loses a vote in the House of Lords. The American line has ended in elder abuse. Most American dynasties end this way.
The fortune persists for a few generations, thins, then disperses. What was unusual about the Astors was how visible the unwinding was. Fifth Avenue, the House of Lords, court, a passenger liner, a swimming pool. Across two and a half centuries.
If you walk past Fifth Avenue and 34th today, you will not know any of it. The Empire State Building is the building. The lot is the lot.
And that is the story.