On a warm spring afternoon in 1861, Mary Anna Custis Lee stood at the entrance of the grandest house in Virginia, looked over its furniture, portraits, her husband’s letters, and her father’s silver, and made a decision that shaped her family’s life for the next sixty years. She took only what she could carry, believing she would return within weeks. She never set foot in that house again. Her husband was Robert E.
Lee. The house was Arlington. And what happened to his seven children and the wife he left behind did not end with the Confederate surrender at Appomattox. In many ways, their story began at that exact moment.
One daughter did not live to see the war’s end. One son spent seventeen years fighting the United States government in courtrooms, not with a rifle but with a lawsuit that reached the Supreme Court. Another daughter disappeared into Europe for decades, and when she finally returned home, she was arrested. By the end of this story, it becomes clear why historians say the Lee family’s real war did not end in 1865.
It only changed its form. Mary Anna Custis Lee was already crippled by arthritis when Union troops crossed the Potomac and occupied her family estate. She left Arlington believing the separation was temporary, just as most people in April 1861 believed everything about the coming war would be temporary. Within three years, the federal government turned her front yard into a cemetery for Union war dead.
It was a deliberate punishment aimed directly at her husband, so that no Confederate general could ever look at his home again without seeing rows of men his army had killed. For the rest of the war, Mary Lee lived as a refugee, moving between relatives’ homes across Virginia, confined to a wheelchair, watching her husband’s reputation grow into legend while her health quietly deteriorated. When Richmond finally fell and the Confederacy collapsed, she did not get her house back. She got silence, distance, and a husband who spent his final years as a college president in Lexington rather than as the owner of an estate in Arlington, because Arlington no longer belonged to either of them in any way the law recognized.
Robert E. Lee himself lived only five years after the surrender. He died in 1870 without ever recovering the property he had spent thirty years restoring. That fact alone would have ended most family stories.
But for the Lees, it was merely the beginning of the second chapter. There is an irony buried in the grounds of Arlington that most people do not know, and it made the family’s loss even harder to accept. While the front garden filled with graves, the federal government also set aside part of the property for a completely different purpose: a settlement for formerly enslaved people who had fled bondage during the war. For a time, the very hill where Mary Custis Lee once presided over one of Virginia’s largest enslaved workforces became home to hundreds of newly free men, women, and children building their own lives on land the Lee family no longer controlled.
History rarely offers such precise symbolism by accident, but Arlington did regardless of whether the Lee family wanted to confront it. Not every Lee child survived to see the war’s end, let alone its aftermath. Daughter Annie died of typhoid fever in 1862, three years before Appomattox, and was buried in North Carolina in a grave her parents did not see for years. She never watched her father surrender.
She never lived through Reconstruction. In a story about what happened after the Confederacy fell, her absence itself is the point: a reminder that loss for this family did not politely wait for the war to officially end before arriving. The two daughters who survived into peacetime carried their own quiet tragedies. Agnes Lee, the third child, suffered from poor health for years, and by the early 1870s that struggle was constant.
She never married. Neither did her sisters Mary and Mildred, a fact contemporaries noticed and whispered about, as if four unmarried daughters from one of the South’s most prominent families required some special explanation. In October 1873, Agnes died in Lexington at age thirty-two. Her mother, exhausted by decades of illness and grief, followed her to the grave just five weeks later.
Mother and daughter died in the same autumn, a detail that turns the Lee family’s postwar story from a single collapse into something closer to a slow, ongoing unraveling that continued claiming family members for years after the last shot at Appomattox. But the story does not stay quiet for long, because the eldest daughter refused to allow it. Mary Custis Lee, the family’s firstborn, took a completely different path from her sisters. While Agnes and Mildred stayed near home and faded from view, Mary spent most of her adult life traveling through Europe, deliberately distancing herself from the rest of the family, absent from funerals and weddings, living almost entirely on her own terms in an era when that was nearly unbelievable for a Southern woman of her standing.
For years, she was like a family ghost, present in name but absent in reality. Then, in June 1902, nearly forty years after her family fled Arlington, Mary Custis Lee did something that made headlines across the country. While riding a streetcar in Alexandria with her Black maid, she sat in the section legally designated for Black passengers under Virginia’s new segregation laws. When the conductor ordered her to move to the white section at the front of the car, she refused.
She was arrested. The daughter of the Confederacy’s most legendary general spent that night explaining her position to police, while crowds of Confederate veterans gathered outside the station, unsure whether to treat the moment as a scandal or a spectacle. To this day, historians still argue about why she did it. Some insist it was a genuine act of quiet defiance against segregation.
Others argue she simply did not know the law existed and was too stubborn to move her belongings. Either way, the daughter of the man the Lost Cause had turned into a marble saint gave the country a moment no one could fully explain then, and no one has fully explained since. While Mary made headlines in Alexandria, her younger sister Mildred was living a life that was almost a mirror image of restless travel, moving across Europe, Egypt, and the American West for years without settling down or marrying, seemingly determined to build a life as far from the wreckage of Arlington as geography would allow. Family letters describe her as the warmest and most cheerful of the four sisters in childhood, the one her father gave a special nickname.
Yet she is also the sister who spent the least amount of her adult life near the family she supposedly adored. She eventually died abroad in France in 1905. Another Lee daughter spent her final decades choosing distance over roots, leaving behind a pattern that by then looked less like coincidence and more like a shared, unspoken family instinct to simply keep moving. Now to the sons, because their struggle with the past did not resemble their sisters’ in any way, and that struggle produced the most significant outcome for the entire family.
George Washington Custis Lee, the eldest son and his father’s heir in spirit if not fully in name, inherited something far heavier than money when Robert E. Lee died in 1870. He inherited his father’s job. Custis became president of what is now Washington and Lee University, a position he held for more than two decades, walking the same campus grounds where his father spent his final years trying to rebuild a shattered region through education rather than weapons.
But Custis carried something else as well: a legal claim to Arlington itself, inherited from his mother upon her death in 1873. Unlike his sisters, he chose to fight for it in the only arena still open to a defeated Confederate family: the American courtroom. The case dragged on for years, working its way through the federal court system while Custis continued teaching, administering, and waiting. It finally reached the United States Supreme Court in 1882, seventeen years after the guns fell silent, and the ruling stunned nearly everyone who thought the matter had been settled the moment Union soldiers buried their first dead on that hill.
The Court found that the federal government’s seizure of Arlington had been improper, that the wartime tax sale used to justify taking the property was void, and that legal ownership of Arlington belonged to Custis Lee and had always belonged to him. For a brief moment in December 1882, the son of Robert E. Lee legally owned Arlington National Cemetery, with its headstones, monuments, and everything on it. It could not last, and everyone knew it.
Thousands of Union dead were already buried in that ground, and no realistic scenario existed in which the Lee family could evict the United States military from its national cemetery. So in 1883, Custis Lee agreed to sell the property to the government for $150,000, a huge sum at the time, and walked away from the case legally victorious and practically defeated in the longest lawsuit connected to the entire Civil War. He never lived in Arlington again. He had simply proven, decades later, that the government had taken it from his family illegally in the first place, and forced them to pay to keep it.
Custis also never married. A fact rarely mentioned alongside his sisters’ unmarried lives, but equally striking. He spent nearly all his remaining decades serving an institution and a name that were not entirely his own, running a university built around his father’s memory and sleeping in rooms a short distance from the chapel where his father was ultimately buried. Where his sisters scattered across continents to escape the weight of the family name, Custis built his entire adult identity around carrying it.
While Custis fought in courtrooms, his younger brother Rooney was fighting to rebuild something closer to home. His path was the most brutal of all the Lee siblings. Captured by Union forces in 1863 while recovering from a battle wound, Rooney spent months as a prisoner in a Northern jail, and the family’s desperate desire to get him back led to one of the most extraordinary offers of the entire war. Custis, still serving in the Confederate army, volunteered to take his brother’s place so Rooney could be released to see his dying wife.
The Union refused the offer outright. Rooney remained imprisoned. Charlotte died of what family letters described as severe anxiety and exhaustion over his captivity, and Rooney could not get home in time to say goodbye. He returned from the war to find his farm, White House, burned to the ground, and both of his young children dead.
Most people would have been forgiven for simply giving up. Instead, Rooney rebuilt the farm from ashes, remarried, started a new family, and slowly worked his way toward public life, first as a state senator in Virginia, and eventually by 1887 as an elected member of the United States Congress, representing the same district his father had once tried to tear away from the nation Rooney now helped govern. He served in office until his death in 1891. He was the only Lee son to hold federal office in the government the Confederacy had fought against.
The youngest son, Robert E. Lee Jr. , took the quietest path of all. He inherited Romancoke, another estate that had belonged to his mother’s family, and spent the rest of his life there as a farmer, far from the spotlight compared to his brothers.
Decades later, in 1904, he published a memoir of his father, drawn from personal memories and letters, offering a rare and intimate window into what life at Arlington was like before the war tore it apart. He died a decade after that book appeared, the last of Robert E. Lee’s sons to pass away, having spent his postwar life working the same land his family had lost everything to hold onto. So what actually happened to Robert E.
Lee’s family after the Confederacy fell? It was not one fate but eight completely different fates, scattering between courtrooms, farms, European cities, a seat in Congress, and at least one police station in Alexandria. A mother who never returned home. A daughter who did not live to see peace.
Two other daughters who chose exile over roots, one of whom unintentionally made headlines forty years after the surrender. A son who beat the United States government in its own Supreme Court and still lost. Another who literally rebuilt from ashes and ended up serving in the very Congress his father had resigned from in 1861. And the youngest son, who simply stayed on the land and wrote everything down before time folded over it.
None of them got Arlington back. None of them got their old lives back either. Instead, they got nearly forty more years of a war that had technically already ended, fought this time through courtrooms, silence, distance, and the slow, painstaking effort of figuring out who they were after the uniform, the plantation, and the world that produced both had disappeared. What makes the Lee family story worth telling is not that they lost a war.
Many families lost that war. It is that their loss did not settle anything for them the way a surrender is supposed to settle a story. There was no clear ending waiting after Appomattox, no return to the world they knew, only decades of scattered, individual attempts to build lives out of what remained. Some tried to win it back through courts.
Some tried to escape it across oceans. Some simply stayed on whatever land was left and quietly held the family together. None of it looked like victory, and none of it looked like the defeat people usually imagine.
It looked like eight different people spending the rest of their lives trying to figure out what to do with a name that had already been carved into history before any of them had the chance to decide what it meant.