On a warm spring afternoon in 1861, a woman stood at the entrance of the finest house in Virginia, looked over its furniture, portraits, her husband’s letters, her father’s silver, and made a decision that shaped her family’s life for the next sixty years. She took only what she could carry. She expected to return within weeks. She never set foot in that house again.
Her husband was Robert E. Lee. The house was Arlington. And what happened to his seven children and the wife he left behind did not end with the Confederate surrender at Appomattox.
In many ways, their story only began at that moment. One daughter would not live to see the war end. One son would spend seventeen years fighting the United States government in court, not with a rifle but with a lawsuit that reached the Supreme Court. Another daughter vanished into Europe for decades, and when she came home, she was arrested.
By the end of this story, it becomes clear why historians say the Lees’ real war did not end in 1865. It only changed form. The mother’s story sets the tone for everything that followed. Mary Anna Custis Lee was already crippled by rheumatoid arthritis when Union soldiers crossed the Potomac and occupied her family estate.
She left Arlington believing the separation was temporary, as most people in April 1861 believed everything about the coming war would be temporary. Within three years, the federal government had turned her front lawn into a cemetery for Union war dead, a deliberate punishment aimed directly at her husband, so that no Confederate general could look at his own home again without seeing rows of men his army had killed. For the rest of the war, Mary Lee lived as a refugee, moving between relatives’ homes across Virginia, confined to a wheelchair, watching her husband’s reputation grow into legend while her health quietly deteriorated. When Richmond finally fell and the Confederacy collapsed, she did not get her house back.
Instead she got silence, distance, and a husband who spent his final years as a college president in Lexington rather than as a plantation owner at Arlington, because Arlington no longer belonged to either of them in any way the law recognized. Robert E. Lee himself lived only five years after the surrender. He died in 1870 without ever recovering the estate he had spent thirty years restoring.
That fact alone would have ended most family stories. For the Lees, it was only the beginning of the second chapter. There is an irony buried in the grounds of Arlington that most people never learn, and it made the family’s loss even harder to accept. While the front garden filled with graves, the federal government also set aside part of the estate for a completely different purpose: a settlement for formerly enslaved people who had fled bondage during the war.
For a time, the very hill where Mary Custis Lee once presided over one of the largest enslaved workforces in Virginia became home to hundreds of newly free men, women, and children building their own lives on land the Lees no longer controlled. History rarely offers such precise symbolism without intention, but Arlington offered it anyway, whether the Lee family wanted to face it or not. Not all of Lee’s children survived to see the end of the war, let alone its aftermath. His daughter Annie died of typhoid fever in 1862, three years before Appomattox, and was buried in North Carolina in a grave her parents did not see for years.
She never watched her father surrender. She never lived through Reconstruction. In a story about what happened after the Confederacy fell, her absence itself is the point: a reminder that loss for this family did not politely wait for the war to officially end before arriving. The two daughters who survived into peacetime carried their own quiet tragedies.
Agnes Lee, the third daughter, suffered from poor health for years, and by the early 1870s that struggle was constant. She never married. Neither did her sisters Mary and Mildred, a fact contemporaries noticed and whispered about, as though four daughters of one of the South’s most prominent families choosing not to marry required special explanation. In October 1873, Agnes died in Lexington at age thirty-two.
Her mother, worn down by decades of illness and grief, followed her to the grave only five weeks later. Mother and daughter died the same fall, a detail that turns the Lee family’s postwar story from a single collapse into something more like a slow, continuing unraveling that kept claiming family members for years after the last shot at Appomattox. The eldest daughter refused to let the story stay quiet. Mary Custis Lee took a completely different path from her sisters.
While Agnes and Mildred stayed close to home and faded from view, Mary spent most of her adult life traveling through Europe, deliberately distancing herself from the rest of the family, absent from funerals and weddings, living almost entirely on her own terms in an era when that was nearly unthinkable for a Southern woman of her standing. For years she was, to most people, something like the family ghost: present by name, absent in fact. Then, in June 1902, nearly forty years after her family fled Arlington, Mary Custis Lee did something that made headlines across the country. While riding a streetcar in Alexandria with her Black maid, she sat in the section legally reserved for Black passengers under Virginia’s new segregation laws.
When the conductor ordered her to move to the whites-only section at the front of the car, she refused. She was arrested. The daughter of the Confederacy’s most legendary general spent that night explaining herself to police while crowds of Confederate veterans gathered outside the station, unsure whether to treat the moment as a scandal or a spectacle. Historians still argue about why she did it.
Some insist it was a genuine act of quiet defiance against segregation. Others argue she simply did not know the law existed and was too stubborn to move her luggage. Either way, the daughter of the man the Lost Cause had turned into a marble saint gave the country a moment no one could fully explain then, and no one has fully explained since. While Mary made headlines in Alexandria, her younger sister Mildred lived a life that was almost a mirror image of relentless travel, moving across Europe, Egypt, and the American West for years without settling down or marrying, as though determined to build a life as far from the wreckage of Arlington as geography would allow.
Family letters describe her as the most affectionate and cheerful of the four sisters as a child, the one her father gave a special nickname. Yet she is also the sister who spent the least time as an adult near the family she supposedly adored. She eventually died abroad in France in 1905. Another Lee daughter spent her final decades choosing distance over roots, leaving behind a pattern that by then looked less like coincidence and more like a shared, unspoken family instinct simply to keep moving.
The sons’ struggles with the past looked nothing like their sisters’. George Washington Custis Lee, the eldest son and heir to his father’s spirit if not his full name, inherited something far heavier than money when Robert E. Lee died in 1870. He inherited his father’s job.
Custis became president of what is now Washington and Lee University, a position he held for more than two decades, walking the same campus where his father spent his final years trying to rebuild a broken region through education rather than arms. But Custis also carried something else: a legal claim to Arlington itself, inherited from his mother when she died in 1873. Unlike his sisters, he chose to fight for it in the only field still open to a defeated Confederate family: the American courtroom. The case stretched for years, winding through the federal court system while Custis continued teaching, administering, and waiting.
It finally reached the United States Supreme Court in 1882, seventeen years after the guns fell silent, and the decision stunned nearly everyone who had assumed the matter had been settled the moment Union soldiers buried their first dead on that hill. The Court found that the federal government’s seizure of Arlington had been improper, that the wartime tax sale used to justify the taking was void, and that legal title to Arlington belonged to Custis Lee and had always belonged to him. For a brief moment in December 1882, Robert E. Lee’s son legally owned Arlington National Cemetery, headstones, monuments, and all.
That could not stand, and everyone knew it. Thousands of Union dead were already buried in that ground, and no realistic scenario existed in which the Lee family could evict the United States Army from its national cemetery. So in 1883, Custis Lee agreed to sell the estate to the government for $150,000, a huge sum at the time, and walked out of the case legally victorious and practically defeated, ending the longest lawsuit connected to the entire Civil War. He never lived at Arlington again.
He had simply proven, decades after the fact, that the government had taken his family’s property illegally in the first place, and forced them to pay for keeping it. Custis also never married, a fact rarely mentioned alongside his sisters’ unmarried lives but striking all the same. He spent nearly all of his remaining decades serving an institution and a name that were never fully his own, running a university built around his father’s memory, sleeping in rooms a short distance from the chapel where his father was eventually buried. Where his sisters scattered across continents to escape the weight of the family name, Custis built his entire adult identity around carrying it.
While Custis fought in the courthouses, his younger brother Rooney fought to rebuild something closer to home. His path was the harshest of all the Lee siblings. Captured by Union forces in 1863 while recovering from a battle wound, Rooney spent months as a prisoner in a Northern prison, and the family’s desperate desire to get him back produced one of the most extraordinary offers of the entire war. Custis, still serving in the Confederate army, volunteered to take his brother’s place so Rooney could be released to visit his dying wife.
The Union refused entirely. Rooney remained imprisoned. Charlotte died of what family letters described as extreme anxiety and grief over his captivity, and Rooney could not get home in time to say goodbye. He returned from the war to find his farm, White House, burned to the ground and his two small children dead.
Most people would have been forgiven for simply giving up. Instead, Rooney rebuilt the farm from the ashes, remarried, started a new family, and slowly made his way into public life, first as a state senator in Virginia and eventually, by 1887, as an elected member of the United States Congress, representing the same district his father had once tried to tear away from the nation Rooney now helped govern. He served until his death in 1891. He was the only Lee son to hold federal office in the government the Confederacy had fought.
The youngest son, Robert E. Lee Jr. , chose the quietest path of all. He inherited Romancoke, another property that had belonged to his mother’s family, and spent the rest of his life there as a farmer, out of the spotlight compared to his brothers.
Decades later, in 1904, he published a memoir of his father, drawn from personal memories and letters, offering a rare and intimate window into life at Arlington before the war tore it apart. He died a decade after that book appeared, the last of Robert E. Lee’s sons to go, having spent his postwar life working the same land his family had lost everything to hold. So what really happened to Robert E.
Lee’s family after the Confederacy fell? Not one fate, but eight entirely different fates, scattering among courtrooms, farms, European cities, a seat in Congress, and at least one police station in Alexandria. A mother who never returned home. A daughter who did not live to see peace.
Two other daughters who chose exile over roots, one of whom made unintended headlines forty years after the surrender. A son who beat the United States government in its own Supreme Court and still lost. Another who rebuilt from literal ashes and ended up serving in the very Congress his father had resigned from in 1861. And the youngest son who simply stayed on the land and wrote it all down before time erased him.
None of them got Arlington back. None of them got their old lives back either. Instead, they got nearly forty more years of a war that had technically already ended. This time they fought it through courtrooms, through silence, through distance, and through the slow, painful effort of figuring out who they were after the uniform, the plantation, and the world that had produced both disappeared.
What makes the Lee family story worth telling is not that they lost a war. Many families lost that war. It is that their loss settled nothing for them in the way a surrender is supposed to settle a story. There was no clear ending waiting for them after Appomattox, no return to the world they had known, only decades of scattered, individual attempts to build lives out of what remained.
Some tried to win it back through the courts. Some tried to escape it across the oceans. Some simply stayed on whatever land was left and quietly held the family together. None of it looked like victory, and none of it looked like the defeat people usually imagine.
It looked like eight different people spending the rest of their lives trying to decide what to do with a name that had already been carved into history before any of them ever had a chance to decide what it meant.