On a warm spring afternoon in 1861, a woman stood at the entrance of the finest house in Virginia, looked over its furniture, its portraits, her husband’s letters, and her father’s silver, and made a decision that would shape her family’s life for the next sixty years. She took only what she could carry. She believed she would return within weeks. She never set foot in that house again.
Her husband was Robert E. Lee. The house was Arlington. And what happened to his seven children and the wife he left behind did not end with the Confederate surrender at Appomattox.
Their story, in many ways, began at that exact moment. One daughter did not live to see the war’s end. One son spent seventeen years fighting the United States government in the courts, not with a rifle this time, but with a lawsuit that reached the Supreme Court. Another daughter vanished into Europe for decades, and when she finally came home, she was arrested.
By the end of this story, it becomes clear why historians say the Lee family’s real war did not end in 1865. It only changed its shape. The mother’s story sets the tone for everything that followed. Mary Anna Custis Lee was already confined by rheumatoid arthritis when Union soldiers crossed the Potomac and occupied her family estate.
She left Arlington believing the separation was temporary, just as most people in April 1861 believed everything about the coming war was temporary. Within three years, the federal government turned her front yard into a cemetery for Union war dead, a deliberate punishment aimed directly at her husband, so that no Confederate general could ever look at his home again without seeing rows of men his army had killed. For the rest of the war, Mary Lee lived as a refugee, moving between relatives’ homes across Virginia, confined to a wheelchair, watching her husband’s reputation grow into legend while her health quietly deteriorated. When Richmond finally fell and the Confederacy dissolved, she did not get her house back.
She got silence, distance, and a husband who spent his final years as a college president in Lexington instead of a plantation owner in Arlington, because Arlington no longer belonged to either of them in any way the law recognized. Robert E. Lee himself lived only five years after the surrender, dying in 1870 without ever recovering the property he had spent thirty years restoring. That fact alone would have ended most family stories.
For the Lees, it was only the beginning of the second chapter. There is an irony buried in the grounds of Arlington that most people do not know, one that makes the family’s loss even harder to accept. While the front garden filled with graves, the federal government also set aside a portion of the estate for a completely different purpose: a settlement for formerly enslaved people who had fled bondage during the war. For a time, the very hill where Mary Custis Lee had once presided over one of the largest enslaved labor forces in Virginia became home to hundreds of newly free men, women, and children building their own lives on land the Lee family no longer controlled.
History rarely offers such precise symbolism without intent, but Arlington offered it anyway, whether the Lee family wanted to face it or not. Not all of Lee’s children survived to see the end of the war, let alone its aftermath. His daughter Annie died of typhoid fever in 1862, three years before Appomattox, and was buried in North Carolina in a grave her parents did not visit for years. She never saw her father surrender.
She never lived through Reconstruction. In a story about what happened after the fall of the Confederacy, her absence is itself the point, a reminder that loss for this family did not politely wait for the war to officially end before arriving. The two daughters who survived into peacetime carried their own quiet tragedies. Agnes Lee, the third daughter, had suffered from poor health for years, and by the early 1870s that struggle had become constant.
She never married. Neither did her sisters Mary and Mildred, a fact contemporaries noticed and whispered about, as if four daughters of one of the South’s most prominent families choosing not to marry required a special explanation. In October 1873, Agnes died in Lexington at the age of thirty-two. Her mother, worn down by decades of illness and grief, followed her to the grave just five weeks later.
Mother and daughter died in the same autumn. That detail transforms the Lee family’s postwar story from a single collapse into something more like a slow, continuous unraveling that kept claiming family members for years after the last shot was fired at Appomattox. The story does not stay quiet for long, because the eldest daughter refused to allow it. Mary Custis Lee, the family’s firstborn, took a completely different path from her sisters.
While Agnes and Mildred stayed near home and faded from view, Mary spent most of her adult life traveling across Europe, deliberately distancing herself from the rest of the family, absent from funerals and weddings, living almost entirely on her own terms in an era when that was nearly unbelievable for a Southern woman of her standing. For years, she was to most people something like the family ghost, present in name but absent in fact. Then in June 1902, nearly forty years after her family had fled Arlington, Mary Custis Lee did something that made headlines across the country. While riding a streetcar in Alexandria with her Black maid, she sat in the section legally designated for Black passengers under Virginia’s new segregation laws.
When the conductor ordered her to move to the white section at the front of the car, she refused. She was arrested. The daughter of the Confederacy’s most legendary general spent that night explaining herself to the police, while crowds of Confederate veterans gathered outside the station, unsure whether to treat the moment as a scandal or a spectacle. To this day, historians still argue about why she did it.
Some insist it was a genuine act of quiet defiance against segregation. Others argue she simply did not know the law existed and was too stubborn to move her belongings. Either way, the daughter of the man the Lost Cause had turned into a marble saint gave the country a moment no one could fully explain then, and no one has fully explained since. While Mary was making headlines in Alexandria, her younger sister Mildred was living a life that was almost a mirror image of restless travel, moving across Europe, Egypt, and the American West for years without settling down or marrying, seemingly determined to build a life as far from the wreckage of Arlington as geography would allow.
Family letters describe her as the most affectionate and cheerful of the four sisters in childhood, the one her father had given a special nickname. Yet she is also the sister who spent the least time in her adult life near the family she supposedly adored. She eventually died abroad in France in 1905. Another Lee daughter spent her final decades preferring distance over roots, leaving behind a pattern that by then looked less like coincidence and more like a shared, unspoken family instinct to simply keep moving.
The sons’ struggles with the past did not resemble their sisters’ in any way, and they produced the most significant outcome for the entire family. George Washington Custis Lee, the eldest son and his father’s heir in spirit if not entirely in name, inherited something far heavier than money when Robert E. Lee died in 1870. He inherited his father’s job.
Custis became president of what is now Washington and Lee University, a position he held for more than two decades, walking the same campus grounds where his father had spent his final years trying to rebuild a broken region through education rather than arms. But Custis also carried something else: a legal claim to Arlington itself, inherited from his mother upon her death in 1873. Unlike his sisters, he chose to fight for it in the only field still open to a defeated Confederate family, the American courtroom. The case dragged on for years, working its way through the federal judicial system while Custis continued to teach, administer, and wait.
It finally reached the United States Supreme Court in 1882, seventeen years after the guns had fallen silent, and the ruling stunned nearly everyone who had assumed the matter was settled the moment Union soldiers buried their first dead on that hill. The Court found that the federal government’s seizure of Arlington had been improper, that the wartime tax sale used to justify taking the property was void, and that legal ownership of Arlington belonged to Custis Lee and had always belonged to him. For one brief moment in December 1882, Robert E. Lee’s son legally owned Arlington National Cemetery, headstones, monuments and all.
It could not last, and everyone knew it. Thousands of Union dead were already buried in that ground, and there was no realistic scenario in which the Lee family could evict the United States Army from its national cemetery. So in 1883, Custis Lee agreed to sell the property back to the government for $150,000, a huge sum at the time, and walked away from the longest lawsuit connected to the entire Civil War legally victorious and practically defeated. He never lived at Arlington again.
He had simply proven, decades after the fact, that the government had taken it from his family illegally in the first place, and forced them to pay for keeping it. Custis never married either, a fact rarely mentioned alongside his sisters’ unmarried lives, but equally striking. He spent nearly all of his remaining decades serving an institution and a name that were never fully his own, running a university built around his father’s memory, sleeping in rooms a short distance from the chapel where his father was eventually buried. Where his sisters scattered across continents to escape the weight of the family name, Custis built his entire adult identity around carrying it.
While Custis fought in the courtrooms, his younger brother Rooney was fighting to rebuild something closer to home, and his path was the most brutal of all the Lee siblings. Captured by Union forces in 1863 while recovering from a battle wound, Rooney spent months as a prisoner in a Northern jail, and the family’s desperate desire to get him back produced one of the most extraordinary offers of the entire war. Custis, still serving in the Confederate army, volunteered to take his brother’s place so Rooney could be released to see his dying wife. The Union refused the offer entirely.
Rooney remained imprisoned. Charlotte died of what family letters described as severe anxiety and exhaustion caused by his captivity, and Rooney could not get home in time to say goodbye. He returned from the war to find his plantation, White House, burned to the ground, and his two young children dead. Most people would have understood if he simply gave up.
Instead, Rooney rebuilt the plantation from the ashes, remarried, started a new family, and slowly made his way into public life, first as a Virginia state senator, and eventually, by 1887, as an elected member of the United States Congress, representing the same district his father had once tried to tear away from the nation Rooney now helped govern. He served until his death in 1891. He was the only son of Lee to hold federal office in the government the Confederacy had fought against. The youngest son, Robert E.
Lee Jr. , chose the quietest path of all. He inherited Romancoke, another property that had belonged to his mother’s family, and spent the rest of his life there as a farmer, far from the spotlight compared to his brothers. Decades later, in 1904, he published memoirs of his father, drawn from personal memories and letters, offering a rare and intimate window into what life at Arlington had been like before the war tore it apart.
He died a decade after that book appeared, the last of Robert E. Lee’s sons to pass away, having spent his postwar life working the same land his family had lost everything to hold onto. So what really happened to Robert E. Lee’s family after the fall of the Confederacy?
It was not one fate but eight completely different fates, scattered between courtrooms, farms, European cities, a seat in Congress, and at least one police station in Alexandria. A mother who never returned home. A daughter who did not live to see peace. Two more daughters who chose exile over roots, one of whom unintentionally made national headlines forty years after the surrender.
A son who defeated the United States government in its own Supreme Court and still lost. Another who rebuilt from the ashes, literally, and ended up serving in the very Congress his father had resigned from in 1861. And the youngest son, who simply stayed on the land and wrote it all down before time erased him. None of them got Arlington back.
None of them got their old lives back either. Instead, they got nearly forty more years of a war that was technically already over. They fought it this time through courtrooms, through silence, through distance, and through the slow, painstaking effort of figuring out who they were after the uniform, the plantation, and the world that produced both had disappeared. What makes the Lee family story worth telling is not that they lost a war.
Many families lost that war. It is that their loss settled nothing for them in the way a surrender is supposed to settle a story. There was no clear ending waiting for them after Appomattox, no return to the world they had known, only decades of scattered individual attempts to build lives out of what remained. Some tried to win it back through the courts.
Some tried to escape it across the oceans. Some simply stayed on whatever land was left and quietly held the family together. None of it looked like victory, and none of it looked like the defeat people usually imagine.
It looked like eight different people spending the rest of their lives trying to figure out what to do with a name that had already been carved into history before any of them had the chance to decide what it meant.