They remember Andrew Johnson as the stubborn man who followed Abraham Lincoln into the presidency and survived impeachment by a single vote. But far from the Senate chamber, there was another arena where no parliamentary rules protected him: his own home in Greeneville, Tennessee. There, the man who rose from illiterate tailor’s apprentice to president tried to be something history never trained him for and never forgave him for failing at—a steady husband and father. Johnson married Eliza McCardle when he was a poor young tailor, barely educated and scarred by poverty.
She was quiet, literate, and determined. Legend says she helped teach him to read and write more fluently, sitting with him over books after long days at the workbench. Together they built a hard life. Long hours in the shop, debts, and political fights in a town that did not always welcome Johnson’s relentless ambition.
In the middle of that grinding climb, five children arrived in quick succession: Martha, Charles, Mary, Robert, and much later, Andrew Jr. , called Frank. As Johnson climbed from local office to Congress, governor, senator, and finally vice president, his children watched their world stretch from Greeneville’s dusty streets to Washington salons and back again. The Civil War tore their universe open.
Johnson’s decision to stay loyal to the Union while remaining a slaveholder put his family in a dangerous middle ground, despised by Confederates and distrusted by many Unionists. When Lincoln was assassinated in 1865 and Johnson suddenly became president, the family moved into a White House still draped in grief, a place that had never been meant for them. Frank, the youngest, was born in 1852, nearly a decade after his older siblings, at the moment when his father was rising from local politician to national figure. Unlike his brothers and sisters who remembered the cramped tailor shop and early years of poverty, Frank was born into a home already tasting national prominence.
By the time he began forming memories, his father was serving as military governor of Tennessee. The Johnson name had become a target—admired by Unionists, despised by Confederates. For a sensitive, bookish child, it was a strange burden. The turning point came when his father became vice president, then president within weeks.
Still only thirteen, Frank suddenly lived in the White House, a place vibrating with grief, suspicion, and political fury. His older siblings took on ceremonial or support roles, but Frank remained the child in the background, the quiet observer in hallways where cabinet secretaries argued and whispers of impeachment began long before formal charges were filed. Andrew Johnson wanted at least one child to rise above obscurity, to claim a place in business or politics that might soften the stain of his tumultuous presidency. Frank tried to answer that hope.
After leaving the White House, he pursued business in Tennessee and later Washington, his father granting him responsibility for family estates. But Frank struggled with the quiet discipline required for financial life—intelligent but inconsistent, earnest but easily discouraged. In 1879, at just 26 years old, Frank died of tuberculosis. Andrew Johnson had pinned his hopes on his youngest son, believing he might redeem the family legacy after the losses of the older boys.
Instead, Frank’s death left the Johnson line without the successor Andrew imagined and without the stability the family desperately needed. Robert Johnson, born in 1843, carried the quietest footprint in public memory. He grew up in the cramped rooms above the tailor shop, raised in poverty in a household taut with Andrew’s pride and temper but also Eliza’s steadiness. Where his older siblings inherited their father’s force of will, Robert inherited his mother’s gentleness.
He was the calmest of the children, the least drawn to confrontation. When Tennessee split between Union and Confederate sympathies and Andrew Johnson chose to remain loyal to the Union, Robert joined his brother Charles in military service, fighting for the Union at a time when their home state branded such loyalty as betrayal. His military career was not marked by fame or scandal, but by steady, dutiful service. After the war, he returned to Tennessee and tried to manage the family’s business affairs, but his health, always fragile, deteriorated.
The strain of wartime service, compounded by the stress of living under his father’s political shadow, left him weakened. Tuberculosis took hold, the same disease that would later claim his youngest brother. Robert died in 1869 at just 26 years old, only a few months after his father left the presidency. Mary Bell Johnson, born in 1850, was the quieter of the two daughters.
While her sister Martha became Andrew Johnson’s public-facing hostess, Mary remained deeply embedded in the private emotional world of the family. Eliza, frail in health but unyielding in spirit, leaned heavily on Mary as she grew. During the Civil War, as Confederate sympathizers grew hostile, Mary endured threats, property damage, and the isolation that came from being aligned with a father despised by many in their region. While her brothers went off to fight, Mary became part of the domestic defense, protecting the home and supporting her mother.
When the family moved to Washington, she preferred to remain out of the spotlight, managing personal affairs and helping stabilize the household during one of the most turbulent presidencies in American history. She tended to her mother’s declining health, mourned the deaths of her brothers, and watched Frank fight and lose his battle with tuberculosis. Mary never married. Her adult life remained tied to Greeneville, where she lived with her parents as both caretaker and companion.
Charles Johnson, born in 1830, was the eldest son, the child who most directly mirrored the turbulence and fatal costs of his father’s political world. He inherited Andrew’s intensity, his desire to prove himself, and his instinct for confrontation. But unlike his father, Charles never found a stable channel for that drive. When Tennessee splintered and Andrew Johnson declared loyalty to the Union, Charles enlisted in the Union Army, serving as assistant surgeon with the 10th Tennessee Infantry.
His decision was both political and deeply personal—the Johnson men would not be seen as cowards. But military service exposed him to the brutal reality of violence and disillusionment. Those who knew him said the conflict hardened him further, darkening the edges of a temperament already prone to volatility. Alcohol became both companion and accelerant.
When his father became president, Charles carried into Washington the rawness of a man used to battlefields and taverns, frequently clashing with his father over his drinking. In April 1869, two months after Andrew Johnson left the presidency, Charles died from injuries sustained in a drunken fall from a horse. He was 39. Martha Johnson Patterson, born in 1832, was the eldest child, the daughter who became Andrew Johnson’s public shield and emotional anchor.
Of all the children, she absorbed her mother’s strength most fully—steadiness, silence used as strategy, an instinct for how to manage a difficult man without provoking or enabling him. She married David T. Patterson, a Tennessee judge, in 1855. Then came April 1865.
Eliza, stricken by chronic illness, could not serve as White House hostess. Martha stepped into the position instantly, without ceremony or training. She brought her dairy cow to the White House lawn to ensure her children had fresh milk. She repaired tattered furnishings, reorganized staff, and created a domestic environment that quietly communicated stability.
During the impeachment crisis of 1868, Martha stood closest to her father. When Andrew Johnson survived removal from office by a single vote, White House staff recalled Martha weeping in relief—not for political victory, but because her father would not be remembered as the first president cast out in disgrace. After the presidency ended, Martha returned to Greeneville and continued managing the household, especially after her brothers’ deaths and her mother’s decline. When Andrew Johnson died in 1875, Martha oversaw his funeral and preserved his legacy with fierce loyalty.
The name Andrew Johnson is usually invoked as a cautionary tale—the president who clashed with Congress and resisted Reconstruction. But when the story shifts from the Senate chamber to the Johnson household, a different narrative emerges. Each of his five children carried a piece of the burden he could not resolve. Each bore the shadow of a man whose strengths and weaknesses were carved into the foundations of their lives.
The consequences of political leadership ripple far beyond the nation, reaching into the illnesses and heartbreaks of the children who never chose the path that history forced upon them.