He'd never heard of the Civil War either, until he reached the age of 8 or 9 and his parents began taking him, his brother, Donald, and his grandmother, Betty Graiffius, on Sunday afternoon trips to Antietam National Battlefield near Sharpsburg.
"My grandmother used to get me interested. She'd say, 'Dickie,' ? she always called me 'Dickie' ? over there, that's the old Dunker Church ...'
"'Course back then, it was just an old pile of brick. It blew down in 1921. I didn't even know what the
Civil War was, but I could see it in her eyes and it just kind of stuck with me over the years," Clem said.
The story of Isaac Avery
Clem was a young man, working as a cabinetmaker, when his budding interest in Civil War history took the form of relic hunting and, eventually, of writing articles about it. As the subject became his passion, the story of Isaac Avery grew in his mind.
Avery, in his early 30s when the Civil War began in 1861, left one of his father's plantations in North Carolina to help form a unit of soldiers, Company E, 6th North Carolina Infantry, for the Confederacy.
Appointed captain, Avery and his regiment were sent to defend Richmond, Va., in 1862. The Union was driven back, but Avery was wounded and still was recovering when his men suffered heavy losses at
Antietam that September, Clem wrote in an article published this past spring by The Washington Times.
The following year, Avery ? by now, a colonel ? led a brigade in the South's decisive victory at
Chancellorsville, Va. Among the Rebel casualties was Gen. Thomas J. "Stonewall" Jackson.
Within weeks, Gen. Robert E. Lee was leading the Rebels on to Gettysburg, Pa. In the resulting historic clash between the Northern and Southern armies, Col. Avery's unit was ordered on July 2 to attack a heavily fortified Union position on East Cemetery Hill, Clem wrote.
On horseback, leading his men, Col. Avery was mortally wounded and, after the fighting, he lay dying when a close friend, Maj. Samuel McDowell, reached his side. With his friend's help, Avery, now unable to speak, was given a piece of paper.
Using a stick or some other sharp object, Avery dipped its point into his own blood and wrote: "Major, tell my father I died with my face to the enemy."
Avery died early the next day, July 3.
The document written in his blood is preserved in historical archives in Raleigh, N.C.
Avery's father lost two other sons in the Civil War, and both of their bodies were returned to North Carolina.
But until this year, the family never knew what had become of Isaac Avery's remains.
They only knew that after the fighting at Gettysburg, Avery's slave had buried his body on land overlooking the Potomac River at Williamsport as the Rebel troops made their long march back home.
Members of Avery's family made repeated trips right after the war and as recently as the 1960s to Williamsport, trying to find his grave site.
Unraveling the mystery
What they didn't know, Clem said this past week, was that a few years after the war, Maryland Gov. Oden Bowie had appropriated $5,000 to find and rebury the thousands of Confederate soldiers buried in shallow graves near Sharpsburg, Williamsport and other areas of Washington County.
Aware of the anger in many Northern states at even the thought of burying the Rebels in the national cemetery at Antietam, Gov. Bowie bought three acres for what became known as Washington
Confederate Cemetery inside Hagerstown's Rose Hill Cemetery, Clem said.
Col. Avery's were among the remains that were moved, but no one realized that until Clem put the evidence together.
"What really kept me fascinated with Avery was the story of his death," said Clem, who lives off Wagaman Road south of Hagerstown.
"Seems like it's always been in the back of my mind. And I knew that a good many of those Averys had kept coming up and looking for him."
Clem began by obtaining Bowie's list of the 346 identified Confederate bodies that had been reinterred.
A total of 2,122 unidentified Confederate soldiers also are buried at Rose Hill.
On the list, Clem found a notation, "Buried in the public graveyard at Williamsport," and with it, "Col. J.E. Ayer, 6th N.C.S.T., July 3, 1863."