Another of the diaries Richards is working from belonged to a young woman, Lutie Kealhulfer, he said.
"I want to show the civilian side, and not just battle, battle, battle," he said.
Richards' son, friends and other relatives appear in the film. More than 100 people were involved on the set Thursday, he said.
During a break in filming, a dozen Confederate re-enactors sat in a row along a stone wall in front of a Prospect Street home. One or two appeared to be napping.
Richards is working with a budget of about $10,000. Actors from across the country are volunteering their time for the project, he said.
Stone came from West Virginia for Thursday's filming. Stone started depicting Lee about 15 years ago, he said from atop a horse made to look like Traveller.
He has worked with Richards before, and this film is a good project, he said.
"I think it's a lot of storytelling," Stone said.
During a scene filmed Thursday morning, Stone rode along Prospect Street before stopping to greet civilians and hold a baby.
Luke Utterback, only 6 months old, is the baby who appears in that scene. Luke and his mother, Amy Utterback, arrived for filming at about 8 a.m. Thursday, she said.
Her father-in-law, Chris Utterback, also appears in the film. He has worked with Richards before and wanted his daughter-in-law and grandson to appear in this film, Utterback said.
Filming moved to City Park in the afternoon.
Two more days of filming are planned at Michaux State Forest in Franklin County, Pa.
Kent Courtney, a musician from Hanover, Pa., depicts several Confederate soldiers in the film and has a speaking part where he leads soldiers in singing. He said he has worked with Historical Entertainment for years.
"I'm in a movie," said 6-year-old Kylee McKenrick, who appears in a scene filmed Thursday wearing a bright blue hoop dress. She is Richards' cousin, she said.
The girl was one of several extras portraying a crowd of women and children gathered beside the street to greet Lee as he rode into town. She stood patiently on the sidewalk between takes.
Mulch was spread on the street, covering asphalt where the crew was filming. After one of the directors yelled, "Action," Confederate re-enactors marched ahead of Stone and two other re-enactors riding horses. The two other mounted re-enactors portrayed Brig. Gen. George E. Pickett and Lt. Gen. James Longstreet.
Gary Hubb, a member of Historical Entertainment's management team, is directing the film.
Richards has wanted to film this project since he read books by local historian Roger Keller in the 1990s, he said.
Keller found the diaries kept by local people and "brought them to life," Richards said.