Wagaman said Carson does not have a permit for the sign.
The penalty is a $100 fine, according to court records. Carson's trial date is set for April 28.
The Carsons had until mid-February to remove the sign from in front of the 104 S. Mechanic St. establishment, Wagaman said.
The well-lighted sign had bothered some neighbors, but Paul Carson said he turned the lights off last month. He said the sign was in place for a year before anyone complained.
Carson's request to put a sign at Town Square was denied, and he relies on the sign in front of the restaurant to attract patrons, he said.
Many people think the restaurant building is a library or courthouse, he said. The building housed American Legion Post 236 until late 1991 when the legion moved to 16609 Shepherdstown Road.
The sign has helped him attract tourists visiting nearby Antietam National Battlefield, Carson said.
He said Wagaman had told him at various times to keep the sign, to take it down or to remove the letters from the sign.
"I don't know what the man wants. I tried to work with him," Carson said.
Carson said he applied for a permit, but it wasn't approved.
Carson did not attend the December meeting of the Board of Zoning Appeals, so the permit application was denied, Wagaman said during February's Town Council meeting.
Carson said he will let the judge decide what to do.