Then he bought another one - the Hagerstown Suns in a sale made public last Monday.
"Two are better than one," Rayburn said from Ohio during a telephone interview last week.
Rayburn's company, Chagrin Falls, Ohio-based Big Game Capital, may even buy another team, but he's not in a hurry.
While Rayburn loves sports, he has a particular passion for baseball.
"I really think a baseball team is the most important team to a city," said Rayburn, citing summer nights, kids out of school and the high volume of games during a season.
As a kid growing up in a Cleveland suburb, Rayburn and his friends would pay a quarter to ride the rapid transit train to Municipal Stadium, buy a bleacher ticket for a dollar and watch the Cleveland Indians play for the day. They had no adult supervision.
When he wasn't watching the pros, he was playing ball in a field across from his home from morning until nightfall. He would stop long enough to eat dinner, but wouldn't miss his next at-bat.
Rayburn said he thought about buying a team for three to four years, but didn't get serious about it until he sold Flexalloy, the fastener distributing firm his father started in 1967.
After graduating from Dartmouth College with a bachelor's degree in French, Rayburn went to California and worked various jobs. He returned to Ohio at the age of 24 to take a job in Flexalloy's warehouse.
He swept trash in the warehouse, as he did when he was a youth, but progressed quickly from packing to inventory control to purchasing and sales.
"I was basically training to take it over," Rayburn said.
He did, in 1981 when his father, James Rayburn, died suddenly shortly after being diagnosed with leukemia.
"I was 26, taking it over considerably earlier than I planned to," he said.
He also was taking over a company during a difficult economic climate as the recession was just hitting the country.
His father had left a financially strong company, with $8 million in annual sales and operations in Cleveland and Indianapolis. He also left the company with a considerable cash balance, which generated more profit through investment than the company did for three years, Rayburn said.
In 1984 as Rayburn saw the company's main market, the heavy-duty truck market, make a return, he expanded to three shifts seven days a week to boost production of a special nut used to hold dual truck wheels.
The move paid off when the heavy-duty truck market - including Mack Trucks in Hagerstown - exploded and Flexalloy was the only fastener distributor with a huge inventory of those nuts, Rayburn said.
Sales grew 60 percent in 1984 and averaged more than 25 percent through 1999, when Rayburn sold the company to Textron Inc. That year, Flexalloy had $230 million in revenue before expenses.
Rayburn wouldn't disclose details of the sale or his net worth, but said he is a millionaire and co-owns a Lear jet.
Turning to baseball
Rayburn stayed on with Textron for a year after the sale before leaving the business and consummating the business end of his love affair with baseball.
Selling Flexalloy was about timing.
The offer was good, and Rayburn wanted to spend more time with his three children - 6-year-old twins John and Annabelle and daughter Iris, 8.
"They bat right and throw right and they love baseball," Rayburn said.
A studious businessman, Rayburn talked to approximately 10 minor league owners, learning the business and checking out promotional ideas.
Rayburn was specifically interested in Class A baseball, the first rung on the ladder to the Major Leagues.
"Everything is more affordable, from the tickets to the hot dogs to the teams themselves," Rayburn said.
He also likes the idea of watching young guys playing against the odds for a shot at the big leagues.
"There's a purity there we can really enjoy," he said.
"The big rush for me is to sit back in the stands, in the bleachers, and just watch all these kids running around having a good time with their parents knowing everything is safe and clean and it doesn't cost a lot of money," Rayburn said.
On June 19, 2000, Rayburn announced he had bought the Daytona Cubs, an upper level Class A team in the Florida State League.