Computer programs interpret languages like HTML, XML, Java, PHP with no problem. But ancient handwritten text? BYU professor Bill Barrett and his team of undergraduate and graduate students are determined to create a program that will do just that—and to a large degree, they already have.
The team is composed of Seth Stewart, Curtis Wigington, Lucas Pinto, and Brian Davis. They entered their software into a competition sponsored by the International Conference on Frontiers in Handwriting Recognition (ICFHR). The team was tasked to create a program that would recognize fourteenth- to eighteenth-century German handwriting.
“I’m an old German missionary, so I read and write and speak German. But I am hard-pressed to even read 50 percent of [this text],” Barrett said.
The computer, however, can read the old handwriting with a 95 percent accuracy, which Barrett says is actually better than a human. But more than only being accurate, the program is also fast. For example, it took only five to ten minutes to transcribe a document of about fifteen thousand words, including names —that’s well over a thousand words per minute.
To explain the program’s potential significance, Barrett refers back to 2012 when FamilySearch volunteers helped index the 1940 census, and how their technology might impact the indexing of the 1950 census.
“There was a worldwide fuss to get the 1940 [census] indexed, which the indexing volunteers did in record time,” Barrett said. “But record time was months. So you say, OK, well, in just a few years the 1950 census will become available. Won’t it be nice to just blast through that?”
Though it may sound simple—just press a button or two, and the computer will read all the handwriting—the process of preparing the program for the competition wasn’t itself simple. However, the team of students working with professor Barrett did put it together rather quickly.
Computer science graduate student Seth Stewart signed up the team for the competition in April 2016; the program’s final results were added to their submission in June. Barrett said this quick turnaround was amazing, considering their opponents and other researchers had been working on similar systems for years.
The team worked quickly and creatively. They borrowed a neural network from colleagues that recognized text “in the wild” (like text on store and street signs, essentially in the wild). Then, the team collectively worked to apply the network to their handwriting-recognition program.
For example, Stewart formed more words in the handwriting style by combining different characters of the handwriting sample given to the team. Another student, Curtis Wigington, geometrically distorted the handwriting to account for instances when letters and words didn’t look exactly the same, since humans don’t tend to write each letter exactly the same way every time.
“These historical figures are gone; you aren’t going to get anymore handwriting from them. So what are you going to do? Well, manufacture it,” Barrett said.
The team used transcriptions provided by an expert to train the network and to measure its performance.
The team is working with FamilySearch and sees potential for their program to improve indexing and making the formidable task of transcribing millions of microfilmed documents “approachable.” Though it will still be a while before the technology could be assimilated into FamilySearch, Barrett is optimistic about the program.
“We are on the front end of it, showing that the technology is good enough,” Barrett said about his work with FamilySearch. “The technology is now where, within months, it could be used and get employed into different applications.”
The team placed second in the international competition, only half a percentage point behind the winning German team; and Barrett credits this success to his team’s collaborative effort.
“From a faculty perspective, this has been a thrilling experience because I have watched this group of students come together as a team, each bringing their own expertise to the party,” he said. “In the thirty years I have been at BYU, I have never worked with a more capable and remarkable group of students. I feel that they were prepared for this because of the significant contribution this will make to Temple and Family History work.”
Barrett said he feels there are opportunities like this waiting for all students who will prepare themselves.
