The formula that changed the future
Late UD professor’s 50-year-old statistical feat finds new life in world of Big Data
For the past several decades, and onward into this new age of Big Data, good results demand good mathematics — formulas that allow experts to reel in meaningful conclusions from churning rivers of numbers.
That’s when they turn to the magic of a 50-year-old statistical analysis, co-developed by a UD professor and still regarded worldwide as a foundational tool in Big Data’s number-crunching advance.
Called “Ridge Regression,” this statistical marvel came from the minds of the late Prof. Arthur Hoerl and UD alumnus Bob Kennard, then a DuPont Co. research statistician. In terms simplified for mere-mortal readers, the formula essentially allows researchers to cope with problemsome “correlated” variables in mathematical models — variables like temperature and pressure, which frequently change in unison, but which tended to skew predictions in ways that seem to contradict known scientific realities.
What the formula also did was prompt immediate academic skepticism from the high-brows of numbers science: It seemed like a statistical sleight-of-hand to the establishment, says Hoerl’s son Roger, himself a dual-degree UD graduate and now a chaired statistics professor at Union College in New York.
“In some sense, he was challenging a method that was considered beyond reproach,” Roger Hoerl says. “It’s sort of like challenging the common notion that someone should exercise and eat a good diet to fight heart disease. If you’re challenging something so well-established, it kind of seems like quackery.”
Part of the establishment’s doubt was grounded in academic skepticism, but scholarly elitism simmered just beneath the surface. Arthur Hoerl held degrees in engineering and mathematics, but not a single one in statistics, making his claims immediately suspect to territorial insiders.
But Arthur Hoerl was undaunted, championing his technique for years despite all doubters. “He was a courageous and sometimes stubborn man, especially when he was convinced he was right,” Roger says of his father, who would leave a lucrative position at DuPont to teach at UD. “He could dig in his heels. He was definitely not the person who was gonna ‘go along to get along.’ ”
Once its utility had been established and the naysayers swayed, Ridge Regression analysis enjoyed something of a golden era through the 1980s, then slipped somewhat out of favor among forward-leaning statistical theorists, Roger says. Then, the old became new again: As computing capabilities grew, and as the use of “machine learning” as a scientific tool began to gain momentum, Ridge Regression has enjoyed a wonky renaissance of sorts.
“When the data revolution came along, it led people to re-look at my father's work, and that led to a couple of enhancements,” says Roger, a 1981 and 1983 UD graduate who presented a short history of his father’s contributions at a recent webinar, co-sponsored by Lerner College’s Institute for Financial Services Analytics and the Department of Electrical and Computer Engineering’s Distinguished Speaker Series.
Data-diving statistical papers began to refer to Ridge Regression as a foundational tool, and younger scholars began to re-evaluate and re-engage with the technique, sensing that its framework could help lead to bigger and better ways to pry open numbers’ secrets.
From his professor’s chair situated firmly in the 21st century, Roger Hoerl sees the ongoing renaissance with pride, sensing that in some ways, his father — who died in 1994 — is still with us all. He’s also proud to know that there’s even more richness to be found in his family’s storied past, reaching back into the first half of the 20th century, when the Hoerl name was involved in a faraway land known as Hollywood, and a then-mysterious project cryptically known as “Manhattan.”
Those chapters of family history serve as mere tangents to the Ridge Regression legacy, but do perhaps add a deliciously quirky edge to the broader Hoerl legacy: Long before he arrived at UD or made his mark with the formula, Arthur Hoerl and his wife-to-be also happened to be working members of the Manhattan Project, the super-secret U.S. program that led to the creation of the nuclear bomb.
The story started in the anxious early days of World War II, when military higher-ups noticed draftee Hoerl’s mathematical prowess, quickly nixed his deployment to the European theatre, and ordered him to report to Los Alamos, New Mexico. Once at Manhattan’s desert development center, he would meet a young data-input worker named Marguerite, fall in love — and ultimately marry the woman who would become Roger’s mom.
“I never judged him in that sense,” Roger said of his dad’s work on weapons of war-ending mass destruction. “He actually worked on what they call bombing tables, which had to do with figuring out what elevation they should ignite the bomb.”
The other curious tangent of the Hoerl story reaches even further back into the American saga: Arthur’s father, Arthur Hoerl Sr., was a luminary of sorts in another field entirely, and his mark on civilization is arguably more well known today than the formula his son would create.
This story took place in the early days of the American movie-making industry, when Arthur Sr. was working as a perpetually struggling screenwriter, churning out scripts for a slew of since-forgotten movies, from The Drums of Jeopardy (1923) to The Singing Cowgirl (1938). But it was his script for one ostensibly civic-minded reel that would be forever remembered, attaining a cult status that endures to this day.
The 1936 movie was called Tell Your Children, but millions of B-movie, old-school schlock fans know it best by its later title: Reefer Madness, now regarded with high amusement for its sensationalized account of marijuana’s psychic and familial perils.
Such successes would bring the Hoerl family into the Hollywood limelight and lift the family’s fortunes considerably, taking them from a lackluster lifestyle in Brooklyn to the glitter of Beverly Hills. And even Arthur Jr.’s Ridge Regression partner, Newark High graduate Bob Kennard, would carry a memorable early moment into his later life in Delaware: He worked with the cryptographic unit that would finally crack the “Purple” cipher, the one of Japan’s “unbreakable” WWII codes.
“That was a big deal,” Roger says. “Both of them played really significant roles in WWII.’’
None of them could have known that even more far-ranging, far-reaching impacts lay just ahead — and on into our still-evolving, data-driven, 21st-century future.
“My father and I had a great relationship,” Roger says. “I always thought highly of my father, but it wasn’t until I started seeing his name in textbooks that I realized he’s not just a good statistician — he was really onto something there.”