Many of us now incorporate one of the popular AI tools, maybe even two of them, into our work lives. In your case, maybe it has to do with running your business, how to turn machine feedback into insights for the future. In my case, I use ChatGPT to explore topics before writing about them. I did so with this article. I enjoy it.
As AI becomes almost as universal as word processing, we are learning to explore it and apprehend its usefulness in day-to-day business life, and even in manufacturing life—right on the shop floor. AI, sometimes with machine vision and an ability to move, is finding its place inside and outside machine tools but definitely in the mix of accomplishing something.
How do we evaluate it?
Three methodologies have tried to do just that. The first was the Turing test, designed by Alan Turing, and was called the “imitation game,” included in his 1950 paper, “Computing Machinery and Intelligence.” The basic setup was this: Can an interrogator tell the difference between responses from a human and responses from a computer with language software? Testing went on for generations, and was covered occasionally in newsstand magazines like Scientific American and more hard-core scientific journals like Nature.
In a July 2023 article in Nature, Celeste Biever wrote that “ChatGPT broke the Turing test.” Eight months later, Stanford researchers reported that ChatGPT passed the test. Whether you believe the first results or the second results, it took 73 or 74 years to pass the Turing test of non-differentiation of replies from a human vs. AI.
However…much as a computer’s responses may be well-thought-out, well expressed, and even show an emotional quotient, an AI program might convince you of a human-derived basis yet politely select the wrong tooling for an upcoming job. Also, older testing for “humanity,” such as the countless times we’ve clicked on squares with traffic lights or motorcycles, works well against bots but it’s easy pickings for AI.
The second methodology is the Lovelace test, named for researcher Ada Lovelace but invented by fellow experts Selmer Bringsjord, Paul Bello, and David Ferrucci in 2001. The gist of this methodology is testing for a computer’s ability to originate things, to be creative, to surprise with an original idea—as part of a repeatable process, not as a fluke or juxtaposition. It must not be able to be explained via the system’s construction. You can’t explain the output by using the system’s architecture, knowledge, and functions. It must be something new. Think of a computer having an “aha moment.” Add to that a difficult or impossible task of explaining the genesis of the idea, and you’ll understand what the test is all about.
To date, I am unable to find any reference to a successfully run Lovelace test—ChatGPT can’t find one either. If it could, I might be surprised.
However…the United States Patent and Trademark Office currently rules out naming AI software (or any AI product) as a creator of an innovation. Instead, it needs the name of the human creator who used AI software, much as a human might use lab equipment or a database or spreadsheet to assist in the invention process. In practical terms, even if the Lovelace test were successful, there would be no evidence that the output would be useful or valuable. Example: a doorstop that sings opera all day and removes stubborn denture stains at night. Surprising, yes, but a good idea?
The third test is the Chinese Room test, more of a thought experiment than a real test. Suppose you do not understand Chinese. You are given instructions for placing Chinese symbols next to each other. Results might convince people outside the room that the person manipulating the symbols understands Chinese, while the person understands none of it. The argument is that executing a program is insufficient means for understanding.
It can produce results. I am in a partial Chinese Room every day of my life, because like many men, I have some color blindness. As a young man I quietly accepted compliments about my green suit and didn’t tell a soul I thought it was brown when I was fitted for it and purchased it.
This came into play more strikingly when I ran a technology business and would run twisted-pair Ethernet cable for companies. Ethernet cable has four twisted pairs of wire, and I always had trouble seeing the difference between green and brown. To make things worse, depending on the manufacturer, the hues were different. My choices were to ask someone at my customer’s business which was which, or remember another characteristic, like maybe the green is a little darker than the brown. If I look at a spectrum, I must remember ROY G BIV. It is a partial Chinese Room test.
So as long as I had some external help, I could do that part of my job.
However…the Chinese Room test is a bit philosophical, maybe a bit like the infinite number of monkeys on an infinite number of typewriters test (would they create all literature?). It can be logically divided into partial rooms (as my version of color blindness required).
How we evaluate AI in today’s environment depends very much on the tasks entrusted to it, and the presentation of it to the problem. By that I mean the scope and type of its interface to the task. I once saw a washing machine in a store advertised with AI. It used the size of the load and the opacity of the water to judge how best to wash the clothes. A good evaluation for this is to look at the clothing later and see how clean they are.
And I’m sorry to report that most of our evaluation must take on this same methodology. The reason I’m sorry to report it is that there are many islands of artificial intelligence in the archipelago of your shop and your entire business. How can we find mental tweezers tiny enough to pick out the AI parts of programs or even within machine tools, in order to evaluate those efforts? For example, some machine tool actions are more like the washing machine’s usage of AI (e.g. let color and sound dictate the speed of a laser cut or even a weld). Others are more systemic (like the umbrella-type software systems that the larger providers offer; they can see entire weekly schedules at once and have all the variable influences taken into account).
The first example with color and sound is sort of a no-brainer; our grade is “A” until something goes haywire (and it probably never will). As for the software, well, we tend to register complaints with companies that they did not think of this instance or that situation. It might affect that product’s grade in your book but you know the answer to your complaints will be in the next release of the software.
I have something to propose as we move forward with AI.
The Dark Cockpit: For specialized actions like the washing machine and some machine tool monitoring/acting AI applications, my suggestion is to use the dark cockpit approach. It’s used in airliners and most modern small airplanes. It works like this: go ahead and light the gauges in the cockpit, but show me as few lights as possible unless they are alarms that something is not right. Machine tool makers should adopt this philosophy in reporting errors in AI. I think many already employ the concept, and many others will agree that it makes sense.
However, there is an implied responsibility to doing things this way and that is to find the best way to violate the dark cockpit. In a jetliner, a minor problem gets a minor alert. As the problems become more major (like an impending stall), the visual multi display alerts, a speaker voice callout of “STALL” alerts, and a haptic alert shakes the yoke or stick of the aircraft. The problems in your shop might range from a report of increasingly more dust on the nozzle of a laser cutter to every third job being rejected from a laser/punch machine. A third ramp up might be something in the welding or finishing department; it’s worse to start over from the end. You will have to decide which alarms are more urgent than others.
Meeting review: for the larger systems, it makes sense to include AI in your weekly review meetings. If there is a problem, it can affect a lot of things in your business. And it should be more obvious as to the cause, but it doesn’t work that way in many cases. The problem may be sporadic and mysterious. It may require looking at actions, coding, logs, results…many things that are not AI but are touched by AI.
You’ll be able to track AI’s performance by the results of the things it touches, and for now, that is the best way to track AI’s ROI. We must rely on our providers to keep systems open.