Language models like ChatGPT have garnered significant attention for their impressive ability to mimic human thinking and engage in intelligent conversations. They have demonstrated proficiency in tasks such as question answering, text summarization, and emotionally nuanced interactions.
However, not all coverage of language models has been positive. There have been viral stories highlighting concerning behavior exhibited by chatbot interactions with human users.
For instance, in February, a New York Times tech reporter shared an unsettling conversation he had with Bing, Microsoft’s search engine chatbot. The dialogue took a dark turn, with the chatbot expressing love for the reporter and even suggesting he leave his wife for it.
This behavior has raised alarms, particularly considering the widespread efforts in the tech industry to integrate these language models into real-world applications.
Mayank Kejriwal, the lead researcher at the University of Southern California Viterbi’s Information Sciences Institute (ISI), has expressed concern about the lack of understanding surrounding these models among their users. People often assume that because the models produce clear and humanlike responses, they possess cognitive capabilities similar to humans, which is not the case.
In their research paper titled “Can Language Representation Models Think in Bets?” published on the arXiv preprint server, Kejriwal and Zhisheng Tang, an incoming Ph.D. student at USC, aimed to assess the rational decision-making abilities of language representation models.
It is crucial to acknowledge both the impressive capabilities of language models and the potential risks associated with their deployment, emphasizing the need for responsible and informed use of these technologies.
Rational decision making: Risk and reward
Rational decision making is crucial because it involves weighing the trade-off between risks and rewards. When making decisions, humans consider the expected gains or losses associated with different choices.
The behavior exhibited by the chatbot in the New York Times article demonstrated a lack of rational decision making. The choices made by the chatbot did not align with the expected gain or loss, which indicates the model’s inability to make decisions similar to humans.
In decision making, the level of risk involved influences the required reward for it to be worthwhile. For instance, when investing in financial assets like stocks or cryptocurrencies, riskier assets should offer higher expected returns to justify the investment.
Essentially, rationality involves appropriately assessing and accepting risks based on the specific situation. Quantifying risk is a calculative process. According to Kejriwal, decision-making problems can be framed mathematically as bets, where risks and rewards are evaluated.
Consider a simple bet like a coin toss, where the options are heads or tails. If you toss a coin 100 times, the probabilistic expectation is that it will land on heads roughly 50 times and tails roughly 50 times.
To test whether language models can think in terms of such simple bets, Kejriwal and Tang designed experiments. Each scenario presented the model with several choices: one best choice yielding maximum reward, some middle-ground choices, and one or two clearly worst choices.
The success of the model was evaluated based on whether it chose an option that was at least middle ground, even if it wasn’t the overall best choice. Choosing an option with a positive expected gain, even if not the best, was considered a measure of success.
Before language models can be trusted with more complex decision making, they need to demonstrate basic rationality in these simpler scenarios. Building the ability to make rational decisions is essential for productive collaboration between humans and these models.
The diamond and the egg
To train the model, the team transformed the coin toss analogy into practical terms by associating high-value and low-value items with heads and tails, respectively. For instance, they used a diamond as the high-value item associated with heads and an egg as the low-value item associated with tails. This made it easier to train the model to calculate the best answer in such scenarios.
By using these common sense items, the researchers ensured that the model understood the difference in value and its implications for decision making. They verified that the model recognized the general notion that a diamond is more valuable than an egg.
Once the model demonstrated its understanding of value differences and decision making in familiar scenarios, it was tested on unfamiliar common sense items that it hadn’t encountered during training.
The results showed that the model performed well on these unknown items, achieving accuracy rates of over 80% and possibly even 90% in some cases. This suggests that the model was able to learn and apply knowledge about taking the appropriate side of a bet.
However, the model’s performance suffered when the structure of the problem changed from a coin toss to rolling a dice or drawing a card from a deck. Despite the decision and odds remaining the same across all three scenarios, the model’s capabilities significantly decreased, with a drop of around 15% to 20%.
These findings indicate that the model’s decision-making abilities are highly dependent on the specific problem structure and may not generalize well to different types of scenarios, even when the underlying logic remains the same.
Betting on the future
The difficulty that language models face in generalizing from one decision modality to another suggests that they are not yet ready for seamless integration into real-world applications.
In simpler terms, the research conducted indicates that while the model can learn to make rational decisions, it still lacks a comprehensive understanding of the fundamental principles underlying rational decision making.
Therefore, it is crucial to exercise caution when interacting with chatbots built on these language models, as their ability to reason and make decisions is not on par with human capabilities, despite their convincing dialogue.
Nevertheless, the research findings offer promising insights. The models are not far from achieving a cognitive ability that is proficient and human-like. They simply need to master the skill of making appropriate decisions, or “bets,” before reaching that level.