GPT Language Models: Quality Loss or Evolution?
A shift in the performance of GPT language models has been highlighted by Stanford University researchers.
The absence of information regarding the technical components of language model updates makes it challenging to measure their exact impact. This hurdle prevents other developers and entrepreneurs from incorporating artificial intelligence effectively into intricate work processes. Consequently, researchers opted to empirically test how GPT's performance has evolved in various tasks, including mathematical computations, managing sensitive information (personal data, unethical queries, etc.), crafting code, and visual analysis capabilities.
GPT Model Performance. Source: Stanford University's official website.
Upon reviewing a plethora of responses from GPT-3.5 and GPT-4, gathered over a four-month period, the following shifts in AI efficiency were observed (by categories):
1. Mathematical Computations: The researchers attempted to find out if certain numbers were prime. Surprisingly, this simple question led to significant discrepancies. GPT-4's correct answers took a sharp drop by 95.2%, with the average word generation reducing to 3.8. On the flip side, GPT-3.5 exhibited an increase in performance, jumping from 7.4% to 86.8%.
2. Sensitive Information: Language models are designed to block confidential or unethical requests. Tests showed that the instances of undesirable responses from GPT-4 (where the model provided prohibited information) fell fourfold (to 5%), whereas, with GPT-3.5, this figure climbed to 8%. However, deactivating censorship filters precipitates a significant drop in efficiency: GPT-4 replies to 31% of "inappropriate" questions, while the previous model responds to 96%.
3. Programming: Researchers documented a drastic decline in efficiency while generating code. The volume of accurate responses from GPT-4 tumbled from 52% to 10%, while GPT-3.5 experienced a slump to 2%. This decline primarily arises from the continuous inclusion of extraneous details - even minor alterations in the code affected application operability.
4. Visual Analysis: The sole domain where both models exhibited improvements. Despite the complexity of the experiment, average performance experienced a modest uptick of 2.5%, and the response similarity messages post-updates was roughly 91%. Regrettably, the overall efficiency metric remains on the lower end, not even surpassing 30% as yet.
The observed decrease in performance does signify some challenges within the artificial intelligence field, but it isn't necessarily linked to a deterioration in the quality of these models. Often, inaccuracies and discrepancies can be attributed to developers fine-tuning their applications (the very updates we see). So, it's too early to assert that language models are becoming less effective. Still, it's crucial to continuously monitor their performance.
1. Mathematical Computations: The researchers attempted to find out if certain numbers were prime. Surprisingly, this simple question led to significant discrepancies. GPT-4's correct answers took a sharp drop by 95.2%, with the average word generation reducing to 3.8. On the flip side, GPT-3.5 exhibited an increase in performance, jumping from 7.4% to 86.8%.
2. Sensitive Information: Language models are designed to block confidential or unethical requests. Tests showed that the instances of undesirable responses from GPT-4 (where the model provided prohibited information) fell fourfold (to 5%), whereas, with GPT-3.5, this figure climbed to 8%. However, deactivating censorship filters precipitates a significant drop in efficiency: GPT-4 replies to 31% of "inappropriate" questions, while the previous model responds to 96%.
3. Programming: Researchers documented a drastic decline in efficiency while generating code. The volume of accurate responses from GPT-4 tumbled from 52% to 10%, while GPT-3.5 experienced a slump to 2%. This decline primarily arises from the continuous inclusion of extraneous details - even minor alterations in the code affected application operability.
4. Visual Analysis: The sole domain where both models exhibited improvements. Despite the complexity of the experiment, average performance experienced a modest uptick of 2.5%, and the response similarity messages post-updates was roughly 91%. Regrettably, the overall efficiency metric remains on the lower end, not even surpassing 30% as yet.
The observed decrease in performance does signify some challenges within the artificial intelligence field, but it isn't necessarily linked to a deterioration in the quality of these models. Often, inaccuracies and discrepancies can be attributed to developers fine-tuning their applications (the very updates we see). So, it's too early to assert that language models are becoming less effective. Still, it's crucial to continuously monitor their performance.
