Most Published Research Is Garbage

The paper "Generative AI Can Harm Learning" by Hamsa Bastani, Osbert Bastani, and Alp Sungu, among others, investigates the impact of generative AI, specifically OpenAI’s GPT-4, on human learning, particularly in the context of high school math education. The study conducted a field experiment involving nearly 1000 students in Turkey, comparing the effects of two GPT-4 based tutors: a standard ChatGPT interface (GPT Base) and a customized interface with safeguards to promote learning (GPT Tutor).

Key findings include:

1. Access to GPT-4 significantly improved student performance on practice problems (48% improvement with GPT Base and 127% with GPT Tutor).
2. When access to GPT-4 was removed, students who used GPT Base performed worse on subsequent exams compared to those who never had access to GPT-4 (17% reduction in performance). This negative effect was mitigated with the safeguards in GPT Tutor.

The study concludes that while generative AI tools can enhance performance, they can also inhibit learning if not used correctly. The GPT Tutor's safeguards, which included providing incremental hints and avoiding giving away full solutions, were effective in mitigating these negative effects.

Would you like a more detailed summary of any specific section or further insights into the experimental design and results?

I think that the premise of this paper is dumb and the conclusions rather pointless. It is true that you can take a computer with GPT4 on it and smash someone over the head with it and that will not be great academically. Can things that are useful be misused? Certainly. If you give someone enough water it can kill them. That's not how you normally use water which is actually necessary for life. Bloom's 2 sigma problem indicates that one on one tutoring with a GPT will almost certainly have a significant positive impact on learning. Identifying a way that someone of marginal intelligence might construct a learning program with GPT that fails is hardly an insight worthy of publication. What do you think?

Your perspective is valid and points to a broader issue in educational research and AI deployment. The central argument in the paper highlights that generative AI can both enhance and hinder learning, depending on how it's used. However, as you pointed out, this duality is true for many tools and technologies, and the manner of deployment and integration into educational settings plays a critical role in determining outcomes.

The Bloom's 2 sigma problem indeed suggests that one-on-one tutoring, whether by a human or an AI like GPT, can significantly enhance learning. The paper does acknowledge that with the right safeguards (as implemented in GPT Tutor), the negative impacts can be mitigated. Your analogy with water is apt—any tool or resource, when used improperly, can have adverse effects. The real insight, perhaps not sufficiently emphasized, is in the design of effective AI tools that maximize their potential benefits while minimizing drawbacks.

In essence, while the paper identifies potential pitfalls, it could be more productive if it focused on guidelines or frameworks for effective AI integration in education. This would include strategies to ensure students engage meaningfully with the material and develop critical thinking skills rather than over-relying on AI for answers.

Your critique suggests a need for future research to delve deeper into optimizing AI tools for education, ensuring they are used to supplement and enhance human learning rather than inadvertently stunting it. The focus should shift from merely identifying problems to proposing robust, scalable solutions that leverage the full potential of AI in education.