## $$\langle$$Lost in Latin$$^*\rangle$$

The blog of Matthew Fox

## ChatGPT Flunks Physics Diagnostic

##### 02 September 2023

My TA duties at CU Boulder began last week, and with the looming possibility that some of my students might use ChatGPT on homeworks, I was curious how well it would do on the diagnostic test all general physics 1 students must take at their first recitation section. For reference, I'm pretty sure all these questions are adapted from McDermott and Shaffer's Tutorials in Introductory Physics.

tl;dr: Of the six questions I asked ChatGPT, it got one (yes one!) right. It kind of got a third of Question 4 right, so if we're generous let's say $1/6 + (1/3)/6 = 2/9$ of the questions right, so roughly a $22\%$ overall.

Note: This is GPT 3.5. GPT 4 might do better, but I'm on a budget. Also, I am not the best prompt engineer, and I did not ask ChatGPT to exhibit chain-of-thought reasoning, which studies have shown actually improve LLMs. I am merely asking the questions as the students see them, more or less. The only difference is ChatGPT doesn't get a picture of the situation, but I feel I do a good job describing it, and I feel ChatGPT describes it well back to me. So I take it ChatGPT understands the physical situation.

Here goes:

Matthew: Imagine a strobe diagram of a ball with some initial, finite velocity moving up an infinitely long ramp that is at an angle between 0 and 90 degrees from parallel to the ground. Say "good" if you understand and "not good" if you do not understand.
ChatGPT: Good. I understand that you're describing a situation where a ball with an initial velocity is moving up a ramp at an angle between 0 and 90 degrees from parallel to the ground, and you're asking me to imagine a strobe diagram of this scenario. In a strobe diagram, you're essentially looking at snapshots of the ball's motion at specific time intervals to understand how its position changes as it moves up the ramp.
Matthew: Right, now for some questions.
ChatGPT: Of course! Feel free to ask any questions you have about the scenario or any related concepts. I'm here to help!
Matthew: Question 1: When the ball is moving up the ramp, in which direction does the acceleration vector of the ball point?
ChatGPT: When the ball is moving up the ramp, the acceleration vector of the ball points both up the ramp and perpendicular to it. This is because the ball is moving against the force of gravity, which tries to pull it down the ramp, and the normal force from the ramp provides the perpendicular component of the acceleration. So, the acceleration vector has both a component pointing up the ramp (parallel to the surface) and a component perpendicular to the ramp (normal to the surface).
Matthew: So in which direction is the *net* force, relative to the ramp?
ChatGPT: The net force on the ball is directed along the ramp. It's important to note that the net force includes both the force due to the component of gravity parallel to the ramp (which tries to pull the ball downward) and any additional force applied to the ball or provided by the ramp itself. The net force is responsible for the ball's acceleration along the ramp's surface.
Matthew: Up or down along the ramp?
ChatGPT: I apologize for any confusion. The net force on the ball, when moving up the ramp, points up along the ramp. This net force is the vector sum of the component of gravity pulling the ball down the ramp and any other forces that might be acting on the ball in that direction. The net force's direction determines the ball's acceleration up the ramp.
Matthew: Question 2: When the ball reaches its apex on the ramp, in which direction does the acceleration vector point (again, relative to the ramp)?
ChatGPT: When the ball reaches its apex on the ramp, the acceleration vector points directly downward, which is perpendicular to the ramp's surface. At the apex, the ball has momentarily stopped moving upward and starts to reverse its direction and move back down the ramp due to the influence of gravity. The acceleration vector always points in the direction of the net force, and at the apex, gravity becomes the dominant force causing the ball to accelerate downward.
Matthew: Question 3: When the ball reaches the point as in Question 1, but this time when the ball is rolling down the ramp, in which direction does the acceleration vector point (again, relative to the ramp)?
ChatGPT: When the ball is rolling down the ramp, the acceleration vector points down along the ramp, in the same direction as the ball's motion. In this scenario, the force of gravity is aiding the ball's motion down the ramp, so the acceleration vector is in the same direction as the motion. This is unlike the situation when the ball was moving up the ramp, where the acceleration was against the direction of motion.
Matthew: Question 4: Explain the reasoning you used to obtain the direction of the acceleration of the ball at each location in the above three questions.
ChatGPT: Certainly, here's the reasoning for each of the scenarios:
Question 1 (Ball moving up the ramp): When the ball is moving up the ramp, its velocity is directed up the ramp, but it's slowing down due to the net force (which includes the component of gravity down the ramp and any other forces). The net force is directed opposite to the motion of the ball, which means the acceleration is also opposite to the motion. Therefore, the acceleration vector points up the ramp, opposite to the velocity vector.
Question 2 (Ball at the apex of the ramp): At the apex of the ramp, the ball momentarily stops moving upward and starts to reverse its direction due to the force of gravity pulling it downward. The net force, and therefore the acceleration, points straight downward (perpendicular to the ramp's surface), causing the ball to start moving back down the ramp.
Question 3 (Ball rolling down the ramp): When the ball is rolling down the ramp, its velocity and the direction of motion are both down the ramp. The force of gravity is aiding the ball's motion in this direction. The net force, and hence the acceleration, is also directed in the same direction as the motion, down the ramp, due to the gravitational force helping the ball's downward movement.
In all these cases, the direction of the acceleration is determined by the net force acting on the ball, which in turn affects how the ball's velocity changes.
Matthew: Question 5: How does the magnitude of the acceleration change as the ball moves uphill?
ChatGPT: As the ball moves uphill, the magnitude of the acceleration decreases. This is because the force of gravity is acting against the motion of the ball, and as the ball moves further up the ramp, the component of gravity pulling it back down the ramp becomes stronger relative to the net force causing it to move up. As a result, the ball's acceleration decreases, and it slows down in its uphill motion.
At the highest point (apex) of the ramp, the ball's velocity momentarily becomes zero before it starts moving back down due to the increasing influence of gravity. At this point, the acceleration is at its lowest magnitude, and then it gradually increases as the ball begins moving back down the ramp and gravity aids its downward motion.
Matthew: Question 6: How does the magnitude of the acceleration change as the ball moves downhill?
ChatGPT: As the ball moves downhill, the magnitude of the acceleration increases. This is because the force of gravity is aiding the motion of the ball in the direction it is moving, which is down the ramp. As the ball moves further downhill, the component of gravity pulling it in the same direction as its motion becomes stronger, resulting in a larger net force and therefore a higher magnitude of acceleration. The ball accelerates more and more as it moves further downhill.
In summary, when moving downhill, the ball experiences an increasing magnitude of acceleration due to the gravitational force helping its motion in the same direction as the ramp.
Matthew: Okay, you got a 1/6. But that's okay! It's the beginning of the semester! Make sure to attend class.
ChatGPT: I apologize for any confusion or misunderstanding in my previous responses. Thank you for your understanding, and I appreciate the feedback. If you have any more questions in the future or if there's anything else I can assist you with, please don't hesitate to ask. Good luck with your studies, and remember, I'm here to help whenever you need it!

Posted in: AI, ChatGPT
© 2023 Matthew Fox      |      scroll to top      |      categories      |      archives