He said the TTCT is protected proprietary material, so ChatGPT couldn’t “cheat” by accessing information about the test on the internet or in a public database.

Of course no proprietary data at all could ever permeate through the sampling of LLM creators, well known for scrupulously following all licensing terms, and thoroughly checking that scrapped data might not be accidentally licensed.

-_____________________________________________________________________________-

The premise that a standardized test could truly measure originality or creative thinking seems dubious to me in the first place. Just because a response is common amongst humans does not suggest to me that it is uncreative or even unoriginal (strictly speaking as long as you come up with it independently it is still both).

That being said I am curious as how a Text Based Machine could have completed these portions of the test:

• Verbal tasks using verbal stimuli
• Verbal tasks using nonverbal stimuli (eg: visual input, if being extremely generous, they might have access to the “multi-modal” version of stochastic parrot 4, but this is not actually described as such in any source I could find)
• Non-verbal tasks (eg: completely visual input and output)

Until I see the methodology, I am not even sure I buy the result as advertised.

As far as I can tell, the Torrance Tests of Creative Thinking primarily targets children. So the text predictor trained on the entire internet can outperform literal children, and was on par with college-age testers. Incredible.

I can’t find the paper, looks like it was presented at a conference and isn’t available. But I will point out it was the College of Business that put this out.

A bit related. There was an episode of Behind The Bastards podcast where they talked about generative AI and how rather than help non-creative people feel creative by generating art they should be used to generate bot fans on social media that are convincingly impressed by that person’s uncreative output. I thought it was a genius idea that was nice to imagine as a reality.