It’s very simple: I’ve played for several hours with ChatGPT and yet, despite all the hype, I have not been able to find one non-trivial task that it can actually accomplish. I want to understand the power and the limitations of LLMs, so I’d be grateful if someone could point me to any example of a non-trivial question that ChatGPT can actually answer. For the purposes of this question, non-trivial means roughly “well-defined question whose answer is not found in <1min of googling". So no non-well-defined tasks ("write X in the style of Y") and no copy-paste tasks (for example, programming tasks which have a ready-made answer in stackoverflow). Here is a (non-exhaustive) list of such tasks I've tried, with no success. - Write a program that solves the TOV equations - Write a program that decodes a Posit - Write a program that scores a trick in a simple card game - Simulate a python terminal (fucked up immediately when I asked to slice a string or to bit-shift an int). - Tell me the melody of a given Mozart piano sonata - Prove a basic fact about metric spaces (diameter of open ball is twice the radius) - Prove a basic fact about quantum gates - Identify a chord given the pitch classes - Write a poem with a well-defined rhyme - Write a poem with a well-defined metric - Tell me who developed an (obscure) software - Give me a short biography of an (obscure) singer - Summarise a research paper - Translate a joke - Simulate a Zork-like game - Summarise the Europa League final 2013 - Summarise the Counterstrike Boston Major final 2018 In every single one of these it failed. Now it's great that it can answer the question "how to I make an http request in js", but that's a google search, not really impressive in my book. As you can see, even some questions that are google searches it failed. Suggestions and discussion about LLMs welcome! Also note I only have access to GPT 3.5, not 4.
Story Published at: April 13, 2023 at 05:33PM
Story Published at: April 13, 2023 at 05:33PM