• QuaternionsRock@lemmy.world
      link
      fedilink
      English
      arrow-up
      0
      ·
      6 months ago

      I mean, it’s not shit at everything; it can be quite useful in the right context (GitHub Copilot is a prime example). Still, it doesn’t surprise me that these first-party LLM benchmarks are full of smoke and mirrors.

        • QuaternionsRock@lemmy.world
          link
          fedilink
          English
          arrow-up
          1
          ·
          6 months ago

          That GitHub Copilot and friends are useful? I would argue that their utility is rather subjective, but there are indications that it improves developer productivity.

          I’m unsure if you’ve used tools like GH Copilot before, but it primarily operates through “completions” (“spicy autocorrect” in its truest form) rather than a chatbot-like interface. It’s mostly good for filling out boilerplate and code that has a single obvious solution; not game-changing intelligence by any means, but useful in relieving the programmer of various menial tasks.

          May I ask, what evidence are you hoping to see in particular?