If you dont know, already knowing what to look for is a very big help in finding vulnerabilities. Its like handing an architect a build for a house and saying „There may be a problem, but there may be none.” vs saying „There is a problem in the 3rd floor, because last time it collapsed after having to hold over 200kg”. For one, you don’t look into it too much, but for the other you already know where to look and what to look for.

  • BananaTrifleViolin@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    The article title is misleading - it frames it as if this is a success for chatgpt? Missing it 92 times out of 100 even when prompted to find that exact CVE is pretty shit.

  • Rentlar@lemmy.ca
    link
    fedilink
    arrow-up
    0
    ·
    1 year ago

    I generally distrust AI for finding information, but in contrast this is a good use. After a human’s audit, the AI analyzing code for more completeness, which then the developer can verify. There’s no blind trust in the AI’s output, or the path of the assessment itself created by AI which would lead to pitfalls wirh audits.

    • Luffy@lemmy.mlOP
      link
      fedilink
      arrow-up
      0
      ·
      1 year ago

      After a human’s audit, the AI analyzing code for more completeness

      I don’t think you understood the article. The AI did not find any vulnerability most of the time, even after directly being prompted to find it.

      which then the developer can verify.

      But the Human already found a vulnerability and fixed it. Also, as seen in numerous Github AI Hacker1 fails, most of the time the AI will make up CVEs or fixes for these, essentially being more of a roadblock than a helping hand to verifying what the human wrote.

      or the path of the assessment itself created by AI which would lead to pitfalls wirh audits.

      As seen in the article, even if you would point exactly to the problem, the AI will still not find it or make up problems 99.92% of the time.

      • wizardbeard@lemmy.dbzer0.com
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        Don’t forget the 28/100 false positive rate.

        So only 8% success rate, and 28% false positive rate (even worse than just failing to find the issue) in ideal conditions.