• ViatorOmnium@piefed.social
    link
    fedilink
    English
    arrow-up
    3
    ·
    edit-2
    25 days ago

    LLMs suck at counting and therefore suck at writing valid lisp. (Source: I’ve seen people trying to use code agents with clojure))

    Sorry it seems I miscounted the (parenthesis)

    You are totally right my fix made the syntax valid but doesn’t work…

        • dihutenosa@piefed.social
          link
          fedilink
          English
          arrow-up
          1
          ·
          24 days ago

          Citation needed.

          I asked a LLM to start a stdio REPL, write code and then copy the code into the REPL. It closely mirrors actual Lisp development process - you write a couple of lines, eval it, see why it didn’t work, and repeat. Here’s what it came up with:

          Starting Lisp

          $ : > /tmp/lisptest-repl.commands; : > /tmp/lisptest-repl.log; (tail -n +1 -f /tmp/lisptest-repl.commands | sbcl --noinform > /tmp/lisptest-repl.log 2>&1) & echo $! > /tmp/lisptest-repl.pid; sleep 1; printf '(require :asdf)\n(asdf:load-asd
          #P"/home/user/lisptest/lisptest.asd")\n(asdf:load-system "lisptest")\n' >> /tmp/lisptest-repl.commands; sleep 2; tail -40 /tmp/lisptest-repl.log (timeout 10s)
          
          * ("ASDF" "asdf" "UIOP" "uiop")
          * #<ASDF/FIND-SYSTEM:DEFINE-OP >
          #<ASDF/PLAN:SEQUENTIAL-PLAN {1204C49563}>
          * T
          *
          
          Took 3.1s
          

          Invalid input

          Feeding a single ) into the REPL. The LLM later recovered it by sending the abort command to the debugger.

          $ printf ')\n' >> /tmp/lima-repl.commands; sleep 1; tail -25 /tmp/lima-repl.log (timeout 10s)
          
          * #<ASDF/FIND-SYSTEM:DEFINE-OP >
          #<ASDF/PLAN:SEQUENTIAL-PLAN {1204C49563}>
          * T
          * #<SB-THREAD:THREAD tid=151 "lima-echo" RUNNING {1203EB14E3}>
          * Echo server listening on 127.0.0.1:5555
          T
          * NIL
          * #<SB-THREAD:THREAD tid=0 "lima-echo" RUNNING {1201C0DDC3}>
          * Echo server listening on 127.0.0.1:5555
          T
          *
          debugger invoked on a SB-INT:SIMPLE-READER-ERROR in thread
          #<THREAD tid=142 "main thread" RUNNING {12003D8003}>:
            unmatched close parenthesis
          
              Stream: #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDIN* {12014C9E73}>
          
          Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
          
          restarts (invokable by number or by possibly-abbreviated name):
            0: [CONTINUE] Ignore it
            1: [ABORT   ] Exit debugger, returning to top level.
          
          (SB-IMPL::READ-RIGHT-PAREN #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDIN* {12014C9E73}> #<unused argument>)
          0]
          
          Took 1.0s
          

          Hotpatching

          This thing activates a new function definition. Next time it’s called, the new one gets executed.

          printf '(load #P"/home/user/lisptest/src/server.lisp")\n' >> /tmp/lisptest-repl.commands
          
          • ViatorOmnium@piefed.social
            link
            fedilink
            English
            arrow-up
            1
            ·
            24 days ago
            (defn process-scores [user & scores]
              (println "User:" user) (reduce + scores  (filter pos? scores))
            

            can be either

            (defn process-scores [user & scores]
              (println "User:" user)
              (reduce + scores)
              (filter pos? scores))
            

            or

            (defn process-scores [user & scores]
              (println "User:" user)
              (reduce + scores (filter pos? scores)))
            
            • dihutenosa@piefed.social
              link
              fedilink
              English
              arrow-up
              1
              ·
              24 days ago

              Yep, can happen. Do we know if this tends to happen more often with lispy (than curlybracy) languages?