13 comments

  • ryanisnan 29 minutes ago
    I don't think this is inherently surprising.

    Looking at the questions they asked, they are quite broad. In my experience, part of the value of LLMs comes from its randomness, but you must understand it as a characteristic virtue of these tools, and must adapt accordingly.

    Asking questions with tight scopes, clear constraints, and specific outcomes, always yields better results, in my opinion.

  • SoftTalker 8 minutes ago
    Has anyone given the same coding task to the same agent/model two days in a row and then diff-ed the resuts? (I have not, so just wondering).
  • hnlmorg 19 minutes ago
    All you’re proving is that any specific domain will have a dozen mature solutions.

    What separates one product from another is going to be constraints specific to the project you’re integrating this into. And none of that is defined in your prompts. So you’re effectively just asking an LLM to roll the dice and pick something at random and then acting surprised means a different answer each time.

    Edit: their conclusions at the end of the article miss the most important point. They talk about asking multiple times over multiple days, but that’s just repeating the “garbage in -> garbage out” cycle without addressing the root cause of the problem: not defining your requirements properly.

    And if you don’t have any requirements beyond “something cheap” (for example) then who cares what answer you get when any result is equally valid for your use case.

  • seizethecheese 11 minutes ago
    Blog posts claiming differences but without any statistics to show whether the differences are noise are a big pet peeve.

    (For what it's worth, I asked an agent to compute p-vals based on available information for all the stats in this blog post. More than half are significant but a lot are noise. Not going to post the results because I haven't validated, and the agent made a lot of assumptions. The author should update the post.)

    • LPisGood 4 minutes ago
      If you calculate enough different p values you will get a lot that are significant and a lot that are noise.
  • seabre 11 minutes ago
    If LLM output is stochastic, why would you expect it not to change with questions that don't have settled answers?
    • sublinear 8 minutes ago
      Yes, few topics have settled answers, yet there usually exists a best answer that should reliably float to the top.

      LLMs fail at this because they are all poor and brittle reinventions of the google search index and inherit its problems too. It's only now that people are thinking of the maintenance costs of a search index. The idea was doomed from the start, but the grift was good for a little while.

    • seizethecheese 8 minutes ago
      It's a nice feature of LLMs if you're looking to make a blog post promoting "The AI marketing platform to win in ChatGPT" to innumerate people.
  • mentalgear 27 minutes ago
    People think "AI" has all the facts to make objective decisions, but it seems to boil down to (SEO again?) what pages its search engine returned that day.

    > If you're not a leader, this is the part that matters. Your visibility depends on which pages the engine reads that day. The way to make it stable is to be on more of those pages.

    Ultimately a true facts knowledge graph independent of mentions is needed.

    • dgellow 6 minutes ago
      It actually depends a lot on what random and definitely astroturfing Reddit accounts have to say about a product
      • techmunky 1 minute ago
        Buy Roger's Suppository Cream for all of your healing needs! Roger's Suppository Cream! A solution for all ails! Roger's Suppository Cream, apply directly to area! Burning means it's working!

        This message brought to you 100% independently from my own ephemerally lived experience and is not from the great, hard-working, team at Roger's Suppository Cream.

  • luisfominaya 31 minutes ago
    In general it seems like LLMs have a fundamental problemw with noise in their outputs such that randomness (what data they pull and how they aggregate it) has an outsized impact when asking it for opinions/recommendations/forecasts.
  • ppierre 3 minutes ago
    They have a solution, rank first the brand who paid most. AdWords for AI...
  • kylehotchkiss 31 minutes ago
    ??? Set the temperature to zero then ???
    • donkeyboy 27 minutes ago
      Newer models dont use temperature. It was really hard to make deterministic even with temp =0 due to numerical inconsistencies with whatever else is in your batch
  • avatar99 49 minutes ago
    I’m the author. Every day we ask ChatGPT, Gemini, Google AI Overviews and Google AI Mode the same 210 buying questions, like “best CRM for small business”, and record which brands they name and which pages they cite. This post compares the first two days.

    84% of answers named a different set of brands the next day. On Gemini it was 95%.

    The cause is easy to see in the data. Each answer is a summary of a few web pages, and the engine picks those pages fresh each time. Fewer than half of the pages cited on day one were cited again on day two. When the pages change, the brands change.

    One example: asked for the “easiest CRM for a team that has never used one”, Google AI Mode read Zapier’s list and put Less Annoying CRM first. The next day it read a Slack blog post instead, and Less Annoying CRM wasn’t mentioned at all.

    Who survives depends on how many pages you’re on. A brand ranked #1 was named again the next day 90% of the time. A brand ranked sixth or lower: 61%. Big brands are on most of the pages, so whichever ones the engine reads, they’re there. Smaller brands are on a few, so they come and go.

    • jerf 21 minutes ago
      You are asserting that these changes are somehow tied to time, by the very construction of your experiment in that you ask the same questions sequentially day-by-day. However, that leaves you unable to be sure whether the changes are actually tied to the time because it is not being controlled for, as near as I can tell.

      As near as I can tell from your "Method and Limitations" from the previous day's run, you are only asking a given AI a given question once. You need to ask the same AI the same question a number of times, preferably as close to simultaneously as possible, to see if there's as much variance within a single day as there is in a week.

      As your experiment is constructed, it is not possible for you to collect the data adequate to prove the claim you are trying to make. As I think it is likely that the variance you are seeing is the randomness in the answer process itself and not tied to time at all; two people can probably ask the same question at the same time and get two different answers.

      If nothing else, have an AI go back over your experiment, perhaps feed it this comment as well, and ask it what else may be flawed in your experiment. That doesn't mean you have to adopt every suggestion; every experiment can always gather more data and always have an expanded scope and always cost more money. However this is a more-or-less fatal problem with your experiment and there may be others, this is just all I'm considering. An AI can bang on the problem in a more dedicated fashion than I'm doing here.

    • tetraodonpuffer 26 minutes ago
      have you tried doing the same with web search disabled (to see what is actually in the training, and see if answers vary due to temperature)
  • josefritzishere 14 minutes ago
    [dead]
  • fwip 29 minutes ago
    I don't think "84% of answers changed" is a useful way to describe it. It looks like the content stayed pretty much the same, with sometimes one or two brands switching position or dropping off the list.

    If you asked me what my top six favorite shows were, and then again 2 days later, I'm sure I would have one or two series that were new to the list or changed spots.

    Edit: If this is something you're actually interested in measuring, there are several statistical tools meant for comparing the similarity of rank-ordered lists. I'd probably reach for a rank-biased overlap measure first.