7 comments

  • woah 37 minutes ago
    The heavy reliance on eval datasets showcased here and also evidenced by the frontier lab's high payments for such data cast doubt on the claims that AI will become self-improving soon
  • xlayn 1 hour ago
    Snap, no gguf to download, will wait for one to appear to test, any improvement over an already good model is good news.

    "18 days on 144 H200 GPUs" insane.. didn't thought so much hardware would be needed to train Qwen.

    • drdaeman 10 minutes ago
      Rather than waiting, ask your assistant to quantize for you. It could be not the best quantization possible but it’s relatively quick, and I had an overall positive experience.
  • jacquesm 21 minutes ago
    I always wonder what you lose when you start maxing out some specific benchmark. It is really hard to do this in a balanced way and I would love if everybody that did these kind of efforts would take the time to also list the before/afters of a number of other benchmarks that they did not use in their training.
    • nblintao 3 minutes ago
      Good point! First of all, none of the benchmark tasks were used in training, though the training data was designed to improve capabilities on tasks like TB4.1. If you consider TB4.1 the main target, the blog actually shows before/after results on several other benchmarks, with no regressions and even some improvements.
    • gardnr 7 minutes ago
      Check the article. It’s the first graphic that has other benchmarks.
  • robertkarl 1 hour ago
    I look forward to testing this against my private eval set to determine if it's benchmaxxing or actual improvement based on just 1200 tasks. downloading it now
  • XCSme 1 hour ago
    Is it post-training or bench-maxxing?
  • octoberfranklin 1 hour ago
    FWIW that is about $331,000 on low-onboarding-grief clouds (e.g. vast.ai) right now using the SXM variants they used.

    You might be able to shave that down using a different flavor of H200 (like PCIe) but it would take more hours, so benefits are unclear.

    • lumost 24 minutes ago
      Savings would likely be larger with alternative accelerators such as trainium, no?
      • octoberfranklin 11 minutes ago
        Savings would likely be larger with magic beans such as floobarbaz, no?

        But seriously. If there were a cheaper and reliable way, I'm sure the authors would have mentioned it instead of H200 SXM modules. They're trying to brag about cheap training.

        Maybe you wanted to use an opportunity to shill tranium.