Josherich's Blog

HOME SHORTS PODCAST SOFTWARES DRAWING ABOUT RSS

The Case Against Transformer-based AI

30 Jul 2024

Strategy to be right about Transformer-based AI

Or any kind of promising technology.

  • Keep an updated list of evidences that collectively have almost zero chance of missing the trend. This is more effective than the opposite, which is cherrypicking a list of evidences that show the real thing is coming soon (they will not).
  • Focus on the evidences of what’s going on right now or has happened, ignore predictions that are based on or includes any projections of the current ability with a curve of improvement in a short period of time.
  • You only need one strong evidence, and then try really hard to interpret it correctly. One good example would be self driving ability gets a decent jump that’s in the level of previous autoregressive language generation to GPT3 with RLHF. It starts to interpret 99.9% of open roads and make every right decisions in highway situations that could lead to severe crashes. It starts to predict dangerous situations and take actions in advance. Its driving behaviors can be fine-tuned without hurting the general ability, just like LM fine-tuning.
  • Crushing a hard benchmark is not a strong evidence.

https://www.reddit.com/r/math/comments/19fg9rx/some_perspective_on_alphageometry/

AlphaGeometry’s core component, DD+AR, is not AI but a traditional algorithm that deduces geometric relationships using angle-chasing, cyclic quadrilaterals, and similar triangles. The AI aspect is limited to suggesting new points to add when DD+AR fails to solve a problem. Surprisingly, many IMO problems can be solved by DD+AR alone or with minimal AI input.[Claude-3.5]

https://www.sequoiacap.com/article/ais-600b-question

This updates a previous analysis on the gap between AI infrastructure investments and actual revenue growth in the AI ecosystem. The gap has widened significantly, with the “hole” growing from $125B to $500B annually. Despite easier access to GPUs and growing stockpiles, AI revenue remains concentrated primarily with OpenAI, while the market anticipates continued investment in next-generation chips like Nvidia’s B100. The author cautions against the belief in quick riches from AI, highlighting potential issues such as lack of pricing power, rapid depreciation of hardware, and the historical pattern of capital loss during speculative technology waves, while acknowledging that AI will create significant economic value in the long term.[Claude-3.5]

vast.ai supply and demand

Not looking good since June 17, 2024

Labor Productivity labor productivity 2024

Productivity gain boost will be seen, since people claim it is general purpose intelligence.