A short history of AI, part 3: transformers and large language models (2017–2026)

Created on October 22, 2026

2026   ·   artificial-intelligence   history-of-science   large-language-models   deep-learning   ·   notes

Part 1 followed AI from Alan Turing to Deep Blue, and Part 2 from Deep Blue to the Transformer, the attention-based network that Google researchers proposed in June 2017. This last part covers the nine years since then. They contain more change than the previous eighty: language models grew from a research curiosity into tools used by more than a billion people, and from tools into agents that write and run code on their own. It ends with the models of late 2026, including Claude Opus 5.5 and Gemini 4 Argon. After this, the blog turns back to the Earth for a while.

A timeline of AI from 2017 to 2026

Figure 1. Milestones in AI from 2017 to 2026, discussed in this post. Rows are evenly spaced rather than drawn to scale in time. Figure by Daeha Kim, based on the sources cited below.

Pre-training: learning language by predicting words (2017–2019)

The Transformer was built for translation, but its real impact came from a different use. In 2018 OpenAI trained a Transformer on a large collection of books simply to predict the next word in a sentence, and then fine-tuned it on specific tasks; this “generative pre-training” (the GPT in ChatGPT) improved results on most of the language-understanding tasks the authors tried (Radford et al. 2018). A few months later Google’s BERT learned by filling in randomly hidden words using the context on both sides, and set new records on a wide range of benchmarks (Devlin et al. 2019).

The idea behind both is that predicting text well requires knowing a lot about the world the text describes. Nobody labels the data: the next word is its own answer, so any text on the internet becomes training material. GPT-2, trained on millions of web pages, could write passages of startling fluency and perform some tasks it had never been trained on, simply because they appeared in its training text (Radford et al. 2019).

The scaling era (2020–2022)

In January 2020 a group at OpenAI reported that the error of a language model falls as a smooth power law as the model, the data and the computing budget grow, with some trends holding over more than seven orders of magnitude (Kaplan et al. 2020). For a field used to unpredictable progress, this was a remarkable finding: it said that bigger would reliably be better, and by how much. Four months later GPT-3, with 175 billion parameters, showed what that meant in practice. Given just a few examples of a task in its prompt, without any further training, it could translate, answer questions or write simple code (Brown et al. 2020).

The race that followed is visible in Figure 2. Model size grew about 2,500-fold in five years, reaching 540 billion parameters with Google’s PaLM (Chowdhery et al. 2022). Then DeepMind showed that most of these models had been trained on too little data for their size: Chinchilla, with 70 billion parameters but four times more training text, outperformed models several times larger (Hoffmann et al. 2022). The recipe changed from “make it bigger” to “balance size and data.”

Number of parameters of landmark language models, 2017–2023

Figure 2. Number of parameters of landmark language models, as reported in the papers that introduced them. Green points are models deliberately trained on more data relative to their size. From 2023 most developers of frontier models stopped publishing model size, so the recent models cannot be plotted. Figure by Daeha Kim, drawn from numbers reported in Vaswani et al. (2017), Devlin et al. (2019), Radford et al. (2019), Brown et al. (2020), Chowdhery et al. (2022), Hoffmann et al. (2022) and Touvron et al. (2023).

The empty right side of Figure 2 is itself part of the history. As language models became products, the information that would let outsiders judge them (size, training data, compute) largely disappeared from public view. The GPT-4 technical report, for example, gives no parameter count (OpenAI 2023). Since then, most of what we know about frontier models comes from benchmarks, which, as my post on vibe coding discussed, have their own problems.

From predicting text to answering people (2022–2023)

A model trained only to continue text is not an assistant. Asked a question, it may continue with more questions. The step that turned language models into assistants was fine-tuning them on human preferences: people wrote examples of good answers and ranked the model’s outputs, and the model was then trained to produce the kind of answers people preferred, a method known as reinforcement learning from human feedback. In OpenAI’s InstructGPT study, people preferred the answers of a 1.3-billion-parameter model trained this way to those of the 175-billion-parameter GPT-3 (Ouyang et al. 2022).

On 30 November 2022 OpenAI released ChatGPT, a conversational model trained with the same methods, as a free research preview (OpenAI 2022). An analyst note estimated that it reached 100 million monthly users by January 2023, which would make it the fastest-growing consumer application recorded at the time (Hu 2023). Within a year the field had changed shape:

  • GPT-4 (14 March 2023) handled images as well as text and, according to OpenAI, scored around the top 10% of test takers on a simulated bar exam (OpenAI 2023).
  • Claude, Anthropic’s first public model, was released the same day (Anthropic 2023).
  • LLaMA (February 2023) and Llama 2 (July 2023) from Meta made capable models available as open weights, and Llama 2 was free for research and commercial use (Touvron et al. 2023; Meta 2023). An ecosystem of open models grew around them.
  • Gemini (December 2023) was Google’s first model designed from the start to work with text, images, audio and video together (Google 2023).

Thinking longer, and acting (2024–2025)

Researchers had noticed that language models solve harder problems when prompted to write out their reasoning step by step, a technique called chain-of-thought prompting (Wei et al. 2022). In September 2024 OpenAI’s o1 built this into training: the model was trained with reinforcement learning to produce a long internal chain of reasoning before answering, and it performed far better on mathematics and programming problems (OpenAI 2024). In January 2025 the Chinese company DeepSeek released R1, a reasoning model trained in a similar way, with its weights openly available; its Nature paper reported that the reinforcement-learning stage cost about US$294,000 in rented computing time, although that excludes the much larger cost of training the base model it started from (Guo et al. 2025). Reasoning was no longer the preserve of a few companies.

The next step was to let models act. Instead of answering a question, an agent is given a goal and tools (a terminal, a browser, a code editor) and works toward the goal over many steps, checking its own results. Anthropic released Claude Code, an agent that reads, writes and runs code in a developer’s own environment, for general use in May 2025 (Anthropic 2025), and similar tools followed from other companies. METR’s measurements, discussed in my earlier post on writing and coding with LLMs, suggest that the length of task such agents can complete on their own has been doubling every few months (Kwa et al. 2025). The vibe coding of my previous post is a direct product of this period.

AI for science, and for the Earth

The same years brought AI into the natural sciences, in ways that matter more to a hydrology lab than any chatbot. AlphaFold 2 predicted the three-dimensional structure of proteins from their amino-acid sequences with accuracy competitive with experiments, solving a problem that had been open for fifty years (Jumper et al. 2021). In 2024 the Nobel Prize in Chemistry went half to David Baker for computational protein design and half to Demis Hassabis and John Jumper for protein structure prediction, and the Nobel Prize in Physics to John Hopfield and Geoffrey Hinton for foundational work that enabled machine learning with artificial neural networks (Nobel Prize Outreach 2024a, 2024b).

Weather forecasting was transformed next. GraphCast, trained on decades of reanalysis data, produced ten-day global forecasts in under a minute and outperformed the world’s leading operational deterministic forecast on 90% of 1,380 verification targets (Lam et al. 2023). Its probabilistic successor GenCast beat the European Centre for Medium-Range Weather Forecasts’ ensemble on about 97% of 1,320 targets (Price et al. 2025).

Hydrology moved too. A single LSTM network trained on 531 catchments at once outperformed conceptual models calibrated for each catchment individually, and learned catchment similarities that matched hydrological understanding (Kratzert et al. 2019). In 2024 Grey Nearing and colleagues showed that a global AI model could predict extreme floods in ungauged basins up to five days ahead with reliability similar to or better than the current global system’s same-day nowcasts, and the model now feeds a free public flood-warning service in more than 80 countries (Nearing et al. 2024). The question that Hsu, Gupta and Sorooshian raised in 1995, which Part 1 described, is now being asked of operational systems.

Where things stand in late 2026

The pace has not slowed. In June 2026 Anthropic released Claude Fable 5, which it described as state-of-the-art on nearly all the benchmarks it tested (Anthropic 2026a). On 22 September came Claude Opus 5.5, which Anthropic says performs at the level of its top model, Claude Fable 5.1, on most work while costing 40% less to run than Opus 5 (Anthropic 2026b). On 30 September Google announced Gemini 4 Argon, its new frontier model, reporting a state-of-the-art score on the DeepSWE software-engineering benchmark; it reached the Gemini app on 13 October (Google 2026). OpenAI’s GPT-5 family continued in parallel, and in August 2026 OpenAI described ChatGPT as having more than a billion weekly active users (OpenAI 2026).

All of these claims come from the companies themselves, and none of the models’ sizes, training data or full evaluations are public. That is the honest summary of where the story stands: the systems are more capable than ever, more widely used than any previous technology of their kind, and harder for outsiders to examine than at any point in this history.

What ninety years teach

Turing’s question has not gone away. In 1950 Turing replaced “can machines think?” with a test of behavior. Today’s models pass versions of that test daily, and the argument has moved, exactly as it did after Deep Blue, to what passing the test does and does not show.

Rules lost to learning, but not completely. The learning tradition that began with McCulloch, Pitts and Rosenblatt won decisively. Yet the most useful systems of 2026 are learned models that use rule-based tools: they call calculators, run tests and follow procedures. The two traditions of Part 1 have merged.

Data and computing power decided the timing. Almost every key idea in this series was older than the moment it worked. What changed, again and again, was the amount of data and computing power available to make it work at scale.

Measurement is the bottleneck. From the imitation game to ImageNet to today’s agent benchmarks, each era has had to invent a way to measure progress, and each measure has eventually been outgrown or gamed. For hydrologists, who have argued about model evaluation for decades, this is familiar ground, and it is where our field can contribute.

Bottom line

  • From 2018, pre-training on large amounts of text, with no labels, became the basis of language AI.
  • Scaling laws (2020) made progress predictable; models grew about 2,500-fold in five years until Chinchilla showed that data mattered as much as size.
  • Fine-tuning on human preferences turned language models into assistants; ChatGPT (2022) brought them to the public, followed by GPT-4, Claude, LLaMA and Gemini in 2023.
  • Reasoning models (2024–2025) and coding agents (2025) extended what the models can do on their own.
  • AI transformed protein science and weather forecasting and is entering operational flood forecasting; the 2024 Nobel Prizes recognized both the methods and their scientific use.
  • In late 2026 the frontier includes Claude Opus 5.5 and Gemini 4 Argon, but most of what we know about such models comes from their developers.

Questions to think about

  1. Predicting the next word turned out to require a great deal of knowledge about the world. Is predicting tomorrow’s streamflow from yesterday’s weather the same kind of problem? What would a “pre-trained” hydrological model learn?
  2. Model developers stopped publishing model sizes in 2023. What would you need to know about an AI model before trusting it in an operational flood warning system?
  3. Looking back over the three parts of this series, which idea do you think took longest to go from proposal to practical use, and why?

한 줄 요약 (Korean summary). 인공지능 역사 연재의 마지막 편으로, 트랜스포머 이후 지금까지(2017–2026)를 다룬다. 2018년부터 다음 단어를 예측하도록 대량의 텍스트로 미리 학습시키는 방식(GPT, BERT)이 언어 AI의 기반이 되었다. 정답 라벨이 필요 없어 인터넷의 모든 글이 학습 자료가 되었다. 2020년 스케일링 법칙은 모형 크기, 데이터, 계산량이 늘수록 오차가 예측 가능하게 줄어든다는 것을 보였고, 언어모형은 5년 만에 약 2,500배 커졌다. 그러다 친칠라가 데이터의 양이 크기만큼 중요하다는 것을 보였고, 2023년부터는 대부분의 개발사가 모형 크기를 공개하지 않게 되었다. 사람의 선호로 미세조정하는 방법(RLHF)이 언어모형을 대화형 비서로 바꾸었고, 2022년 11월 ChatGPT가 공개된 뒤 2023년 GPT-4, Claude, LLaMA, Gemini가 잇따랐다. 2024–2025년에는 답하기 전에 길게 추론하는 모형(o1, DeepSeek-R1)과 스스로 코드를 쓰고 실행하는 에이전트(Claude Code)가 나왔다. 같은 시기 AI는 단백질 구조 예측(알파폴드, 2024년 노벨 화학상)과 날씨 예보(GraphCast, GenCast)를 바꾸었고, 미계측 유역의 홍수를 5일 앞서 예측해 80여 개국에 무료 경보를 제공하는 수준에 이르렀다. 2026년 9월에는 Claude Opus 5.5와 Gemini 4 Argon이 나왔지만, 이런 성능 주장은 대부분 개발사 자체 발표이고 모형의 크기와 학습 자료는 공개되지 않고 있다. 90년의 역사가 남긴 교훈은 튜링의 질문이 여전히 유효하다는 것, 규칙과 학습의 두 전통이 결국 합쳐졌다는 것, 아이디어가 실현되는 시점을 정한 것은 데이터와 계산 능력이었다는 것, 그리고 발전을 어떻게 측정할 것인가가 여전히 가장 어려운 문제라는 것이다.

References

Anthropic (2023). Introducing Claude. Company announcement, 14 March 2023. https://www.anthropic.com/news/introducing-claude

Anthropic (2025). Introducing Claude 4. Company announcement, 22 May 2025. https://www.anthropic.com/news/claude-4

Anthropic (2026a). Claude Fable 5 and Claude Mythos 5. Company announcement, 9 June 2026. https://www.anthropic.com/news/claude-fable-5-mythos-5

Anthropic (2026b). Introducing Claude Opus 5.5. Company announcement, 22 September 2026. https://www.anthropic.com/claude-opus-5-5

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., et al. (2020). Language models are few-shot learners. arXiv:2005.14165 (NeurIPS 2020). https://doi.org/10.48550/arXiv.2005.14165

Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., et al. (2022). PaLM: Scaling language modeling with pathways. arXiv:2204.02311. https://doi.org/10.48550/arXiv.2204.02311

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 4171–4186. https://doi.org/10.18653/v1/N19-1423

Google (2023). Introducing Gemini: our largest and most capable AI model. Company announcement, 6 December 2023. https://blog.google/technology/ai/google-gemini-ai/

Google (2026). Gemini 4 Argon: our next era of frontier intelligence. Company announcement by K. Kavukcuoglu, 30 September 2026. https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-4-argon/

Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., et al. (2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081), 633–638. https://doi.org/10.1038/s41586-025-09422-z

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., et al. (2022). Training compute-optimal large language models. arXiv:2203.15556. https://doi.org/10.48550/arXiv.2203.15556

Hu, K. (2023). ChatGPT sets record for fastest-growing user base: analyst note. Reuters, 1 February 2023. https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/

Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873), 583–589. https://doi.org/10.1038/s41586-021-03819-2

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., et al. (2020). Scaling laws for neural language models. arXiv:2001.08361. https://doi.org/10.48550/arXiv.2001.08361

Kratzert, F., Klotz, D., Shalev, G., Klambauer, G., Hochreiter, S., & Nearing, G. (2019). Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets. Hydrology and Earth System Sciences, 23(12), 5089–5110. https://doi.org/10.5194/hess-23-5089-2019

Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025). Measuring AI ability to complete long software tasks. arXiv:2503.14499. https://doi.org/10.48550/arXiv.2503.14499

Lam, R., Sanchez-Gonzalez, A., Willson, M., Wirnsberger, P., Fortunato, M., Alet, F., et al. (2023). Learning skillful medium-range global weather forecasting. Science, 382(6677), 1416–1421. https://doi.org/10.1126/science.adi2336

Meta (2023). Meta and Microsoft introduce the next generation of Llama. Company announcement, 18 July 2023. https://ai.meta.com/blog/llama-2/

Nearing, G., Cohen, D., Dube, V., Gauch, M., Gilon, O., Harrigan, S., et al. (2024). Global prediction of extreme floods in ungauged watersheds. Nature, 627(8004), 559–563. https://doi.org/10.1038/s41586-024-07145-1

Nobel Prize Outreach (2024a). The Nobel Prize in Chemistry 2024. https://www.nobelprize.org/prizes/chemistry/2024/summary/

Nobel Prize Outreach (2024b). The Nobel Prize in Physics 2024. https://www.nobelprize.org/prizes/physics/2024/summary/

OpenAI (2022). Introducing ChatGPT. Company announcement, 30 November 2022. https://openai.com/index/chatgpt/

OpenAI (2023). GPT-4 technical report. arXiv:2303.08774. https://doi.org/10.48550/arXiv.2303.08774

OpenAI (2024). Learning to reason with LLMs. Company announcement, 12 September 2024. https://openai.com/index/learning-to-reason-with-llms/

OpenAI (2026). A milestone in expanding access to AI. Company announcement, 31 August 2026. https://openai.com/index/expanding-access-to-ai-with-chatgpt-ads/

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155 (NeurIPS 2022). https://doi.org/10.48550/arXiv.2203.02155

Price, I., Sanchez-Gonzalez, A., Alet, F., Andersson, T. R., El-Kadi, A., Masters, D., et al. (2025). Probabilistic weather forecasting with machine learning. Nature, 637(8044), 84–90. https://doi.org/10.1038/s41586-024-08252-9

Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. OpenAI technical report. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI technical report. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., et al. (2023). LLaMA: Open and efficient foundation language models. arXiv:2302.13971. https://doi.org/10.48550/arXiv.2302.13971

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. arXiv:1706.03762. https://doi.org/10.48550/arXiv.1706.03762

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. arXiv:2201.11903 (NeurIPS 2022). https://doi.org/10.48550/arXiv.2201.11903

Both figures are original. The Python scripts that produce them are available on GitHub: Figure 1 and Figure 2.