Generative modelling has moved on from being just "next-token" predictors to including systems that are now able to summarise documents, draft replies to customers, generate code, and engage in multi-turn conversations. This development is not simply due to the availability of more data; it is the result of a series of architectural improvements which have enhanced memory, training stability, and compute efficiency. Although you are taking a course on
generative ai course, understanding this history will explain why modern models scale, where their costs arise from, and why new sequence architectures keep reappearing.
RNNs: the first practical sequence generators
Recurrent neural networks (RNNs) were an early mainstay in the field of text generation. They handle a sequence token by token and maintain a hidden state which represents "what has happened so far"; this approach corresponds to the way language develops, but it does so at the cost of two practical limitations.
The first thing to note is that the computation involved in RNNs is naturally sequential since token t relies on token t1; this limits the degree of parallelism and causes training to take longer as the size of the datasets increases. The second issue is that it is difficult to learn long-range dependencies because the gradients tend to vanish or explode when backpropagated over a large number of time steps. In actual text generation, this usually manifested itself as sentences that were locally fluent but which drifted in topic or lost coherence in the case of longer outputs.
LSTMs: gates that made long memory usable
LSTM networks enhanced simple RNNs by introducing gates which determine what information should be written out, stored, and forgotten at each step. Rather than pushing all of the historical information through a single vulnerable hidden state, LSTMs keep a cell state which can pass on information over many steps while the gates regulate the updates. The reason for this design was to make it easier to learn long dependencies and to stabilise training when carrying out sequence tasks.
LSTMs came to be the standard backbone in many generation pipelines since they were reliable and relatively easy to train. However, the fundamental limitation still existed in that recurrence had to process the tokens one at a time. With both the size of the models and the amount of data increasing, this characteristic of LSTMs proceeding step by step became a bottleneck for both the speed of training and for hardware utilisation.
Transformers: self-attention unlocks scalable pretraining
The way attention works has changed sequence modelling by allowing the model to refer directly to the relevant tokens rather than compressing all of them into a single state. Transformers go even further in this regard: they eliminate the use of recurrence and instead depend on self-attention layers together with feed-forward networks, positional encodings being used to capture the order. Since each token can attend to the other tokens it is possible to create context-aware representations and the model is able to process many tokens in parallel when training.
A major factor in Transformers becoming the core of today's large language models is the fact that they work very well with GPUs and TPUs, which allows for large-scale pretraining using self-supervised objectives followed by adaptation to a variety of downstream tasks. The main point in a course on generative AI is that architecture and scaling are related: the types of designs that can be trained efficiently on a large scale tend to be the ones that end up defining the ecosystem.
The result is higher costs when dealing with long contexts. Since standard self-attention is proportional to the square of the sequence length, every token is able to attend to all of the other tokens. When handling long documents, streaming logs, or long audio files, this cost becomes important and therefore encourages research into alternatives that maintain quality while reducing the amount of computation required for long contexts.
State Space Models: linear-time sequence processing returns
State Space Models (SSMs) are derived from control theory and signal processing; they represent sequences by means of a latent state which evolves with time according to learned dynamics and then generates the observations. Since the appropriate structure is used, SSMs are able to capture long-range behaviour using a computation that scales linearly with the length of the sequence, a feature that is advantageous when quadratic attention becomes a bottleneck.
The Structured State Space sequence model (S4) showed that it is possible to make SSMs both efficient and competitive by using parameterisations which allow for fast computation without losing their ability to handle long sequences. More recently, selective SSMs such as Mamba have made state updates dependent on the input, thereby improving adaptivity when dealing with information-dense data such as text while still maintaining linear scaling. SSMs are worth studying in a generative AI course since they provide an alternative efficiency profile compared to attention-based models.
Looking at the issue from a systems point of view, this provides a useful range of options: Transformers continue to serve as strong general-purpose frameworks, whereas SSMs become attractive in cases where you need very long contexts, high inference throughput, or predictable latency. Nowadays, many groups also assess hybrid approaches, combining attention for selectivity with state-space type updates for efficiency.
Conclusion
The progression from RNNs through LSTMs to Transformers and now to SSMs is mainly concerned with striking a balance between expressiveness and efficiency; LSTMs made it possible to handle long dependencies more stably, Transformers allowed for parallel training and for large-scale modern pretraining, and SSMs are once again introducing linear-time sequence modelling for extremely long contexts. A good course on generative AI should present these different approaches as practical design decisions, selecting the appropriate backbone according to your sequence length, latency requirements, data type, and the quality needs of your application.
