What is transformer in machine learning?

Introduction to Transformers in Deep Learning

In recent years, Transformer models have completely transformed (literally) the field of Artificial Intelligence, Machine Learning, and Deep Learning. If you have heard names like ChatGPT, BERT, GPT, T5, or LLaMA, all of them are built on a single powerful architecture—the Transformer.

Before Transformers, most sequence-based problems such as language translation, text summarization, speech recognition, and question answering relied on Recurrent Neural Networks (RNNs) and LSTM models. While these models worked well, they had serious limitations when dealing with long sequences and large datasets.

Transformers solved these problems and became the backbone of modern NLP (Natural Language Processing) and Generative AI systems.

This article explains:

• What Transformers are
• Why Transformers were introduced
• How the Transformer architecture works
• Attention and Self-Attention
• Encoder and Decoder structure
• Practical examples
• Python code snippets for implementation

Why Do We Need Transformers?

Problems with RNNs and LSTMs

RNNs and LSTMs process text sequentially, word by word. This causes several issues:

1. Slow training – cannot be parallelized efficiently
2. Long-term dependency issues – information from far-away words is hard to retain
3. High computational cost for long sequences

Example sentence:

“The book that you gave me last summer while we were traveling in Europe was amazing.”

To understand the word “book”, the model must remember context from far earlier in the sentence.
Even LSTMs struggle when sequences become very long.

Key Idea Behind Transformers

The Transformer model introduces a revolutionary idea:

“Attention is all you need.”

Instead of reading sequences step by step, Transformers:

• Look at all words at once
• Decide which words are important
• Capture long-range dependencies efficiently

This makes Transformers:

• Faster to train
• More accurate
• Highly scalable

What Is a Transformer?

A Transformer is a deep learning architecture based entirely on attention mechanisms, without using recurrence (RNN) or convolution (CNN).

It consists of:

• Encoder
• Decoder
• Self-Attention Mechanism
• Positional Encoding

High-Level Transformer Architecture

Input Text → Encoder → Decoder → Output Text

• Encoder: Understands the input sequence
• Decoder: Generates the output sequence

Example:

• Input: English sentence
• Output: French translation

Understanding Attention Mechanism

What Is Attention?

Attention allows the model to focus on important words while processing a sentence.

Example:

“The animal didn’t cross the street because it was tired.”

What does “it” refer to?

• The animal

Attention helps the model connect these words correctly.

Self-Attention Explained Simply

Self-attention means:

• Each word looks at other words in the same sentence
• Assigns importance scores
• Builds a contextual representation

Every word asks:

• “Which other words should I pay attention to?”

Mathematical Intuition (Simplified)

Each word is converted into:

• Query (Q)
• Key (K)
• Value (V)

Attention formula:

Attention(Q, K, V) = softmax(QKᵀ / √d) × V

This computes:

• Similarity between words
• Importance weights
• Context-aware word embeddings

Multi-Head Attention

Instead of one attention mechanism, Transformers use multiple attention heads.

Why?

Each head learns different relationships:

• Grammar
• Meaning
• Word dependencies
• Position

This improves understanding of complex language patterns.

Positional Encoding

Transformers do not process sequences in order, so they need a way to understand word position.

Solution: Positional Encoding

• Adds position information to word embeddings
• Uses sine and cosine functions

This allows the model to know:

• Word order
• Relative positions

---

Transformer Encoder Explained

The Encoder consists of multiple identical layers.

Each encoder layer has:

1. Multi-head self-attention
2. Add & Normalize
3. Feed-forward neural network
4. Add & Normalize

Encoder Purpose

• Converts input text into rich contextual representations

---

Transformer Decoder Explained

The Decoder also has multiple layers.

Each decoder layer includes:

1. Masked self-attention
2. Encoder-decoder attention
3. Feed-forward network

Decoder Purpose

• Generates output one word at a time
• Uses both past output and encoder context

---

Real-World Applications of Transformers

Transformers power most modern AI systems:

1. Natural Language Processing (NLP)

• Machine Translation
• Text Summarization
• Sentiment Analysis
• Question Answering

2. Generative AI

• ChatGPT
• Code generation
• Content creation

3. Computer Vision

• Vision Transformers (ViT)
• Image classification

4. Speech Processing

• Speech-to-text
• Voice assistants

Simple Transformer Implementation (PyTorch)

Below is a basic Transformer model example using PyTorch.

Step 1: Import Libraries

import torch
import torch.nn as nn

---

Step 2: Define Transformer Model

class TransformerModel(nn.Module):
    def __init__(self, vocab_size, embed_dim, num_heads, hidden_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.transformer = nn.Transformer(
            d_model=embed_dim,
            nhead=num_heads,
            num_encoder_layers=2,
            num_decoder_layers=2
        )
        self.fc = nn.Linear(embed_dim, vocab_size)

    def forward(self, src, tgt):
        src = self.embedding(src)
        tgt = self.embedding(tgt)
        output = self.transformer(src, tgt)
        return self.fc(output)

---

Step 3: Initialize Model

vocab_size = 5000
embed_dim = 256
num_heads = 8
hidden_dim = 512

model = TransformerModel(vocab_size, embed_dim, num_heads, hidden_dim)

---

Step 4: Forward Pass

src = torch.randint(0, vocab_size, (10, 32))
tgt = torch.randint(0, vocab_size, (10, 32))

output = model(src, tgt)
print(output.shape)

Transformer Implementation Using Hugging Face (Practical)

For real projects, we use pre-trained Transformers.

Example: BERT Text Classification

from transformers import BertTokenizer, BertForSequenceClassification
import torch

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForSequenceClassification.from_pretrained("bert-base-uncased")

inputs = tokenizer(
    "Transformers are powerful models",
    return_tensors="pt"
)

outputs = model(**inputs)
print(outputs.logits)

---

Advantages of Transformers

• Parallel processing
• Handles long-term dependencies
• Highly scalable
• State-of-the-art performance
• Works across text, images, and audio

---

Limitations of Transformers

• Requires large datasets
• High computational cost
• Memory-intensive
• Needs careful optimization

---

Transformers vs LSTM (Quick Comparison)

Feature	LSTM	Transformer
Sequence Processing	Sequential	Parallel
Long Dependencies	Limited	Excellent
Training Speed	Slow	Fast
Scalability	Medium	High

---

Are Transformers the Future of AI?

Absolutely. Transformers are:

• Foundation of Large Language Models (LLMs)
• Core of Agentic AI systems
• Used in search engines, chatbots, recommendation systems

Understanding Transformers is essential for any student pursuing:

• Machine Learning
• Artificial Intelligence
• Data Science

---

Conclusion

The Transformer model is one of the most important breakthroughs in Deep Learning and Artificial Intelligence. By replacing recurrence with attention mechanisms, Transformers achieved unmatched performance in handling sequential data.