Mastering Transformers: Build state-of-the-art models from scratch with advanced natural language processing techniques

A few other key words that come to mind to describe this book: Foundational, Hands-on, Practical, Crisp, Concise, Depth & Breadth, Tremendous Value.With the continued accelerating explosion in the growth of unstructured data collected by enterprises in texts and documents – the need to be able to analyze and derive meaningful information is more critical than ever – and will be the competitive advantage that distinguishes future winners from losers in the marketplace of solutions. This book is an investment in expanding your awareness of the techniques and capabilities that will help you navigate those challenges.From the book: “Transformer models have gained immense interest because of their effectiveness in all NLP tasks, from text classification to text generation….[and] effectively improve the performance of multilingual and multi-task NLP problems, as well as monolingual and single tasks.”This book is a practical guide to leveraging (and applying) some of the leading-edge concepts, algorithms, and libraries from the fields of Deep Learning (DL) and Natural Language Processing (NLP) to solve real-world problems – ranging from summarization to question-answering.In particular, this book will serve as a gentle guided tour of some of the important advances that have occurred (and continue to evolve) as the transformer architecture gradually evolved into an attention-based encoder-decoder architecture.What I particularly liked:The deep subject-matter experience and credentials of the authors (“Savaş Yıldırım graduated from the Istanbul Technical University Department of Computer Engineering and holds a Ph.D. degree in Natural Language Processing (NLP). Currently, he is an associate professor at the Istanbul Bilgi University, Turkey, and is a visiting researcher at the Ryerson University, Canada. He is a proactive lecturer and researcher with more than 20 years of experience teaching courses on machine learning, deep learning, and NLP.”, “Meysam Asgari-Chenaghlu is an AI manager at Carbon Consulting and is also a Ph.D. candidate at the University of Tabriz.”)The companion “Code In Action” YouTube channel playlist for the book, and the GitHub repository with code examples.The excellent quality/conciseness/crispness of the writing.The extensive citation of relevant research papers – and references at the end of chapters.The authors’ deep practical knowledge – and discussions – of the advantages and disadvantages of different approaches.The exquisitely balanced need for technical depth in the details covered by a given chapter – with the need to maintain a steady pace of educating & keeping the reader engaged. Some books go too deep, and some stay too shallow. This book is exceptionally well balanced at just the right depth.The exceptional variety of examples covered.The quality of the illustrations used to convey complex concepts – Figures 1.19, 3.2, 3.3, 7.8, 9.3 are just a few examples of the many good diagrams.Chapter-1’s focus on getting the reader immediately involved in executing a hello-world example with Transformers. The overview of RNNs, FFNNs, LSTMs, and CNNs. An excellent overview of the developments in NLP over the last 10 years that led to the Tranformer architecture.Chapter-2’s guidance on installing the required software – and the suggestion of Google Colab as an alternative to Anaconda.Chapter-2’s coverage of community-provided models, benchmarks, TensorFlow, PyTorch, and Transformer - and running a simple Transformer from scratch.Chapter-3’s coverage of BERT – as well as ALBERT, RoBERTa, and ELECTRA.Chapter-4’s coverage of AR, GPT, BART, and NLG.Chapter-5’s coverage of fine-tuning language models for text classification (e.g., for sentiment analysis, or multi-class classification).Chapter-6’s coverage of NER and POS was of particular interest – given the effort that I had to expend last year doing my own deep-dive to prepare some recommendations for a client – I wish I had had this book then.Chapter-7’s coverage of USE and SBERT, zero-shot learning with BART, and FLAIR.Chapter-8’s discussion of efficient sparse parsers (Linformer, and BigBird) – as well as the techniques of distillation, pruning, and quantization – to make efficient models out of trained models. Chapter-8 may well be worth the price of the book, itself.Chapter-9’s coverage of multilingual and cross-lingual language model training (and pretraining). I found the discussion of “Cross-lingual similarity tasks” (see p-278) to be particularly interesting.Chapter-10’s coverage of Locust for load testing, fastAPI, and TensorFlow Extended (TFX) – as well as the serving of solutions in environments where CPU/GPU is available.Chapter-11’s coverage of visualization with exBERT and BertViz – as well as the discussion on tracking model training with TensorBoard and W&BThe ”Other Books You May Enjoy” section at the end of the book (“Getting Started with Google BERT”, and “Mastering spaCy”)Suggestions for the next edition:The fonts used for the text in some figures (e.g., 3.8, 3.10, 3.12, 3.13, 3.14, 4.5, 4.6, 6.2, 6.7, 8.4, 8.6, 9.4, 9.5 ) appear to be a bit fuzzy in the PDF version of the book. Compare those with the clarity of figure 6.6.

The first part of the book is a long introduction (around 80 pages). The book starts with a review and some code related to the early concepts that led to Transformers (e.g., PCA, bag-of-words, embeddings, RNNs, LSTMs) and jumps right into the attention mechanism and the Transformer architecture in several pages. The architecture is explained through various pictures and visualizations in order to help people understand why this model works so well. The few references included at the end of various chapters can always provide the additional details. While the first chapter is mostly theoretical, the following chapter does contain the gritty details related to progamming - from installing the various libraries to various tricks needed in order to make certain models work for solving a particular problem. The author's habits of adding good images everywhere helps a lot in this endeavour.The second part of the book is focused on two big issues: a) understanding the architectural differences between autoencoding models (e.g., BERT) and autoregressive models (e.g., BERT); and b) applying such models for classification tasks (e.g., text or token classification). This is the part in which the authors teach you how to fine-tune such models.The last part of the book is dedicated to advanced topics like efficient Transforms, multilingual models or production-related issues (e.g., serving Transformers, visualizing their attention layers or outputs, etc.). This is the part that links directly to advanced research, not unlike the kind of research that was submitted to top conferences in the field a couple of years ago (e.g., up to 2019, there were hardly any Transformer visualizations!).I particularly liked the ease with which the authors jumped straight into a difficult subject and teached it up to research level. There are very few books able to do this (e.g., thinking of the Dive into Deep Learning books or Grokking Deep Learning). It is great to see publishers introducing such books extremely fast (e.g., it took 5 years to have the first D3.js book published, but only 2 years to have a book about BERT or GPT-2/3). Make no mistake, while the title is still about Transformers, the real stars are the newer models discussed here. I consider this a five star book and I would recommend you to read it along the various examples of Transformer / BERT / GPT-2 explained source code that are available online. You will thank yourself later!

I love that book! It gives me the introduction and understanding on the very important topic that is Transformers. The book gives a bit of history and the current standard implementation of transformers. The book also delves in the advanced topic of Attention algorithms. This book is a "must have" for the seasoned AI partitioner of Language Model Transformers.

Currently, I have more experience as a data engineer but Machine Learning(ML) and Deep Learning(DL) are not some of my strangers. However, I have an attraction for NLP so that I had the opportunity to read the Mastering Transformers.This is a fantastic book all in one about Transformers. Transformers are the principal NLP state of the art. Very futuristic. The principal features of the book are to give good examples step by step of which I can mention BERT the most popular Transformer model by Google, another nice feature is that you can learn the origins of NLP, the traditional techniques, ML and DL evolution.You can find information about the principal BERT variants like ALBERT, ELECTRA, RoBERTa. I missed maybe some examples or information about BioBERT and ClinicalBERT for medical domains and medical apps.Finally, there is one of my favorites topics about ML. How to deploy and serving your model to production for real or experimental APPs.

To start with this book is really comprehensive in its coverage of various transformers models like autoencoding models(BERT ,ROBERTa, etc.) and autoregressive models(GPT, T5, etc.), serving models, etc. with relevant code snippets and use cases to play around with.Besides that most of the code examples and text in the book appears to be straight out picked from hugging face website, and in some cases adapted for their example datasets.Therefore, this book will be a great resource for beginners or someone new to transformers models or huggingface website. However, for people who are somewhat familiar with transformers models or familiar with huggingface models, I don't think this book to be a great resource for them to spend money on, since this book appears to be just a curation of freely available articles or code snippets available on huggingface website or other blogs.On the finishing note for the people starting out in NLP or transformers, or for the people who want to save on trouble of going through articles on hugginface or across the web, I will suggest this book as a great resource. However, for the people familiar with huggingface models or who have some experience using them, I think they can freely find the information presented in the book.