Mastering Transformers: Build state-of-the-art models from scratch with advanced natural language processing techniques

What do you get with Print?

Instant access to your digital copy whilst your Print order is Shipped

Paperback book shipped to your preferred address

Redeem a companion digital copy on all Print orders

Access this title in our online reader with advanced features

DRM FREE - Read whenever, wherever and however you want

AI Assistant (beta) to help accelerate your learning

Key benefits

Explore quick prototyping with up-to-date Python libraries to create effective solutions to industrial problems

Solve advanced NLP problems such as named-entity recognition, information extraction, language generation, and conversational AI

Monitor your model's performance with the help of BertViz, exBERT, and TensorBoard

Description

Transformer-based language models have dominated natural language processing (NLP) studies and have now become a new paradigm. With this book, you'll learn how to build various transformer-based NLP applications using the Python Transformers library. The book gives you an introduction to Transformers by showing you how to write your first hello-world program. You'll then learn how a tokenizer works and how to train your own tokenizer. As you advance, you'll explore the architecture of autoencoding models, such as BERT, and autoregressive models, such as GPT. You'll see how to train and fine-tune models for a variety of natural language understanding (NLU) and natural language generation (NLG) problems, including text classification, token classification, and text representation. This book also helps you to learn efficient models for challenging problems, such as long-context NLP tasks with limited computational capacity. You'll also work with multilingual and cross-lingual problems, optimize models by monitoring their performance, and discover how to deconstruct these models for interpretability and explainability. Finally, you'll be able to deploy your transformer models in a production environment. By the end of this NLP book, you'll have learned how to use Transformers to solve advanced NLP problems using advanced models.

What you will learn

Explore state-of-the-art NLP solutions with the Transformers library

Train a language model in any language with any transformer architecture

Fine-tune a pre-trained language model to perform several downstream tasks

Select the right framework for the training, evaluation, and production of an end-to-end solution

Get hands-on experience in using TensorBoard and Weights & Biases

Visualize the internal representation of transformer models for interpretability

What do you get with Print?

Instant access to your digital copy whilst your Print order is Shipped

Paperback book shipped to your preferred address

Redeem a companion digital copy on all Print orders

Access this title in our online reader with advanced features

DRM FREE - Read whenever, wherever and however you want

AI Assistant (beta) to help accelerate your learning

Frequently bought together

Transformers for Natural Language Processing

$89.99

Machine Learning for Time-Series with Python

$54.99

$54.99

Total $ 199.97

Kelvin D. Meeks Sep 15, 2021

A few other key words that come to mind to describe this book: Foundational, Hands-on, Practical, Crisp, Concise, Depth & Breadth, Tremendous Value.With the continued accelerating explosion in the growth of unstructured data collected by enterprises in texts and documents – the need to be able to analyze and derive meaningful information is more critical than ever – and will be the competitive advantage that distinguishes future winners from losers in the marketplace of solutions. This book is an investment in expanding your awareness of the techniques and capabilities that will help you navigate those challenges.From the book: “Transformer models have gained immense interest because of their effectiveness in all NLP tasks, from text classification to text generation….[and] effectively improve the performance of multilingual and multi-task NLP problems, as well as monolingual and single tasks.”This book is a practical guide to leveraging (and applying) some of the leading-edge concepts, algorithms, and libraries from the fields of Deep Learning (DL) and Natural Language Processing (NLP) to solve real-world problems – ranging from summarization to question-answering.In particular, this book will serve as a gentle guided tour of some of the important advances that have occurred (and continue to evolve) as the transformer architecture gradually evolved into an attention-based encoder-decoder architecture.What I particularly liked:The deep subject-matter experience and credentials of the authors (“Savaş Yıldırım graduated from the Istanbul Technical University Department of Computer Engineering and holds a Ph.D. degree in Natural Language Processing (NLP). Currently, he is an associate professor at the Istanbul Bilgi University, Turkey, and is a visiting researcher at the Ryerson University, Canada. He is a proactive lecturer and researcher with more than 20 years of experience teaching courses on machine learning, deep learning, and NLP.”, “Meysam Asgari-Chenaghlu is an AI manager at Carbon Consulting and is also a Ph.D. candidate at the University of Tabriz.”)The companion “Code In Action” YouTube channel playlist for the book, and the GitHub repository with code examples.The excellent quality/conciseness/crispness of the writing.The extensive citation of relevant research papers – and references at the end of chapters.The authors’ deep practical knowledge – and discussions – of the advantages and disadvantages of different approaches.The exquisitely balanced need for technical depth in the details covered by a given chapter – with the need to maintain a steady pace of educating & keeping the reader engaged. Some books go too deep, and some stay too shallow. This book is exceptionally well balanced at just the right depth.The exceptional variety of examples covered.The quality of the illustrations used to convey complex concepts – Figures 1.19, 3.2, 3.3, 7.8, 9.3 are just a few examples of the many good diagrams.Chapter-1’s focus on getting the reader immediately involved in executing a hello-world example with Transformers. The overview of RNNs, FFNNs, LSTMs, and CNNs. An excellent overview of the developments in NLP over the last 10 years that led to the Tranformer architecture.Chapter-2’s guidance on installing the required software – and the suggestion of Google Colab as an alternative to Anaconda.Chapter-2’s coverage of community-provided models, benchmarks, TensorFlow, PyTorch, and Transformer - and running a simple Transformer from scratch.Chapter-3’s coverage of BERT – as well as ALBERT, RoBERTa, and ELECTRA.Chapter-4’s coverage of AR, GPT, BART, and NLG.Chapter-5’s coverage of fine-tuning language models for text classification (e.g., for sentiment analysis, or multi-class classification).Chapter-6’s coverage of NER and POS was of particular interest – given the effort that I had to expend last year doing my own deep-dive to prepare some recommendations for a client – I wish I had had this book then.Chapter-7’s coverage of USE and SBERT, zero-shot learning with BART, and FLAIR.Chapter-8’s discussion of efficient sparse parsers (Linformer, and BigBird) – as well as the techniques of distillation, pruning, and quantization – to make efficient models out of trained models. Chapter-8 may well be worth the price of the book, itself.Chapter-9’s coverage of multilingual and cross-lingual language model training (and pretraining). I found the discussion of “Cross-lingual similarity tasks” (see p-278) to be particularly interesting.Chapter-10’s coverage of Locust for load testing, fastAPI, and TensorFlow Extended (TFX) – as well as the serving of solutions in environments where CPU/GPU is available.Chapter-11’s coverage of visualization with exBERT and BertViz – as well as the discussion on tracking model training with TensorBoard and W&BThe ”Other Books You May Enjoy” section at the end of the book (“Getting Started with Google BERT”, and “Mastering spaCy”)Suggestions for the next edition:The fonts used for the text in some figures (e.g., 3.8, 3.10, 3.12, 3.13, 3.14, 4.5, 4.6, 6.2, 6.7, 8.4, 8.6, 9.4, 9.5 ) appear to be a bit fuzzy in the PDF version of the book. Compare those with the clarity of figure 6.6.

Amazon Verified review

rockstar82 Nov 14, 2021

The first part of the book is a long introduction (around 80 pages). The book starts with a review and some code related to the early concepts that led to Transformers (e.g., PCA, bag-of-words, embeddings, RNNs, LSTMs) and jumps right into the attention mechanism and the Transformer architecture in several pages. The architecture is explained through various pictures and visualizations in order to help people understand why this model works so well. The few references included at the end of various chapters can always provide the additional details. While the first chapter is mostly theoretical, the following chapter does contain the gritty details related to progamming - from installing the various libraries to various tricks needed in order to make certain models work for solving a particular problem. The author's habits of adding good images everywhere helps a lot in this endeavour.The second part of the book is focused on two big issues: a) understanding the architectural differences between autoencoding models (e.g., BERT) and autoregressive models (e.g., BERT); and b) applying such models for classification tasks (e.g., text or token classification). This is the part in which the authors teach you how to fine-tune such models.The last part of the book is dedicated to advanced topics like efficient Transforms, multilingual models or production-related issues (e.g., serving Transformers, visualizing their attention layers or outputs, etc.). This is the part that links directly to advanced research, not unlike the kind of research that was submitted to top conferences in the field a couple of years ago (e.g., up to 2019, there were hardly any Transformer visualizations!).I particularly liked the ease with which the authors jumped straight into a difficult subject and teached it up to research level. There are very few books able to do this (e.g., thinking of the Dive into Deep Learning books or Grokking Deep Learning). It is great to see publishers introducing such books extremely fast (e.g., it took 5 years to have the first D3.js book published, but only 2 years to have a book about BERT or GPT-2/3). Make no mistake, while the title is still about Transformers, the real stars are the newer models discussed here. I consider this a five star book and I would recommend you to read it along the various examples of Transformer / BERT / GPT-2 explained source code that are available online. You will thank yourself later!

Amazon Verified review

Colbert Philippe Jun 29, 2022

I love that book! It gives me the introduction and understanding on the very important topic that is Transformers. The book gives a bit of history and the current standard implementation of transformers. The book also delves in the advanced topic of Attention algorithms. This book is a "must have" for the seasoned AI partitioner of Language Model Transformers.

Amazon Verified review

Daniel Eduardo Portugal Revilla Sep 15, 2021

Currently, I have more experience as a data engineer but Machine Learning(ML) and Deep Learning(DL) are not some of my strangers. However, I have an attraction for NLP so that I had the opportunity to read the Mastering Transformers.This is a fantastic book all in one about Transformers. Transformers are the principal NLP state of the art. Very futuristic. The principal features of the book are to give good examples step by step of which I can mention BERT the most popular Transformer model by Google, another nice feature is that you can learn the origins of NLP, the traditional techniques, ML and DL evolution.You can find information about the principal BERT variants like ALBERT, ELECTRA, RoBERTa. I missed maybe some examples or information about BioBERT and ClinicalBERT for medical domains and medical apps.Finally, there is one of my favorites topics about ML. How to deploy and serving your model to production for real or experimental APPs.

Amazon Verified review

Sachin Kumar Sep 16, 2021

To start with this book is really comprehensive in its coverage of various transformers models like autoencoding models(BERT ,ROBERTa, etc.) and autoregressive models(GPT, T5, etc.), serving models, etc. with relevant code snippets and use cases to play around with.Besides that most of the code examples and text in the book appears to be straight out picked from hugging face website, and in some cases adapted for their example datasets.Therefore, this book will be a great resource for beginners or someone new to transformers models or huggingface website. However, for people who are somewhat familiar with transformers models or familiar with huggingface models, I don't think this book to be a great resource for them to spend money on, since this book appears to be just a curation of freely available articles or code snippets available on huggingface website or other blogs.On the finishing note for the people starting out in NLP or transformers, or for the people who want to save on trouble of going through articles on hugginface or across the web, I will suggest this book as a great resource. However, for the people familiar with huggingface models or who have some experience using them, I think they can freely find the information presented in the book.

Amazon Verified review

FAQs

What is the delivery time and cost of print book?

Shipping Details

USA:

Economy: Delivery to most addresses in the US within 10-15 business days

Premium: Trackable Delivery to most addresses in the US within 3-8 business days

UK:

Economy: Delivery to most addresses in the U.K. within 7-9 business days.
Shipments are not trackable

Premium: Trackable delivery to most addresses in the U.K. within 3-4 business days!
Add one extra business day for deliveries to Northern Ireland and Scottish Highlands and islands

EU:

Premium: Trackable delivery to most EU destinations within 4-9 business days.

Australia:

Economy: Can deliver to P. O. Boxes and private residences.
Trackable service with delivery to addresses in Australia only.
Delivery time ranges from 7-9 business days for VIC and 8-10 business days for Interstate metro
Delivery time is up to 15 business days for remote areas of WA, NT & QLD.

Premium: Delivery to addresses in Australia only
Trackable delivery to most P. O. Boxes and private residences in Australia within 4-5 days based on the distance to a destination following dispatch.

India:

Premium: Delivery to most Indian addresses within 5-6 business days

Rest of the World:

Premium: Countries in the American continent: Trackable delivery to most countries within 4-7 business days

Asia:

Premium: Delivery to most Asian addresses within 5-9 business days

Disclaimer:
All orders received before 5 PM U.K time would start printing from the next business day. So the estimated delivery times start from the next day as well. Orders received after 5 PM U.K time (in our internal systems) on a business day or anytime on the weekend will begin printing the second to next business day. For example, an order placed at 11 AM today will begin printing tomorrow, whereas an order placed at 9 PM tonight will begin printing the day after tomorrow.

Unfortunately, due to several restrictions, we are unable to ship to the following countries:

Afghanistan
American Samoa
Belarus
Brunei Darussalam
Central African Republic
The Democratic Republic of Congo
Eritrea
Guinea-bissau
Iran
Lebanon
Libiya Arab Jamahriya
Somalia
Sudan
Russian Federation
Syrian Arab Republic
Ukraine
Venezuela

What is custom duty/charge?

Customs duty are charges levied on goods when they cross international borders. It is a tax that is imposed on imported goods. These duties are charged by special authorities and bodies created by local governments and are meant to protect local industries, economies, and businesses.

Do I have to pay customs charges for the print book order?

The orders shipped to the countries that are listed under EU27 will not bear custom charges. They are paid by Packt as part of the order.

List of EU27 countries: www.gov.uk/eu-eea:

A custom duty or localized taxes may be applicable on the shipment and would be charged by the recipient country outside of the EU27 which should be paid by the customer and these duties are not included in the shipping charges been charged on the order.

How do I know my custom duty charges?

The amount of duty payable varies greatly depending on the imported goods, the country of origin and several other factors like the total invoice amount or dimensions like weight, and other such criteria applicable in your country.

For example:

If you live in Mexico, and the declared value of your ordered items is over $ 50, for you to receive a package, you will have to pay additional import tax of 19% which will be $ 9.50 to the courier service.
Whereas if you live in Turkey, and the declared value of your ordered items is over € 22, for you to receive a package, you will have to pay additional import tax of 18% which will be € 3.96 to the courier service.

How can I cancel my order?

Cancellation Policy for Published Printed Books:

You can cancel any order within 1 hour of placing the order. Simply contact [email protected] with your order details or payment transaction id. If your order has already started the shipment process, we will do our best to stop it. However, if it is already on the way to you then when you receive it, you can contact us at [email protected] using the returns and refund process.

Please understand that Packt Publishing cannot provide refunds or cancel any order except for the cases described in our Return Policy (i.e. Packt Publishing agrees to replace your printed book because it arrives damaged or material defect in book), Packt Publishing will not accept returns.

What is your returns and refunds policy?

Return Policy:

We want you to be happy with your purchase from Packtpub.com. We will not hassle you with returning print books to us. If the print book you receive from us is incorrect, damaged, doesn't work or is unacceptably late, please contact Customer Relations Team on [email protected] with the order number and issue details as explained below:

If you ordered (eBook, Video or Print Book) incorrectly or accidentally, please contact Customer Relations Team on [email protected] within one hour of placing the order and we will replace/refund you the item cost.
Sadly, if your eBook or Video file is faulty or a fault occurs during the eBook or Video being made available to you, i.e. during download then you should contact Customer Relations Team within 14 days of purchase on [email protected] who will be able to resolve this issue for you.
You will have a choice of replacement or refund of the problem items.(damaged, defective or incorrect)
Once Customer Care Team confirms that you will be refunded, you should receive the refund within 10 to 12 working days.
If you are only requesting a refund of one book from a multiple order, then we will refund you the appropriate single item.
Where the items were shipped under a free shipping offer, there will be no shipping costs to refund.

On the off chance your printed book arrives damaged, with book material defect, contact our Customer Relation Team on [email protected] within 14 days of receipt of the book with appropriate evidence of damage and we will work with you to secure a replacement copy, if necessary. Please note that each printed book you order from us is individually made by Packt's professional book-printing partner which is on a print-on-demand basis.

What tax is charged?

Currently, no tax is charged on the purchase of any print book (subject to change based on the laws and regulations). A localized VAT fee is charged only to our European and UK customers on eBooks, Video and subscriptions that they buy. GST is charged to Indian customers for eBooks and video purchases.

What payment methods can I use?

You can pay with the following card types:

Visa Debit
Visa Credit
MasterCard
PayPal

What is the delivery time and cost of print books?

Shipping Details

USA:

Economy: Delivery to most addresses in the US within 10-15 business days

Premium: Trackable Delivery to most addresses in the US within 3-8 business days

UK:

Economy: Delivery to most addresses in the U.K. within 7-9 business days.
Shipments are not trackable

Premium: Trackable delivery to most addresses in the U.K. within 3-4 business days!
Add one extra business day for deliveries to Northern Ireland and Scottish Highlands and islands

EU:

Premium: Trackable delivery to most EU destinations within 4-9 business days.

Australia:

India:

Premium: Delivery to most Indian addresses within 5-6 business days

Rest of the World:

Premium: Countries in the American continent: Trackable delivery to most countries within 4-7 business days

Asia:

Premium: Delivery to most Asian addresses within 5-9 business days

Unfortunately, due to several restrictions, we are unable to ship to the following countries:

Afghanistan
American Samoa
Belarus
Brunei Darussalam
Central African Republic
The Democratic Republic of Congo
Eritrea
Guinea-bissau
Iran
Lebanon
Libiya Arab Jamahriya
Somalia
Sudan
Russian Federation
Syrian Arab Republic
Ukraine
Venezuela

Mastering Transformers: Build state-of-the-art models from scratch with advanced natural language processing techniques

What do you get with Print?

Key benefits

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Product Details

Frequently bought together

Recommendations for you

Customer reviews

Filter reviews by

People who bought this also bought

About the authors

FAQs

Mastering Transformers: Build state-of-the-art models from scratch with advanced natural language processing techniques

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Key benefits

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Product Details

Packt Subscriptions

Frequently bought together

Recommendations for you

Customer reviews

Filter reviews by

People who bought this also bought

About the authors

FAQs