What Happened to Bert and T5?

Yi Tay discusses the shift from encoder models like BERT to a new era of large language models (LLMs), explaining technical nuances like Encoder-Decoders, PrefixLM architectures, and Denoising objectives. He presents insights on why BERT-style models are deprecated, how models like T5 offer more flexibility, and the role of denoising objectives in complementing causal language modeling (CLM).

• Yi Tay authored the post.
• Discusses encoder-decoder models.
• BERT-style models are deprecated.
• Denoising objectives complement CLM.
• New era of LLMs explored.