Natural Language Processing with Deep Learning

  • CS224N-2019
  • CS224N
  • Playlist

    • What is this course about?

    Natural language processing (NLP) is one of the most important technologies of the information age, and a crucial part of artificial intelligence. Applications of NLP are everywhere because people communicate almost everything in language: web search, advertising, emails, customer service, language translation, virtual agents, medical reports, etc. In recent years, Deep Learning approaches have obtained very high performance across many different NLP tasks, using single end-to-end neural models that do not require traditional, task-specific feature engineering. In this course, students will gain a thorough introduction to cutting-edge research in Deep Learning for NLP. Through lectures, assignments and a final project, students will learn the necessary skills to design, implement, and understand their own neural network models. This year, CS224n will be taught for the first time using PyTorch rather than TensorFlow (as in previous years).


    Schedule

    Lecture slides will be posted here shortly before each lecture. If you wish to view slides further in advance, refer to last year's slides, which are mostly similar.

    The lecture notes are updated versions of the CS224n 2017 lecture notes (viewable here) and will be uploaded a few days after each lecture. The notes (which cover approximately the first half of the course content) give supplementary detail beyond the lectures.

    This schedule is subject to change.

    Date Description Course Materials Events Deadlines
    Tue Jan 8 Introduction and Word Vectors
    [slides] [video] [notes]

    Gensim word vectors example:
    [code] [preview]     		Suggested Readings:
    1. Word2Vec Tutorial - The Skip-Gram Model [pdf ⬇️]
    2. Efficient Estimation of Word Representations in Vector Space (original word2vec paper)
    3. Distributed Representations of Words and Phrases and their Compositionality (negative sampling paper)
    		 Assignment 1 out
    [code] [preview]
    Thu Jan 10 Word Vectors 2 and Word Senses
    [slides] [video] [notes]     		Suggested Readings:
    1. GloVe: Global Vectors for Word Representation (original GloVe paper)
    2. Improving Distributional Similarity with Lessons Learned from Word Embeddings
    3. Evaluation methods for unsupervised word embeddings
    Additional Readings:
    1. A Latent Variable Model Approach to PMI-based Word Embeddings
    2. Linear Algebraic Structure of Word Senses, with Applications to Polysemy
    3. On the Dimensionality of Word Embedding.
    Fri Jan 11 Python review session
    [slides]     		1:30 - 2:50pm
    Skilling Auditorium [map]
    Tue Jan 15 Word Window Classification, Neural Networks, and Matrix Calculus
    [slides] [video]
    [matrix calculus notes]
    [notes (lectures 3 and 4)]     		Suggested Readings:
    1. CS231n notes on backprop
    2. Review of differential calculus
    Additional Readings:
    1. Natural Language Processing (Almost) from Scratch
    		 Assignment 2 out
    [code] [handout]     		Assignment 1 due
    Thu Jan 17 Backpropagation and Computation Graphs
    [slides] [video]
    [notes (lectures 3 and 4)]     		Suggested Readings:
    1. CS231n notes on network architectures
    2. Learning Representations by Backpropagating Errors
    3. Derivatives, Backpropagation, and Vectorization
    4. Yes you should understand backprop
    Tue Jan 22 Linguistic Structure: Dependency Parsing
    [slides] [scrawled-on slides]
    [video] [notes]     		Suggested Readings:
    1. Incrementality in Deterministic Dependency Parsing
    2. A Fast and Accurate Dependency Parser using Neural Networks
    3. Dependency Parsing
    4. Globally Normalized Transition-Based Neural Networks
    5. Universal Stanford Dependencies: A cross-linguistic typology
    6. Universal Dependencies website
    		 Assignment 3 out
    [code] [handout]     		Assignment 2 due
    Thu Jan 24 The probability of a sentence? Recurrent Neural Networks and Language Models
    [slides] [video]
    [notes (lectures 6 and 7)]     		Suggested Readings:
    1. N-gram Language Models (textbook chapter)
    2. The Unreasonable Effectiveness of Recurrent Neural Networks (blog post overview)
    3. Sequence Modeling: Recurrent and Recursive Neural Nets (Sections 10.1 and 10.2)
    4. On Chomsky and the Two Cultures of Statistical Learning
    Tue Jan 29 Vanishing Gradients and Fancy RNNs
    [slides] [video]
    [notes (lectures 6 and 7)]     		Suggested Readings:
    1. Sequence Modeling: Recurrent and Recursive Neural Nets (Sections 10.3, 10.5, 10.7-10.12)
    2. Learning long-term dependencies with gradient descent is difficult (one of the original vanishing gradient papers)
    3. On the difficulty of training Recurrent Neural Networks (proof of vanishing gradient problem)
    4. Vanishing Gradients Jupyter Notebook (demo for feedforward networks)
    5. Understanding LSTM Networks (blog post overview)
    		 Assignment 4 out
    [code] [handout] [Azure Guide] [Practical Guide to VMs]     		Assignment 3 due
    Thu Jan 31 Machine Translation, Seq2Seq and Attention
    [slides] [video] [notes]     		Suggested Readings:
    1. Statistical Machine Translation slides, CS224n 2015 (lectures 2/3/4)
    2. Statistical Machine Translation (book by Philipp Koehn)
    3. BLEU (original paper)
    4. Sequence to Sequence Learning with Neural Networks (original seq2seq NMT paper)
    5. Sequence Transduction with Recurrent Neural Networks (early seq2seq speech recognition paper)
    6. Neural Machine Translation by Jointly Learning to Align and Translate (original seq2seq+attention paper)
    7. Attention and Augmented Recurrent Neural Networks (blog post overview)
    8. Massive Exploration of Neural Machine Translation Architectures (practical advice for hyperparameter choices)
    Tue Feb 5 Practical Tips for Final Projects
    [slides] [video] [notes]     		Suggested Readings:
    1. Practical Methodology (Deep Learning book chapter)
    Thu Feb 7 Question Answering and the Default Final Project
    [slides] [video] [notes]     		Project Proposal out
    [instructions]

    Default Final Project out [handout] [code]     		Assignment 4 due
    Tue Feb 12 ConvNets for NLP
    [slides] [video] [notes]     		Suggested Readings:
    1. Convolutional Neural Networks for Sentence Classification
    2. A Convolutional Neural Network for Modelling Sentences
    Thu Feb 14 Information from parts of words: Subword Models
    [slides] [video]     		Suggested readings:
    1. Minh-Thang Luong and Christopher Manning. Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models
    		Assignment 5 out
    [original code (requires Stanford login) / public version] [handout]     		Project Proposal due
    Tue Feb 19 Modeling contexts of use: Contextual Representations and Pretraining
    [slides] [video]     		Suggested readings:
    1. Smith, Noah A. Contextual Word Representations: A Contextual Introduction. (Published just in time for this lecture!)
    2. The Illustrated BERT, ELMo, and co.
    Thu Feb 21 Transformers and Self-Attention For Generative Models
    (guest lecture by Ashish Vaswani and Anna Huang)
    [slides] [video]     		Suggested readings:
    1. Attention is all you need
    2. Image Transformer
    3. Music Transformer: Generating music with long-term structure
    Fri Feb 22 Project Milestone out
    [instructions]     		Assignment 5 due
    Tue Feb 26 Natural Language Generation
    [slides] [video]
    Thu Feb 28 Reference in Language and Coreference Resolution
    [slides] [video]
    Tue Mar 5 Multitask Learning: A general model for NLP? (guest lecture by Richard Socher)
    [slides] [video]     		Project Milestone due
    Thu Mar 7 Constituency Parsing and Tree Recursive Neural Networks
    [slides] [video] [notes]     		Suggested Readings:
    1. Parsing with Compositional Vector Grammars.
    2. Constituency Parsing with a Self-Attentive Encoder
    Tue Mar 12 Safety, Bias, and Fairness (guest lecture by Margaret Mitchell)
    [slides] [video]
    Thu Mar 14 Future of NLP + Deep Learning
    [slides] [video]
    Sun Mar 17 Final Project Report due [instructions]
    Wed Mar 20 Final project poster session
    [details]     		5:15 - 8:30pm
    McCaw Hall at the Alumni Center [map]     		Project Poster/Video due [instructions]