CAP 4613 / CAP 5619

Deep and Reinforcement Learning Fundamentals

Administrivia

  • 📢 Instructor: Dr. Shangqian Gao (sg24bi[at]fsu[dot]edu)
  • 🎒 Format: In person
  • 📅 Time: Tuesday & Thursday, 3:05 pm-4:20 pm (ET)
  • 🏫 Location: Lov Building 101
  • 🔍 Office Hour: Thursday, 4:30 pm-6:30 pm (ET)
  • 💡 Teaching Assistants: TBD

Course Overview

This course covers fundamental principles and techniques in deep and reinforcement learning. Topics include convolutional neural networks, recurrent and recursive neural networks, transformers, backpropagation, regularization and optimization techniques, and fundamental reinforcement-learning algorithms. Reinforcement-learning topics include dynamic programming, Monte Carlo methods, temporal-difference learning, and function approximation. The course also examines applications and active research topics in deep and reinforcement learning.

The emphasis is on understanding, implementing, and analyzing the algorithms behind modern AI systems, rather than only using existing models.

Learning Objectives

After completing the course, students should be able to:

  • Implement and use backpropagation algorithms to train deep neural networks
  • Apply regularization techniques to training deep neural networks
  • Apply optimization techniques to training deep neural networks
  • Construct and train convolutional neural networks
  • Construct and train recurrent neural networks
  • Construct and train transformers
  • Implement and analyze deep-learning algorithms for object recognition
  • Implement and analyze deep-learning algorithms for natural-language processing
  • Implement and apply policy iteration and value iteration
  • Implement and apply Monte Carlo reinforcement-learning algorithms
  • Implement and apply temporal-difference reinforcement-learning algorithms

Prerequisites

Senior or graduate standing in science or engineering, or permission of the instructor. Students should be familiar with basic linear algebra, probability, algorithm design, and programming. Examples and programming assignments will primarily use Python and PyTorch.

Textbooks

Deep Learning

Authors: Ian Goodfellow, Yoshua Bengio, and Aaron Courville

Reinforcement Learning: An Introduction, 2nd ed.

Authors: Richard S. Sutton and Andrew G. Barto

Additional papers and lecture notes will be provided for transformers, large language models, modern reinforcement learning, and current applications.

Software

The primary framework for examples and programming assignments is PyTorch. Students are encouraged to become comfortable with Python, tensors, automatic differentiation, neural-network modules, optimizers, and GPU-based training.

Assessment

  • Homework Assignments (30%): Six assignments, 5% each
  • Programming Assignments (20%): Two programming assignments, 10% each
  • Midterm Exam (25%)
  • Final Project (20%)
  • Participation (5%)

The official syllabus and Canvas are authoritative for assignment requirements, deadlines, and any semester-specific changes.

Tentative Schedule

Week Topic Suggested Reading
Week 1 Introduction Deep Ch. 1 and 5; RL Ch. 1

Course Policies

Grade of “I” Policy: Incomplete (“I”) grades should be recorded only in exceptional cases when a student, who has completed a substantial portion of the course and who is otherwise passing, is unable to complete a well-defined portion of a course for reasons beyond the student’s control. Students in these circumstances must petition the instructor and should be prepared to present documentation that substantiates their case.

University Attendance Policy: Excused absences include documented illness, deaths in the family and other documented crises, call to active military duty or jury duty, religious holidays, and official University activities. These absences will be accommodated in a way that does not arbitrarily penalize students who have a valid excuse. Consideration will also be given to students whose dependent children experience serious illness.

Academic Honor Policy: The Florida State University, Academic Honor Policy, outlines the University’s expectations for the integrity of student’s academic work, the procedures for resolving alleged violations of those expectations, and the rights and responsibilities of students and faculty members throughout the process. Students are responsible for reading the Academic Honor Policy and for living up to their pledge to . . . be honest and truthful and . . . [to] strive for personal and institutional integrity at Florida State University. (Florida State University Academic Honor Policy, found at http://fda.fsu.edu/Academics/Academic-Honor-Policy).

For this course, in particular, every student must complete his/her assignments, quizzes, and exams independently. Showing your work to your peers or making it accessible to them is considered academic dishonesty. You are responsible for ensuring that your work is adequately protected and not accessible to others.

Americans with Disabilities Act: Students with disabilities needing academic accommodation should: (1) register with and provide documentation to the Office of Accessibility Services; (2) bring a letter to the instructor indicating the need for accommodation and what type; (3) meet (in person, via phone, email, skype, zoom, etc…) with each instructor to whom a letter of accommodation was sent to review approved accommodations. Please note that instructors are not allowed to provide classroom accommodation to a student until appropriate verification from the Office of Accessibility Services has been provided. This syllabus and other class materials are available in an alternative format upon request. For more information about services available to FSU students with disabilities, contact the: Office of Accessibility Services, 874 Traditions Way, 108 Student Services Building, Florida State University, Tallahassee, FL 32306-4167; (850) 644-9566 (voice); (850) 644-8504 (TDD), oas@fsu.edu, https://dsst.fsu.edu/oas/

Confidential Campus Resources: Various centers and programs are available to assist students with navigating stressors that might impact academic success. These include the following:

  • Victim Advocate Program University Center A, Room 4100, (850) 644-7161, Available 24/7/365, Office Hours: M-F 8-5 https://dsst.fsu.edu/vap

  • University Counseling Center, Askew Student Life Center, 2nd Floor, 942 Learning Way. (850) 644-8255 https://counseling.fsu.edu/

  • University Health Services Health and Wellness Center, (850) 644-6230 https://uhs.fsu.edu/

Free Tutoring from FSU: On-campus tutoring and writing assistance is available for many courses at Florida State University. For more information, visit the Academic Center for Excellence (ACE) Tutoring Services’ comprehensive list of on-campus tutoring options at http://ace.fsu.edu/tutoring or contact tutor@fsu.edu. High-quality tutoring is available by appointment and on a walk-in basis. These services are offered by tutors trained to encourage the highest level of individual academic success while upholding personal academic integrity.

Late Policy and Make-up Exams:

  • Late assignments will not ordinarily be accepted. If, for some compelling reason, you cannot hand in an assignment on time, please contact the instructor as far in advance as possible.
  • No credit will be given to late course projects.
  • No make-up exams (except under extremely unusual circumstances).

Syllabus Change Policy: Except for changes that substantially affect the implementation of the evaluation (grading) statement, this syllabus is a guide for the course and is subject to change with advance notice.