Math 595, Special Topics: Machine Learning via Probabilistic Graphical Models

Professor:Kay Kirkpatrick
Office:231 Illini Hall
Contact:kkirkpat(at)illinois.edu
Course site:https://faculty.math.illinois.edu/~kkirkpat/2nd595spring2017.html
Lectures: 2:00-2:50pm MWF in 147 Altgeld Hall, 2 credits for 8 weeks.
Office hours: Mondays 4:00-4:50 pm, and Fridays 11:00-11:50am, or by appointment, in my office, 231 Illini Hall. I would be happy to answer your questions in my office anytime as long as I'm not otherwise engaged, and before and after class are good times to catch me either in my office or in the classroom.
Textbook: The recommended but not required text will be Probabilistic Graphical Models: Principles and Techniques by Koller and Friedman, available in engineering reserves: linked here. Koller and Friedman's Chapters 3 and 4 have been scanned and are available online: http://www.cs.cmu.edu/~epxing/Class/10708-13/reading/Ch%203.pdf and http://www.cs.cmu.edu/~epxing/Class/10708-13/reading/Ch%204.pdf
Grading policy: Homework: 30% of the course grade
Participation: 30% (possibly scribing one lecture at the blackboard, TBD)
Project: 40%, a short midterm presentation on a paper or section of a book closely related to the course. Some suggestions are below, but are not exhaustive.
Inclusion and Justice: I am committed to affirming the identities, realities and voices of all students, especially students from historically marginalized or under-represented backgrounds. I value the use of self-specified pronouns, and I require respect for everybody. Please contact me to receive disability accommodations. My pronouns are she/her/hers.


Homework (due many Fridays in class or to or to my email address or my mailbox in AH 250 by the end of class): I will drop your lowest homework score, and I provide 72 hours total of automatic grace period/s, which can be used, for example, as multiples 24-hour grace periods or one long one. In order to use this, please let me know how many hours of grace period you are taking. (All such requests get automatically approval.) The only time you might want to not take (too many) grace hours is when you want to get timely feedback from me on your project draft.

Week 1: Introduction

If you'd like a glimpse into how I view this field, please check out these slides and also the video of me presenting these slides on 8/28/19 at https://mediaspace.illinois.edu/media/t/1_jdkvslqn

Wednesday lecture
Friday lecture

HW #1, due by 5pm the first Friday of the semester: please send me an email introducing yourself, for instance, what name you prefer to be called, your major and hobbies, why you're interested in this course, or anything else you want to share or have questions about. It would also be helpful to attach a photo of yourself to help me connect your face with your name. An analog alternative is to fill out this info sheet and turn it in by the end of class Friday.

Week 2:
Monday: lecture
Wednesday: lecture
Friday: lecture


HW #2 is due by email at 5pm the second Friday of classes (Jan. 31):
A) Select a talk of your choice to attend (I recommend a colloquium talk in Math, ECE, CS, or Physics), and send me a short email reporting on it: the topic, one thing you liked about it, and one thing you would do differently if you were the speaker. An alternative is to watch a talk, such as https://www.youtube.com/watch?v=mwVKXKlU1GU&app=desktop or to ask for an extension if there's a talk you want to attend in February.
B) Select two of the following readings and write a two-sentence summary of each.

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3488060 file:///Users/kkirkpat/Downloads/SSRN-id3488060.pdf
https://www.newyorker.com/magazine/2019/10/14/can-a-machine-learn-to-write-for-the-new-yorker
https://slate.com/technology/2019/12/tiktok-disabled-users-videos-suppressed.html
https://www.nature.com/articles/d41586-019-03895-5
https://www.youtube.com/watch?v=mwVKXKlU1GU&app=desktop
https://theintercept.com/2019/12/20/mit-ethical-ai-artificial-intelligence/
https://www.vox.com/future-perfect/2019/3/5/18251924/self-driving-car-racial-bias-study-autonomous-vehicle-dark-skin
https://www.seattletimes.com/business/technology/when-convenience-meets-surveillance-ai-at-the-corner-store/?utm_content=bufferd902e&utm_medium=social&utm_source=twitter&utm_campaign=owned_buffer_tw_m
https://reallifemag.com/the-algorithmic-colonization-of-africa/
https://www.nytimes.com/2018/11/05/opinion/artificial-intelligence-machine-learning.html
https://www.nytimes.com/2019/10/08/opinion/ai-hiring-discrimination.html?action=click&module=Opinion&pgtype=Homepage
https://howwegettonext.com/what-our-tech-ethics-crisis-says-about-the-state-of-computer-science-education-a6a5544e1da6

Week 3:
Monday: lecture
Wednesday: lecture
Friday: lecture plus notes

HW #3, your project proposal, is due Fri 2/7 via email. Please type this and aim for a length of about one page (your references can go into a second page), addressing each of the following:
1. In the opening, identify the paper you will focus on, your topic, and your main message, also known as a thesis statement or a motivation that your final project will address.
2. Think about your audience: your classmates, not just me; people in STEM, not just in your field. Write a paragraph addressing some or all of the following questions: Why should your audience care? What do you want them to take away from your presentation(s)? How can you clarify the benefits of your project to your audience?
3. Specifics: What kinds of audiovisual aids will you be choosing to use? Your project should have at least one item of visual interest (picture, simulation, etc.), and at least one item of technical interest (theorem, algorithm, etc.). Which two or three definitions or key ideas will you introduce to your audience? What is a good example (think n=2) that illustrates the main point of your project? Can you find a story that's related to your topic?
4. Annotated bibliography: can go beyond the 1 page limit, and should contain at least 3 references. Please annotate each reference by indicating which sections of the references are most relevant to your project and what their main messages are.

Week 4:
Monday: lecture
Wednesday: lecture
Friday: lecture

https://www.cs.cmu.edu/~epxing/Class/10708-17/readings/Ch9.1-9.4.pdf
https://www.cs.cmu.edu/~epxing/Class/10708-17/readings/Ch10.1-10.3.pdf

HW #4, your midterm title and abstract, due Fri 2/14 by email in plain text. IMPORTANT: Please indicate clearly in your subject line what format (talk, paper, etc.) your project will take. Also: Please read "How to give a good 20-minute math talk" by William Ross.

Week 5:
Monday: lecture
Wednesday: lecture
Friday: lecture



HW #5 is due Wednesday Feb 26 by email. Draft 3 slides for your talk, and revise them according to the Doumont principles in these readings: downloadable booklet on slide design for scientific talks and/or "Slides are not all evil"--both written by Jean-luc Doumont. The homework that you turn in should look something like this example, with six slides: 3 originals and 3 improved versions. ALSO: please let me know which days of Mar 2, 4, 6, 9, 11 work best for you to give a talk, if you're giving a talk. (And if you've already told me, please remind me.) For a paper, you should have at least 1.5 to 2 pages of text (12 pt font; 1 to 1.5 spacing), and an outline of the remainder, a figure, and citations.



Week 6:
Monday: lecture
Wednesday: lecture
Friday: lecture

HW#6, due at least one day before your talk by email or on March 5 if you're writing a paper: Reasonably complete draft of your project. For a talk, you should have an appropriate number of complete slides based on when you're giving your talk, plus the remainder of the talk outlined (e.g., headlines only). If you would like more comments from me, you can turn your draft in as hard-copy or with more lead-time. If you'd like a particular kind of feedback (e.g., if you hand in hard-copy but you only want general suggestions), please let me know. Also, please read "How to Talk Mathematics" by Paul Halmos or Academic talk advice from a Berkeley CS prof.




Midterm presentations will be on a subset of March 2, 4, 6, 9, 11. If you have a conflict, please let me know. Attendance and participation when your classmates are giving their talks is recommended, i.e., attending as many talks as you can and asking one question each of a couple of speakers. If you're giving a computer talk, please test your connection to the A/V system well in advance. If you want to borrow a machine for projecting your slides, please ask me by email to bring a machine and send me the slides early. The classroom desktop computer is also an option (you can log in with your netid). In any case, your final draft talk slides should be sent by email to me before your talk.


HW #7, for speakers due after your talk: Send me an email with a short reflection, discussing what you thought worked well, and what you would do better if you had a chance to do it over. Please attach slides or notes if they're different from previous versions you sent me. If you're writing a paper, the due date is 3/13, and the target length is 3 to 10 pages.


Midterm talk schedule: (20 minutes per speaker plus about 5 minutes for questions and switching machines for the next speaker)

Mon, Mar 2:
Sejeoung Yoo
Adriana Morales

Wed, Mar 4:
Mark Rothermel
Eion Blanchard

Fri, Mar 6:
Yue Wu
Riley Vesto

Monday, Mar 9:
Tejo Nutalapati
Yinan Hu

Wednesday, Mar 11:
Luo Di
Sophie Le




Names, Titles, and Abstracts for Midterm Talks:

Sejeoung Yoo

Title: Introduction to collaborative filtering in recommendation system

Abstract: In this talk, I will introduce a paper “MATRIX FACTORIZATION TECHNIQUES FOR RECOMMENDER SYSTEMS” written by Yehuda Koren, Robert Bell and Chris Volinsky. It has been reported by Netflix that more than 80 percent of movies watched on Netflix came through recommendations, and the value of Netflix recommendations is estimated at more than US$1 billion per year. As such, recommender system is ubiquitous in our modern society, and this paper introduces the most popular technique in modern recommender systems, called ‘collaborative filtering’. Previous methods are mostly based on ‘content-based filtering’ that recommends items based on simple content matching, thereby requiring external auxiliary information. Meanwhile, ‘collaborative filtering’ trains a latent factor model (i.e., matrix factorization) on users’ historical feedback assuming that users that had similar interest in the past will likely have similar interest in the future. Therefore, it would be a great chance to be familiar with the system and broaden our knowledge to apply it on other fields.

Adriana Morales

Title: Tackling drug development with artificial neural networks

Abstract: Virtual screening (VS) has emerged in recent years as a way to expedite drug development. Instead of physically testing every compound in a given library, VS computes the properties of a compound in order to predict which will be most favorable to bind to the desired drug. VS is done through the use of machine learning (ML) methods. This means that we can create a model which predicts if a given compound will bind to a given target after being trained on a data set containing both compounds that are known to bind and compounds that are known to not bind. Getting a good model is not an easy task, so different classifiers are used in order to have have structural risk minimization. We will focus on one classifier: the artificial neural network (ANN). Comparison studies have found that ANNs outperform other classifiers for VS. We will describe the fundamentals of ANNs and proceed with an overview of the specific ways in which ANNs are implemented in VS.

Mark Rothermel

Title: "Sum-Product Networks: A Deep, Exact and Tractable PGM."

Abstract: Probabilistic Graphical Models (PGMs) like Bayesian Networks or Hidden Markov Models generally provide intractable or inexact inference. I am going to present a PGM from 2012 called Sum-Product Network (SPN) which provides both tractable and exact inference while learning is efficient. Like neural networks, SPNs are used in a deep learning setting and can be applied to a wide variety of problems. My presentation will cover the (idea behind the) architecture of SPNs, give an intuitive explanation for the SPN's superiority (incl. tractability, exactness, and flexibility), and show two/three vivid application examples.

Eion Blanchard

Title: "Model Theory and Machine Learning"

Abstract: A goal in query learning is to determine which concepts are learnable using a limited scope of queries and how complex learning given concepts is. When concept classes are models of first-order logical theories, results from model theory grant unique insight into this instance of machine learning. In fact, the learnability of a concept class is directly related to the stability of its underlying theory. This approach through mathematical logic, while not readily implementable, grants researchers yet another angle through which to pursue a rightly interdisciplinary understanding of machine learning, its potentials, and its limitations.

Yue Wu

Title: How to access brand reputation with social network data? -- A method based on Probabilistic Graphical Model

Abstract: Social media is a great platform to connect people around the world and they can share information about different topics that they are interested, in particular personal opinions. With such a fast information exchange mechanism, reputation of individuals, consumer products, or business companies can be quickly built up within a social network. Viewing this property from a business perspective, social media listening can be very valuable to marketing and branding. The applications of social network data mining has been put into practice for while in order to access brand reputation and build business strategy. In fact, Probabilistic Graphical Model (PGM) can be a very use tool in this process. In this talk, we will review a paper which utilizes PGM to access brand reputation and related works. This paper proposes a probabilistic graphical model to collectively measure reputations of entities in social networks. With large amount of data collected from the social network Facebook, the model proposed in this paper can effectively and efficiently rank entities, such as presidential candidates, professional sport teams, musician bands, and companies, based on their social reputation. According to this paper, the proposed model produces results largely consistent with the two publicly available systems - movie ranking in Internet Movie Database and business school ranking by the US news & World Report - with the correlation coefficients of 0.75 and 0.71, respectively. The presenter may will also try to implement this model with the data he collected and test the validity this model if possible.

Riley Vesto

Title: Machine Learning Applications to Crystal Design and Optimization

Abstract: In this presentation, we discuss the application of machine learning to the materials and chemical design process. These applications consist of using trained algorithms to automate the modeling process using large data sets of experimental results. The algorithms can then be pushed further into predicting stable compounds which satisfy desired properties. For this presentation, we will focus on the specific use case in crystal design which has many applications in electronic and photonic device designs. The breadth of this methodology, however, can be applied to many scientific fields and could potentially be disruptive to the scientific method.

Tejo Nutalapati

Yinan Hu

Luo Di

Sophie Le


Syllabus

This is a graduate course on advanced machine learning, a growing field at the intersection of probability, statistics, optimization, and computer science, which aims to develop algorithms for making predictions based on data. This course will cover foundational models and mathematics for machine learning, including statistical learning theory and neural networks, with a project component. Prerequisites: some linear algebra, probability, basic machine learning, and graph theory--or a willingness to catch up through studying.

List of topics we are likely to cover, and the later topics are if time allows:

Intro:
Bayesian Networks, Undirected Graphical Models
Chains, Trees, (Factorial Graphs)
Markov Properties (important probability background)

Exact Inference:
Variable elimination
Multivariate Gaussians, Kalman Filtering, (Smoothing)
Hidden Markov Models, (Forward Backward Algorithm)

Approximate Inference:
Exponential Families, Variational Optimization point of view
Message Passing, Mean Field
Variational Methods
(Mixture Models)

Learning:
Network inference, learning the model given data
Sparse Estimation
(Convex-Optimization-Based Methods)

If there are specific topics that interested you, please let me know and/or choose that topic for your project.




Potential papers/topics for your project (others are possible with my approval):

https://arxiv.org/pdf/1912.03593.pdf
https://arxiv.org/pdf/2001.05046v1.pdf
https://arxiv.org/abs/quant-ph/9508027
Any paper by Timnit Gebru


Emergency information link and the new one.