Prepared by: Joseph Bakarji Logistic Regression Logistic Regression Part I

Given new input, what’s the output? ….. ….. Input Output ? Query ?? Prediction Given new input, what’s the output? Actions: make decisions and act from a set of possible Perceptions: observe the world and one’s own state

Given new input, what’s the output? Assuming  Given the data, find a function , that predicts , given  Given new input, what’s the output? Actions: make decisions and act from a set of possible Perceptions: observe the world and one’s own state

What if  is a label? Size of Tumor Cancer or Not Cancer No Cancer Given the data, find a function , that predicts , given  What if  is a label? Actions: make decisions and act from a set of possible Perceptions: observe the world and one’s own state

What if  is a label? Size of Tumor Cancer or Not Cancer No Cancer Given the data, find a function , that predicts , given  A step function, or threshold What if  is a label? Actions: make decisions and act from a set of possible Perceptions: observe the world and one’s own state

What if  is a label? Given the data, find a function , that predicts , given  A smooth function that returns probability of occurrence Cancer No Cancer Probability What if  is a label? Actions: make decisions and act from a set of possible Perceptions: observe the world and one’s own state

What if  is a label? A smooth function that returns probability of occurrence Cancer No Cancer Probability & What if  is a label? Actions: make decisions and act from a set of possible Perceptions: observe the world and one’s own state

What if  is a label? A smooth function that returns probability of occurrence Cancer No Cancer Probability & Where  What if  is a label? Actions: make decisions and act from a set of possible Perceptions: observe the world and one’s own state

What if  is a label? Cancer No Cancer Probability & Define a predictor: the logistic function ✅ Define a loss: distance between function and data ❓ Optimize loss Test model What if  is a label? Define a predictor: the logistic function ✅ Define a loss: distance between function and data ❓ Optimize loss Test model

How do we pick the best parameters  ? How do we pick the best parameters  ? Actions: make decisions and act from a set of possible Perceptions: observe the world and one’s own state

Logistic Regression Gradient descent  Done! Compute gradient  Logistic Regression Actions: make decisions and act from a set of possible Perceptions: observe the world and one’s own state

Logistic Regression Why not use an ordinary least squares loss? Logistic Regression Actions: make decisions and act from a set of possible Perceptions: observe the world and one’s own state

Higher dimensional inputs For Logistic Regression and beyond Higher dimensional inputs

 discrete values for representing output  One-hot encoding Higher dimensional outputs Multi-class classification - Softmax regression Higher dimensional outputs  is called the base measure (not depend on )  is called the log partition function (not depend on ) ,  and  are scalar.  and  have the same dimensions.

Multi-class classification - Softmax   Speed (?) Size (?) Multi-class classification - Softmax  is called the base measure (not depend on )  is called the log partition function (not depend on ) ,  and  are scalar.  and  have the same dimensions.

How to turn scores into probabilities? What’s the probability it’s a car? Note: This slides explains the intuition. Softmax regression is not ‘exactly’ equivalent to a combination of one-vs-all classification models How to turn scores into probabilities?  is called the base measure (not depend on )  is called the log partition function (not depend on ) ,  and  are scalar.  and  have the same dimensions.

How to turn scores into probabilities? What’s the probability it’s a car? How to turn scores into probabilities?  is called the base measure (not depend on )  is called the log partition function (not depend on ) ,  and  are scalar.  and  have the same dimensions.

How do you train? car plane boat horse 1 Probability Given Label How do you train?  is called the base measure (not depend on )  is called the log partition function (not depend on ) ,  and  are scalar.  and  have the same dimensions.

How do you train? car plane boat horse 1 Probability Prediction Logit Train with Gradient Descent! 1 2 3 4 How do you train?  is called the base measure (not depend on )  is called the log partition function (not depend on ) ,  and  are scalar.  and  have the same dimensions.