# Lesson 2: Deep Learning 2019 – Data cleaning and production; SGD from scratch

And then we’re going to do this: [email protected] What is “[email protected]”? [email protected], in python, means a matrix product between x and a. It actually is even more general than that. It can be a vector-vector product, a matrix-vector product, a vector-matrix product or a matrix-matrix product. And then actually in PyTorch, specifically, it can mean even more general things where we get into higher rank tensors, which we will learn all about very soon. Right? But this is basically the key thing that’s going to go on in all of our deep learning. The vast majority of the time our computers are going to be basically doing this: multiplying numbers together and adding them up, which is a surprisingly useful thing to do. Ok, so we basically are going to generate some data by creating a line and then we’re going to add some random numbers to it. But let’s go back and see how we created “x” and “a”. So I mentioned that you know, we’ve basically got these two coefficients, 3 and 2, and you’ll see that we’ve wrapped it in this function called “tensor()”. You might have heard this word ‘tensor’ before. Who’s heard the word tensor before? About 2//3 of you. Okay, so it’s one of these words that sounds scary and apparently, if you’re a physicist, it actually is scary, but in the world of deep learning it’s actually not scary at all. Tensor means ‘array’. Okay? It means array. So specifically it’s an array of a regular shape, right? So it’s not an array where row 1 has two things and row 3 has three things and row 4 has one thing what you call a ‘jagged’ array. That’s not a tensor. A tensor is any array, which has a ‘rectangular’ or ‘cube’ or whatever… you know, a shape where every element every row is the same length, and then every column is the same length so a 4×3 matrix would be a tensor. A vector of length 4 would be a tensor. A 3D array of length 3 x 4 x 6 would be a tensor. That’s all a tensor is. Okay? And so we have these all the time. For example, an image is a three dimensional tensor. It’s got number of rows by number of columns by number of channels; normally red green blue. So for example, a kind of a VGA texture would be 640 by 480 by 3 or actually… we do things backwards, so when people talk about images they normally go width by height but when we talk mathematically we always go a number of rows by number of columns So it’d actually be 480 by 640 by 3 That will catch you out We don’t say ‘dimensions’ though, with tensors, we use one of two words: We either say ‘rank’ or or ‘axes’. ‘Rank’ specifically means how many axes are there? How many dimensions are there? So an image is generally a “rank 3 tensor”. So what we’ve created here is a “rank 1 tensor” or also known as a ‘vector’, right? But like, in math people come up with slightly different words or actually no; they come up with very different words for slightly different concepts. Why is a one dimensional array a ‘vector’ and a two dimensional array’s a ‘matrix’ and then a three dimensional array… Does that even have a name? Not really. It doesn’t have a name. Like, it doesn’t make any sense. We also you know with computers we try to have some simple consistent naming conventions. They’re all called ‘tensors’. Rank 1 tensor, rank 2 tensor, rank 3 tensor. You can certainly have a rank 4 tensor If you’ve got 64 images then that would be a rank 4 tensor of 64 x 480 x 640 x 3, for example. So tensors are very simple. They just mean arrays. And so, in PyTorch, you say tensor and you pass in some numbers and you get back, in this case, just a list. I got back a ‘vector’, okay? So this, then, represents our coefficients: the slope and the intercept of our line. And so, because remember, we’re not actually going to have a special case of “ax + b” instead, we’re going to say there’s always this second x value which is always 1 (you can see it here, always 1), which allows us just to do a simple ‘matrix vector product’. Ok, so that’s ‘a’ and then we wanted to generate this ‘x array’ of data which is going to have we’re going to put random numbers in the first column and a whole bunch of ones in the second column. So to do that, we basically say to PyTorch: “create a rank 2 tensor, Actually no, sorry, let’s say that again. We see to PyTorch: “we want to create a tensor of ‘n x 2’. So since we passed in a total of 2 things we get a rank 2 tensor. The number of rows will be ‘n’ and the number of columns will be 2. And in there, every single thing in it will be a 1.
That’s what torch.ones() means. And then, this is really important, you can index into that, just like you can index into a list in Python, but you can put a colon (:) anywhere. And a colon means – “every single value on that axis”. Or “every single value on that dimension”. So this here means every single row. And then this here means column 0. So this is every row of column 0, I want you to grab a uniform, random number. And here’s another very important concept: in PyTorch, anytime you’ve got a function that ends in an underscore, it means “don’t return to me that uniform random number but replace whatever this is being called on, with the result of this function”. So this takes column 0 and replaces it with a uniform random number between -1 and 1. So there’s a lot to unpack there, right? But the good news is those two lines of code, plus this one (which we’re coming to), cover 95% of what you need to know about PyTorch. How to create an array, how to change things in an array, and how to do matrix operations on an array, okay? So there’s a lot to unpack but these small number of concepts are incredibly powerful. So I can now print out the first 5 rows, okay? So “:5” is standard python ‘slicing’ syntax, to say ‘the first five rows’. So here are the first five rows, two columns looking like my random numbers, and my ones. So now I can do a matrix product of that x by my a, add in some random numbers to add a bit of noise, and then I can do a scatter plot. And I’m not really interested in my scatter plot in this column of ones, right? There just there to make my linear function more convenient, so I’m just going to plot my 0-index column against my “y”s and there it is. “plt” is what we universally use to refer to the plotting library ‘matplotlib’. And that’s what most people use for most of their plotting in python. In scientific python we use matplotlib. It’s certainly a library, you’ll want to get familiar with because being able to plot things is really important. There are lots of other plotting packages. Lots of them, the other packages, are better at certain things than matplotlib, but like matplotlib can do everything reasonably well. Sometimes it’s a little awkward, but you know, for me, I do pretty much everything in matplotlib because there’s really nothing it can’t do (even though some libraries can do other things a little bit better or a little bit prettier). But it’s really powerful, so once you know matplotlib, you can do everything. So here I’m asking matplotlib to give me a scatterplot with my x’s against my y’s and there it is, okay? So this is my my dummy data representing like, you know, of temperature and ice cream sales So ,now what we’re going to do is we’re going to pretend we were given this data and we don’t know that the values of our coefficients are 3 and 2. So we’re going to pretend that we never knew that we have to figure them out, okay? So how would we figure them out? How would we draw a line to fit to this data? And why would that even be interesting? Well, we’re going to look at more about why it’s interesting in just a moment, but the basic idea is this: if we can find (this is going to be kind of perhaps, really surprising) but if we can find a way to find those two parameters to fit that line to those (how many points were there? – ‘n’ was 100) if we can find a way to fit that line to those 100 points, we can also fit these arbitrary functions that convert from pixel values to probabilities. It’ll turn out that there’s techniques that we that we’re going to learn to find these two numbers, works equally well for the 50 million numbers in resnet34. So we’re actually going to use an almost identical approach. So that’s (this is the bit that I found in previous classes, people have the most trouble digesting), like, I often find even after week 4 or week 5, people will come up to me and say “I don’t get it, how do we actually train these models?” – and I’ll say “It’s SGD. It’s that thing we throw in the notebook with the 2 numbers”. It’s like “Yeah, but but we’re fitting a neural network”. So “I know, and we can’t print the 50 million numbers anymore, but it is literally, identically, doing the same thing”. And the reason this is hard to digest is that the human brain has a lot of trouble conceptualizing of what an equation with 50 million numbers looks like and can do.
So you just kind of, for now, will have to take my word for it. It can do things like recognize Teddy Bears. And all these functions turn out to be very powerful. Now we’re going to learn a little bit more in just a moment, about how to make them extra powerful, but for now, the thing we’re going to learn to fit these two numbers is the same thing that we’ve just been using to fit 50 million numbers. Okay, so we want to find what PyTorch calls ‘parameters’. Or in statistics, you’ll often hear called ‘coefficients’. These values a1 and a2. We want to find these parameters such that the line that they create minimizes the error between that line and the points. So in other words, you know, if we created, you know, if the a1 and a2 we came up with resulted in this line, then we’d look and we’d see like how far away is that line from each point? I would say “Oh, that’s quite a long way”. And so maybe there was some other a1 or a2 which resulted in this line and they would say, like, “oh, how far away is each of those points”? And then eventually we come up with Blue We come up with this line and it’s like, “Oh, in this case, each of those is actually very close”. All right? So you can see how in each case we can say how far away is the line at each spot away from its point and then we can take the average of all those and that’s called the ‘loss’. And that is the value of our loss, right? So you need some mathematical function that can basically say how far away is this line from those points? For this kind of problem, which is called a ‘regression’ problem ,a problem where your dependent variable Is ‘continuous’, so rather than being “Grizzlies” or “Teddies”, it’s like some number between -1 and 6, this is called a regression problem. And for regression the most common loss function is called ‘mean squared error’, which pretty much everybody calls ‘MSE’. You may also see RMSE just ‘Root Mean Squared Error’. And so the mean squared error is a loss, it’s the difference between some prediction that you’ve made, okay, which you know is like the value of the line, and the actual number of ice cream sales. And so, in the mathematics of this, people normally refer to the actual, they normally call it “y” and the prediction, they normally call it “y hat”, as in they they write it like that. And so what I try to do like when we’re writing something like, you know, mean squared error equation, there’s no point writing ice cream here and temperature here because we wanted to apply it to anything. So we tend to use these like mathematical placeholders. So the value of mean squared error is simply the difference between those two, squared! All right? And then we can take the mean. Because, remember, that is actually a ‘vector’ or what we now call it, a “rank 1 tensor” and that is actually a rank 1 tensor, so it’s the value of the number of ice cream sales at each place. And so when we subtract one vector from another vector, (and we’re going to be learning a lot more about this), but it does something called element-wise arithmetic in other words It subtracts each each one from each other, and so we end up with a vector of differences, and then if we take the square of that, it squares everything in that vector. And so then we can take the mean of that to find the average square of the differences between the actuals and the predictions. So, if you’re more comfortable with mathematical notation what we just wrote there was the “sum of…” (which way round did we do it?) y hat minus… y… squared, over… n”, right? So that equation is the same as that equation. So one of the things I’ll note here is, I don’t think this is, you know, more complicated or unwieldy than this, right? But the benefit of this is you can experiment with it like once you’ve defined it, you can use it you can send things into it and get stuff out of it and see how it works, alright? So, for me, most of the time I prefer to explain things with code rather than with math. Right? Because I can actually…they’re the same, they’re doing, in this case at least, in all the cases we’ll look at, they’re exactly the same, they’re just different notations for the same thing. But one of the notations is executable, it’s something that you can experiment with, and one of them is abstract, so that’s why I’m generally going to show code. So the good news is, if you’re a coder, with not much of a math background, actually, you do have a math background because code is math. Right? Now if you’ve got more of a math background and less of a code background, then actually a lot of the stuff that you learned from math is going to translate very directly into code, and now you can start to experiment really with your math. Okay, so this is a ‘loss function’. This is something that tells us how good our line is. So now, we have to kind of come up with: “What is the line that fits through here?” Remember, we don’t know (we’re going to pretend we don’t know) so what you actually have to do is you have to guess. You actually have to come up with a guess: what are the values of a1 and a2? So let’s say we guess that a1 and a2 are both 1. So this is our tensor. ‘a’ is (1.0, 1.0), right? So here is how we create that tensor. And I wanted to write it this way because you’ll see this all the time. Like, written out it should be “1.0…” (sorry…it should be -1)… Written out fully it would be “-1.0… 1.0”. Like that’s written out fully. We can’t write it without the point, because that’s now an ‘int’, not a floating point. So that’s going to “spit the dummy” if you try to do calculations with that in neural nets, all right? I’m lazy, I’m far too lazy to type “.0” every time. python knows perfectly well that if you add a dot next to any of these numbers, then the whole thing is now floats, right? So that’s why you’ll often see it written this way, particularly by lazy people like me. Okay, so ‘a’ is a tensor. You can see it’s floating-point – you see like, even PyTorch is lazy, they just put a “.” they don’t bother with a 0, right? But if you want to actually see exactly what it is. You can write “.type()” and you can see it’s a ‘float’ tensor, okay? And so now we can calculate our predictions with this, like, random guess [email protected] (matrix product of x and a), and we can now calculate the mean squared error of our predictions and their actuals and that’s our loss. Okay, so for this regression, our loss is 8.9.
And so we can now plot a scatter plot of x against y and we can plot the scatter plot of x against y-hat (our predictions) and there they are. Okay, so this is the (1 , -1) line …sorry, the (-1, 1) line and here’s actuals. So that’s not great, not surprising, it’s just a guess. so SGD, or “gradient descent” more generally (and anybody who’s done any engineering or probably computer science at school will have done plenty of this, like Newton’s method what all the stuff that you did… university – if you didn’t, don’t worry, we’re going to learn it now)… It’s basically about taking this guess and trying to make it a little bit better. So, how do we make it a little bit better? Well, there’s only two numbers right and the two numbers are and the two numbers are the intercept of that orange line and the gradient of that orange line. So what we’re going to do with gradient descent is we’re going to simply say: “What if we change those two numbers a little bit, what if we made the intercept a little bit higher…?” or a little bit lower? What if we made the gradient a little bit more positive or a little bit more negative? So there’s like four possibilities. And then we can just calculate the loss for each of those four possibilities and see what see what worked. Did lifting it up or down make it better? Did tilting it more positive or more negative make it better? And then all we do is we say, okay, well, whichever one of those made it better that’s what we’re going to do. And that’s it. Right? But here’s the cool thing for those of you that remember calculus – you don’t actually have to move it up and down and round about, you can actually calculate the ‘derivative’.
black bears) and then when we’re done, we check the loss function and the accuracy to see how good is it on a bunch of images which were not included in the training. And so, if we do that, then if we have something which is too wiggly, it’ll tell us. “Oh, your loss function and your error is really bad”, because on the bears that it hasn’t been trained with, the wiggly bits are in the wrong spot. Where if it was underfitting, it would also tell us that your validation set’s really bad. So, like, even for people that don’t go through this course and don’t learn about the details of deep learning, like if you’ve got managers or colleagues or whatever, at work, who are kind of wanting to, like, learn about AI, the only thing that you really need to be teaching them is about the idea of a validation set. Because that’s the thing they can then use to figure out, you know, if somebody’s selling them snake oil or not, you know, they’re like, hold back some data and then they get told, like, “oh here’s a model that we’re going to roll out” and then you say “okay, fine… I’m just going to check it on this held out data to see whether it generalizes.” There’s a lot of details to get right when you design your validation set. We will talk about them, briefly, next week, but a more full version would be in Rachel’s piece on the fast.ai blog called “How (and why) to create a good validation set”. And this is also one of the things we go into in a lot of detail in the ‘Intro to Machine Learning’ course. So we’re going to try and give you enough to get by, for this course, but it is certainly something that’s worth deeper study as well. Any questions or comments before we wrap up? Okay, good. All right, well, thanks everybody. I hope you have a great time building your web applications. See you next week.

1. Kelvin Chan says:

Excellent introduction and many practical technical tips. One thing, I suspect at 1:55:51, those 3 charts may come from Andrew Ng Machine Learning MOOC at Coursera. Unsure if the Quora author properly credited things.

2. wolfer I says:

thanks Jeremy

3. Peter Treese says:

Can you please share the code for the FileDeleter or the colaboratory that you used?

4. Mike G says:

5. shreyanshvalentino says:

48:31 in what universe, do you think the error rate gets better?
With default lr the error rate was 2percent and with 1e-5 lr, the error rate is 40percent

6. Davinder Chandhok says:

I think your analogy at 16:00 about soccer is far off. First, you don't give the kid a ball to watch soccer, but you do give them some initial ideas on how the game works. Second, when you suggest someone might go academic on soccer, you're exaggerating to the absurd. Of course we're not suggesting you teach us basic arithmetic 1 + 1 = 2 so that we can learn deep learning, but it is certainly handy to have an overview of a neural network. After all, if we're going to exaggerate to the absurd, your videos are basically magic shows without Houdini…

EDIT: and need I say that most people don't expect to spend years learning this stuff before they write production-level code (which your analogy to learning soccer for years suggests)?

7. Naushil Bheda says:

I love this course, but really wish the audio was a bit clearer.

8. Sandeep Challa says:

Hi Jeremy. The javascript doesn't seem to generate the text file for me. I am using google chrome.

"urls = Array.from(document.querySelectorAll('.rg_di .rg_meta')).map(el=>JSON.parse(el.textContent).ou);
window.open('data:text/csv;charset=utf-8,' + escape(urls.join('n')));"

This is the script that I am running. Am I doing anything wrong?

9. Yasser Salem says:

Thank you very much!! I just have one question.
How did loss get the backward function? It is a torch tensor and does not have a grad function

Thanks

10. cloogshicer says:

Please improve the audio quality for the next course! A better microphone would be great.

11. YuJin Dev says:

https://youtu.be/ccMHJeQU4Qw?list=PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn&t=3728

Thank you so much Sir. Highly indebted.

13. Aiden Still says:

Thanks! What do I need to know to create my own Python deep learning framework? What are the books and courses to get knowledge for this?

14. fen1x_ says:

How did Jeremy infer the learning_rate to be 3e-5 by looking at the graph? @28:40
I'm surely missing something. Any help is appreciated. Thanks!

15. 1Life says:

This man is a wizard!!!

16. Nagendra Puthane says:

46:31

17. Will K - Politics/Culture says:

I did different types of cucumbers (English, Field, and Lemon) by getting the images from Google. So fun! Sadly the GCP web deploy didn't work due to some fastai library changes I think. Would be nice if it worked out of the box so it should probably be updated.

18. Nigel Higgs says:

I made a YouTube video of a YouTube video specifically this video lol I made an image classifier with a custom dataset in google Colab if anyone is interested.
https://youtu.be/ubY_x2MMPuQ

19. Bhoomireddy Akhil kumar reddy says:

20. Luis Ernesto Morales Cordova says:

see video on 2x and u will get doble dose of jerremy wisdom

21. nofreewill says:

I love this course.

However, I think that human creativity should play a role in experimentation while you experiment. When you are done experimenting and you want to show your results, you should make the notebook so that it runs top to bottom, or at least making it clear how to run the code.

Second, setting random.seed so that anyone can reproduce the same result as you did may be very important. What if you didn't set it, you got good results and someone else tries to reproduce it but is not able to. He has now a hard time to tell if he has some bug in his code or the worse results are just because the randomness. First you should make it reproducible. Then you can see if it is robust. I think.

22. snippletrap says:

Overfit == the wiggles are far too wiggly

23. snippletrap says:

"If you stopped listening now — please don't, that would be embarrassing…" LOL