Let's Learn TensorFlow.js!
What is TensorFlow.js, what can you use it for, and how can you get started with it? Bekah Hawrot Weigel will teach us all about it!
Links & Resources
- https://www.tensorflow.org/js
- https://cocodataset.org/
- https://codepen.io/jasonmayes
- https://codelabs.developers.google.com/?cat=tensorflow
- https://www.oreilly.com/library/view/learning-tensorflowjs/9781492090786/
- https://www.learnwithjason.dev/machine-learning-with-tensorflow-in-javascript
- https://virtualcoffee.io/
Full Transcript
Click to toggle the visibility of the transcript
Captions provided by White Coat Captioning (https://whitecoatcaptioning.com/). Communication Access Realtime Translation (CART) is provided in order to facilitate communication accessibility and may not be a totally verbatim record of the proceedings
Hello everyone. And welcome to another episode of Learn with Jason. Today on the show, we have Bekah Hawrot Weigel. Thank you for joining us today. How you doing.
Thank you for having me. I'm excited to be here.
I'm thrilled about today because we're doing things I don't understand. Which is always fun. Machine learning and TensorFlow and TensorFlow.js. But a little about you. Who are you, what are you up to, a little background.
I don't have a traditional background. I taught college English part-time for ten years before coming into tech. And so I just made the transition a couple of years ago. So I've been a front-end developer doing contract work and work with react. When I graduated boot camp I thought I would never work with react. It was a challenge to me but I really enjoyed doing it now. I also started the community, the developer community virtual coffee, virtual coffee.io. I hang out with cool developers all the time and learn lots of new things. This is one of the things that I was inspired. Gant comes to virtual coffee and he has a book out. I dove into this book and it's been a great experience to learn something new and be able the talk about it.
That is awesome.
Thank you Rochel for the sub. I dropped a link for the virtual coffee into the chat for everybody. Let's talk a little bit about Gant is in the chat. What's up, good to see you. We've had Gant on the show before and done a little with TensorFlow. It's been like, yes, yes, TensorFlow, let's do something else. We don't like talk about TensorFlow itself, we tend to jump right into a project. I'm really excited for today to just talk about it in the more abstract. So what is TensorFlow?
Maybe I'm going to back up just a little bit if that is okay and talk about machine learning and how I got here. When we're thinking about machine learning, we're creating a place where computers can take datasets and label them and analyze them and they can come out with some usable data, I guess you can say. So of course, we as humans can do all of that but computers can do it on a much larger and faster scale. The potential for what can happen through machine learning is really, I think very fascinating because this is how we can make advances in so many different places and industries. I think starting there, and seeing, okay, what is the goal that you have in mind and what path should you go down. And that is kind of how I started being interested in machine learning because I went to a talk from Guy Royce at Revolution Conf. It was on Dungeons & Dragons. I didn't know anything about that. He gave the talk in a way, I know how this could have been used in my life to prevent some of the challenges that I had been facing through this idea of well, you know, we could take these images and we can find a way to label them and to detect certain problems that someone might be facing. For me, I had gone through an issue where I had some of my organs ruptured. And I thought, this, in that moment, I was like this was predictable. As he was talking about Dungeons & Dragons, I thought this is my story in a very personal way. It took me down a path of, this is something that I know I want to learn and I'm very excited about this. For me, it's been this whole personal journey of understanding, like, okay, we can use this to solve problems for so many more people because what happened to me, is going to happen to other people as well. So how do we prevent these things from continuing to happen. Deep medicine is the name of a book by Dr. Tople and he talks about the medical applications for it. It's one of my favorites: I visit right behind me.
This is what I like to refer to. That is excellent, I did a good job of getting that in the camera and not blinding everybody. Sorry. It's deep medicine, how artificial intelligence can make healthcare human again. And so for me, I'm not a data scientist. I think data is fascinating and I love to be able to keep track of data. And I am, it's not that I'm not a big fan of dungeon's and dragons, I just never played it. The potential is there, one day I will be cool enough to play Dungeon & Dragons. I don't have huge experiment working with data. At that point I was a front-end developer. I'm like, okay, how can I still figure out what something that is meaningful and something that takes me closer to this idea of being able to create an impact on projects that can help a number of people or that can allow access to information that hasn't been provided before. And so this is kind of like where TensorFlow came along. So TensorFlow is software that can create machine learning models for desktop, web, mobile, and cloud. And so when we talk about TensorFlow.js, I'm trying to follow along in the chat. Sorry, I get so distracted. I love the chat and then I get distracted on what I'm saying. When we talk about TensorFlow.js, what that means is we can run this in the browser. And if we're talking about it on the larger scale of what is really interesting about being able to run it in the browser. Well, it provides access to so many more front-end developers that haven't had access to it. Historically it's been something that's belonged more to back-end languages and Python. For me to have access to it would mean learning another language. I think that we all have potential to do that but to be able to explore it from this perspective, I think is really great because also, I don't have the typical perspective of someone what's been working with Python for years. My approach to how we can use it for front-end development is, I lost an earring -- is really something that I think allows more people to contribute and to shape what it's going to look like. And for me, that's super important. Because the more voices we have out there, the better. I'm not a Fortnite gamer, either. I will play some card games, Euchre.
All right.
Do you play Eucher.
I play Canasta which I think is almost the same
rules. That is how my family spends time together. We can't just sit around and talk but we can play cards for four hours straight.
That is exactly my family too. We're moving or we're playing cards. It's such a regional thing. Everybody has their own card game I think.
I really do think, I feel like somebody taught me Euchre and this is just Canasta and then this, wait, that is not Canasta.
I'm writing down Canasta, I'm going to check it out.
This is interesting, one question that was interesting, does machine learning require math and calculus knowledge or can you just start with the coding knowledge you have now?
This is another reason that I love TensorFlow.js. It allows you to almost immediately access it. You, if you understand some JavaScript and there are so many great tutorials out there that you can work through and learn as you go. There's lots of different approaches to learning and for me, I'm still learning TensorFlow.js. But it's important to be able to learn about these things and talk about them. But some people say, well, you have to learn JavaScript before you learn react. I think that is fair for some people that might work, but for others like learn in the way that is most accessible to you. And for a lot of people, that might be starting with react and then as things click, you're like, okay, now I understand a little more about JavaScript. So there's nothing to say that you should do it one way or another.
Yes, I think like, there's always this discussion of what should you learn and what shouldn't you learn. At the bottom of it is, sure, if you learn everything, your life is going to be easier. If you learn the thing underneath the thing, of course, you'll have a better understanding of how it works. But you don't need to learn the thing under the thing to use it. You can just learn the thing you have to do and that's o.k. That's been a big, I think that's a big thing especially for me as I don't like it when people are like, well, you can't do that until you learned X. I remember when I was first getting into coding, I was trying to write some HTML stuff and I asked question about flash, and somebody was like, well, before I tell you that, can you tell me this piece of trivial about HTML.
And I'm look, oh, now I hate you. I'll go Google it I guess.
Some people have great memories. I do not. I'm a really good note taker and I will refer back to that. But for me, if you put me on the spot, I'm not going to be able to answer all the questions. But I can for sure, figure out how to look them up or say I know I have this in these notes here, let me go and grab that. I think getting back to TensorFlow.js, this idea is we can get these things up and running. There are models created for us that allow us to do this. And I think that the most important part is allowing yourself the ability to explore this and how you are doing it and getting excited about it because that's what's ultimately going to lead to what you're going to do next. Do you want to deep dive. The more you know it, obviously, the better. Then you can start training your own models or adjusting existing models or working off of datasets to do something new and different. But I say, get started with something that excites you and see how you go from there.
Yes, absolutely. Russian from LA, what's a good test project to learn with TensorFlow. We're going to try that out today, so just stick with us. And another question is around doing ML in the browser is it way more limited than a back end server with all the extra gear? I think the answer is yes, if you have more equipment you can do more work. But that doesn't mean we can't do cool stuff.
Exactly. You can always be building and doing new things and exciting things with what you're doing. Start at a place that you can learn and understand and then start moving forward and being able to do those other things. Like, you know, AirBNB uses TensorFlow for things and there's lots of cool things you can do that build off of the understanding.
All right. So is now a good time to start trying this and seeing what we can get built?
Yes, I think we can certainly go in that direction.
I'm going to move us over into pair programming view here. Before we start, I want to do a shout out, we have live captioning on the show like every show, that's on the home page, learn with Jason.DEV. That is provided by White Coat Captioning. Thank you for being here. It makes a real difference to have a real person doing the captioning. Thank you, White Coat. And that is made possible
through the support of our sponsors. They all kick in a little cash, we have Netlify, Fauna, Hasura, and Auth0 all kicking in to make this show more accessible to more people. While checking things our check out Bekah on the Tweeter. That is a good account to follow. You'll learn a bunch of cool things, VirtualCoffee.IO. Here is another link to that. Today we're talking about TensorFlow but TensorFlow.js. So here is a link to TensorFlow.js. So the difference if I'm understanding correctly, TensorFlow is the way that you do machine learning. You train models using the TensorFlow setup and TensorFlow.js is a way to access the TensorFlow stuff but the way that Java operators.
There are versions of TensorFlow that you can do different things with. This will let us run it in the browser. You can use node JS with this. I'm pretty sure I haven't done that because does not play nicely with my Mac book. But I'm getting there. I'm moving in that direction.
Great. Okay. So if we're ready to start here, what's the first thing I should do?
We are going to be working with CocoSSD today. They have this nice COCO dataset.org and they show you how it works. We're going to do some image detection and looking through that. This is a good way to just test it out. If you want to click maybe a couple of those images and then run a search, you'll see what we've got going on.
Here is an umbrella. And I see pizza. Let's do it. Okay. So this one is detecting all sorts of stuff including there is pizza, this is cool. So you hover over the thing and it shows you the thing. And oh, there are people. You can see the people. Here is an umbrella. That's a tricky, that is just like a tricky identification in general. That's a blurry umbrella in the backgrounds.
Right. Sometimes you'll find it's more accurate than others depending on what you're doing and how you're looking at it. But it's -- I don't know, this feels like a playground to see actually I found one with a refrigerator and a bear in it just clicking around to see the most ridiculous things.
Yes, you have pizza and have umbrellas in the background and a bench somewhere. There's the bench. This tiny bit is identified as a bench. That is impressive. Cars, a truck. It can identify a car around his
nose. That is impressive.
There's 80 classes of objects that they're looking at here. So it's obviously not going to detect everything in the image but what it does is allow you to look at and classify certain things.
Now let's see if it has an opinion on whether or not hot dogs are a sandwich. There is no hot dog. Wait, this is the hot dog and that's the sandwich.
Okay. Is that a hot dog?
Is that a hot dog? It looks like a sandwich to me?
There you go, there is your answer, a hot dog is a sandwich.
Yes, it's definitely got some -- oh, that one, wait, these are both the same thing, right. It thinks one is a hot dog and one is a sandwich. There is our answer, we have no idea.
It doesn't know, it does not know either, nobody knows for sure.
I love it. Okay. This is great. This is great. So this is the dataset that we're going to use that we'll do this identification.
Yes. Each of those classes of objects has a name and an ID and label. And so what you're seeing here with things like pizza, that would be your label for that class of objects. And that's how it goes through and identifies it and you'll see as we make our pre-discs with the image, that it will identify those things as well.
Interestingly, this thinks a hot dog is pizza.
I feel like that just blew my mind.
A new mystery. Okay. So to use this then, we can use this dataset with TensorFlow.
Yes.
And to do that --
We are going to spin up a new project and we're going to need TensorFlow.js and we're going to have to load that model and we're also going to be using parcel to -- I hope I said that right because I've only read it, also to be looking at this in the browser.
Is it just parcel like that.
It's parcel-bundler.
Okay. And then I need, come on, buddy.
The last one is TensorFlow dash models slash COCO dash SSD.
I'm going to need you to walk me through those one more time. So it was TensorFlow.
Dash models.
Okay. One second.
TensorFlow models.
Slash.
COCO.
Dash.
And then the other one we needed is TensorFlow, is it prefix like that.
It is TensorFlow slash TFJS.
TFJS. Is there another one I needed.
That was the three. That is all we need.
Okay. Let's get these in here. Thank you for the elevator music, Nicki. There we go. We have these ready, let me open this up here. It's going to open in the wrong window so we'll move it over here. Inside of this we have nothing yet. It is just set up with parcel bundler and our dependencies set up here for TensorFlow. What is my next step?
I would say our next step is probably to create an index.html file.
Okay.
There are two different ways you can run this. You can run this.
Sorry, that's the sound effects. I forgot to warn you about the high level of chaos on this show.
Um, we can go ahead and run this with a script and I probably should have sent this to you beforehand. Let's see here. Or we can go ahead and use parcel to run it. We're going to go ahead -- I put that in the Zoom chat.
I see. We'll have to transcribe between the two but that is okay. We have CDN.JS deliver.Net. We're not going to install it, we're going to pull it in from a CDM which is great.
Yes, we'll move to that next. This for me was the easiest way to get things up and running. One of the biggest frustrations I had getting started learning TensorFlow was getting started, getting TensorFlow running. And so I think there's really nice community out there, if you need help, definitely reach out to people for help and check existing issues for things. But being able to run it with the script tag is a really great way to get up and get started. And actually, you can find a code pen, there is TensorFlow organization I think has some things to get you up and running and started. Let's see here.
Npx serve, I think.
So what we'll do here, the first thing we're doing with that script, the TensorFlow TFJS script, that is required. We need it to run the Coco-SSD model. The second one is loading the model. That's the one that is TensorFlow models slash COCO SSD. If you go to the TensorFlow site, there are lots of different models you can use and work through. Image detection is just one of the small things that we can do with that. What does CoCo-SSD stand for? COCO is the name of the dataset, SSD is single shot multi-box detection.
COCO is common objects in context. COCO.
There we go. We're going to work with these together to identify objects in an image like you saw that we did and we're using the TensorFlow object detection API as well.
Okay. I have this basic page, it's loading from the CDN so we can see in here that we've got those models being loaded. So we are, I think we're ready to try something.
Excellent. So one of the things that you'll see with a lot of the models and examples is the answers are going to be con soul logged. We can add to what we're doing to have it printed. But a lot of the things have the predictions going through there and that is what we're going to be learning as well. Gant is always coming with the jokes. He put an SSD hard drive in his COCO. That is not the answer to learning how to do this. Okay. What we want to do next is, trying to think if we should start, we should probably get an image that we want to look at and see if it can identify those things. Like you saw in the pictures that we went through and on the COCO site, it's nice to have something that has a number of different things happening. Having restaurant people.
Here is a bunch of stuff. There's food, there's, well that one's got -- that is all stock. How do I not use those? Are these all stock? I don't want premium. Here is some not stock images. I forgot they got acquired. Here's a good one. There's lights, people, glasses, we'll use this image.
Okay. Where he want to go ahead and save that and just import that into our repository that we're working on. Was that Gant's voice.
Yes. That was Gant's voice. That is a sound
effect back from when he was on the show.
That is wild. It's great for my ADHD.
I have the image link or I want the image itself.
Go with the image itself. Sometimes there's an issue with grabbing the link and being able to analyze it.
I have this and I need to save this in my, in the project.
Yep. Just save it right there in the project. We're not going to do anything super-complicated here: You can just add that in right there just so we can grab it and take a look at what we're doing.
Okay. Now we have, here is the image. It's pretty big I think. That's why it's not doing it. But that is okay. We will, I think I broke my VS code. I'm really excited about that. It's going to be a fun problem to debug.
That's great.
Super looking forward to it. We have an image here and we should be able to grab it and console log something.
We have the model loaded and the next thing is you add the image. So using, we're going to give it an ID because we're going to need to grab that ID and give it to the model. Let's see here, it should be an image element, give it an ID of image and then give it your source, exactly. Import it there.
Okay. Then if I can figure out why my -- ut-oh -- it just decided to run it 1,000 percent. That is always a good sign. Is it frozen everywhere? It is frozen everywhere. This is fun. Good.
This is excellent. Now you have learned how to run TensorFlow.js.
Okay. Yep, yep. Do I need to force quit it? Come on, we're going to force quit it. Okay. Let's try that again. We're going to bring back Edge. When it arrives -- it's just straight up crashing. This is promising. Let's try Chrome, how you doing today, Chrome?
I want to say, too, there are a lot of great resources for TensorFlow.
Now that I started opening another browser and it was like -- no, I work, I swear. Can I just restore -- wait, this is my, that is my work profile. How about we use Chrome today? How about that?
Sounds like a good place to be.
Okay. So this is -- you hackers, you dirty
hackers.
Okay. Now we've got ourselves an image, it is working. I'll adjust the tiniest bit of style to make that a little easier to look at. Okay. Now we can actually see the image, we've got it in here. We're looking at the console and we're ready to do stuff.
Okay. Now, we're going to place a script tag.
Okay.
That's going to go below that image. We want to make sure that everything all gets loaded up and then we can grab that image and console log the prediction. Now we're going to create a variable that's called, we'll call it image IMG this time.
Okay.
You're going to say document, get element by ID and guess what the ID is? Because there is one on the page. That's what we want there. Now we're going to go ahead and load the model right below that. Go ahead and click enter. To do that, we're going to use, it's going to go COCO capital S and a lower case sd. There you go, dot load. We're calling that as a function so you have the two parentheses there. And then a dot then. And we're going to go, we'll call it model and then, there we go, error function. We're going to go ahead and detect the model. We do that by saying model dot detect and pass an image.
Okay.
Yep, there we go. And then we're going to go make another then so we can predict what is happening here. Dot then, we'll call it predictions here. And then another arrow function, console log the predictions. And we'll go ahead and give this a save and we should see something happening. Oh no.
From pixels cannot be null. I did something wrong.
It is not --
What up? What up? Oh, I need to -- one second. Let me open -- I have my whole chat shut down and everything because it crashed. Let me get this pulled up again so I can see what everybody is saying. Okay. What was wrong? Image instead of image. No, this one, the ID is image. So we're pulling it in like that. Oh, but wait, it doesn't need the hash. If we're using image.
Okay. Go ahead and get rid of that. That should work. Nice call who said that.
Thanks Tony and everyone else. They're like look, Doofus. Okay. So we found six people.
No this will work, this will work.
It is a bit okay. It's like 77 percent sure that's a person. 73 percent sure that's a person. All right. All right. We're getting -- it looks like, yes, we're 50 percent sure that's a person. We're at least half way sure that is a person.
It scores between a zero and one and I think the default is .5 or 50 percent. So that last one is just barely sure. You see there's a B box there, that's a bounding box, we'll talk about that in a little bit. I think we'll get to that. We'll draw the boxes around the people in the image so we can identify, okay, yes, that definitely was a person. And that's what it was looking at. So we'll get a little more specific here.
Nice, very cool. Okay. So we've got basic image detection. If I swap this out for something else, like, let's go here and look up pizza. Here is a pizza. And let's download. Then we'll go to this one, I think. People in a restaurant. There it is. So if I copy this one and change it to be pizza, then when we come out here and hit refresh, it has a picture of pizza. It identifies pizza. Okay. We've got ourselves --
What does it see? It sees pizza twice.
Sees pizza twice. This is promising. This is very cool.
Yes, it is. And pretty simple to get up and running and you can see, this is pretty accessible. You see that you're loading that model and then you're detecting and making the prediction. So there are these steps there and some of them, some of them make it easier than others to walk through. But I think we get a general understanding of what we're looking at and how that process is working.
Yeah.
So the next thing I think maybe we want to do is try and do this without the script tags and using NPN instead. Does that sound like a --
Sure.
-- place to go. Let's see, what do we need? We're going to have to add more dependencies here.
Okay.
We are going to need to add stuff to handle back end. We'll go back to TensorFlow, add another, it's going to be, let's see here, TensorFlow slash TFJS
dash back end dash CPU.
Okay.
Sorry, I couldn't see it on my screen. We're going to add, it's going to be same thing except instead of dash CPU, it's going to be web GL.
Web GL, like that.
It's coming in slower on my screen so I'll let you know. Yes, that's right. Then we're going to do the next time it's TensorFlow TFJS dash converter.
Okay.
Then let's see, the next one is TFJS dash core. We're going to see --
Like TensorFlow TFJS core like that.
It's TensorFlow and slash TFJS dash core.
Okay.
Let's go ahead and install these. I think it should work from here. I know I had some errors and had to do some other stuff to get it up and running. Hopefully, this works but if not, I have an idea of what we should be doing.
It looked all of those. So they are in.
Yeah. All right. We'll go back to that index, okay, so now what we want to do, we're going to be doing a little bit of work on this and we're going to do it simply. We're going to add in index dot JS and index dot html and we'll get rid of the script stuff we see in there.
Does the index JS go in a source folder?
You can put it in a source folder, yes.
Okay. All right. And then out here we get rid of these.
Yes.
And then we can also get rid of this?
You know what, we might be grabbing that for the index dot JS file.
Okay, I'll comment it out for now.
Yes, that's -- go ahead and comment that out. We'll clean up the index dot html file first I think. We will start with, we want to add -- we have the body going on in there and then within the body, we'll want a Div with an ID of app.
Okay.
And then that -- you can close the Div there, that is fine. Below that, this is where we're going to add our image. So where we were detecting it before, we're going to add that image ID, the image element, there we go. That should be good. Let's see, below that, we'll add the script source pointing to our index dot JS file where we'll handle
all of the loading and the detection now.
Okay. So we have our script and I'm assuming we need to tell parcel to bundle this?
Yes. You should, it should be fine if you just run NPM install. If we want to add a script to the package JSON, it will be, let's see, so we want to build it, run the build script which is parcel build index dot html.
Parcel dash build.
Just a space.
Parcel build index --
Dot html.
Okay.
Then we want to make that main says index dot html, I think.
Okay. Let's try it.
Then it also would be useful to have a start, I think in the script.
So it did do what we were after which is it grabbed all the things we needed. Granted, there is nothing in this. This is an empty file but that is good. I start script.
Yes. Might as well. And then that will be parcel index dot html and then space dash, dash open. So that will let us get it up and running, I think or that's how I get it up and running.
I might just pick this up. Let's see what it does. It goes and does the thing and opens. Tada. Okay. Now we have our --
We've got some pizza.
It's running in both Chrome and Edge now. Good. I'm going to click Chrome.
This won't break anything at all. This is excellent.
We're fine, everything is fine.
Now we want to jump into that. Jump into that index dot JS file and tell it to do something. We need to import COCO SSD. So you can do import star import all. As COCO capital SSD, there you go, from, there you go. Grab that one. Then we want to import all -- I don't know if we need to do this or not. I don't think we need to import, well, oh, no. We don't need -- sorry, I'm arguing with myself, this is excellent. Maybe we want to import TensorFlow.js. But we definitely want to import the back end CPU, backend web GL at the top there.
So import everything as backend CPU.
No, you don't need to import it as. So those packages that we installed, you can just skip the, yes, just import and then start with the TensorFlow part.
Oh, we're just --
Yes, we don't need to. We just need to make sure they're there.
Got it. Okay.
Let's see, I think this should be good enough to get us up and running. If not, we'll make sure that we -- oh look, it says there's an error already. Cool.
Do we need core for this to work? Also, this syntax is does it need --
Does it need double?
Not sure.
Quotes.
The single or double quotes shouldn't impact it. Do I need to configure parcel. Is there a parcel config or something.
If we have to add the other package, it should be up and running like this. I think. Wait, let me double check. COCO SSD, we have that, we have TensorFlow.js. All right, maybe let's add the -- what ended up working for me was adding regenerator run time. That is another package we'll have to add.
Okay.
Let's see here, it's regenerator dash run time.
Failed to install.
Backend.
Hold on, did I screw something up?
Yes, maybe.
Web GL.
Oh, jeez, okay. The problem was me all along.
Typing.
As it is. As it usually is. What is GPL, what was in my head when I decided that was a thing?
That's a tough one, GPL, those are three very different letters, gigantic purple lion.
So that's exciting.
Cool. All right. Well, I'm always happy when the answer is I'm incompetent. That is so much easier to dole with than the computer is arguing.
Great. That is a lot of what I've been dealing with. We are going to -- oh, we want to import that image -- do we want to import the image? Yes. Let's go ahead and import that image.
Okay.
Below that we're going to run an async function here. We'll go ahead -- I think this is essentially the same thing that we did in the index of html.
This one here?
Yes. So we, yes, exactly. Just grab all of that stuff.
Okay.
We're going to load. And then model.
Like that?
Go ahead and run MC, I think that can work.
Regenerator runtime is not defined. I will have to install the regenerator runtime. That is okay. Remind me, is that regenerator runtime like this?
And slash runtime dot JS after that.
This is what is in the package JSON, regenerator runtime like this.
Yes, sorry about that. That is how we import it as the runtime dot JS.
We're going to import, regenerator runtime dot runtime dot JS. Save that. Run this thing again. The regenerator runtime is -- yes, it's a poly fill is really at the core of it. I think it adds some support for battle leaf things, I think. I screwed something up. The thing that I screwed up is: Import image -- hum ... Hum ... Run pixels.
Okay. That's a TensorFlow thing. Pixels passed to TensorFlow -- must be in html video element.
I'm wondering if this import isn't compatible with this call.
Okay. Let's think about this here.
Because we can do the same thing and, like, const image equals document, query selector and then we'll get image and then we can just drop that out. Then this will work because that's going to be the html element. There it is. There is our pizza.
Our little pizza. Good.
I don't know how parcel imports images. That is always configured differently.
Yeah, you know what --
Not this time, Tony because I'm using query selectors.
We could have kept that import and just said image dot source equals image, right. I think that could have worked the same way, too.
Oh, yes. So we could have built an element for it.
Yes.
But you know what, that is easy to show. Let's do it. We will go with image equals new image and then
image dot source equals --
Pizza.
Image source, right.
Yes.
And then this one should work. Let's give it a shot. There it is. So basically we're getting, we're getting the image but we have to put it into a Dom element like that, an html element, and it makes TensorFlow able to parse it. It's looking for this and not just the source. That makes sense, I get that.
There we go. We have this running this way now, I feel like this is the second step. We can run it using the script. Now we're running it using NPN. And I think the next step would be adding the boundary boxes. So we can see, okay, it's saying there is two pizzas here, what are the two pizzas this we're seeing. For this, we are going to be using canvas, the canvas element, I really never work with those. This was my first time diving into working with that. But I think that MDM had a lot of good resources on what it means and how do do it. If people are interested in really seeing the break down of that, you can definitely, I have some links that I'll try to drop in the chat later. Okay. We are here. We have this thing going on. What do we want to do next? We want to -- let's see -- we're going to use the get context method. Wait a minute, we need to add a canvas. That is in the index dot html file. Right there under image, right, do a canvas, give it an ID of canvas. And let's set the width and height just so we can do that.
Are we looking for a certain width and height here?
Why don't we go with 600 by, I think we can do
Okay.
Now we have that canvas element there, there is two images after we run this. Not yet, I don't think.
I think we have to put it into the canvas, right.
Yes, yes. One of them will be the original image and the other will be the one with the bounding box. And then the box is going to label what images it sees.
Okay.
So if it was an umbrella, it would say umbrella. Hot dog, it would say hot dog next to it. We're going to go in that direction and try to get that
up and running here. So the next thing that we need to do is, we're going to use the get context method and that is going to draw it on the canvas. We're also going to pass a type.
Okay, giver me --
Which is two dimensional.
Give me just a second here. We got to start with, like we're going to draw the image, right. So we start by getting the canvas which is document query selector and we get the canvas. And then I need to canvas get context you said.
Yes. You can do it that way. Dot get context and pass it 2D as a string. So you're telling it it's a two dimensional image that we're seeing.
Do I need to save this.
Yes, you can do it that way for sure. We can probably -- yes, do it all in here. I was trying to think. We're going to draw the image, we're going to label the image. And I'm trying to think of, okay, what is the best way -- do we just put it all in here? Maybe?
Then we can call it from here, right, with like --
Yes.
Pass in the predictions, we'll get the predictions here. That might make it a little easier to follow.
For sure, that does a really nice job of breaking it down. The next thing that we'll want to do is draw the actual image. There's a draw image method in the canvas 2D API. So there are three things we're going to do here. We're going pass in the image and then the X axis coordinate and Y axis coordinate.
Image source like this one or I need to pass through the html element?
I think --
Let's just try one and if we need to we'll fix. We want it to be top left, right.
Exactly. Also, we know that we're going to want to draw labels here. We want these labels to be next to the pictures so we can see what it's identifying. So we can add a, let's see dot font. So context dot font.
The provided value is not a -- TSS image value -- we do need to pass in the image. We'll pass image and pass in image -- okay, this should do it now. Okay. Here is an important thing. It doesn't
auto resize so our image is blown off the side here. I'm going to switch this over to be the -- did that just work -- the other one, because it's the right aspect ratio.
As we go through, it will be good as well to see, this has more people and more things that we can potentially see there.
It's also important to note that we don't need this anymore. This might make our lives a little bit easier. We don't need that image anymore.
There we go, nice.
So that's that. If we really wanted to cheat, we can go resize this image to be 600 by 400. Should I do that just for ease of use.
Yes, I think that makes sense.
Okay. Let me, let's go to Figma. Crap, that was the work one, nobody look at that, that's all secret.
You hackers, you dirty hackers.
Let's make a little 400 by 600. Then we can drop our restaurant in here if we can reveal in finder which is over here. There you have it.
Never done that in Figma. That is useful.
It's really, really handy.
Now that I got that, I'm going to drop this one in here and rename, maybe I'm going to rename to restaurant so it's easier for us to find. In here we'll use restaurant instead. So this should now fit the canvas perfectly. The reason we're doing that is because I think our alternative is to do a bunch of math to convert the boundaries, and I don't want to do that.
This is a good thought. All right. Let's keep making -- our boundary box. We want that text in there and we're just going to specify what we want that font to be. So if you go back to, where you have context. Right below line 12, context dot font. And then we can specify what that is. So let's make it 12 pixel and then we can just go with Arial and then a space.
Like this?
Yes, that should work. Now we need to tell it where to start the line pass and make sure it has the opportunity to label all of the images, we can see here we have 7 there. We are going to create -- sorry, I'm looking at what you're doing.
Like loop through each one.
Yes exactly. Predictions and then we want to, we have to begin
the path.
Okay.
To do that, there's a method called begin path.
Like that?
And that is -- so we're going to call that, yes. The next thing, okay, now the next thing -- there is a lot of context dot things. I needed to look and see what is going on with these things. There is dot rect. This is a method in the API that adds the rectangle to that current path.
Is it like next, like context dot -- say it again.
Rect, like rectangle but not with the rest of it. Right. Nice, it tells you what we're looking for there. We want to make sure that we grab all of those predictions, right. Let's go ahead and use a spread operator. Can we do that there?
So basically we want all of the --
All of the images.
It's prediction dot BBOX. And that would be all four points. Okay. Is that right?
I'm looking at what you're doing here. Predictions dot BBOX, yes, that looks like that should work.
Okay.
And then what we also want to do, okay, we have to finish drawing the line. Go ahead and give it a line width.
Line width like that.
Exactly. And then we'll make that one.
Okay.
Then we want a stroke style. We're just going to make a color there. Go ahead and make it green so we can identify what we're looking at there and then the fill style. Let's go ahead and make that blue to make it easier to differentiate what we're looking at.
End path or anything.
Yes. We do that by doing context dot stroke.
Like that.
Yep, exactly. So that should end the path. So we've got like the style created for what we want our bounding box to be. Then we can probably go ahead and, okay, do we need to create fill text for this? Hum. Yes. How do we do that?
BBOX is for bounding box. That's like the outline around a person.
So we need to make sure that each of those, that we can grab each of those and give it a score and a class.
Okay. We have the score and the class in prediction here. Sorry, trying to read the chat and the code and think at the same time is proving to be difficult for me now. We want the score and we want the class and we want the bounding box to go around those. So the Y axis for the text according to the image. That is what we're looking for here. So we want, how do we grab the predictions here? So prediction -- it's not going to be.
It's prediction dot class and prediction dot score.
Right. But we should be able to do just that, right.
How do you add text to the canvas?
Prediction dot score. There we go. And then let's fix that to three.
Where am I putting that? Is it context dot text?
No. I think we should be able to do that. Wait, context dot fill text. I'm sorry. So the inside of there, no, we're going to, that's just, that's a method that we're calling. We're going to put that inside the parentheses. Sorry. My brain had to reset. Prediction dot score and go to fixed there. So we have, let's do three. So we don't have to look at all of the different things. And then we want the predictions class. So we know what we're looking at. Yes, there we go.
Okay.
And then we need the bounding box predictions, let's get the X axis. So we need -- here we go, predictions dot BBOX zero. So we're going to call that first item in the array, I think.
Where do I put that?
Let's back up a second, I didn't look at everything that was going on. All of this can go in the fill text, that first one. We don't need to keep doing that here. You can grab that prediction dot class.
I see. Prediction dot BBOX, zero.
And grab that prediction dot class that is right below on 24.
Okay. So we're kind of making --
Labeling it. So we'll be able to see the score and we'll be able to add the class to that.
So we can put the score in there and try that whole thing. All right. I'm just going to save this
and see what happens.
I feel like that's 90 percent of what I do all day.
We got the first one and it didn't like three arguments required and only two present. I missed the third argument which is --
We need, okay we have the X axis and now we need the Y axis.
That is going to be predictions, actually I can go right up here and just put the one.
Yes.
Let's give it a shot. And look at all of these people. Okay. Because of the -- I don't know how, we're probably going to need to --
This is a rough one to do.
It's still not great but we can see what's happening. It sees, here is a person and here's a person and here's a person and here's a person. It's definitely doing the identification which is very cool. If we swap this out for another image, so let's put the pizza one in here, or let's get a different one, let's try something new. Go to Unsplash and we'll get dogs. This is fun. Okay. Let's get these dogs. Then I'm going copy the image and put it right in here.
So the next thing that you can do with TensorFlow is you might have a model that identifies different types of animals and then you can work with that dataset to train it to be more specific. So this will give us a number of different dogs but using, I think it's called transfer, you can teach it to say this is this type of dog. This is a chihuahua, this is a pug. There are lots of things you can work with. This is a good way to get it up and ready. It's interesting. It's a bear.
I love it. This is great. This dog will be thrilled to learn it's a bear.
It didn't grab the rest of them which is interesting. I think that the -- it can do up to 20 at once. I'm not positive about that.
These are low res images and I'm copying from the previews and not using the full size. This is a 400 by 600 image. There is a lot going on here. Am I even zoomed in. No. That is 100 percent. Yes, it's a pretty small image and you can see down here, this does not look very much like a dog, right. This one looks like a dog, that one should have been categorized.
Yes. Some kind of animal for sure. I think there is a lot of different things that can effect it. I think that's just kind of the larger concept of data identification in general. Like, how is the dataset created. Did it look at different angles. Did it look at different lighting sources. Is there a wide variety of data. Because we find that a lot of the things that go into datasets that can effect them are biases that we might have and that we don't even recognize it. So one of the things about data in general and identification is making sure that we question it and we figure out what the methods and the processes have been and as we develop our own, the same thing goes for always asking ourselves, you know.
Yes. Well, you bring up a good point, right. It's really important to not look at any of these tools as a magic bullet. It's not like you're going to add machine learning to your website and suddenly you have no problems. You have opted into a different class of problems. You're now making sure your data is accurate. You're making sure the detection isn't like, I know Gant had, he has been in the chat today, he has a library called NSFW dot JS for detecting whether or not an image that is up loaded is inappropriate. And early on in the training process, it was identifying things that weren't actually inappropriate as inappropriate. And you have to be careful of that because that's how you end up with things like the misclassification that comes off as super racist because somebody didn't look at their dataset. The classic example is there is a machine learning algorithms that have identified black people as monkeys. Those types of classifications are really dangerous if we're not paying attention.
There was another example that I heard recently, where I think prisoners were using machine learning to identify who got early parole or something like that. But the questions that were being asked were things like, did your parents stay married.
Oh no.
Maybe did you have a parent in prison. So a lot of the questions actually had nothing to do with whether or not they would be okay to be released early. But what they did is mainly identify black males to not be receiving early parole and more white, affluent or middle class people were receiving it because of the way that the questions were created. So, identifying those things is so important.
Absolutely.
Every picture of Jeff Goldblum is slightly inappropriate. But I mean, as a general thing, maybe don't use it to answer ethical questions, maybe use your human brain for that. But for more like automation applications, this can be a really good way to quickly sort things or, like I'm imagining now that if we were trying to do a, like, user submitted anything. This wouldn't necessarily be the judge but it could flag things for human review, right. And say, hey, this one looks like it might need another set of eyes on it and then a person looks and says that is definitely inappropriate, delete it or it's not and you check the box and it goes through. It's very, I can see a lot of really exciting uses for this to just cut down some of the kind of mechanical labor that happens in moderation, in review. Are there any other really exciting applications that you've thought of as you started learning more about this?
So I know that Airbnb uses TensorFlow on their site. And I think they have the specially created model and system because it makes sense. They have a lot of people who are up loading photos of their Airbnb and not labeling them at all. One of the things their model does is run through those images and detects a toilet, this is probably a bathroom. There is a stove, this is probably a kitchen. That saved them a lot of time in having to go through and making sure that things are labeled at all and identifies the things and can apply the labels to things. There's certainly, I don't think that this was in TensorFlow but there is, let me try to say this correctly. There is a project that is being worked onto help early identify postpartum depression. And they're using -- not national -- natural language processing to do that. And the way they do, I think that you can either use your Twitter feed or Facebook feed. You give it access to that. It's looking at how did you talk before having a baby. And now, identifying patterns that indicated postpartum depression and using that to, you know, assign some kind of warning or indication to somebody that is using the app to say, okay, there's been this shift. And the accuracy was surprisingly high.
For something that is so hard to detect just because, I think there's a lack of a conversation around it, it makes it accessible to more people to get early care which is huge in the long-term mental health of a lot of people.
Yes, absolutely. I mean that kind of stuff is pretty cool. But, yes, that is definitely the, not a way I would have thought it would work. That's pretty cool.
And another image detection example is I think Carnegie Mellon was running a study on machine learning on brains of people that had depression. They were looking at using an F MRI to look at different brains and predicting whether or not certain treatments would work for people. So treatments like electroconvulsive therapy or there's a magnetic resonance, I can never remember the third, there's a third letter in that. What they saw was, okay, these people have brains labeled A and D. And they will be responsive to this treatment. And these people have brains labeled -- what did I say? I said A and D? Whatever the other two are, B and C, these people will not respond to electroconvulsive therapy. It was a small study but it was huge because they were right. So these are things that I think have been kind of behind this veil of we don't really know what's happening with postpartum depression or with depression and this provides us some resources to find the right help and support for people. So I think it starts with this, it starts with being curious about what you're learning, being okay with getting it wrong. And I will probably never be a master at machine learning. But I will know it enough to be able to talk about it, to understand what it can do. And to be able to find people who are experts and are being to go down this journey of something that maybe doesn't seem like it's going to make you tons of money but it's certainly going to help a whole lot of people.
Absolutely. And it's cool to think about the, just the impact it can have on one of the most limited resources we have is attention. And when you start talking about working with people at scale, one of the hardest things to do is pay attention to everything to make sure that things aren't going off the rails. So one of the things that I find exciting about these use cases that you're talking about is some of this stuff are things that you can justify saying, well, there was no way for us to know. There was no
way we could have seen it coming. This is a way to get that early detection. And not to say, like, that a thing is or is not happening. But to say, hey, we're seeing a pattern here, somebody should look at this. That to me is really, that is a positive use of this technology is letting us know where should our limited attention be directed. And making sure that we don't miss things that we should have seen coming. Somebody is talking about self-harming in Instagram algorithms. If somebody posts something that they're contemplating it, it should probably get flagged to humans for moderation if that is what is happening. Don't robo dial the cops, that's a bad way to do something but hey, humans go and look at this post. Does that person need someone to follow up with them.
Yes, I like that stuff a lot.
It provides access to the conversation. A lot of those conversations are really hard to have with other people. To say, hey, there is an indication that I should talk to you. It almost provides a buffer between being very, being very upsetting and saying, okay, there was a step that said maybe you need some help. And allowing something that seems more data driven to identify those things can lead to that, I think. Somebody also mentioned, I have four kids and they are, their ages are almost 5 to 12 right now. I do a lot of Zoom calls. I work from home. So sometimes I get the mom face on they call it. They know when my kids have entered the room. There should be some kind of image detection that identifies when the mom face comes on and auto mutes for me. I don't have to worry about yelling at my kids.
I can imagine a few people who would love that feature. I think that's super fun. I love those sort of out-of-the-box applications of this. I think sometimes we get a little too laser focused on what machine learning can be used for. But getting outside the box on that and thinking about -- even fun stuff. Like I sad Jason Mayes was on the show and we made a password protected -- I can show you this. It was, oh, I have a search now. This is going to be amazing. I forgot. I forget and I'm getting ready to command F. We did machine learning, it's the one where we did it. We built a demo where you had to show a picture of a -- I don't have these objects so I can't do it, you
have to show it a fork, a book, and a banana and that's the password. Not everything needs to be, there are very big far-reaching implications and they're also very fun playful applications. And I think that it's worth digging into both and see what you can do. Just remember, nothing is a magic bullet, right.
Yes. Exactly.
Cool, so Bekah, this has been a lot of fun and I feel like we got, like this is very cool that you can so quickly get through and just get these, just these boundaries. I'm happy just to see this, like this part. Where we're able to draw boxes and identify what's in an image. This is a very cool thing to be able to do. And I can see a lot of ways that we can play with that, that we can have fun with that. Is there anything -- where should somebody go if they've seen what we've done today and they want to go further.
They have a lot of really great resources on the TensorFlow site. They have a lot of different -- it's so funny that you're talking about Jason Mayes because his code pen is one of the ones that I thought was a good example. He has some stuff on his site. Or if you go to code pen, there is TensorFlow topics templates, TensorFlow templates. That would be another one. But let's see, there is this, where did I put that link? Code labs dot developers dot Google.com has a lot of stuff on TensorFlow.js and those code labs are really nice to walk you through everything. Gives you an indication of what time, how much time it's going to take to do it. What you're looking at. You can go through a lot of things. I'm sure you can filter by category to TensorFlow maybe.
Yes. Here is a filtered link, everyone. Lots of, lots in here, holy crap.
Yes, so not all of that I think is TensorFlow.js. There is TensorFlow light there. But there's a lot of good starters. I would say, stick with it. Play around with it. Some are more intuitive than others. Or give really great instruction. But reach out to other people who are talking about it or interested in talking about it because I think it's lots of fun to learn these things together and to play with, okay, what can we do with this. Object identification for our password or whatever. Or build me the mom face mute on Zoom.
Mom face mute is a good name for that. I like that. Algorithms virtual coffee. This is a place
where somebody can go and just start a conversation about this, right.
For sure. We meet every Tuesday and Thursday and that's the way to become a member. We have Devs at all stages doing lots of different things. Gant has been coming and I'm going to do a live stream of Gant's book. I have his sticker on my desk. I'm really good at making this blinding. Learning TensorFlow.js is his book. I think all of the chapters coming up here in a couple of weeks, I'll be diving into twitch myself. And I'm really excited to be able to do that because I've learned a lot already. And it's been a lot of fun.
There's the book. Okay.
I'm sorry, I had a link to that. I just didn't drop it.
Yes, so we've got several episodes with Gant on the show as well. If you're interested in seeing what Gant has done on the show. We have a few here. You can search for Gant and you'll find, here is Gant, here is some fun with machine learning. I think we put some masks on people for this one and then with this machine learning one, we did a time warp filter. Which was just chaos, super fun. Lots of fun things you can do with that for sure. So head over to the site and search for Gant and you'll find lots and lots of fun stuff there. With that, Bekah, I think we have hit a great stopping point. We have places to go. We got something running. We're in solid shape here. Any parting words for the chat before we call this one wrapped up.
I want to say thank you to everyone who has been here with me. This has been such a fun journey to talk through this and to do it in front of people is terrifying and I appreciate everybody's kindness here.
Absolutely. Well, we appreciate you being here. Chat, why don't you hang on out. We're going to go and raid somebody here. So we're going to find somebody to raid and with that, make sure, well, before we do, that why don't we head over to the learn with Jason site and talk about our live captioning which we've had rolling all day. Thank you to White Coat Captioning for making that available. That is available on the home page and it's made possible through our sponsors. While on the home page, all of these are click able. You can head over to the schedule. We have so much good stuff coming up. Next week Lynn is coming on
the show. We'll talk about UI and UX. This is really, really good stuff if you want to make the things you build look better which we all should. Then teamwork flows in notion. I use notion all the time for work and personal stuff. Really excited about that. We have white panther coming onto teach us next. We're going to do personalization. We have observability in JAMstack sites and the list just goes on. There is so many amazing people coming onto the show. Head over to the schedule. Hit this add on Google calendar button you see it on your calendar when shows are going to happen. I can't wait to hang out again, we will see you all next time. Thank you so much.
Bye.
Learn With Jason is made possible by our sponsors: