Video instructions and help with filling out and completing Who Form 2220 Toggle

Instructions and Help about Who Form 2220 Toggle

What is going on everybody welcome to part four of our web scraping with beautiful soup 4 tutorial series in this tutorial we're going to be talking about is how to scrape dynamically updated information from a web page so to begin I have added some information to the Parkes mEEMIC parse face page underneath this picture you can see this JavaScript dynamic data test and it just says look at you shine in it says that because we're viewing it with a client in a browser and the browser is actually doing something that makes that show up let's look further so viewing the source code swimmin in' scrolling down here is what we're looking for so this is what we I was just showing you whoops okay so what's happening is this paragraph text is of class JavaScript test it's got an ID of yes or no or yes no js' and the starting text is why you bad though but we didn't see that that's just the starting text underneath it we've got a script so JavaScript and what it's doing is it's finding the element by an ID that ID is yes no js' just here and when it finds that it's going to say the dot enter HTML so we're going to say okay now the HTML says look at you shine it that's what we want it to say so when Google Chrome or whatever browser are you using browses this page it says look at you shine it it requested data from the server and the initial information that the server said was no no between the paragraph tags this is what you want to say but then it also said here's a script though by the way and then chrome ran that script so actually you run that script the server didn't run that script you ran that script so closing this out if we wanted to parse those tags we would use the following code you don't need to write this out I mean you can if you want we're going to use it in a moment well towards the end but hopefully if you've been following along you already have everything up to this just without these two lines anyway this is basically the code that we would use we're going to parse paragraph tags with the class J's test simple enough so let's save and run that real quick and we get why you bad though that's not even what we thought we were supposed to get so a lot of times maybe you're going to parse a webpage that maybe you're going to parse a table you're expecting that table to have some data and lo and behold it has nothing like without so the problem is you aren't a client you're not a browser so what we have to do is mimic being a client or a browser and actually run that JavaScript which is actually a little bit more involved than you might think or maybe you are thinking that and realizing oh no so so there's a whole lot of options at our disposal for how we're going to do this I think the easiest way to do this is to use PI QT specifically we'll be using QT 4 I'm sure you can do it QT 5 but I'm just going to use QT 4 so I do have a tutorial on PI QT for you don't need to follow this entire tutorial but you should go to the first step the first page and you'll need to get QT 4 if you're on Windows go to this URL here and download the wheel for pi QT 4 there might be an installer now or something really simple you can check that out if you're on Windows you can go to riverbank computing and I think you can install it from there it's been so long I don't remember what method I used on Windows but I have it it's possible if you're having problems leave a comment below and I'll do my best to help you out so once you have QT 4 you're ready to rumble so we're going to go ahead and come up above these imports and we're going to start importing a few new things first we're going to import sis we're doing that because QT applications want to be able to take system arguments and it's going to be angry if you can't so we're going to do it then from PI QT 4 and I'm going to go ahead and just copy this so I don't type of it but it's capital P lowercase Y capital Q lowercase T 4 dot Qt GUI we're going to import Q application then again and in fact actually I might as well do all that from pi q 4 dot QT core we're going to import q your Alec by the way I didn't really explain it q application is probably easy enough but it's the thing for making applications q URL this is how we're actually going to read the URL basically and then finally QT WebKit actually its capital K WebKit import q web page lovely so this is going to let us actually load the page and act like a browser act like a client and run that JavaScript so saving this a ton of programming is that would be involved there and in theory you could actually make the page show up even you can make your own web browsers in QT 4 as you might be able to surmise at this point anyway um cool so we have all the imports we need and now what we what we need to do is we're going to write a client class and if you're not too up to speed on object-oriented programming hopefully this one will be super simple enough I don't yet but very