|
im making a web toy that makes a youtube web player using all the vids in a thread, decided to make this thread to post updates about it instead of making GBS threads up the meta thread so far ive managed to make new endpoints for the unofficial somethingawful API (check it out) for viewing the posts in threads, so that i dont have to write an entire web scraper for the project and can just make an API call thankfully the youtube API seems slightly less useless than i first anticipated so getting the actual player to work shouldnt be TOO horrible
|
![]() |
|
![]()
|
# ? May 27, 2022 15:11 |
|
should mention that the endpoint for viewing full threads absolutely shits itself if the thread is too long (tested it on the cjs thread in yospos and it took so long that cloudflare thought it died) so im probably going to add a version for making playlists out of specific pages and/or page ranges for if you want to grab poo poo from a particularly long thread but first im going to focus on the Main Thing, and do those two things after thats up and running
|
![]() |
|
have got an incredibly simple player up and running but now i need to wrestle with jinja to see how i can import this loving array into js from my flask backend
|
![]() |
|
thanks for the updates hbag. looking forward to seeing this in action
|
![]() |
|
i actually have literally the entire player working but now i just need to add a shuffle function and also make it look nicer than uh this ![]()
|
![]() |
|
![]() seems good enough for now i noticed that a lot of videos seem to not work with the iframe API so they display as "not available" and get skipped, so rip lmao
|
![]() |
|
drat. thats loving sweet. God bless you
|
![]() |
|
seems the API can handle, at most, 5 pages at a time so im probably going to make a new endpoint that just gets the last 5 pages of a thread and use THAT for the playlist generator
|
![]() |
|
lol nvm turns out i can use async functions instead of waiting for each individual page like a psycho
|
![]() |
|
async functions are proving to be dumb and lame so i might just add a function to the site where you can start monitoring a thread and itll build the playlist from a list of links stored in a db or some poo poo
|
![]() |
|
im not much of a computer guy and i have no clue how you set this up on your end. but wouldn't a good way to cut down on scraping and performance problems be to just archive the results and show a list by thread name or some poo poo. then only run a scrape when someone enters a threadid, either to add a new one or update and existing one. this would also make it easier to use because it's kind of a pita on my phone. and this data seems like it should be easy to store as a table or text file of some kind so it's not like it would eat up a bunch of your server space. edit I guess that's kind of exactly what you said in your last post lol
|
![]() |
|
Larry Parrish posted:im not much of a computer guy and i have no clue how you set this up on your end. but wouldn't a good way to cut down on scraping and performance problems be to just archive the results and show a list by thread name or some poo poo. then only run a scrape when someone enters a threadid, either to add a new one or update and existing one. this would also make it easier to use because it's kind of a pita on my phone. and this data seems like it should be easy to store as a table or text file of some kind so it's not like it would eat up a bunch of your server space. yeah lmao kinda already up and running - it doesnt scrape the thread when you search it unless its a brand new thread. if the thread has been scraped before, it'll just load the results from a database and use that
|
![]() |
|
u should make an index so my lazy rear end can just click link ![]()
|
![]() |
|
Larry Parrish posted:u should make an index so my lazy rear end can just click link yeah ill probably do something like that once ive made sure everything's working alright and not doxing me or something
|
![]() |
|
gotmore ideas. scribbling them down for later. loving Diabolical
|
![]() |
|
also lmao at this loving janky-rear end function for retrieving the playlistsPython code:
|
![]() |
|
![]()
|
# ? May 27, 2022 15:11 |
|
the thread scraper that the playlist generator uses has a few bugs that cause it to crash, so im gonna be working on fixing those letting you know because it means new playlists arent gonna be generated beyond the first page for now
|
![]() |