Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
pram
Jun 10, 2001
it sounds like its making rest calls, since he said ajax. its probably some json hoarseshit

Adbot
ADBOT LOVES YOU

ahmeni
May 1, 2005

It's one continuous form where hardware and software function in perfect unison, creating a new generation of iPhone that's better by any measure.
Grimey Drawer

pram posted:

uhh you cant make calls to whatever api is updating the stuff?

this

Cocoa Crispies
Jul 20, 2001

Vehicular Manslaughter!

Pillbug

Shinku ABOOKEN posted:

anybody got any experience making web scrapers in linux?
i got a scraper made with c#+awesomium (ugh at the name) and i want to make port it to headless, gui-less linux. i want a browser engine but i don't want to install xserver, wayland, qt, or gtk. i just want dynamic html+js and they don't even need to render. the site i am scraping is doing some crazy ajax gymnastics with obfuscated js so doing the requests by hand is way too much effort.

any ideas?
chromedriver, capybara, ruby

Workaday Wizard
Oct 23, 2009

by Pragmatica
i am not gonna reverse engineer the logic of a f'ed up piece of early 2000s best engineering practices thank you very much. it has at least three different frameworks running and none of them is easy to reason about. it will take me months to go anywhere with this.

but yeah, i am looking for a full browser engine that will run without having a whole gui running on the linux box. apparently phantomjs can do that despite what some snackoverflow user said (outdated answers :argh:).

i am gonna try that when i go home. thanks ppl.

cowboy beepboop
Feb 24, 2001

Shinku ABOOKEN posted:

anybody got any experience making web scrapers in linux?
i got a scraper made with c#+awesomium (ugh at the name) and i want to make port it to headless, gui-less linux. i want a browser engine but i don't want to install xserver, wayland, qt, or gtk. i just want dynamic html+js and they don't even need to render. the site i am scraping is doing some crazy ajax gymnastics with obfuscated js so doing the requests by hand is way too much effort.

any ideas?

ghost.py+beautifulsoup

theadder
Dec 30, 2011


Notorious b.s.d. posted:

it's much more important that it work well than look good

lol

theadder
Dec 30, 2011


i have a lunix vm and i yearn to replace it with a mac mini

bobbilljim
May 29, 2013

this christmas feels like the very first christmas to me
:shittydog::shittydog::shittydog:

Shinku ABOOKEN posted:

i am not gonna reverse engineer the logic of a f'ed up piece of early 2000s best engineering practices thank you very much. it has at least three different frameworks running and none of them is easy to reason about. it will take me months to go anywhere with this.

but yeah, i am looking for a full browser engine that will run without having a whole gui running on the linux box. apparently phantomjs can do that despite what some snackoverflow user said (outdated answers :argh:).

i am gonna try that when i go home. thanks ppl.

use Lynx

Lysidas
Jul 26, 2002

John Diefenbaker is a madman who thinks he's John Diefenbaker.
Pillbug

ZShakespeare posted:

lol at linux users buying things

$ du -sh .local/share/Steam/
105G .local/share/Steam/

also an extra 20G or so for the gog version of witcher 2 in ~/Games

Subjunctive
Sep 12, 2006

✨sparkle and shine✨

Lysidas posted:

$ du -sh .local/share/Steam/
105G .local/share/Steam/

also an extra 20G or so for the gog version of witcher 2 in ~/Games

an important lesson about the importance of indulging tribal packaging customs for Linux software, to be remembered when someone says that different just-so packages are needed for Linux users to adopt. here we see them paying rare Linux-bux for something that really doesn't give a poo poo whether it uses the currently-fashionable LFS directory structure or whatever.

Lysidas
Jul 26, 2002

John Diefenbaker is a madman who thinks he's John Diefenbaker.
Pillbug
yeah i hit install and steam puts its steam stuff in its steam place and i hit play and it works, its p nice

pram
Jun 10, 2001
lol at the privilege escalation bug on polkit

pram
Jun 10, 2001
desktop linux everyone

ZShakespeare
Jul 20, 2003

The devil can cite Scripture for his purpose!

Lysidas posted:

$ du -sh .local/share/Steam/
105G .local/share/Steam/

also an extra 20G or so for the gog version of witcher 2 in ~/Games

I, too, own some windows games that run (poorly) on my idiot piss garbage work pc.

Dairy Days
Dec 26, 2007

ZShakespeare posted:

I, too, own some windows games that run (poorly) on my idiot piss garbage work pc.

kerbal space program works better on linux than windows

pseudorandom name
May 6, 2007

pram posted:

lol at the privilege escalation bug on polkit

the privilege escalation bug where users in the administration group ("wheel") have administrator privileges?

The Leck
Feb 27, 2001

Subjunctive posted:

I think it's a good idea for "when was this released", "what movie was it in again", "what's the lead singer's name" sorts of stuff. kindle fire has that sort of thing for movies, contextual to who's on the screen, it's pretty neat.
i kind of hate that this exists.

Suspicious Dish
Sep 24, 2011

2020 is the year of linux on the desktop, bro
Fun Shoe
What polkit privilege escalation exploit?

pram
Jun 10, 2001
I just got an email about it, they even gave it a cute media friendly name grinch

pseudorandom name
May 6, 2007

Suspicious Dish posted:

What polkit privilege escalation exploit?

The one where if you're in the wheel group you're allowed to do things.

Series DD Funding
Nov 25, 2014

by exmarx

Lysidas posted:

$ du -sh .local/share/Steam/
105G .local/share/Steam/

also an extra 20G or so for the gog version of witcher 2 in ~/Games

lol

pseudorandom name
May 6, 2007

"Holy poo poo you guys, administrators are allowed to administrate!"
"You think up a media friendly vulnerability name, I'll start the wiki!"

prefect
Sep 11, 2001

No one, Woodhouse.
No one.




Dead Man’s Band

pseudorandom name posted:

The one where if you're in the wheel group you're allowed to do things.

raymond chen just posted something similar: http://blogs.msdn.com/b/oldnewthing/archive/2014/12/17/10581257.aspx

pseudorandom name
May 6, 2007

in conclusion, everybody involved is an idiot, especially pram

pram
Jun 10, 2001
The grinch bug is the hottest exploit of the winter my friend

Gazpacho
Jun 18, 2004

by Fluffdaddy
Slippery Tilde
i thought of a way to steal christmas but it rather involved being at the bottom of this tiny chimney

Cocoa Crispies
Jul 20, 2001

Vehicular Manslaughter!

Pillbug

basically the same post from eight years ago: http://blogs.msdn.com/b/oldnewthing/archive/2006/05/08/592350.aspx

Gazpacho
Jun 18, 2004

by Fluffdaddy
Slippery Tilde

quote:

We couldn’t use Sudo for a variety of reasons (lack of permissions, password, etc.), Yum was inaccessible because it requires root, and DNF wouldn’t work because of FS permission checks; however, PKcon worked flawlessly. In order to exploit this, all we need is a single vulnerability in any package in a repo. There are tons to choose from. If we type ‘PKCon’ or simply ‘man PKCon,’ we can find a list of repos in use and then pull a list of all bins and version numbers. I won’t provide one here because you don’t want everything handed to you.
fukin lmao, they literally wrote "this margin is too small to contain the exploit"

quote:

This simple logic will mostly affect home users who run on an account with wheel. This includes most people, as they need Sudo.
wrap it up folks we've arrived, most people are running lunix on the desktop

Gazpacho fucked around with this message at 19:45 on Dec 17, 2014

Notorious b.s.d.
Jan 25, 2003

by Reene

Shinku ABOOKEN posted:

anybody got any experience making web scrapers in linux?
i got a scraper made with c#+awesomium (ugh at the name) and i want to make port it to headless, gui-less linux. i want a browser engine but i don't want to install xserver, wayland, qt, or gtk. i just want dynamic html+js and they don't even need to render. the site i am scraping is doing some crazy ajax gymnastics with obfuscated js so doing the requests by hand is way too much effort.

any ideas?

"i don't want to install.." is the idiot motto of gentoo users. it's a stupid thing said by stupid people. the only distinguishable phrase in the endless gibbering of beardlords in stained t-shirts

short of the kernel itself, browser engines are the most complicated software you use on a daily basis. go figure that they depend on other frameworks for abstractions. for example: gtk and qt

you can most definitely run a browser completely headless (e.g. phantomjs) but I can about guarantee you will have to give up the 50 MB of disk space to install various gui libraries

Notorious b.s.d. fucked around with this message at 19:43 on Dec 17, 2014

Notorious b.s.d.
Jan 25, 2003

by Reene
btw the correct way to do a web scraper in linux is scrapy

lol at reinventing the wheel with a browser engine and mono

Suspicious Dish
Sep 24, 2011

2020 is the year of linux on the desktop, bro
Fun Shoe

Gazpacho posted:

fukin lmao, they literally wrote "this margin is too small to contain the exploit"

wrap it up folks we've arrived, most people are running lunix on the desktop

This is amazing.

Gazpacho
Jun 18, 2004

by Fluffdaddy
Slippery Tilde
there are headless DHTML browsers (see the whole DHTML test ecosystem) and installing an entire UI stack rather than using one is kinda dumb

Subjunctive
Sep 12, 2006

✨sparkle and shine✨

Notorious b.s.d. posted:

btw the correct way to do a web scraper in linux is scrapy

does scrapy do anything with script-loaded/generated content, or does it still crawl lynx's view of the internet?

pseudorandom name
May 6, 2007

note that Google doesn't even crawl Lynx's view of the Internet anymore

Captain Foo
May 11, 2004

we vibin'
we slidin'
we breathin'
we dyin'

pseudorandom name posted:

note that Google doesn't even crawl Lynx's view of the Internet anymore

now is the winter of our lynxcontent

Cocoa Crispies
Jul 20, 2001

Vehicular Manslaughter!

Pillbug

pseudorandom name posted:

note that Google doesn't even crawl Lynx's view of the Internet anymore

well yeah

consider that a shitdick seo (but you repeat yourself) could jam the lynx view full of google-friendly content and typical seo bullshit and just show ppc ads or whatever in the neurotypical view

Notorious b.s.d.
Jan 25, 2003

by Reene

Subjunctive posted:

does scrapy do anything with script-loaded/generated content, or does it still crawl lynx's view of the internet?

mostly the latter. there is a plugin to do some very basic js stuff, e.g. render the page text like a browser would.

it's a crawler library, not selenium.

Sapozhnik
Jan 2, 2005

Nap Ghost
how long have single page applications been a thing anyway? because i'm guessing Chrome is a natural byproduct of Google having to run a full-fledged headless web browser in a cage in order to scrape all that poo poo. it certainly explains why it has such a strong emphasis on sandboxing.

so it seems liek HotJava was the way to go all along and modern single page JavaScript applications are effectively a bastardized version of HotJava only with a lovely scripting language instead of the JVM. then again given Java applets' security track record that might be for the best.

RIP the semantic web :(

sports
Sep 1, 2012
the web

Adbot
ADBOT LOVES YOU

pram
Jun 10, 2001
semantic web was stupid as poo poo

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply