The Usenet Megathread - No invite discussion!

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Usenet Megathread - No invite discussion!

longview: Dec 25, 2006; heh.

Odette posted:

These speeds are unbelievable. At the beginning of this year, I was downloading at 700kb/s, now the infrastructure's been upgraded .. 1.7mb/s.

If I download my Linux distros in school I can get about 25 MB/s sustained on a good day from Giganews. Still only 1.4 MB/s at home though :sigh:

# ¿ Jul 16, 2011 18:02

Adbot: ADBOT LOVES YOU

# ¿ May 4, 2024 07:52

longview: Dec 25, 2006; heh.

LukeyBoy posted:

On modern systems it's a few percent at most - plus they're talking about seeing faster speeds when using encrypted connections.

Well, assuming your ISP isn't crap it is slower to use SSL since the encryption has overhead. Some ISPs may limit some connection types more than others though.

# ¿ Jul 31, 2011 18:46

longview: Dec 25, 2006; heh.

Thermopyle posted:

I can't stand to watch any SD content on an HDTV. No matter what post processing you do, SD looks like poo poo on a 50+ inch 1080p set.

Hell I can barely stand it on a 32" CRT, I can tolerate the low resolution, but the lovely contrast, noisy blacks and macroblocking just kills it.

# ¿ Sep 24, 2011 20:45

longview: Dec 25, 2006; heh.

Drizzt01 posted:

20. Unless your on a 100mb connection there shouldn't be a speed difference.

I tried it on a >200 mbit connection and found somewhere around 5-10 was ideal, at 20 the speed would oscillate wildly leading to lower average throughput.

# ¿ Mar 24, 2012 23:30

longview: Dec 25, 2006; heh.

So are people setting up SpotWeb servers too? I decided to abandon NewzNab since it requires far too much bandwidth and work for me to run it at home, filling the backlog of spotweb has only taken little over an hour.

# ¿ Dec 17, 2012 15:22

longview: Dec 25, 2006; heh.

copperblue posted:

Install was seriously 5 minutes on Win7, and the back loading was done in a couple hours. It'll automagically check for new spots every hour.

I'm assuming this'll be the preferred route in the future. You're already paying for usenet, why not get your nzbs from there too. Get more people submitting and commenting entries and it'll be better than having an indexer.

Not to mention you'll subconsciously learn dutch.

No joke, even setting it up on a bunch of VMs with reverse proxying it only took me an hour.

I already had to skim a bunch of Dutch documentation and it's surprisingly readable. These are just teething problems, like SpotWebs English translation has weird grammar and translation issues (Image category example). I expect usage to increase a lot, since it's a pretty good distributed system that doesn't involve gigabytes per day and hours of typical CPU-time.

SpotWeb has the worst search engine though, it seems like it ORs together every word by default, so making a search more detailed usually brings back even more useless results that match one odd word.

# ¿ Dec 19, 2012 12:02

longview: Dec 25, 2006; heh.

crm posted:

Can somebody give me a rundown on how Spotweb works? The docs don't really explain what it is.

I've got it installed and had it pull everything, but I'm not exactly sure how it's different than newznab.

Executive Summary: Instead of putting user submitted data in a central database (like Newzbin, Matrix etc.) they post the data back to usenet in a standard format.

SpotWeb checks for new posts in a specific group and adds them to the database. Comments are also implemented by posting them back to usenet.

The protocol is known as SpotNet, for more information. The real advantage is it can be very high quality, and the hardware and bandwidth requirements are much much lower. Disadvantage is new posts have to be added by users instead of automatically popping up as soon as they're posted.

# ¿ Dec 19, 2012 17:04

longview: Dec 25, 2006; heh.

It looks like usage has increased a lot in the last few weeks, a lot of the content is still Dutch but I have been able to find English content easily enough. Often the content is English but the description is Dutch.

# ¿ Dec 19, 2012 17:44

longview: Dec 25, 2006; heh.

They *might* be able DMCA the report posts, but that's a little questionable since I don't think the reports themselves are their content, even if it were to be information on the location of their content.

More likely is that since automated services like Newznab can match posts to content, they could automatically send DMCA requests based on that. Otherwise watching spotnet wouldn't be a problem and sending takedowns based on the nzbs in the reports would probably be doable.

# ¿ Dec 19, 2012 22:04

longview: Dec 25, 2006; heh.

All you need to get either working is some kind of webserver+a database+php, though I recommend using SW as a supplement to a site like nzbs.org since that will pretty much always have more content, being automatic and all.

NewzNab has very detailed guides, SpotWeb has detailed Dutch guides so I'll try to summarize what you need to do here.

I don't have my Newznab setup running but it was pretty simple to get working in Ubuntu Server at least, literally cut and paste commands. Filling a backlog takes forever, it requires lots of disk space and is fairly CPU intensive regexping all the headers to form releases. Since it's pretty much purely header based you can run it for free off an Astraweb block account, headers don't count as usage.

SpotWeb was a bit harder but I set it up in CentOS with nginx as a front-end, php-fpm for CGI and PostgreSQL for a database, it will run about as well with a LAMP or WAMP stack.
All you need is to make a database called spotweb beforehand, preferably with a separate user for it, this is pretty easy to do on the SQL command line.
Download the latest source from the github site, extract it to your www-root.
Open install.php in a browser and it asks for database credentials, walks you through connecting your usenet server, then you make a user and it's done.

Importantly, you should probably have an unlimited account for this, none of the unlimited accounts will let you account share but in my case I use the same account for indexing and downloading, since they're behind NAT they still count as one external IP. I think SW actually downloads release info and nzbs from usenet on the fly when you open a release, there is an option to pre-fetch but it literally makes the process about 50x slower, so I set it to fetch full reports, spots and comments, but not to fetch nzbs, images or full comments.

Only other thing needed is to set up a scheduled task/cronjob to run "php (or path to php.exe etc.) (path to)retrieve.php --force" something like once per hour, I add --force in case a job is killed or exits with warning so it won't stop working until a .pid file is deleted. Filling takes at most a few hours for SW the first time, a few minutes if you do it often, and my database still fits on an 8 GB partition.

SW runs very well on my VM with 4 cores (Q9300) and 1 GB ram allocated, it's fairly CPU intensive to load the front page or search, but it doesn't use a lot of memory at all.

# ¿ Dec 19, 2012 23:20

longview: Dec 25, 2006; heh.

GhostSeven posted:

I have just fixed that as it was winding me up!

Assuming you are using mysql this is the hacky fix to essentially default to an AND search if you have not specified any other search modifiers such as + - etc.

It also works for newznab api so improves Sickbeard et al.

It is hacky so be warned!

Edit /spotweb-install-dir/lib/dbeng/dbfts_mysql.php

BACK UP THE FILE!

Replace / Comment Out :
code:
if (($searchMode == 'match-natural') || ($searchMode == 'both-match-natural')) {
  /* Natural language mode altijd default in MySQL 5.0 en 5.1, but cannot be explicitly defined in MySQL 5.0 */
  $queryPart = " MATCH(" . $field . ") AGAINST ('" . $this->_db->safe($searchValue) . "')";
  $filterValueSql[] = $queryPart;
} # if
With :
code:
$queryAdj = str_replace(array(' '), ' +', $searchValue);
if (($searchMode == 'match-natural') || ($searchMode == 'both-match-natural')) {
  $queryPart = " MATCH(" . $field . ") AGAINST ('" . $this->_db->safe($queryAdj) . "' IN BOOLEAN MODE)";
  $filterValueSql[] = $queryPart;
} # if

Nice work, unfortunately I installed PostegreSQL and it looks like the code is fairly dissimilar between MySQL and that. I found sorting by date makes it a little easier to read though.

# ¿ Dec 21, 2012 12:23

longview: Dec 25, 2006; heh.

monkeybounce posted:

I feel like this is a very stupid question, but has anyone figure out how to stop SpotWeb from showing anything until after someone has logged in? I want to make it accessible to a few friends, but I don't like the idea of it being visible to the world.

Change the group membership of the anonymous user to Anonymous user - closed system.

# ¿ Jan 3, 2013 22:54

Adbot: ADBOT LOVES YOU

# ¿ May 4, 2024 07:52

longview: Dec 25, 2006; heh.

I helped my mate out with incompletes and he was still on Astraweb, complaining about the large number of takedowns.

I'm on SuperNews, and in my download history I have no failed downloads, I have block accounts at Astraweb, Blocknews and now NewsDemon to get Highwinds Media downlads, the block accounts see a total of maybe 100 MB per month vs. ~60GB transferred per month otherwise.

# ¿ Jan 12, 2013 10:19

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Usenet Megathread - No invite discussion!