Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us $3,400 per month for bandwidth bills alone, and since we don't believe in shoving popup ads to our registered users, we try to make the money back through forum registrations.
  • Post
  • Reply
mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Welcome to the BOINC and grid computing thread! Were you a cool kid back in the dotcom era, running GIMPS or SETI@Home on your work desktop? Did you crunch Folding@Home on your PS3? Well, that scene is still going, and in fact it has expanded substantially.

I want to help fight Covid-19! What do?

Here are some projects to join:
  • Folding@Home, which is now the largest computer in the world in terms of TFLOPS. So many people have piled on that researchers are struggling to fill the pipeline with workunits, but that's a good problem.Can run on GPUs, so it makes a good pairing with Rosetta.
  • Rosetta@Home has also gone all-in on COVID research. Their software, the Rosetta biodynamics package, does not run on GPUs, so R@H is a CPU-only project. This makes them a good pairing with F@H.
  • TN-Grid are doing human genome research at the University of Trento in Italy. They are currently targeting ACE2: "...closely related to the SARS-CoV-2 virus. Don't expect we will be able to 'find a cure' but we are trying to add something to better know our enemy."

Rosetta, TN-Grid, and most other distributed projects use a framework called BOINC, so you'll only need to install one piece of software to contribute to them (then you attach your BOINC client to as many projects as you want). F@H does their own thing, and have their own client, so you'll need to install that if you want to contribute to their efforts. This isn't good or bad, it's just the way it is, and setting expectations is always good.

And here are some videos on how to set things up:

And of course, ask questions in this thread if you need any help. Thanks for volunteering your cycles!

What's all this, then?

Grid computing is a way to divide up huge problems and/or datasets among many computers for distributed analysis. If you're into science, medical research, and/or pure mathematics, there is almost certainly a grid computing project out there that you can contribute CPU and/or GPU time to.

Research teams with hugely time-consuming computational tasks (on the order of milennia of compute time on modern hardware) split these tasks into very large numbers of chunks, called workunits (WUs). A WU is an amount of work which should be computable on reasonable hardware in a reasonable amount of time -- anywhere from half an hour to a couple of days. Volunteers then sign up to help with projects, install a client, and their hardware begins crunching WUs.

BOINC is a management framework for distributing and aggregating tasks for grid computing applications, and a terrible acronym standing for Berkeley Online Infrastructure for Network Computing. Most grid projects now use BOINC as their backend and network infrastructure, which is good for users who want to contribute. Instead of having a lot of single-purpose applications installed, you can just install a single BOINC client and attach it to various projects. BOINC also provides a lot of control over how/when/how much of your hardware gets used. You can declare what timeslice each project gets in relation to each other. You can suspend computing between certain hours, or when your machine isn't idle. You can only allow a certain number of active threads, so that you have some CPU reserved for yourself -- and a lot of other controls besides.

Once your client(s) are set up, contributing to modern grid computing projects is basically zero effort. Unless (like me) you're the sort of nerd who likes obsessing over how much you're crunching.

But why?

The people I've talked to, tend to fall into two categories:
  1. Those who want to let their hardware do something useful by contributing to scientific or medical research, and
  2. Those who treat it as a real-world form of incremental/idle game, where you prestige by building new machines and upgrading existing ones, and pay upkeep in the form of your power and cooling bills
And then there's those who (again, like me) are in both categories.

Lol SETI is bullshit and nobody's PS3 cured cancer

I'm not going to weigh in on the merits of any specifc project. What I will say is that the field of projects has grown, and most of the popular ones currently running are dedicated to tangible scientific and medical research and are led by actual scientists from actual universities.

Further, while finding a cure for something would be lovely, that's not actually the focus of most projects because that's (usually) not how science works. Most of the projects oriented toward medicine are taking the approach of identifying good targets for further research. A lot of the utility in the current generation of projects is in winnowing away the possibilities which do not work, leaving researchers with a much more focused set of possibilities for attempts to find things which may accomplish their goal. Right tool for the job, and all that.

Finally, acedemic publications do come out of community work. The team behind the Help Stop Tuberculosis subproject at World Community Grid, as an example, recently announced that they will be publishing a set of papers informed by WCG data by the end of the year. This is real science, and that means it's almost always a long march rather than a sprint.

Sorry bro saving the world is nice but I gotta mine buttcoins

I'm sure you will be absolutely shocked to learn that there is now crossover between grid computing and cryptocurrency. I'm not going to link to them, but the big player here is Gridcoin, who claims to pay out for work done on various BOINC projects.

Also -- though I'm not sure why, and I don't believe they're paying out for BOINC work -- a team based around the Jesus-oriented coin Biblepay (because of course that's a real thing) has recently started competing in community challenges on the World Community Grid project.

Ok, fine. What can I do with my computer machine to make the world a better place?

A lot. Since I'm more intimately familiar with them and can provide more information on them, I'll start with the projects I run.
  • World Community Grid -- Medical/Scientific -- Hosted by IBM, and home to a wide collection of subprojects. At the moment, all are medically oriented (Mapping Cancer Markers, Outsmart Ebola Together (ending soon), Fight AIDS@Home (Phase 2), Help Stop TB, OpenZika, Smash Childhood Cancer (ending soon), and the Microbiome Immunity Project). You can read more about all present and past research here. In the next few months, three new projects will be added, all focusing on climate change.
  • GPUGrid -- this project, like Folding@Home, is focued on protein synthesis and molecular dynamics for medical research. Run by Universitat Pompeu Fabra in Barcelona. As you might expect from the name, most of their subprojects are GPU-enabled (Nvidia only).
  • Primegrid -- While the older and better-known GIMPS (Greater Internet Mersenne Prime Search) only cares about one type of prime number, Primegrid is dedicated to the search for pretty much all other interesting forms of prime numbers. Woodall primes, Cullen primes, Proth primes, Sophie Germain primes, and more, including prime-adjacent areas of mathematical research like very long arithmetic progessions of primes. Everything here is GPU-enabled, for CUDA or OpenCL.

Here's some other popular projects:
  • Einstein@Home -- Analize LIGO data from pulsars to find gravitational waves
  • Asteroids@Home -- Increase our store of knowledge about asteroids via computer vision techniques
  • Rosetta@home -- Another really popular protein folding simulation project, similar to Folding and GPUGrid
  • LHC@Home -- A collection of projects from CERN, using data from the LHC. Sort of weird, in that some of it runs under BOINC but some of it requires custom clients which must be run in a VM

And then there's Wikipedia's list of distributed computing projects, and the BOINC list of projects which includes some more esoteric projects involving things like seismology, underwater acoustics, and cognitive science.

Last but not least, there is Folding@Home, which is something of an odd duck. It's a legit project, run by Stanford, but it does not use the BOINC framework.

Tell me more about the incremental game aspect

Most projects are heavily gamified and track/rank contributors based on:
  • How many WUs you've crunched
  • How much compute time you've burned
  • How many "points" your returned WUs are worth. Points are an abstraction designed to give people with less-powerful hardware earn credit at a rate similar to those with faster gear. Slower machines get more points for a WU, while faster machines chew more WUs per unit time. It's all supposed to balance out.
These stats tend to be tracked per team as well, because you can join up with other volunteers in a team so as to have bigger numbers which go up faster, in addition to your personal numbers.

Several projects also award badges for hitting various milestones of work done. Here are my WCG badges (based on cumulative CPU time):



and my Primegrid badges (based on points awarded for work done):



And then there's the meta-stats provided by BOINCstats, which show you how you're doing across all your projects, compared to everyone else who is doing any BOINC work, worldwide, and against yourself via historical charts and tracking of performance over the past 40 days.

This poo poo is like digital crack to me. I currently have two six machines dedicated to doing nothing but BOINC projects, in addition to my desktop machine which spends about 98% of it's time doing the same thing. And I'm about to start buying parts for crunchbox number four.

Goon Teams

In addition to your solo achievements, BOINC projects also have teams, which are collections of users whose contributions are counted together for bragging rights.

* When you're part of a team, all your points/runtime/etc are counted toward your personal stats, and also toward that team's stats.
* You can belong to only one team at a time.
* Points earned while on a team stay with that team if you leave; you can't move points from one team to another

here are the known Something Awful BOINC teams:


And also the SA Folding@Home team.

Tooting my own horn

If you find yourself getting addicted to this poo poo and falling down a hole of building machines which are completely dedicated to crunching WUs, then maybe you'll also find yourself wanting a lazier way to keep all those machines up to date and synced in configuration. I've been working on a suite of tools for that very thing, called Homefarm.

What now

If you're already crunching, hang out and talk. Keep us up to date on news in your projects and talk about your hardware plans. If you're new or interested, ask questions and get terrible opinions!

mdxi fucked around with this message at 18:54 on Mar 23, 2020

Adbot
ADBOT LOVES YOU

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Tuxide posted:

I've been running BOINC for about a year now, and I've been joining the Something Awful teams for projects such as Asteroids@Home, mainly as a way to show that I have stairs and I'm proud of it. Is there any incentive to not doing that, and joining some other team instead?

Not to my knowledge, no. Teams just let people aggregate their work, for fun.

And you don't take your points with you, if you do switch teams. Whichever team you're on gets credit for the work you do while you're part of that team; nothing from before or after.

There's even a kind of a meta-team on the WCG forums, who expressly jump around and lend CPU time to other teams as they see fit.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

craig588 posted:

I started running F@H about a year ago. Any reason to switch or donated compute power can help anyone equally? I'm medically retired from MS and losing my ability to walk, F@H doesn't have a specific MS project, but if there's one out there I'd switch.

From some quick googling, I couldn't find a MS-specific project.

I found some posts on the Rosetta forums which talked about the possibility of such a thing, but apparently they've never had a researcher who was focusing on MS. Also noted was that the protein folding/molecular dynamics type projects are what an MS-specific project would likely look like; you'd just need an appropriate dataset and a conjecture to investigate.

I did find an MS team on WGC, who have been crunching since 2004 (wow!), and have over half a millennium of CPU time. I also found their forum thread. WCG doesn't actually have any subprojects working on MS though.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

After way too much time, because I am terrible at arts and crafts, I have completed my multi-machine chassis.



These machines do nothing but crunch WGC workunits. More info and pics here.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Version 0.16.0 of my Homefarm project is out. This is the first release candidate, which means there will be no work done except bugfixes between here and v1.0.0. The last big addition is a local repository, which compute nodes will use for installation and update of their OS packages.

In more purely BOINC news, the 13th BOINC workshop is being held in Chicago, in July.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Speaking of GPUs, I just got a GTX 1650, and wrote up my Primegrid results here.

Now that some benchmarking is done, I'm about to let it start crunching einstein@home WUs. And GPUGrid, once they un-break their Linux client.

mdxi fucked around with this message at 04:26 on May 22, 2019

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Oops. Thank you.

Fixed above, and for your convenience: https://firepear.net/grid/gtx_1650_vs_gtx_750_ti/

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Tuxide posted:

The second is to see the Something Awful team climb in ranks, which I just do solely for fun. We don't really have an impressive size, but I just think it's funny that goons can outperform other teams like Microsoft and video game groups that goons would normally be competing against in other areas.

I didn't know there were any goon teams out there

It's kind of shocking and/or depressing how little effort and computational grunt you need to throw at these projects to climb the rankings. An enormous number of people obviously show up, maybe run the client for a week (or a day) and then disappear forever.

I've been into this stuff for close to two years, and have only made it a kind of goal for the past year. I have four machines, all of which are completely ordinary PCs. A year ago I had one. The fourth only got built on 06 Feb of this year. Only two of them have ever had GPUs slotted, and for most of the past two years, none of them have had GPUs.

And still my "team" (which is really just me, though two of my machines run under an account made for my partner, because she loves science but doesn't care to be a computer janitor like myself) is #1259 on World Community Grid by WUs returned. We're #3377 globally on BOINCStats. That shouldn't be the case.

It's depressing, because so many people throw so much effort into buttcoins of various flavors. Last week, someone brought a 6 GPU mining chassis into my office and were like "LOL this doesn't even pay its electricity bills anybody wanna play with it?" When I mentioned running a GPGPU grid computing project on it the response was "Or we could do some hash cracking!"

Sigh.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Earlier today, the first 2000 beta WUs for one of WCG's upcoming climate projects were pushed out. One of my nodes got 2 of them. Here's what they look like while running:

code:
boinc     4263 99.6  4.4 835036 733508 ?       RNl  00:52 903:44 ../../projects/www.worldcommunitygrid.org/wcgrid_beta27_wrf_7.19_x86_64-pc-linux-gnu
boinc     4313 99.6  4.4 835040 733568 ?       RNl  02:07 829:01 ../../projects/www.worldcommunitygrid.org/wcgrid_beta27_wrf_7.19_x86_64-pc-linux-gnu
WCG staff say to expect 20h+ runtimes on these WUs. The runtimes shown in the `ps` output above (904min, 829min) are for WUs at 91% and 84% complete, running on a Ryzen 1600. Here are the final timings from the job_log:

code:
1559236628 ue 2732.665687 ct 58854.040000 fe 13697606073123 nm BETA_ARP1_0000263_000_1 et 59068.865957 es 0
1559240866 ue 2732.665687 ct 58589.830000 fe 13697606073123 nm BETA_ARP1_0000364_000_0 et 58801.453863 es 0
The actual runtimes (in field `ct`, for "cpu time") were 58854s (16h 21min) and 58589s (16h 17min).

It is interesting that while the number in the `fe` (estimated FLOPS) field is very large (13.7 TFLOPS), it's actually small compared to other subprojects (46.2 TFLOPS for a MCM WU; 24 TFLOPS for a Zika WU; 23.8 TFLOPS for a MIP WU). This makes the very long runtime surprising. Possibly this is always way off for new projects?

Staffers also said to expect greater-than-normal memory usage with these WUs. You can see that each of these jobs is using about 734MB of resident memory (that is: memory used exclusively by that process). This is, indeed, more than the WUs of other subprojects. Here's a look all WUs running on that machine, which has 16GB RAM, sorted by memory usage:

code:
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 4313 boinc     39  19  835040 733568  29400 R  99.3   4.5 851:40.72 wcgrid_beta27_w
 4263 boinc     39  19  835036 733508  29400 R  99.7   4.5 926:23.18 wcgrid_beta27_w
 4898 boinc     39  19  414036 354304  53388 R  99.3   2.2  47:05.21 wcgrid_mip1_ros
 4957 boinc     39  19  186492 124328  47896 R  99.3   0.8   4:38.02 wcgrid_mip1_ros
 4946 boinc     39  19  132404  58408   2744 S  99.7   0.4  14:20.48 wcgrid_zika_vin
 4909 boinc     39  19  132376  58376   2752 S  99.7   0.4  31:46.39 wcgrid_zika_vin
 4893 boinc     39  19  132396  58352   2752 S  99.3   0.4  49:54.03 wcgrid_zika_vin
 4852 boinc     39  19   77128  37000   2392 R  99.7   0.2 153:24.09 wcgrid_mcm1_map
 4855 boinc     39  19   77024  36692   2332 R  99.7   0.2 146:04.97 wcgrid_mcm1_map
 4862 boinc     39  19   76800  36504   2392 R  99.7   0.2 134:49.66 wcgrid_mcm1_map
 4850 boinc     39  19   76800  36488   2392 R  99.7   0.2 159:56.44 wcgrid_mcm1_map
 4828 boinc     39  19   74788  34756   2060 R  99.7   0.2 187:57.47 wcgrid_mcm1_map
WCG staff say (with tongue firmly in cheek) that they can "neitherconfirm nor deny" that these WUs are from an upcoming climatology project, but they do point out that the software is a modified version of the Weather Research and Forecasting model software from NCAR/UCAR, thus the "WRF" in the binary name. Based on the WU name, the subproject will be known as ARP1 -- no idea what ARP stands for yet. They also say:

quote:

The work for this project will be broken into small geographical regions, and in the end each region will be simulated for one calendar year. Each individual work unit represents 48 hours calendar time for this simulation. Once a result has been validated for the 48 hours, the output will be used to build the input for the next 48 hours of runtime.

Exciting times!

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

thehustler posted:

I also tweaked boinctui a bit if anyone uses that: https://github.com/mpentler/boinctui-extended

You're the updated boinctui person? That's rad!

Thank you for your work.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

After slacking for some months, I'm working on Homefarm again: v2 is coming in the not-too-distant future.

The big change will be the removal of the requirement for the control node to be a Raspberry Pi. In fact, there won't be a requirement for a physical control node at all; those functions are being moved into a docker container. This uses a disgusting amount of drive space (because it's based on Arch, for consistency with the compute nodes), but removes one whole physical computer from the setup.

I'm also working on containerizing a compute node, but that's intended to be only for testing purposes -- right now I can't test a lot of scripts without either doing an actual reinstall on my farm, or having spare nodes lying around for testing.

I'm also also streamlining a bunch of stuff, making QoL improvements, improving the docs, etc., etc. Of course, I think I'm the only one actually using it, but I'm still excited about making things better.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

World Community Grid has launched a new project, the Africa Rainfall Project. Its aim is to produce better, localized weather models for sub-Saharan Africa, which will increase crop yields (and be generally good for everyone).

Unfortunately, it's off to a bit of a rough start. It turns out that climate modelling is not hugely parallelizable in the way that some other problems are. You can split the problem up into localized forecast areas, and run each of those in parallel, but you can't parallelize along the time axis; the results of the first chunk of the simulation for each forecast area becomes the initial conditions for the next time-chunk for that area.

WCG staff anticipated this being problematic, and set the "WUs per machine" limit to 1 for this project, rather than the usual "unlimited". They also asked people to leave it set at 1. Predictably, people did not, and pretty much the entire initial allocation of WUs was gone by the time most of the US woke up. In three days, with 6 machines crunching, I've seen one ARP1 WU, and some people still haven't seen any at all.

WGC staff said that the average completion time for an ARP1 WU, during the beta test period, was 20 hours. So this process is slow on top of everything else. Wingman WUs are slow; reworks are slow; it's all slow. It's frustrating.

Some people are also complaining about the dataset size. Each WU is a 100MB download and a 60MB upload. I think those people are just being whiny though, because

1) I was routinely doing Debian installs and updates over a 56kbps modem back in the day, which was a lot more data

2) Deselect that project then dude. No one is forcing you to participate in it, and the other subprojects haven't gone anywhere.

...Except the ones that have. Smash Childhood Cancer is on a hiatus while the most recent batch of returned data is analyzed and the researchers decide on their next targets.

OpenZika is very, very close to being completely finished. Until 27 October, there were about 300k ZIKA WUs being returned per day. The six days since then have seen 166k, 103k, 82k, 71k, 56k, and 40k.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Tuxide posted:

On Asteroids@Home, Team Something Awful isn't that big and we're nowhere near the top ten, but for some odd reason I do find a bit of pride in our recent credit score:



I don't know who is actually in charge of these goon teams, looks like it's mainly inactive goons...but perhaps they should be linked to the OP, so that goons actually know that they're there in case they want to join them.

Added! If you know of any others, let me know and I'll be happy to update the OP.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Hello, New Captain!

I just joined Asteroids and the goon team as well, because yay science and gently caress 4chan. No workunits so far. I can't actually tell from the very sparse homepage: is this a CPU project, or a GPU project?

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Tuxide posted:

These are the goon teams I'm aware of. Note that most of these, but not all of them, are called "Something Awful". I find them by looking for team names with the word "awful" in them, though sometimes this returns non-teams registered by spambots.

Updated. Thank you.

In unrelated news, I've started posting monthly subproject summaries from the WCG forums to r/boinc. Here's the most recent one: https://www.reddit.com/r/BOINC/comm..._november_2019/

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Cool article on Rosetta@Home: http://www.dailyuw.com/wellness/art...d11958b0cf.html

The following isn't BOINC stuff, but I figure there's a good chance that anyone here is also interested in general scientific computing news.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

WCG's OpenZika subproject has finished compute work: https://www.worldcommunitygrid.org/...o?articleId=614

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

OhFunny posted:

Cracked 1,000 results returned on WCG.



In other news, Navi GPUs appear to have a broken OpenCL implementation. This has raised a ruckus on r/BOINC and r/Amd, and is especially interesting given the very strong compute performance from pre-Navi AMD GPUs. I guess that's some evidence that RDNA really isn't Yet Another GCN Retread.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

WCG Project Updates, December 2019

Another month of research team updates from the World Community Grid forums. The text of all updates is from Caiti Larkin, WCG Admin.

Note: WCG entered a lull this month, with one project reaching completion, and three on hiatus while researchers do analysis and work out what their next targets will be. Additionally, the Africa Rainfall Project has been fully online for a month, but its structure means that there are comparitively few in-flight workunits at any given time -- around 1900 per day. (For comparison, MCM1 averages 675kWU/d)

Africa Rainfall Project

We plan to start mini-updates for ARP in January.

Microbiome Immunity Project

We wrapped up our call with the research team just now.
  • They're hoping to get their paper (mentioned in last month's update) submitted before the end of 2019.
  • They're also working on a project update for us; however, they also have some grant applications due that are taking precedence.
  • They're generating more work units--no ETA yet. We'll let everyone know if there's a gap between the time we run out of work and the time we receive new work.
  • They may be presenting at a Protein Data Bank conference in the US in February--more info to come if this happens.

Smash Childhood Cancer

[This update is from November but happened late, so I missed it last month.]

Highlights from our monthly SCC research call:
  • There is going to be a change in the project leadership. We expect to have more details and an announcement next year.
  • The researchers are looking for additional team members to help create work and analyze data that comes back from WCG.
  • The SCC team is international, and there has been continued political unrest in one of the countries where some of the team is located which has affected universities and research work.
Thanks to everyone who has supported this project. We'll release more news as details are confirmed. It's important to note that the researchers are very committed to continuing to work with WCG, and there's transition going on to help solidify the project.

Help Stop TB

Had a quick call with the researchers today, with no significant updates except that they're working on a project update for us and continuing to get their new team member up to speed

Fight AIDS @ Home Phase 2

Updates from our monthly call with the researchers:
  • WCG just started beta testing on the new work units, after a few weeks of alpha testing. We will let everyone know once beta testing is over and we have new work units ready to go out to the general volunteer group.
  • The two phases are planning their annual face-to-face meeting early next year, as mentioned in the last update.
  • The Phase 1 team continues to plan for more work on WCG next year--this is still in the early planning stages and we don't have an ETA yet.

Mapping Cancer Markers

We just wrapped our monthly call with the researchers.

  • We had a status check on work units. We've been running about 28 batches per day on average.
  • WCG has begun alpha testing on sarcoma work units. This may run for a few more weeks. We'll make an announcement as usual when beta testing starts.
  • They are working on a project update for us.
  • The project PI will be speaking at several meetings early next year (Boston, San Diego, etc.) that are related to computational research, and he will be mentioning WCG when he presents.

OpenZika

Just wrapped up our monthly call with the research team.
  • We've confirmed that there's no more work coming to WCG. We'll update the progress bar soon.
  • The project PIs have recently presented on OpenZika at two conferences in Brazil.
  • They have two accepted papers, which should be published next year. An additional two are in the writing stage.
  • They've prepared a project update for us, which we plan to publish later this week or early next week.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

After finding some inspiration and spare time, I have written and released Homefarm v2!

If you're in the market for a system to manage your BOINC machines, check it out: https://www.reddit.com/r/BOINC/comm...arm_2_released/

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

I tried joining Asteroids today, but the auth server appears to be busted in some way.

I did rejoin crunching on Einstein@Home after doing a PSU swap, getting a node back online, and slotting my spare GTX 750 Ti into it. I attached that machine to GPUGrid as well, but only to crunch "small" jobs, and I think that perhaps none exist at the moment. Either that, or it's still busted on Linux?

Pushed out a Homefarm update as well: v2.0.1 has fixes for Arch switching the compression on their packages. Previously the local repository tooling had assumed eveyrthing would be an xz file.

Thank you for attending my TED talk.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

OhFunny posted:

I passed passed 2,000 results returned on WCG

quote:

My new AMD 3600 has tripled my daily results.
I love Zen 2.

Edit: WCG's FAH2 is shipping WUs again. Has been since the day before Christmas, but I got my first one late last night, and ran to the forums for confirmation.

mdxi fucked around with this message at 15:54 on Jan 5, 2020

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

It took 30 minutes of running commands by hand and adding debug info to a script, but I have finally joined 2 of my machines to Asteroids@Home. I'll be contributing to the SA team as soon as I get some WUs.

The problem this whole time has been that I typoed my email address in the config file

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

yummycheese posted:

Are there other project with more ppd incentives I could target? Have been looking at climateprediction.net and possibly WCG.
I didn't know there were any projects at all with incentives for points. That said, in my opinion, it's really best to treat BOINC points like the points in Whose Line Is It Anyway. There's a lot of fudging in what a "point" is between machines; a WGC point is inflated compared to a "standard" BOINC point; and apparently those aren't so standard either, because the BOINCstats people have their own weighting for RAC calculations. I wouldn't take them too seriously.

Also, your points per day aren't actually the points for the work done by your machine for any specific day from your point of view. It's the points credited for any workunit on which you were the primary or a wingman, validated in the last 24 hours from the point of view of the project server. There can be a decent amount of fluctuation day-to-day, even if your machine is crunching the same project, 24/7. So BOINC points are only generally useful as a performance indicator, when considered as a trendline across timescales of a week or more.

Anyway, what kind of incentive are you after?

quote:

I seem to be unable to create an account on climateprediction.net via the BOINC manager at the moment. Its been telling me account creation is disabled.
I don't have any useful info on this, or on dual-booting. Sorry.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

yummycheese posted:

Number go up. Numbers that go up faster are more entertaining. It feels good to build a machine and modify it a bit so that you get good value for all this computing.
Well, I certainly understand that motivation. It's (a part of) why I went from a single dual-core machine to 4 12-core machines and 2 8-core machines in the space of two years.

quote:

The WCG project seems like an interesting challenge since its
1.) Linux x86_64 only for the OpenIFS work units.
2.) The work units is this silly weather modeling software written in FORTRAN
3.) The memory usage per WU is approaching 4GB/unit which makes its difficult to run 8+ WU's on a machine unless you have gobs of ram.
4.) The WU's take 7+ days to execute
WCG is a project of subprojects. The weather subproject (ARP), which you seem to be talking about, is definitely the "heaviest" of them in terms of resource requirements. But I don't believe being written in FORTRAN has anything to do with that

According to WCG staff, the average time for an ARP WU to complete was 24h, and I believe they said to expect usage of about 1GB/WU, so I'm unsure why you're seeing such high overhead. You can turn off ARP under "My Projects" on the WGC site if you don't want to run it.

Other subprojects definitely require far less memory, in any case. Right now I'm seeing Mapping Cancer Markers WUs using ~35MB each, and Microbiome Immunity Project WUs using around ~350MB each.

quote:

It seems like you need serious dedication to meaningfully contribute. Lots of linux janitoring and probably running retired server hardware at home.
I suppose that depends on your definition of "meaningful". I think that as long as the work is getting done before WUs are late, it's meaningful. If you want to compare RAC (recent average credit) with the seriously big users, then yes, you'll need lots of modern cores.

I can't argue with you about the linux management part. I created a whole project to take care of that for me (but I enjoy that sort of thing). I absolutely do not use any rackmount stuff, as I have neither the space for it nor the desire to have a constant 90dB of fan noise in the background. A lot of people do, because a lot of people who get into this seem to dig the homelab thing. Personally, I use desktop-grade CPUs and 140mm fans.

quote:

Whether this is a good use of any ones resources I have no idea.
I wouldn't be here if I didn't think it was a good use of resources. But they're your resources, so it's your decision.

quote:

Also its outside the reach of a lot of casual boinc users who only run the windows gui in the background.
There's nothing wrong with running BOINC in the background and/or on Windows.

quote:

So far the return seems abysmal. Ill burn through the $300 credit in a few months and maybe only get a few thousand points for the effort.
Cloud instances are provisioned with the assumption that you'll be running something very, very transient and/or bursty -- because that's how most apps are these days. There is massive oversubscription. My very small bit of experimenting with this showed similar results to yours. For 24/7 computing at full load, the ROI is way better on real hardware, assuming you don't live somewhere with cripplingly high electricity costs.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Another month, another WCG update digest.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Progressive JPEG posted:

I have a small cluster of 8 Raspberry Pi 4's running a Kubernetes (k3s) cluster on stock Raspbian.

That's pretty cool.

quote:

Now on the hardware side I need to see if I can bump the PSU voltage enough to fix the low voltage warnings I'm now seeing in dmesg

What kind of power supply are you using? I've thought about this kind of thing before, but I am very dumb with electricity and therefore terrified of custom PSU type stuff.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

WCG stats for my team as of this evening:
code:
Total Run Time (y:d:h:m:s) (Rank)    100:002:00:45:21 (#1,100)
Points Generated (Rank)              254,878,839 (#617)
Results Returned (Rank)              410,516 (#696)
First century of CPU time!

* Started with one machine, on 14 Jan 2018
* Took just over one year to reach 25 years of compute (19 Jan 2019; up to 3 machines at that point)
* Hit 50 years just under 7 months later (12 July 2019; 4 machines online; one upgraded to 3900X)
* A little under 4 months later (3 Nov 2019) was 75 years. All 4 machines were 3900X by this point, the final 2700 having been swapped out a week before
* Just over 3 months later, today, 100 years. 6 machines crunching; four 3900Xs and two 2700s

I'm effectively out of space and power, so from here on out I'm only getting more CPU through upgrade cycles

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

It's been a progression! I included the quarter-century times to show that I built up this capacity, and that it took me time to do it. This has latched onto some part of my brain and re-ignited the hardware obsession that I thought went away in my late 20s.

I think what's driven it for me is having a sense of purpose to attach things to. I had gotten over hardware-for-hardware's sake (I'm not saying there's anything wrong with that sort of exploration; I'm saying that I have been there and done that, and no longer found it compelling). My personal needs were so slight, compared to the capabilities of modern hardware that I had mostly become interested in low-power/high-efficiency computing. But getting into grid computing changed my personal definition of efficiency from "does what i need in as few watts as possible" to "does as much science as possible per unit time, for a reasonable number of watts". Of course, that hinges on the definition of "reasonable" and currently I'm defining that as "around 100W for the overall system, plus another 100W for systems with GPUs".

This is a huge amount of electricity compared to, say, a Raspberry Pi, but the results are going to be hugely scaled up as well. And what really made me OK with that sort of power usage was realizing that until the mid-1990s when CFLs started to become mainstream, most American homes were lit by incandescent lamps in the 60-100W class. My compute farm is a massively better use of that energy than 7 100W light bulbs.

I'm looking forward to Zen 3 and 4, and seeing what kind of improvements I can manage to get from them. I'll also be happy when Intel finally gets their poo poo together and comes back with a reasonably priced high core count part, now that AMD has changed the game. Historically no one has been able to touch them in the lower power arena (without going down a performance bracket to ARM), so I'd be happy for them to be back in the game to keep AMD from getting lazy again. Really, it's just fun to be paying attention to advances in hardware again, and to have found something genuinely useful to do with computing as a hobby.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Somehow it's almost March, so here's my monthly roundup of WCG project updates.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Folding and Rosetta both have COVID-19 research work ongoing.

https://www.ipd.uw.edu/2020/02/rose...ng-coronavirus/

https://foldingathome.org/2020/02/2...d-19-2019-ncov/

So if you wanna get in on that, you know what to do.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

WCG's Smash Childhood Cancer subproject restarted work today after a three month hiatus, with new targets. I'm super stoked about this.

Can confirm that WUs are being sent out. My farm has crunched just under 60 since whenever the tap opened back up.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret


I've added links to FAH and the SA FAH team to the OP. It wasn't an intentional oversight; I only recently discovered that it is not a BOINC project, and so is not included in the BOINC project lists.

quote:

Also I took my Auto OC Afterburner OC off my 1080 FTW should I try it with it on?

Running hot is the last thing you want. The only clock or voltage tweaks I would recommend would be underclocking and/or undervolting. If you're not comfortable with that, then don't worry about it and just run everything stock.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Tuxide posted:

I encourage everyone to give COVID-19 work priority.

The overwhelming majority of my cores/threads are crunching on biomedical research of one sort of another, including Rosetta. Edit: And I've suspended Einstein and Asteroids for the time being; now my GPUs are dedicated to GPUGrid.

Also, while there aren't any hard details yet (and it takes them months to spin up projects), WCG staff have announced that they're looking for teams who want resources for generalized epidemiology research -- because it looks like this is maybe going to be a thing more than it used to be.

mdxi fucked around with this message at 18:58 on Mar 19, 2020

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

I haven't posted about it in a while, but Homefarm has continued to have (mostly minor) updates. Inspired by the burst of BOINC activity going on these days, I just did a bit of work that I've been putting off for a while and have pushed out v2.4.0.

This is the first non-bugfix release in a while, and it adds the 'query' subcommand, which lets you generate simple statistics on the projects your machines are attached to. You can read about it here (scroll down a bit to "Getting project statistics"). Here's the sample output for people who are a tiny bit interested, but not enough to want to click a link:
code:
# farmctl query community
--------------------------------------------------------------
node01
WUs in past 24 hours: 202
        Min runtime: 00h 21min 18s
        Max runtime: 03h 18min 03s
        Avg runtime: 02h 18min 48s
WUs by quintile:
        <= 00h 56min 39s         31      (15.3%)
        <= 01h 32min 00s         23      (11.4%)
        <= 02h 07min 21s         47      (23.3%)
        <= 02h 42min 42s         6       (03.0%)
        <= 03h 18min 03s         95      (47.0%)
If this looks familiar, that's because 'query' uses the script that I wrote to do benchmarking when the 3900X was released. It's actually been included with Homefarm since v2.0.0, but you had to notice that it was there and figure out how to call it manually. Now it's nicely integrated into `farmctl`.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Prescription Combs posted:

Joined the Rosetta team and fired up my mostly idle cloud server to cwunch it up.




Quaint Quail Quilt posted:

I thought overclocking could cause problems with your submissions in grid computing that you wouldn't notice and maybe waste time when someone double checks your results

Mostly you don't want to overclock because when you're doing this kind of work, you're fully utilizing the silicon: all cores, all threads, all the time. The sort of workload that OCers and benchmarkers tend to refer to as "torture testing". Only the goal here isn't to see how fast you can do one run of Cinebench, or to remain stable for 90 minutes of Prime95. The goal is years of nonstop computational work. This is what the chips are designed to do, but they're designed to do it at the rated speed and voltages.

quote:

hmm, I'm number 2 (active member contributions) rosetta@home BOINC and #281 folding@home after ~1 week.

Yep. You'll fly up through all the inactive users and people who never actually returned a single result very quickly. It's fun!

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

I've added a top section to the OP about Covid-19 efforts, including a link to some human genome work now being done by TN-Grid in Italy.

I'll do my best to keep that up to date. Please let me know if you have any ideas for stuff that should be there.


Tuxide posted:

There's a phishing scam going around targeting healthworkers who are being asked through email to install "Folding@Home" to help fight COVID-19, but are instead given a link to malware.

Good to know that there's always people being as lovely as possible, in any situation.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Frontera at TACC has been working on Covid-19, it turns out: https://www.eurekalert.org/pub_rele...a-cms032320.php

TACC host some grid projects, but they don't let you select what work you'll be doing, and they do a LOT of work for the petrochemical industries.

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

Unclean posted:

I haven't had a somewhat decent GPU for awhile, got an XFX 7850 in today for $38 and now computing like it's 2012! Glad to see the first WU I got was for Coronavirus.

Welcome to the party

One of my machines is still crunching with a GTX 750 Ti. Older GPUs can still do lots of good work.

Adbot
ADBOT LOVES YOU

mdxi
Mar 13, 2006

to JERK OFF is to be close to GOD... only with SPURTING



Wedge Regret

World Community Grid has announced their generalized pandemic research project. It's being led by Scripps on the science side.

https://www.ibm.org/OpenPandemics

There are a couple of threads on the WCG forums about this. Questions that have been answered so far:
  • No initial Android support (Windows, Mac, Linux)
  • This is because the underlying software will be Autodock, which is developed by Scripps (the org running the project)
  • "At launch, the project will run on CPU"
  • There will be a beta, which will be announced (this is pretty standard for WCG)
WCG forums threads, in case you're interested in keeping yourself up to date:

Sci/Tech thread
WUs and badges thread

If you're attached to WCG and have "Join new projects when they become available" selected in your prefs (I remember it being on by default), then you'll start getting WUs as soon as they start pushing them out.

Edit: Tuxide, I'll fold your OP recommendations in soon. I started a new job a couple days ago, so I'm a bit more scattered than usual.

mdxi fucked around with this message at 05:51 on Apr 2, 2020

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply