Go: Google's Plan9 refugees are staging a breakout

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Go: Google's Plan9 refugees are staging a breakout

«‹›14 »

InAndOutBrennan: Dec 11, 2008

robostac posted:

The for loop is setting each element of iface to be the address of itself. At this point iface is a slice of interface objects containing pointers to the addresses of that interface object. row.Scan() then gets one row from the database. It then stores each value in *iface[i] which gets rid of the pointer and leaves you with a slice of values (each one stored in an interface object). The next time round the loop you need to reset all of these pointers, otherwise it doesn't know where to put the result.

Thanks, I did some further digging and reading after posting but this puts it clearly.

It seems a bit roundabout that I have to reset the interfaces in the slice back each loop. I'm sure there's a good reason for it but it feels kinda weird that it "changes" under me in this case, even though it's probably a feature.

Is there a better or more idiomatic way of accomplishing what I'm after?

# ? Jan 30, 2019 14:40

Adbot: ADBOT LOVES YOU

# ? Apr 26, 2024 07:11

ErIog: Jul 11, 2001

InAndOutBrennan posted:

Is there a better or more idiomatic way of accomplishing what I'm after?

I don't know, but it sounds like you're trying to implement something similar to what SQLx does. SQLx allows you to scan SQL query results into structs the same way encoding/json and other things in the standard library allow you to do.

I believe it's handled there using reflection rather than relying solely on empty interfaces, but I haven't source-dived it in a while.

# ? Jan 31, 2019 00:33

InAndOutBrennan: Dec 11, 2008

Thanks guys.

Next question, how do I keep track of 0-n unique combos of strings? N being only known at run time.

map[[]string]myStruct was my first guess but slices can't be map keys. Interface{} can be map key but as soon as it turns into a slice it can't. A struct can be map key but as soon as I add a slice to the struct it can't anymore. Which again makes some kind of sense I guess.

C++ allows a vector as map key, in Java i could roll my own equality functions etc but here I'm kind of stumped. I'm not that keen on byteing them with some kind of separator and dealing with that.

Ideas?

# ? Jan 31, 2019 10:27

Joda: Apr 24, 2010; When I'm off, I just like to really let go and have fun, y'know?; Fun Shoe

InAndOutBrennan posted:

Thanks guys.

Next question, how do I keep track of 0-n unique combos of strings? N being only known at run time.

map[[]string]myStruct was my first guess but slices can't be map keys. Interface{} can be map key but as soon as it turns into a slice it can't. A struct can be map key but as soon as I add a slice to the struct it can't anymore. Which again makes some kind of sense I guess.

C++ allows a vector as map key, in Java i could roll my own equality functions etc but here I'm kind of stumped. I'm not that keen on byteing them with some kind of separator and dealing with that.

Ideas?

Make the key a large int and hash your list in stead. You should be doing that anyway, and it's almost definitely how string keys are implemented in the language

Go even comes with a lot of useful hashing functionality out of the box

Joda fucked around with this message at 12:14 on Jan 31, 2019

# ? Jan 31, 2019 12:11

robostac: Sep 23, 2009

The handling of hash collisions would be different in that case - if you let the map handle the hashing then distinct values that hash to the same value get stored in the same bucket but can still be retrieved separately. If you are providing the hash value yourself this wouldn't be the case.

Having said that I've not got any idea how you'd do it without either having a fixed size array with a limit on n (and just having blank strings for smaller n's) or converting them to a single joined string.

# ? Jan 31, 2019 13:57

Joda: Apr 24, 2010; When I'm off, I just like to really let go and have fun, y'know?; Fun Shoe

Oh yeah, I hadn't thought of collisions.

# ? Jan 31, 2019 14:18

Hadlock: Nov 9, 2004

Best sound library? I see beep, and a few others but nothing is really jumping out at me

I'm trying to build a specific waveform given a digital input

# ? Feb 1, 2019 07:59

InAndOutBrennan: Dec 11, 2008

robostac posted:

The handling of hash collisions would be different in that case - if you let the map handle the hashing then distinct values that hash to the same value get stored in the same bucket but can still be retrieved separately. If you are providing the hash value yourself this wouldn't be the case.

Having said that I've not got any idea how you'd do it without either having a fixed size array with a limit on n (and just having blank strings for smaller n's) or converting them to a single joined string.

Yeah, after looking around a bit and also thinking of collisions I'll probably end up joining the strings as keys and store the original slice in a struct as values, including the other things I'm really after storing. That way I can get back the original slice easily enough.

# ? Feb 1, 2019 13:19

ErIog: Jul 11, 2001

Hadlock posted:

Best sound library? I see beep, and a few others but nothing is really jumping out at me

I'm trying to build a specific waveform given a digital input

Are you trying to output it to a file or directly to speakers? Encoding LPCM manually isn't that hard, and it looks like beep would be able to handle that.

You might look into building your own Beep.Streamer implementation that plays your own generated tones instead of things from files.

# ? Feb 2, 2019 00:49

Hadlock: Nov 9, 2004

I'm outputting directly to line out, yeah

It's a radio beacon so i only need to transmit maybe 5 x 256 byte packets per 6 minutes, averaging closer to one packet per 3 minutes, so the packet can be pre rendered and played back on demand, more or less. Packet generation is async but playback needs to be in real time.

# ? Feb 2, 2019 05:02

Vanadium: Jan 8, 2005

Hey, does anyone know if it's safe to try to close the same socket multiple times and/or from multiple goroutines, without risk to close an unrelated socket that reused the first socket's file descriptor after the first close?

I'm trying to untangle some messy websocket setup with separate reader/writer goroutines that currently leaks sockets unless we do the graceful websocket shutdown, and I'm trying to figure out if I can just try to close in some codepath that always gets taken or if I need to track across goroutines whether the graceful shutdown has taken place.

# ? Feb 4, 2019 13:40

homercles: Feb 14, 2010

Yeah it's safe, eg net.TCPConn.Close -> net.conn.Close -> net.netFD.Close -> internal.poll.FD.Close, which itself wraps each close operation in a call to fd.fdmu.increfAndClose which safely guards against multiple closes of the same FD.

You can navigate this stuff fairly easily in just godoc source view mode if you run it with godoc -analysis type -http :7070. strace this example code and you get this output, demonstrating closes on the old socket struct don't impact the newly allocated socket despite it reusing fd 3.

I can't speak for your websocket library, but the core library socket impl looks safe.

# ? Feb 5, 2019 00:29

Vanadium: Jan 8, 2005

Thanks! I'd followed a websocket Close call with delve until I got to increfAndClose, where I figured it looks spooky enough that I wouldn't be confident in my own understand of the synchronization there either way. :shobon:

# ? Feb 5, 2019 10:58

Xik: Mar 10, 2011; Dinosaur Gum

What is the benefit of using a

code:

map[string]SomeStruct

code:

map[string]*SomeStruct

Or put another way: Why bother with the mental gymnastics of deciding if the struct is "large enough" to replace it with pointers to the struct and just always do so? Is there some overhead to resolving the pointer compared to not doing so?

# ? Feb 11, 2019 00:34

Joda: Apr 24, 2010; When I'm off, I just like to really let go and have fun, y'know?; Fun Shoe

Xik posted:

What is the benefit of using a
code:
map[string]SomeStruct
vs
code:
map[string]*SomeStruct
Or put another way: Why bother with the mental gymnastics of deciding if the struct is "large enough" to replace it with pointers to the struct and just always do so? Is there some overhead to resolving the pointer compared to not doing so?

It depends entirely on your use case. If your map owns the structs, use values, if it has references to structs managed by some other data structure, use pointers. In terms of raw performances, a value map is probably faster than a reference map on iteration or repeated access due to lower rate of cache aliasing for objects contiguous in memory, but the reference map will be faster on mutation, due to having to move, allocate and deallocate less memory (e.g. It wont have to reorder all the data in the map, just the references.)

Joda fucked around with this message at 00:46 on Feb 11, 2019

# ? Feb 11, 2019 00:42

Xik: Mar 10, 2011; Dinosaur Gum

Joda posted:

In terms of raw performances, a value map is probably faster than a reference map on iteration or repeated access due to lower rate of cache aliasing for objects contiguous in memory

Ok, so as a general rule if you just plan to frequently read from a map of immutable structs then there are benefits to just using a value map. Otherwise mutating the map or the structs themselves use pointers?

# ? Feb 11, 2019 01:06

Joda: Apr 24, 2010; When I'm off, I just like to really let go and have fun, y'know?; Fun Shoe

I'm just referring to mutating the map itself. The structs don't matter. But yeah that's the gist of it.

# ? Feb 11, 2019 01:15

NtotheTC: Dec 31, 2007

Odd question but whats Go like as a language to learn for career development? I have 6 years of experience with web development using python, but I want to make myself a little more well-rounded and break out of my comfort zone.

I like the idea of learning a lower level language than python bit I dont know of there are any practical benefits to it or whether I should be looking instead to expand into a discipline like machine learning or other data science

# ? Feb 16, 2019 08:45

Walked: Apr 14, 2003

NtotheTC posted:

Odd question but whats Go like as a language to learn for career development? I have 6 years of experience with web development using python, but I want to make myself a little more well-rounded and break out of my comfort zone.

I like the idea of learning a lower level language than python bit I dont know of there are any practical benefits to it or whether I should be looking instead to expand into a discipline like machine learning or other data science

I mean; I'm not sure how to answer your question.

Fundamentals are kinda the key to career growth and development; languages are tools. You're not going to hurt yourself by learning a tool; but look at the end goal of what that tool is going to let you accomplish better than what current tool(s) you have at your disposal.

What do you want to be doing with your career? If it's machine learning, Go might not be (probably isnt) the most optimal use of your time right now. I happen to use Go heavily for a lot of "DevOps" tool building (gently caress I am so tired of that word), but it just happens to be a well suited tool for the job. Often I'll end back up in python because it's more suited for a specific task.

In the end; I'd say:
- Think about what you want to be doing in terms of your career.
- Look into what specific languages and toolchains are most heavily utilized in that space
- Spend your time really honing your understanding of the fundamentals conceptually; generally focus on what's widely used in the area(s) you're interested in, but dont be afraid to pick up new tools / languages if and when it is appropriate.

But Go is awesome, too. So if you're looking for a challenge or just bored and want to mix it up, go for it!

# ? Feb 16, 2019 12:30

spiritual bypass: Feb 19, 2008; Grimey Drawer

Anyone ever seen the SQL library panic under high concurrency? I'm running a load test on a reasonably simple API backend and I start getting lots of panics once I pass about 250 req/s. Seems kinda bizarre to me, but there's still more investigation to do...

# ? Feb 20, 2019 01:48

homercles: Feb 14, 2010

Does building your code with the race detector shed any light?

# ? Feb 20, 2019 05:56

spiritual bypass: Feb 19, 2008; Grimey Drawer

No, it doesn't detect anything in this case.

But it turns out doing defer rows.Close() before handing the error on the Query() means you're invoking a method on a potentially nil rows which is obviously a good time to panic :pwn:

Now it's time to move the error handling sooner in the sequence and run my load test again. Then I'll know what the actual database errors are and handle them appropriately. Feels good to get over this speedbump after crossing so many little hurdles on this particular idiot spare time project.

# ? Feb 20, 2019 22:49

waffle enthusiast: Nov 16, 2007

Don�t remember if this was posted before but it�s making the rounds now. Dave Cheney posted an article constructed from his practical go talk. It�s really, really good. Goes into the philosophy of Go, as well as idiomatic concepts. Well worth your time.

# ? Feb 24, 2019 15:08

Hadlock: Nov 9, 2004

Looking for a drop in replacement for Django/Rails

Looked at Revel and Beego, but looks like Buffalo is somewhat newer, fully batteries included and excellent community support. The two Buffalo tutorials I looked at looked recognizably like Django but with go syntax.

Any thoughts on these web frameworks for 2019? Community support for go libraries tends to vary over time. Thanks.

# ? Mar 21, 2019 13:00

Sylink: Apr 17, 2004

I'm trying to pick up some Go for fun and I cannot loving get channels and goroutines.

So I have this based on some boilerplate and it always results in a panic or deadlock like

fatal error: all goroutines are asleep - deadlock!

code:



package main

import "fmt"


func print_nums(number int,messages chan<- int) {
	messages <- number
}

func main() {

    messages := make(chan int,10000)
	for i := 2; i < 1000; i+=1 {
	    go print_nums(i,messages)
	}

	for elem := range messages {
        fmt.Println(elem)
    }
}

Basically, I want to run a function over a bunch of values and collect the results so I can parallel process a given problem. Also buffered vs unbuffered channels make no difference.

I've read a ton of examples but they are all simple Println output in the goroutine, and its not clear how I can collect the total results for analysis or whatever after all the goroutines are finished.

I'm sure this is simple.

# ? Mar 25, 2019 23:16

Pham Nuwen: Oct 30, 2010

The second for loop is trying to read from a channel that never closes, so it deadlocks. Here's one way to get around that:

code:

package main

import (
	"fmt"
	"sync"
)

func print_nums(number int, messages chan<- int, wg *sync.WaitGroup) {
	messages <- number
	wg.Done()
}

func main() {
	messages := make(chan int, 10000)

	var wg sync.WaitGroup
	for i := 2; i < 1000; i += 1 {
		wg.Add(1)
		go print_nums(i, messages, &wg)
	}

	wg.Wait()
	close(messages)

	for elem := range messages {
		fmt.Println(elem)
	}
}

Every time we kick off a new processing routine, we increment the waitgroup. Every time a processing goroutine exits, it decrements the waitgroup. The main thread waits until everyone has exited, then closes the channel and then reads all the poo poo off the channel.

# ? Mar 25, 2019 23:26

Sylink: Apr 17, 2004

Are wait groups the proper way in Go , then, if you need to wait for everything to finish ? I.e. everything isn't contained within the goroutine itself that you need done?

Because I've noticed trying to close the channel before the for loop, or the beginning while checking to see if the channel is still open first, still doesn't work. Like

code:


	c := make(chan int, 10)
	go someshit(...)
	for i := range c {
		_, ok := <-c
		if ok == false {
		close(c)
....

# ? Mar 26, 2019 01:09

Pham Nuwen: Oct 30, 2010

Sylink posted:

Are wait groups the proper way in Go , then, if you need to wait for everything to finish ? I.e. everything isn't contained within the goroutine itself that you need done?

Because I've noticed trying to close the channel before the for loop, or the beginning while checking to see if the channel is still open first, still doesn't work. Like
code:
	c := make(chan int, 10)
	go someshit(...)
	for i := range c {
		_, ok := <-c
		if ok == false {
		close(c)
....

If you're going to use range, somebody needs to close the channel as per the spec: "For channels, the iteration values produced are the successive values sent on the channel until the channel is closed. If the channel is nil, the range expression blocks forever. "

Now, since you know how many goroutines you're going to run, you could do this:

code:

c := make(chan int)
for i := 0; i < 1000; i++ {
  go calc(c)
}

for i := 0; i < 1000; i++ {
  fmt.Println(<-c)
}
close(c)

Edit: also, in the example you show above, you're reading from the channel twice in each loop iteration, but you're discarding the value the second time you read it.

Pham Nuwen fucked around with this message at 16:38 on Mar 26, 2019

# ? Mar 26, 2019 16:34

spiritual bypass: Feb 19, 2008; Grimey Drawer

Why can't I successfully compare two strings from a url.Hostname()?

When I run this code:

code:

if base.Host != linkUrl.Host {
	log.Printf(`%+q does not match %+q`, base.Host, fullUrl.Host)
	continue
}

I get this sort of output:

code:

2019/06/07 16:07:38 "localhost:8000" does not match "localhost:8000"
2019/06/07 16:07:38 "localhost:8000" does not match "localhost:8000"
2019/06/07 16:07:38 "localhost:8000" does not match "localhost:8000"
2019/06/07 16:07:38 "localhost:8000" does not match "bcbsdharma.org"

That string comparison ought to be pretty straightforward, right? Am I doing something stupid or is Go broken somehow? It's hard to imagine the latter :pwn:

e: welp, upon posting this I realize that linkUrl is not fullUrl

# ? Jun 7, 2019 21:13

Twerk from Home: Jan 17, 2009; This avatar brought to you by the 'save our dead gay forums' foundation.

I'm having mysteriously bad HTTPS serving performance, specifically TLS handshakes. Real applications are using the http.ListenAndServeTLS, right? I profiled my application and... it's spending all of its time doing TLS handshakes on a simulated load that we believe represents realistic traffic.

This is running on AWS, in ECS behind an ALB, but I'm under a contractual requirement to encrypt all traffic end to end, so the service needs its own self-signed cert. Any suggestions before I bolt on a sidecar proxy that will terminate SSL inside the same ECS task? That sounds like more headache than I want right now, and I'm disappointed that NodeJS seems to be able to TLS handshake significantly more efficiency than Go's built in http server.

# ? Dec 15, 2019 03:07

consensual poster: Sep 1, 2009

Twerk from Home posted:

I'm having mysteriously bad HTTPS serving performance, specifically TLS handshakes. Real applications are using the http.ListenAndServeTLS, right? I profiled my application and... it's spending all of its time doing TLS handshakes on a simulated load that we believe represents realistic traffic.

This is running on AWS, in ECS behind an ALB, but I'm under a contractual requirement to encrypt all traffic end to end, so the service needs its own self-signed cert. Any suggestions before I bolt on a sidecar proxy that will terminate SSL inside the same ECS task? That sounds like more headache than I want right now, and I'm disappointed that NodeJS seems to be able to TLS handshake significantly more efficiency than Go's built in http server.

There appears to be an open issue related to what you're seeing: https://github.com/golang/go/issues/20058

Have you tried using ECDSA instead of RSA for your certs? Discussion in that issue seems to indicate that would speed up the handshake quite a bit.

# ? Dec 15, 2019 06:07

Vanadium: Jan 8, 2005

Does the ALB not reuse connections? Maybe I'm naive here but I'd have expected that session spinup gets amortized away immediately.

# ? Dec 15, 2019 11:06

Startyde: Apr 19, 2007; come post with us, forever and ever and ever

I thought they muxed keepalives to the backhaul. Cranking the keepalive timeout up would help save you from needless renegotiation if that's the case.

# ? Dec 15, 2019 15:08

Twerk from Home: Jan 17, 2009; This avatar brought to you by the 'save our dead gay forums' foundation.

All of this is helpful, thanks.

I'll definitely move to an ECDSA key for the cert. I had read they were faster at some point, but didn't realize that optimizations on one path but not the other mean it's an order of magnitude.

The other issue that I've noticed contributing is that the ALB is demuxing HTTP/2 connections onto a much higher number of HTTP/1.1 connections between ALB and backend. I hadn't read the docs yet, and assumed that "HTTP/2" support on ALBs meant both client-facing and backend, but it's client-facing only. I'm going to turn up the keep-alive timeout in hopes of re-using connections more often. The traffic to this is spiky, and the default 60s timeout isn't long enough.

Twerk from Home fucked around with this message at 16:17 on Dec 15, 2019

# ? Dec 15, 2019 15:58

Startyde: Apr 19, 2007; come post with us, forever and ever and ever

I tend to a legacy FCGI product and yea that is definitely what their /2 support is. Honestly it's great since it's breathed a little more performance life out of the thing. For that we've got timeout at like 300sec iirc. (that's not a go listener, just saying)

# ? Dec 15, 2019 17:31

duck.exe: Apr 14, 2012; Nap Ghost

Is there a way to hook up a middleware of type �func(http.Handler) http.Handler� (like gorilla/mux uses) to a Gin router?

duck.exe fucked around with this message at 00:32 on Jul 2, 2020

# ? Jul 1, 2020 22:05

skul-gun: Dec 24, 2001; I got this account for Xmas.

DukeDuke posted:

Is there a way to hook up a middleware of type �func(http.Handler) http.Handler� (like gorilla/mux uses) to a Gin router?

I have not used gin, but it seems like you should be able to wrap an http.Handler:

code:

func wrap(h http.Handler) gin.HandlerFunc {
	return func(c *gin.Context) {
		h.ServeHTTP(c.Writer, c.Request)
	}
}

# ? Jul 4, 2020 07:43

Blame Pyrrhus: May 6, 2003; Me reaping: Well this fucking sucks. What the fuck.; Pillbug

Dumb question about dealing with json in GO.

In powershell, let's pretend the json body I am working with is:

code:

{
    "value": "thing",
    "number": 3
}

In powershell I can store that json body as say, $data and then simply:

code:

$array = $data | convertFrom-Json

and BAM, I have an object where I can say, express the value of number via $array.number

Similarly in python I can store that json statement as data and then:

code:

array = json.loads(data)

and BAM, I have an object I can do stuff with.

How can I do this same thing in Go?

I've googled and sure, I can declare a struct, but I'm writing this to service API calls to dozens of different endpoints, all with different json responses, I shouldn't need to define dozens of structs as well. Googling how to do this more dynamically leads me to use map[string]interface{} which just pollutes the data. with "map" prefixes and brackets all over the place.

This seems like such a basic thing, I'm shocked that I can't more easily do this.

Blame Pyrrhus fucked around with this message at 08:52 on Jul 9, 2020

# ? Jul 9, 2020 04:50

Hadlock: Nov 9, 2004

I was having shower thoughts about powershell the other day; despite being a Microsoft product, it's remarkably well thought out project and the true balls to the walls level of object oriented-ness is quite impressive.

# ? Jul 9, 2020 09:48

Adbot: ADBOT LOVES YOU

# ? Apr 26, 2024 07:11

geonetix: Mar 6, 2011

There's typicall two schools of thought about the json stuff in go as far as I've seen: declare your expected responses explicitely and unmarshal into that, or use the map method.

I'm a strong believer of the first method with predefined expected structs/types. Not only do you need to know what you're getting anyway, so you can actually do something meaningful with it, but it also helps you find changes, malformed input, reject nonsense, etc etc etc. All the code that follows from the json unmarshalling ends up a lot easier afterwards.

Using map[string]interface{} (note, not all json starts with an object, it can be an array, or just a string, or many other valid json types, too. so that would break immediately as well), leads you down a road straight into reflection hell, and personally I think you should avoid that at all cost, or at least isolate it in some tailored libraries so it gives you... structured data that you can predictably work on.

# ? Jul 9, 2020 10:43

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Go: Google's Plan9 refugees are staging a breakout

«‹›14 »