|
Cybernetic Vermin posted:i increasingly suspect that a lot of what is happening with chatgpt is a good old eliza effect. with a bunch of gpt-generated text sure, but i generally suspect that there's a bunch of hard logic layered on top which crudely parses a couple of classes of questions (e.g. "but in the style of x", "explain y", "that's not correct, should be z") tied up with rigid ways of (re-)querying the model. I thought you were going to go for the Eliza effect of humans attributing more capability to systems that interact via chat than they're actually showing. That, or just anthropomorphizing the "reasoning" behind the outputs as human like when it's not. Also, I've had some difficulty in either finding or believing the size on disk the gpt3+ model versions take. Latest i have seen is 800GB, which is actually larger than it takes to store the entirety of some versions of the massive common crawl dataset. I do wonder what fraction of the observed performance would have been possible by efficiently organizing this data for search and layering on top of it some summarization, text mixing/recombining, and style/format translation capabilities. Functionally, with a less than 3-1 compression ratio of training tokens to model weights and the known capability of these models to memorize training elements, this may very well be what is actually happening, just obfuscated within the mass of opaque model weights.
|
#
¿
Jan 24, 2023 13:47
|
|
|
|
| # ¿ Apr 1, 2023 11:13 |
|