This essay is missing the words “cause” and “causal”. There is a difference between discovering causes and fitting curves. The search for causes guides the design of experiments, and with luck, the derivation of formulae that describe the causes.
Norvig seems to be confusing the map (data, models) for the territory (causal reality).
This essay frequently uses the word "insight", and its primary topic is whether an empirically fitted statistical model can provide that (with Norvig arguing for yes, in my opinion convincingly). How does that differ from your concept of a "cause"?
We note that the practice in
applied research of concluding that a model with a
higher predictive validity is “truer,” is not a valid inference. This paper shows that a parsimonious but less
true model can have a higher predictive validity than a
truer but less parsimonious model.
> But it must be recognized that the notion of "probability of a sentence" is an entirely useless one, under any known interpretation of this term.
He was impressively early to the concept, but I think even those skeptical of the ultimate value of LLMs must agree that his position has aged terribly. That seems to have been a fundamental theoretical failing rather than the computational limits of the time, if he couldn't imagine any framework in which a novel sentence had probability other than zero.
I guess that position hasn't aged worse than his judgment of the Khmer Rouge (or Hugo Chavez, or Epstein, or ...) though. There's a cult of personality around Chomsky that's in no way justified by any scientific, political, or other achievements that I can see.
I won't try to defend Chomsky. (Not really a big fan even before this.) But if the mere mention of him is sus to you then I advise you to not study either linguistics or computer science because it's Chomsky normal forms and Chomsky hierarchies all the way down. There's even still people clinging to some iteration of the universal grammar despite the beating it has taken lately.
He's also one of the most prominent political thinkers on the American hard left for the last half century.
There's a joke going around for a while now that you either know Chomsky for his politics, or for his work in linguistics and discrete mathematics, and you are shocked to discover his other work. I guess we can extend that to a third category of fame, or infamy.
https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf
To Explain Or To Predict?
Nice quote
We note that the practice in applied research of concluding that a model with a higher predictive validity is “truer,” is not a valid inference. This paper shows that a parsimonious but less true model can have a higher predictive validity than a truer but less parsimonious model.
Hagerty+Srinivasan (1991)
*like TFA it's a sorta review of Breiman
> But it must be recognized that the notion of "probability of a sentence" is an entirely useless one, under any known interpretation of this term.
He was impressively early to the concept, but I think even those skeptical of the ultimate value of LLMs must agree that his position has aged terribly. That seems to have been a fundamental theoretical failing rather than the computational limits of the time, if he couldn't imagine any framework in which a novel sentence had probability other than zero.
I guess that position hasn't aged worse than his judgment of the Khmer Rouge (or Hugo Chavez, or Epstein, or ...) though. There's a cult of personality around Chomsky that's in no way justified by any scientific, political, or other achievements that I can see.
There's no point minimizing his intelligence and achievements, though.
His linguistics work (eg: grammars) is still relevant in computer science, and his cynical view of the West has merit in moderation.
The question then becomes on of actual novelty versus the learned joint probabilities of internalised sentences/phrases/etc.
Generation or regurgitation? Is there a difference to begin with..?
It's of rather limited use for natural languages.
The article by Peter Norvig is still interesting.
honestly, I'm surprised Noam is even still alive (aged 97), he is not long for this world and will be gone very soon.
He's also one of the most prominent political thinkers on the American hard left for the last half century.
There's a joke going around for a while now that you either know Chomsky for his politics, or for his work in linguistics and discrete mathematics, and you are shocked to discover his other work. I guess we can extend that to a third category of fame, or infamy.
(I don't like Chomsky for other reasons, but having an obituary ain't no reason to disregard someone's thoughts.)