What's Wrong with NLP, Part 4 - an Opinion from a Paper at IJCAI-13 and a Comment by Todor

Why can't computer understand me.. 

On our best behaviour - Hector J. Levesque

 http://www.cs.toronto.edu/~hector/Papers/ijcai-13-paper.pdf  -

I'd generalize what the author tries to say: yes, the machine does need *imagination* (sensory memories and spaces, see many, for example [7]) and to incrementally learn and play with it. That subsystem is required!* I do agree also that many AI-ers, especially in NLP community didn't understand or accept that, this paper suggests that there's no common progress in the acceptance.

Todor:

Some of the reasons are simply the academic and people's urge to produce papers and "results" quick, to demonstrate that you "do something". Papers, papers, papers - who cares about the real progress.

I guess some have understood that very well, the "common sense" problem is cited from decades, it is obvious, yet in the NLP they've been insisting to push a bunch of words with no relation to the real world and then ask "why it can't do proper word-sense disambiguation, given only a corpus?"

Also, IMO it's  the researchers in NLP etc. who don't understand language understanding [4],  rather than the poor computers, which, as many AI-use to say "do what they are told to".

Language teaching of a machine should be done incrementally with sensory-motor feedback and interaction, like teaching a child and not in sensory-less batch mode with a huge corpus with zero interaction, no coordinate space, just a bunch of words, because:

Todor: Natural language is a hierarchical redirection/abstraction/generalization/compression of sequences of multi-modal sensory inputs and motor outputs, and records and predictions for both.

A mere corpus has no imagination, even if it's 1 Terrawords. A small corpus, built by systematic interaction and mapped to sensory-motor system is intelligent and can explore and learn further on its own. Such as - the toddlers.

I support also the criticism of the Turing Test. Right, it is a test for fooling people - not a test of intelligence, and as I myself claimed back in 2001 (see [5] "Човекът и мислещата машина..." below) - a true, honest and too smart or quick AGI will quickly FAIL the test, because its output is too complex, too quick, too deep, she doesn't have some required memories (shuld lie for her childhood etc.) etc.

Finally, I will save some of my thoughts from myself...

Credits to Aaron for sharing the link!

* More about that when I complete the first milestones of mine...

See also: (And many others...)


[1] Faults in Turing Test and Lovelace Test. Introduction of Educational Test (2007)

[2] What's wrong with Natural Language Processing? Part 2. Static, Specific, High-level, Not-evolving... (2009)

[3] What's wrong with Natural Language Processing? Part I (shorter) (2009)

[4] Part 3: The NLP Researchers cannot understand language. Computers could. Speech recognition plateau, or What's wrong with Natural Language Processing? (2010)

[5] Анализ на смисъла на изречение въз основа на базата знания на действаща мислеща машина. Мисли за смисъла и изкуствената мисъл (2004)

[6] Човекът и мислещата машина. Анализ на възможността (...), 2001 г.

[7] Embodiment is just coordinate spaces, interactivity and modalities - not a mystery (2011)

Read More

Cognitive Science's Failure to Become an Interdisciplinary Field - the Multi-Interdisciplinary Blindness

Discussion of mine regarding the paper:

"Perspective | What happened to cognitive science?,  10 June 2019

https://www.nature.com/articles/s41562-019-0626-2?fbclid=IwAR2mMzO4qzIKINXT2BUcMZAS1ZI4PONd04SHQeF7FLgvH5VTc1pwR4DcLow

Available on Github: https://github.com/rdgao/WH2CogSci/blob/master/nunezetal_final.pdf?fbclid=IwAR17HIosUS-7EKdFT4a--SesVuwb3aPp-1a0yE4Rbk_q8io8w5S6nFVrWvY

https://www.facebook.com/groups/RealAGI/permalink/1230556097152988/

IMO a big share of the problem lays in the whole researchers's and overall intellectual direction in the academic circles (and power-and-profit driven societies, modern slavery). It is narrow knowledge and world view, specialization is promoted, ones who obey and execute instructions of their superiors grow the ladder and become leaders, doing the same. The creative, wide-minded and really original ones are not leaders of the research.*

That is related to multi-interdisciplinary blindness, related to insufficient working memory capacity and faculties for understanding and representing the inputs generally enough so that one can encompass the concepts from different domains and contexts and think of them together.

BK calls it too simply "depth of structure".

---

* In the past there were exceptions, such as Alan Kay
** That survey paper reminds me of my cycle "What's wrong with Natural Language Processing" some 10 years ago, because to me NLP/Computational Linguistics should have been a part of the AGI, not what they were.

** Sorry for the sick formatting, I had to write it in external editor etc., this one is annoying, but not now.

See elaborate related discussions:

---

#1

What's wrong with NLP? 

http://research.twenkid.com/agi/2010/Whats_wrong_with_NLP_links.htm

Circa 2009-2010 - series of 3 "perspective" articles

---

What's wrong with NLP, part I:

http://artificial-mind.blogspot.com/2009/02/whats-wrong-with-natural-language.html

Monday, March 23, 2009

---

What's wrong with Natural Language Processing? Part 2. Static, Specific, High-level, Not-evolving...

http://artificial-mind.blogspot.com/2009/03/whats-wrong-with-natural-language.html



Note: now in NLP there are impressive results in NLG (generation), BERT etc. with such "mindless" vector representations, using current methods of machine learning, convolutions, "transformers" etc. however  it probably more or less emulates virtual sensory-motor interactions - by traversing and comparing huge corpora and how different texts/segments (mappings of sensory records) map and relate to each other, what's reasonable in what context. It is more advanced than as it was in the earlier simple frequency-based representations and inverse-frequency - frequency/probability of a token in current document, compared to average in the other documents etc.

#2 Friday, January 1, 2010

I will Create a Thinking Machine that will Self-Improve 

An Interview with Todor, "Obekty" magazine, issue November-December 2009

http://artificial-mind.blogspot.com/2010/01/i-will-create-thinking-machine-that.html   


- Where does the researchers' efforts should be focused in order to achieve Artificial General Intelligence (AGI)?

First of all, research should be lead by interdisciplinary scientists, who are seeing the big picture. You need to have a grasp of Cognitive Science, Neuroscience, Mathematics, Computer Science, Philosophy etc. Also, creation of an AGI is not just a scientific task, this is an enormous engineering enterprise – from the beginning you should think of the global architecture and for universal methods at low-level which would lead to accumulation of intelligence during the operation of the system. Neuroscience gives us some clues, neocortex is “the star” in this field. For example, it's known that the neurons are arranged in sort of unified modules – cortical columns. They are built by 6 layers of neurons, different layers have some specific types of neurons. All the neurons in one column are tightly connected vertically, between layers, and are processing a piece of sensory information together, as a whole. All types of sensory information – visual, auditory, touch etc. is processed by the interaction between unified modules, which are often called “the building blocks of intelligence”.

- If you believe that it is possible for us to build an AGI [Since you do believe], why we didn't manage to do it yet? What are the obstacles?

I believe that the biggest obstacle today is time. There are different forecasts, 10-20-50 years to enhance and specify current theoretical models before they actually run, or before computers get fast and powerful enough. I am an optimist that we can go there in less than 10 years, at least to basic models, and I'm sure that once we understand how to make it, the available computing power would be enough. One of the big obstacles in the past maybe was the research direction – top-down instead of bottom-up, but this was inevitable due to the limited computing
(...)

---

#3

Tuesday, August 27, 2013

Issues on the AGIRI AGI email list and the AGI community in general - an Analysis

https://artificial-mind.blogspot.com/2013/08/issues-on-agiri-agi-email-list-and-agi.html
"- Multi-intra-inter-domain blindness/insufficiency [see other posts from Todor on the list] - people claim they are working on understanding "general" intelligence, but they clearly do not display traits of general/multi-inter-disciplinary interests and skills.

[ Note: Cognitive science, psychology, AI, NLP/Computational Linguistics, Mathematics, Robotics … – sorry, that's not general! General is being adept, fluent and talented in music, dance, visual arts (all), acting, story-telling, and all kinds of arts; in sociology, philosophy; sports … (…) ... + all of the typical ones + as many as possible other hard sciences and soft sciences and languages, and that is supposed to come from fluency in learning and mastering anything. That's something typical researchers definitely lack, which impedes their thinking about general intelligence. ](...) "

Discussion:
http://artificial-mind.blogspot.com/2014_08_05_archive.html


Tuesday, August 5, 2014

---

#4

The Super Science of Philosophy and Some Confusions About it - continuation of the discussion on the "Strong Artificial Intelligence" thread at G+

http://artificial-mind.blogspot.com/2014/08/the-super-science-of-philosophy-and.html

---

#5
Working memory capacity, traces of thoughts ... - continuation of the thread on G+ "Strong Artificial Intelligence" [Continues]

http://artificial-mind.blogspot.com/2014/08/working-memory-capacity-traces-of.html

---

#6

Thursday, March 22, 2012
Human-Level AGI Within ~10 Years and AGI in the Formal Academia
A comment on Ben Goertzel's post: Ray Kurzweil's (Sometimes) Wrong Predictions

Todor "Tosh" Arnaudov said... , 21/3/2011, 8:53 PM
https://artificial-mind.blogspot.com/2012/03/human-level-agi-within-10-years-and-agi.html

Read More

What's wrong with Natural Language Processing?

A short philosophical essay...

NLP?
Do you know about Machine Translation - either rule-based or Statistical, the Lexical Databases like WordNet? Statistical Parsers, POS-Taggers, Parallel Corpora with 1 billion words. Machine Learning, N-grams, Hidden-Markov-Models. And Blah-blah-blah...

What's wrong with NLP?

The main issue I find in the paradigm of NLP today is, I think, embedded in the mindset of the researchers in general, and in the research tradition. IMHO, usually researchers are mathematicians and too nerdy persons.

Usually researchers are mathematicians - not artists, not creative enough and not brave enough to dive into too deep imaginative directions.

Science do also pushes the typical researcher not to invent too much. If he does use his imagination too much and does create "imaginary structures", he might be unable to prove their creation and existence "scientifically" enough to the other members of the sect. He would not be acknowledged etc.

Too nerdy, too mathematical, too obvious and directly "provable" by the raw output data.

As an example of this I would mention Statistical MT.

These distributions work to a certain degree, but this is a trick. It is not really original. There are pure mathematical parameters, which are pretty obvious.

Sorry, but isn't Statistical Machine Translation a mathematical trick?
What are scientific basis of it?

Flat parameters, which can be derived by the distributions of words.

- Take a problem.
- Divide it it into "items" with which you can do something.
- Take the items which you can do something with, and find the combinations which make a difference.
- Take the items and do combinations in order to see what is the difference.

OK, research is a process of exhausting anyway.

Research is exhausting anyway, but if you do not invent structures which are outside and above the obvious ones, you cannot reach too far. However, the relations and combinations which are obvious or easily derived by the raw data without auxiliary "unreal" structures are easier to prove and to be acknowledged in the sect; pardon, I mean science.

This reminds me Quantum Mechanics and the hidden variables. I don't now about physics, but in NLP definitely hidden variables do exist.

Sorry, but in NLP there are hidden variables, out of the text.


I would mention also, that for any creative writer or poet, words are much more than distributions. Writing of a novel is an imaginative process, you are building a world with actors, with laws etc.. Then you simulate that world and record in text what happens.

Creative writing is imagination and simulation, not probability distribution of words.


Actually your dreams about the world you describe are very much more detailed, because the mind is not based on words.

And those Statistical techniques (at least that I know) are flat, because they lack imagination and will, they do not simulate worlds.

Do standard statistical techniques do simulate worlds? Don't think so.

Mind needs to build a virtual world and fit the text to the virtual world, which is simulated during understanding. And I believe that the simulated world in mind is not built by NP, VP etc. NP, VP etc. can be mapped to some aspects of the "simulator" or to modify that simulator in a particular way, but I don't think they are the simulator.

CON: There is no simulator, brain is very slow etc.

(Mostly) memory-based reconstruction is also a simulation. Those 100-level or so of neurons in one "pass" might be pretty enough, if the task is subdivided in a smart way.

Does the "simulator" in mind is built by NP, VP etc.? I don't think so.*

Rather mind can fit its simulator to work with these structures if it needs/wishes.

A prove: babies do not know about words, and in early age when they know something about words, their mind is not based on words anyway. They are pretty intelligent without NP-VP etc., later on they fit NP-VP to something else. I guess so.


To be continued:

- Reflect on explanation of the reason Statistical translation give useful results. ?
- Finally start doing some kind of "simulators" yourself and stop just talking philosophycally! :))

Continues:

What's wrong with Natural Language Processing? Part 2. Static, Specific, High-level, Not-evolving...


* Edit, 1.2.2024: "Do the "simulator"... --> Does the ...; Continue: ... --> Continues

Read More

What's wrong with Natural Language Processing? Part 2. Static, Specific, High-level, Not-evolving...

By Todor Arnaudov

Independent Researcher - Twenkid Research
Independent Filmmaker - Twenkid Studio
ASIC Engineer - (as of March 2009)

MS Software Engineering, Plovdiv University, 2008
BS Computer Science, Plovdiv University, 2007
Internship in RIILP, University of Wolverhampton, 2007


What's wrong with Natural Language Processing?
Part 2. Static, Specific, High-level and many more!


In brief: Static, Specific, High-level, "Short-chain of intelligent operations", Lack of evolution of the systems, Not enough experimental approach and research for new paradigms

Part I: http://artificial-mind.blogspot.com/2009/02/whats-wrong-with-natural-language.html


Overall, I'm disappointed by the state-of-the-art of NLP. I think the approaches are too shallow, too obvious, too static... And many more...


[Critics] Who the hell are you to be disappointed from "the state-of-the-art"? Who cares? When did you finish school?!


And I think after taking into account all of the "wrong" parts, it is not surprizing that the progress is slow.

We are walking on almost the same road.

The road does go to somewhere, but I think - not right there where me and other are aiming to. OK, call us yet "dreamers".

[Critics] Crazy mad "scientists"!

If the approach doesn't change, I can't see how the "normal" researchers would go there . The same stuff which is not going to evolution is being done over and over again...

...

In the beginning, let me note that I believe that any research or research direction, including mainstream NLP research is supposed to be a mirror, indication, derivation, reduction, output or whatever of operation of mind. This implies that there is *something right* in the paradigm, the paradigm maps some aspects of researchers' mind and is supposed to solve some problems. Science proves this "scientifically" - OK, no doubt that, what I call "mainstream NLP" does solve many problems...

However, one of the basic mistakes I see, is that the abstractions of these problems are at so high-level and so dispersed, that it is impossible to "ignite" the engine to run on its own.
Say, it is not an engine, but just a tool. A tool for another mind to "push buttons" and see the output, not an engine or an organism to be born and to develop.


*** What mainstream NLP is doing wrongly, in my opinion ***


-- Trying to do a reverse engineering of language, starting from words, text and very abstract and not based on "good physics" linguistic constructs.

[Critics] What the hell is "Good physics?"

"Good physics" is a basis that allows you to build an "engine", "ignite" it and make it running on its own. :) (Arnaudov, 2009)

I believe text is one of the reductions of the operation of mind, which is dynamic, based on multiple inputs simulation of multiple virtual universes at different levels of details/abstraction. Some of them are at very low level (say sets of images, sets of "video", sets of relations between sensory inputs). Words are pointers to very low-level models in mind.

Language, viewed as a bunch of static text, could not contain enough information to rebuild-back intelligence in the low level. Low level of mind is massively reduced and "cleaned", when converting to text, mind is reconstructing the missing part using its internal rich models.

-- The focus of the models is output - models are based too closely on text itself and on structures which are derived from the text itself and the output words.

Again - too obvious and cheap. Text is a reduction of mind in action. Language is not just a flat bunch of words with tags and a boring set of numbers for distribution and frequency.

Mind operates with images, relations and dynamics of virtual universes/systems that it simulates, and then reduces representation of this simulation into text, which actually is a system of pointers to the items and rules, which represent the real structures in mind. The structures in mind are dynamic, they are not 1 billion word corpora.

[Critics:] And how those "dynamic models" supposed to be modeled? You're stupid theorist! You don't define concepts you use! You're doing nothing!

I did define some of them, but years ago and OK, in pieces yet only in Bulgarian.
Anyway this reminds me a "practical" AI professor of mine I spoke with a year ago. He didn't make a difference between McCarthy and Minsky (who cares anyway?) and haven't even heard of Numenta or other advanced research directions, like simulation of neocortex columns...

I am a theorist, because I've been busy doing stuff for a living, besides theorizing. In fact I haven't been really theorizing since my teenage years. I'm tired of being only theorist, so be prepared!


-- Lack of freedom in researcher's imagination and lack of will to test more imaginative, complex, growing and dynamic models than obvious flat static relations between "scientifically linguistically proven" structures.

Researchers are walking on the same old "paved" road. Freedom? Imagination? Quotations rule this world - not freedom of imagination.

You want to do a radically new experiment? Get lost! If your research was not based on your supervizor's, on the best known researcher in the field so far etc. - then you're not a scientist and your research is not scientific.

Young researchers are trained that way. Fine - knowledge, history, respect, methodology, etc...
That's good. However, then they become experienced researchers, they quote their own papers, which are quoting the previous ones, which are accepted to be in the right direction.

Of course, revolutionary research is hard to be done in such conditions.

[Critics:] What revolutionary research are you talking about? You stupid dreamer! Come down and step on Earth! Learn the real NLP! Join the mainstream and you will be forgiven!

Who told you that I don't learn it?Thanks. I may take this option, but let me first try to do it differently.

-- Systems are not general, they are created to solve specific abstract problems, defined in terms of words or other very abstract concepts. That's like dealing with the symptoms, not with the cause of the "desease".

-- Models are not only specific, but static.

Machine learning, Naive bayesian etc. - they seem to be models in development. But what are they actually learning?

Probabilities between some set of "symbols" inside a set.

That's fine, but what is done with those probabilities between symbols later?
What those models want to do later with these probabilities?
Can they want to do, and do anything at all?

This is too flat. Words are pointless without doing something else with them - humans use words to make somebody do, imagine or feel something.

The purpose of "probabilities" in real natural language is to cause something different than words.

-- Lack of will and intentions in models. Lack of effectors. Lack of general feedback loops for self-improvement.

[Critics] Will and intentions? "Desire is irrelevant. They are machines!" And Computational Linguistics is not exactly Artificial Intelligence! Don't mix the fields!

Mind needs will and effectors. Otherwise it is not mind, but a mere number cruncher. And a pure number-cruncher architecture would hardly have capabilities of mind.

[Critics] Oh... Intelligent Agents. Bravo! You reinvented the wheel!

Thanks! You're so sweet!

Here we are - another weak part of mainstream research.

All that dividing of everything, instead of integration.

This division of everything is connected with the tendency of mainstream researchers to solve specific dispersed abstract problems, but not to search a solution for general problems which can solve many specific problems in an elegant way. I suggest you check out Boris Kazachenko's site.

What are the pieces and the mechanisms that can build intelligence up? The general mechanisms and evolved system will be capable to solve all anaphora-resolutions, word-sense-disambiguation, multiword expression recognition and whatever...

Let's search for an engine, not for tools.


-- Lack of continuous development and accumulation of experience. Lack of evolution.

Of course. Models are so much hand-crafted and specific, like tricks. Meet some of my unfortunate disappointments in Computational Creativity:

MEXICA: A Computer Model of Creativity in Writing - "Creativity" Disappointment again

Faults in Turing Test and Lovelace Test. Introduction of Educational Test. (Arnaudov, 2007; suggestion of educational test and analysis of works by Bringsjord, S., Ferrucci, D)


These systems (MEXICA and BRUTUS.1) may seem very good at first sight, but after you look under the hood, you will see how much they are based on word-by-word direction and how weak are they in creative generation of text.

These systems really are not "computationally creative", it is implied by the simplicity of the models.

A nice model is growing on its own by communicating with intelligent environment. You shouldn't be capable to understand it in details after it grow. If you are capable to understand the details and follow them - your model is too simple, it is too "young" or both.

If you use to code everything line by line and direct it... If you can predict everything by hand or in an obvious way... Sorry, but this is - at least - very boring!

[Critics] Theorist!

Thank you! :))


-- The following generation of researchers base their work on the work of the previous ones.

Again... Sure, this is science. It should be like that. Of course the state-of-the-art should be known. And one should use the knowledge, accumulated in the past.

However, I think the efforts spent on this is should be dosed.

Instead of imaging and testing new approaches, most of the time typical NLP researchers do study bibles with models which are proven to lead to very painful and slow progress.

Or the bibles consist of solutions which researchers are supposed to implement.

Or researchers are spending long-long time, building hand-crafted tools and databases, which cannot evolve on their own, later on.

The same path for so many years...


[Critics] Slow progress? Parsing, "Marsing", Syntax 45.4%, 67.4%, POS-Tagging: 96.4%, ...

So...? This progress doesn't lead to intelligent machines.
Those numbers do not map to a genuine general intelligence, but to production of tools.

Hand-crafted tricks with text... If you call this "Natural language processing" - OK, it's great.
This is useful to a certain degree and for particular class of problems.

Yes, mainstream NLP at the moment:

- Is useful.
- Solve some abstract specific problems by heuristics.
- It works to some degree for "intelligent" tasks, because of course language do maps mind.

However, the mainstream still does not lead to a chain of intelligent operations, there are not loops and cumulative development.


-- The lenght of the chain of inter-related intelligent operations in NLP today is very short. This is related to the lack of will and general goals of the systems. These systems are "push-the-button-and-fetch-the-result".

-- Swallowing of a huge corpus of 1 billion of words or so and a computation of statistical dependencies between tokens is not the way mind works.

!!! Mind learns step by step, modeling simpler constructs/situations/dynamics/models before reaching to more complex.
!!! Temporal relations of the input with different complexity is important.
!!! Mind usually uses many sensory inputs while learning. Very important.
!!! Mind has will, uses feedback and can actively and evolutionary test and improve correctness and effectiveness of its operation, including natural-language-related.


I suggest:

1. Hollistic approach - the goal is building an operational mind with long chain of intelligent operations, not completion of a table with values 94.55% 96.5% 90.4% and a long list with quotes in the end of a paper.

2. System must have will and effectors and must evolve. And saying "to evolve", I am not talking about "genetic algorithms", I'm talking about increasing of complexity by fetching "complexity" from the environment and pushing it into the system.

In other words:

Methodology for building very complex systems:
-- Don't do everything by hand, design something which is capable to design parts of it on its own.

3. If doing reverse engineering - let it be reverse engineering in the beginning of mind development and reverse engineering of the evolution of mind. Not reverse engineering of text.

4. Straight Engineering. Experimental engineering. Experimenting with designs of systems which evolve and fetch complexity.


[Final Critics] Who the hell are you, crazy ignorant stupid kid? "50 years of research of the brightest, talented, etc!" And you think you will change the world! Crazy!

If you are walking on the wrong way, you can't reach the right place, even if you were "the brightests" etc. persons. They couldn't find the right way.

I think one of the important issues with NLP research is that it had been lacking persons with the appropriate combination of talents, mindset and personality to go to a different path than those 50-years old one.

It is not easy to state: "I think this is a wrong approach, let's find another one!", especially if you are young.

Most researchers accept "this is correct, because - quote prof. A, prof. B... They are from University C, which has the most publication in journals D, E and F, which are ... (Oops, there are no Nobel prizes in NLP).

Anyway - therefore, this is the best, because it is quoted there and has 89.95% in this measure, which is accepted by.... Also, the paper suggests 94.34% in the test of "interrelated multipart tagging of coverage structures" etc., so this is real!." and so on.

Or they just want to have their PhD now, and the easiest and fastest way is to fetch a topic from the mainstream and do it the way it is done - these topics are... In Bulgarian it's called "Dissertabilni" - acceptable for a PhD. But mainstream is supposed to be behind the cutting edge.


So I'll say it again:

The reason why so many researchers are doing the same research and progressing so slowly is that they do assume that the others with higher status are right, and base their "original" research too much on it. They do not imagine wildly enough.

The same trivial, not original, not really inventive research, dealing with the old obvious parameters and items, supposed to be "the right ones"....


Conclusion: The paradigm of NLP is wrong.


THE END


To be continued...


Best Regards
Todor Arnaudov


Suggested reading (google): Boris Kazachenko, Jeff Hawkins, Todor Arnaudov (български - http://eim.hit.bg/razum)

Read More

The NLP Researchers cannot understand language. Computers could. Speech recognition plateau, or What's wrong with Natural Language Processing? Part 3

Thanks to my friend O.G.I, for sharing the link below, about the plateau in Speech Recognition software.

Rest in Peas: The Unrecognized Death of Speech Recognition

If you check this out (specialized NLP training is not needed to see)

What's wrong with Natural Language Processing? Part 2. Static, Specific, High-level, Not-evolving...

And this simple generalisation: Language is a hierarchical redirection/abstraction/generalization/compression of sequences of [multi-modal] sensory inputs and motor outputs, and records and predictions for both.  (me) 

You'll get what causes the plateau - why NLP, parsing, speech recognition or whatever would stay at their dead-end forever if it doesn't change radically.

I especially enjoy this one:

"...To some, these developments are no surprise. In 1986, Terry Winograd and Fernando Flores audaciously concluded that “computers cannot understand language.” In their book, Understanding Computers and Cognition, the authors argued from biology and philosophy rather than producing a proof like Einstein’s demonstration that nothing can travel faster than light...."

So silly.  The same goes for any similar sentence of retired AI-niks, because they're actually saying this:

Computers cannot understand language [or think], because computers do exactly what they, those retired AI researchers, program them to do. Besides, machines lack free-will, also you know - Goedel incompleteness, quantum-mechanical blah-blah-blah etc. 

However, this implies that computers just execute their programmers' instructions,  therefore it is not the computers who cannot understand language, it is their incapable programmers.

It is the Programmers and old-fashioned AI-niks doing NLP who cannot understand language, not the computers.

NLP programs are playing with words in dictionaries, while mind is playing with multi-modal pre-processed raw sensory inputs. An AGI is needed to make speech recognition correct, this is what should be worked on.

"...So not everyone agreed. Bill Gates described it as “a complete horseshit book” shortly after it appeared, but acknowledged that “it has to be read,” a wise amendment given the balance of evidence from the last quarter century."

Hmmm, Bill is cool!  :) 

http://www.sadanduseless.com/2010/05/steve-jobs-vs-bill-gates/

Read More

The hardware and resources inequality in AI/AGI: an old story now rediscovered by the worried mainstream — a 2013 & 2009 articles vs 2023 paper

I start to publish in Medium as well - I had to to long ago, as it has a community and "social life", but:  better later than never. I may republish some of the articles here there in order to hopefully extend the appropriate audience reach.

Editing The hardware and resources inequality in AI/AGI: an old story now rediscovered by the worried… – Medium

The hardware and resources inequality in AI/AGI: an old story now rediscovered by the worried mainstream — a 2013 & 2009 articles vs 2023 paper

“Montreal.AI: 23 ч. · Choose Your Weapon: Survival Strategies for Depressed AI Academics Julian Togelius, Georgios N. Yannakakis : https://arxiv.org/abs/2304.06035
#ArtificialIntelligence #DeepLearning #MachineLearning”

While it is true that even 8 years or 10 years ago even regular programmers could have the GPU power ( the well-paid and owning their time on the right target; but usually the ones who make money lack the vision and they buy GPUs/hardware for games, and the ones who had vision and intelligence had no money), the “inequality of opportunities” is of course not a new phenomenon, including in AI. I’ve written about it in 2013 and it was valid for the pioneer AGI researchers one of which was I, publshing substantial works since 2001, aged 17, and being author of the world’s first University courses in AGI in 2010, 2011 with theories and a course program that still stand and are only confirmed and elaborated by more and more researchers and publications. The inequality phenomenon was valid for the AGI researchers versus both the well-"fed" well-funded high-profile and famous academics who “rolled their eyes” when they heard about AGI (ask Hassabis, Legg; and Altman even about 2010-early 2010s in MIT, Altman refers to 2015 when they found OpenAI). It was vlaid versus any researchers from the Academia (with students working for them, “free” laboratories etc.), and of course: the industry.

https://artificial-mind.blogspot.com/2013/08/issues-on-agiri-agi-email-list-and-agi.html

A part of the conclusion of this work:

“… — WORKABLE THEORIES and IMPLEMENTATIONS

Some people try to work on workable theories and implementations, but this list is a home of the poorest and the most lonely ones in the AGI community, even though some of them were some of the pioneers of the new wave of that community, long before the “institutionalized” researchers took it as “prestigious”.

The list’s researchers poorness impedes their opportunities/motivation for concentrated work/producing academic-style materials — many believe the mainstream academic system (including many aspects of the peer-reviewed journals etc.) has intrinsic corruptions and have left it for “political” reasons.

Moreover, even if they do know how or have potential to develop working machines, this is a big effort that may take a lot of time before they could have a complete system — coded and running. If they haven’t produced visible results already, that doesn’t imply they wouldn’t do after years of collection of critical mass, as long as they could work.
Besides they are supposed to be 10, 100 or 1000 times more capable than the normally funded and organized ones from the academic/industrial competition. Current ones can’t afford visiting appropriate conferences or travel around research centers and are alienated.
They should have much broader knowledge and skills, acquire new knowledge and skills in a shorter time and work much faster, because:
 — they can’t afford truly focussed work — too much other troubles, too much sub-problems they should solve alone, a lot of wasted time in attempts to find partners or develop some “booster-funding” technologies, plenty of frustration due to the isolation and helplessness against all the problems [including the dumb financial etc. ones] they have to solve [implement] alone (or give up)
 — they do not have students, partners or “slaves” to give the dirty job to [or barely have, but it’s hard to motivate anyone without funding]

Overall, they should shoot 100 or 1000 targets with one bullet, or they “die out” [in the race]
Welcome to the list of the losers… :))
However some of these “losers”, due to the extreme requirements they face, may really be 50 or 100 times more productive or knowledgeable and non-conventional than the “ordinary” funded and supported competition, and may have guts and balls that the others lack.

Otherwise they should have given up, be part of the existing institutes — “institutionalized” — or from the “AI”. But they are not from those institutes, because when they proclaimed that “AI was wrong” they were outsiders already, heading towards new directions.

Furthermore, those brave ones are supposed to believe and find a way to make thinking machine possible on cheap, old and slow hardware, otherwise they should have another reason to give up to the supercomputer owners and the rich institutionalized researchers…”

The same about NLP:

“What’s wrong with NLP? Part 2”, 3/2009

https://artificial-mind.blogspot.com/2009/03/whats-wrong-with-natural-language.html

One other option for the academics, who are pretty wealthy but complain about that OpenAI, DeepMind etc. are wealthier:

Invent somethign that’s really innovative, different and more efficient. Everybody prefers to just spill in more hardware, make a little change and engrave her name for “new contributions” (what about the credit for the hardware designers and producers?), it was similar in 2000s with NLP: change one bit of some algorithm, produce an increase of 0.1% of some measure/bechnmark and there you are: “a new NLP model”, “moving the SOTA”. Why not building a new paradigm from the ground up. But yes, you can’t, because the dafault is that if you try, you won’t be accepted until you beat the competition and as explained above, in order to do and be accepted, you have to be 1000 times more efficient than them while working on your own with no resources. :)

Read More

Comments on "Chaotic Logic" | Коментар към книгата на Бен Гьорцел "Хаотична логика" - към курса по Универсален изкуствен разум в ПУ

I've been checking out some of Ben Goertzel's works:

Read it at: http://www.goertzel.org/books/logic/contents.html

Chaotic Logic
Language, Thought and Reality
From the Perspective of Complex Systems Science

Ben Goertzel

Chairman and CTO Intelligenesis Corp.
Paper Version published by Plenum Press, 1994

What I liked:

- "Pattern and Prediction" (Ch. 2) 
- "The structure of thought" (Ch. 3) - multilevel control (hierarchical control)
- "Psychology and logic"  (Ch.4)
- Coverage of history of AI/logic/linguistics, good knowledge on particular philosophers, linguists etc.
- Different definitions of complexity

E.g. this lines on patterns: (Ch.2):

...Before getting formal, let us first take a quick intuitive tour through the main concepts to be discussed. The natural place to begin is with the concept of pattern. I define a pattern, very simply, as a representation as something simpler
(...)
In general, a pattern is a short-cut -- a way of getting some entity that is in some sense simpler than the entity itself. A little more formally, suppose the process y leads to the entity x. Then y is a pattern in x if the complexity of x exceeds the complexity of y...., B. Goertzel, 1994
Also, "the amount of structure":

...These concepts may be used to measure the total amount of structure in an entity -- a quantity which I call the structural complexity. The definition of this quantity is somewhat technical, but it is not hard to describe the basic idea.

If all the patterns in an entity were totally unrelated to one another (as, perhaps, with this picture of the square next to the circle discussed above), then one could define the structural complexity of an entity as the sum of the complexities of all its patterns.

But the problem is, often all the patterns will not be totally unrelated to each other -- there can be "overlap." Basically, in order to compute the structural complexity of an entity, one begins by lining up all the patterns in the entity: pattern one, pattern two, pattern three, and so on.

Then one starts with the complexity of one of the patterns in the entity, adds on the complexity of whatever part of the second pattern was not already part of the first pattern, then adds on the complexity of whatever part of the third pattern was not already part of the first or second patterns, and so on.... B. Goertzel, 1994...

I think it's in the right direction of compression and prediction.

A discussion on Nietzsche and logic at Ch. 4  :

Nietzsche declared consciousness irrelevant and free will illusory. He proposed that hiddenstructures and processes control virtually everything we feel and do. Although this is a commonplace observation now, at the time it was a radical hypothesis. 
Nietszche made the first sustained effort to determine the nature of what we now call "the unconscious mind." The unconscious, he suggested, is made up of nothing more or less than "morphology and the will to power." The study of human feelings and behavior is, in Nietszche's view, the study of the various forms of the will to power.
(...)
... Note how different this is from Mill's shallow psychologism. In the Introduction I quoted Mill's "derivation" of the Law of Excluded Middle (which is equivalent to the law of contradiction, by an application of deMorgan's identities).  Mill sought to justify this and other rules of logic by appeal to psychological principles. 

In Mill's view, the truth of "A or not-A" follows from the fact that each idea has a "negative idea," and whenever an idea is not present, its negative is. This is a very weak argument. One could make a stronger psychological argument for the falsity of "A and not-A" -- namely, one could argue that the mind cannot simultaneously entertain two contradictory ideas.

But Nietzsche's point is that even this more plausible argument is false. As we all know from personal experience, the human mind can entertain two contradictory ideas at once. We may try to avoid this state of mind, but it has a habit of coming up over and over again: "I love her/ I don't love her", "I want to study for this test/ I want to listen to the radio instead".

The rule of non-contradiction is not, as Mill would have it, correct because it reflects the laws of mental process -- it is, rather, something cleverly conceived by human minds, in order to provide for more effective functioning in certain circumstances.

 ...One rather simplistic and stilted way of phrasing Nietszche's view of the world is as follows: intelligence is impossible without a priori assumptions and rough approximation algorithms, so each intelligent system (each culture, each species) settles on those assumptions and approximations that appear serve its goals best, and accepts them as "true" for the sake of getting on with life. Logic is simply one of these approximations, based on the false assumption of equality of different entities, and many auxiliary assumptions as well....  B. Goertzel, 1994


Disliked:

What I generally didn't like is too high a level speaking, too much typical syntax-semantics-symbols Linguistics, Psychology, NLP and philosophy; couldn't see "good physics" thus couldn't really sustain reading all in details.

I suggest : What's wrong with NLP... series: http://artificial-mind.blogspot.com/search?q=What%27s+wrong+with+NLP

A curious fact is that a Bulgarian humanoid robot project Kibertron mentions "Chaotic logic computer" in their brief explanation of their "theory of natural intellect", don't know is it related to the book. Unfortunately the company claims they need 5 million Euro to make this "natural intellect" working, together with the robot.

A general advice on reading 

Use your time wise, don't read thoroughly too much. Try to understand when reading more in a topic would be worthless. A very little essence would last in your mind and would make real sense, the rest and most might be just time passed without memories and without work done.

The talk on "saturation of learning" will be continued.

Read More

NLP stands for Natural Language Processing and not Neuro-Linguistic Programming | НЛП означава Обработка на естествен език, а не ...

The more popular NLP is a half-scientific or pseudoscientific psychological methodology for the so called "social engineering".

In my opinion this is an immoral methodology of  how to manipulate and deceive people, especially about making sex or selling something. There are courses about how to pick-up girls and make them believe you love them while you just want sex; how to make people buy your shit and persuade them how brilliant it is etc.

This kind of pseudo NLP sucks, and people who try to use this "methodology" are hard to be trusted. I've met several ones, such as a seller of stolen goods and an abusive network marketing distributor.

Not that the scientific NLP is fine yet - What's wrong with Natural Language Processing? Part 2. Static, Specific, High-level, Not-evolving...

But at least it can be taken as a science.  :)

...

НЛП означава Обработка на естествен език. Има и друго - "невролингвистично програмиране", което според мен представлява неморална методология за манипулация и заблуда - може да си платите за курс на който да ви научат как да заблуждавате хората, да ги накарате да си мислят, че сте им приятел (а да имате единствено користни цели) и т.н.. Веднъж усетиш ли такъв човек, трудно можеш да му имаш доверие след това...

Не че откриват кой знае какво, според мен концентрират добре известни и "работещи" методи на свалячи, ласкатели и други. Ако сте достатъчно умен човек и имате усет за хората, тези "мъдрости" на които ще ви научат, ще си ги знаете от собствен опит. 

Не че Обработката на естествен език е цвете - What's wrong with Natural Language Processing? Part 2. Static, Specific, High-level, Not-evolving... (Къде греши Обработката на естествен език), но това е друга история.

Read More

HyperNEAT in Neural Networks and Ontologies in NLP - Why They Seem Promising?

1. HyperNEAT

Exerpts from the site (bold - mine):

"In short, HyperNEAT is based on a theory of representation that hypothesizes that a good representation for an artificial neural network should be able to describe its pattern of connectivity compactly.

This kind of description is called an encoding. The encoding in HyperNEAT, called compositional pattern producing networks, is designed to represent patterns with regularities such as symmetry, repetition, and repetition with variation.

(...)

The other unique and important facet of HyperNEAT is that it actually sees the geometry of the problem domain. (...) To put it more technically, HyperNEAT computes the connectivity of its neural networks as a function of their geometry.

(...)

NEAT stands for NeuroEvolution of Augmenting Topologies. It is a method for evolving artificial neural networks with an evolutionary algorithm. NEAT implements the idea that it is most effective to start evolution with small, simple networks and allow them to become increasingly complex over generations. That way, just as organisms in nature increased in complexity since the first cell, so do neural networks in NEAT. This process of continual elaboration allows finding highly sophisticated and complex neural networks."

...


That is:

- Compression/Minimal message length

- Repetition as a clue for patterns (symmetry is repetition as well)
- Incrementing (small scale to big scale)

- Coordinates (topology in connectivity)

2. Ontologies in NLP/Computational Linguistics

Basically this is a semantic network, i.e. relations between concepts. WordNet is a sort of ontology. The issue is that they are often designed by hand. There are statistical methods, as well, but they're missing something I've mentioned many times in the series What's Wrong With NLP.

Why this happens to be useful?

- Because it resembles real cognitive hierarchy - it's a "skeleton hierarchy"

Accordingly, they're prone to be too rigid and unable to self-extend.

Read More

Todor's Comments on the Article "AI Is Harder Than We Think: 4 Key Fallacies in AI Research" - no, AGI is actually simpler than it seems and you think

Comment of mine on the article "AI Is Harder Than We Think: 4 Key Fallacies in AI Research" https://singularityhub.com/2021/05/06/to-advance-ai-we-need-to-better-understand-human-intelligence-and-address-these-4-fallacies/

Posted on Real AGI FB group

The suggested fallacies are:

1. Progress in narrow intelligence is progress towards general intelligence
2. What’s easy for humans should be easy for machines
3. Human language can describe machine intelligence
4. Intelligence is all in our heads

(See also the article)

The title reminded me of a conclusion of the "AGI Digest" letters series where after giving the arguments I noted that: "AGI is way simpler than it seems". See the message from 27.4.2012 in "General algorithms or General Programs", find the link to the paper here:

https://artificial-mind.blogspot.com/2017/12/capsnet-capsules-and-CogAlg-3D-reconstruction.html   https://artificial-mind.blogspot.com/2021/01/capsnet-we-can-do-it-with-3d-point-clouds.html.html

 

Summary

⁠In brief: I claim it is the opposite: AI is easier than it seems (if one doesn't unerstand it and confuses herself, it's hard, right). Embodiment is well known and it lays in the reference frames and exploration-causation, stability of the coordinates and shapes and actions, repetitiveness etc. not in the specific "material" substrate of the body. The "easy for humans..." is well known and banal, also the point against machines is funny: in fact humans also can't "apply their knowledge in new settings without training" (see the challenges in the article) etc. IMO progress in "narrow" AI actually is a progress towards AGI and it was so even in the 2000s, as current "narrow AI" ML methods are pretty general and multi-modal and they give instruments to do processes which were attached to "AGI" at least since the early 2000s, such as general prediction and creation, synthesis. Current "narrow AI" does Analysis and Synthesis, but not "generally enough in a big enough and "integrated enough" and "engine-like-running" framework which connects all the branches, modalities and knowledge together, however the branches and "strings" are getting closer. Practically, one can use as many "narrow" NN with whatever glue code and other logic in a system.

Discussion

1. "Progress in narrow intelligence is progress towards general intelligence" [are not progress towards GI] 

— IMO it actually is a progress, because the methods of the "narrow" become more and more general, both in what they solve and in the ambitions of the authors of these solutions. After a problem or a domain is considered "solved" to one degree or another, the intelligent beings direct themselves to another one, or expand the range, or try to generalise their solutions of several problems so far and combine them etc.

One of the introductory lectures in the first university course in AGI back in April 2010, which I authored, was called "Survey of the Classical and Narrow AI: Why it is Limited and Why it Failed [to achieve AGI]?": http://research.twenkid.com/agi/2010/Narrow_AI_Review_Why_Failed_MTR.pdf

 

 

While wrapping up the faults as I saw them, one of the final slides and others in the lecture, matched with one of the main message of the course - hierarchical prediction and generalisation, - suggested that the methods of the advanced "narrow AI" actually converge to the ideas and methods of AGI. Even  image and video compression for example share the core ideas of AGI as a general sensory-motor prediction engine, so MPEG, MPEG2, H264 - these algorithms in fact are "AI". "Motion compensation", the most basic comparison, is related to some of the primary processings in the AGI algorithm CogAlg, all "edge-detections" etc. are something where any algorithm searching for shapes would start or reach in one way or another. Compression - finding matches ("patterns), which is also "optimisation" - reducing space etc.

Two of the concluding slides (translation follows): 

"The winner of DARPA Urban Challenge in 2007 uses a hierarchical control system with multi-layer planing of the motions, a behavior generator, sensory perceptions, modeling of the world and mechatronics".

Points of a short summary, circa early 2010:

What's wrong with NLP? (from articles from 2009) [and "week" AI]: 

* The systems are static, require a lot of manual work and intervention and do not scale 

* Specialized "tricks" instead of universal (geneal purpose) systems 

* Work at a very high symbolic level and lack grounding on primary perceptions and interactions with the environment 

* The neural networks lack a holistic architecture, do not self-organize and are chaotic and heavy. Overall: A good "physics" is lacking, one that would allow creation of an "engine", which to be turned on and then to start working on its own. The systems are instruments and not engines.

Note, 7.2021: The point regarding the NN however can be adjusted:

Many NN can be stacked or connected with anything else in any kind of network or a more complex system - we are not limited to use one or not use any glue code or whatever. The NN and transformers are actually "general" in what they do and are respectively applied for all sensory modalities and also multi-modally. 

Complete or powerful enough for a complex simulated/real world sensory-motor multi-modal frameworks are not good enough and these algorithms may be not the fastest to find the correlations and have unnecessary brute force search which can be reduced by more clever algorithms (and they should), however these models do find general correlations in input.  

 2. "What’s easy for humans should be easy for machines"

—  Isn't that banal, also it is vague (easy/hard). Actually some of the skills of the 3 or 4 years old are achieved by 3 or 4 years long training in supervised settings: humans do not learn "unsupervised" except basic vision/low level physical and sensual stuff (language learning is supervised as well; reading and writing: even more).

Test people who didn't attend school at all, check how good they are in logic for example, in abstract thinking, in finding the essential features of the objects or concepts etc. Even people who have university degrees could be bad in that, especially BAs.

There are no machine learning models with current technology from the "narrow AI" which are trained for that long yet, an year or years with current compute. We don't know what they could achive, even with todays' resources.

On learning and generalising: "If, for example, they touch a pot on the stove and burn a finger, they’ll understand that the burn was caused by the pot being hot, not by it being round or silver. To humans this is basic common sense, but algorithms have a hard time making causal inferences, especially without a large dataset or in a different context than the one they were trained in."  

That's right about training if you use a very dumb RL algorithm (like the ones which played for 99999 hours in order to learn to play the basic games on Atari 2600), however overall the "hardness" of learning this by a machine is deeply wrong and not aware of what the actual solution could simply be:

"An algorithm" would have sensors for temperature which will detect "pain", caused be excessive heat/temperature, which happened at the moment when the coordinates of the sensor (the finger) matched coordinates within the plate of the stove. Also, it could have infrared sensors or detect the increment of the temperature before touching and detecting that there is a gradient of the measurement. The images of the stove when the finger was away didn't cause pain, only the touch. This is not hard "for an algorithm", it's trivial.

4. Intelligence is all in our heads

— Wasn't that clear at least since 20 years? (for me it was always clear) However, taking into account, that the embodiment can be "simulated", "virtual". The key in embodiment are the sensory matrices, coordinates ("frames of reference" in Hawkins' terms) and the capability to systematically explore: cause and perceive/study the world; the specific expressions of the sensory matrices and coordinates could vary.

3. Human language can describe machine intelligence
"Even “learning” is a misnomer, Mitchell says, because if a machine truly “learned” a new skill, it would be able to apply that skill in different settings" 

+ 1. "a non-language-related skill with no training would signal general intelligence"

— I challenge these "intellectuals": can you make a proper one-hand backhand with a tennis racket with "no training"? (Also how long will you train, especially before delivering a proper over-the-head service with good speed or a backhand, while you are facing back to the net; or a tweener (between the legs shot, especially while running back to the base line etc.)

You're not supposed to need explicit training, right? You did move your hands, arms, wrists, elbows;  legs, feet... You've watched tennis at least once on TV sports new, therefore you should be able to just go and play against Federer and be on par with him, right?. If you can't do that even against a 10-year old player, that means "you can't apply your knowledge in new settings"...

Can you even juggle 3 balls: by "applying your knowledge of physics from school and sense of rhythm from listening to music and dance - even the simplest trick.

Can you play the simplest songs on a piano : by applying your understanding of space and motion of the hand and find the correlations with the pressing of the keys and the sound of each of them etc. - can you do it especially if you lack musical talent.

Well, "therefore you lack GI", given your own definitons... I'm sorry about that... 

(In fact the above is true for many humans; humans really lack "general intelligence" by some of the high-bar definitions which a machine is expected to meet before being "recognized") 

...

Слайдът на български (4.2010):

* Какво не е наред в обработката на естествен език [и слабия ИИ]?

●Системите са статични, изискват много ръчна намеса, не се развиват и мащабират.
●Специализирани „трикове“, а не универсални системи.
●Работят на високо символно ниво и нямат основа от първични възприятия и взаимодействия със средата.
●Невронните мрежи нямат цялостна архитектура, не се самоорганизират и са хаотични и тежки.Липсва добра „физика“, която да позволи създаването на „двигател“, който дасе включи и да заработи от самосебе си. Инструменти, а не двигатели.

Read More

Is Natural Language Recursion or Dereferencing? - Comment about the article "Recursive language and modern language were acquired simultaneously ..."

https://phys.org/news/2019-08-recursive-language-modern-simultaneously-years.html?fbclid=IwAR04SqHIJKrbZR52gvllBiM2qJp3-xhZ8aJ0vQXW1-hwEcplsjTw8ZccglE

And the discussion of B.K. from CogAlg: https://www.facebook.com/boris.kazachenko.5/posts/10216534326967246

"a snake on the boulder to the left of the tall tree that is behind the hill,"  

"Prefrontal Synthesis (PFS)."  

" Similarly, nested explanations, such as "a snake on the boulder to the left of the tall tree that is behind the hill,"  force listeners to use PFS to combine objects (a snake, the boulder, the tree, and the hill) into a novel scene. Flexible object combination and nesting (otherwise known as recursion) are characteristic features of all human languages. For this reason, linguists refer to modern languages as recursive languages."

The bold part (bold - mine) is a general characteristic of codes.

I didn't see the capacity of the "working memory" mentioned, while it is crucial and general - both as the definition of  "7+2" and in more general sense of complexity of the representations one could productively work with, which correlates with the complexity of the matters ones could deal with.

Nested and recursive are not synonyms by default IMO. You could call them so, but IMO they *could* be, but are not obliged to be. Recursion is about self-reference, while the example IMO is not "self" reference, unless assuming such "function call" mechanism. It's *chaining* and redirection of relations (a list, graph, network), some of the parts could be not processed the same way, thus not "recursion".

( Sure, one can call "recursion" whatever she wants; e.g. regarding CogAlg there was a time, I don't know if it is still valid, where "recursion" was called "iteration" interchangeably - they are opposite methods in programming. The general of these is "a stage of processing". )

As of redirection, a famous quote in Computer Science says, that "we can solve everything with one more level of redirection". The original uses another word:

Indirection | Dereferencing | Redirection

"We can solve any problem by introducing an extra level of indirection."  

Indirection, dereferencing: https://en.wikipedia.org/wiki/Indirection
https://en.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering

Processor's Indirection

It starts with the CPUs with more advanced methods for addressing, even for simple CPUs like read the value at the address, pointed by the current content of a register, offset by the content of selected index register (read the content of the index register, add the value to the already read value, multiplied by the selected word-size - byte, word, doubleword, quadword, - where those selections are encoded in the opcodes of the instructions - and store the read value in the same source register.

Say:  A - Accumulator, X - IndeX register

MOV @[A+X]

It could be to point to an address, which is used as another address of a table, add to it offset from another register, add something else etc.

This is constantly done also when parsing and expanding high level code and data structures, first load one table with addresses with a key (identifier), search, find where it redirects, get another table etc.

It is similar to the linguistic dereferences like:

"This is the book of the girl that lives on the third floor in your house".

All that is easier to be expressed in code and by graphs/flowcharts with arrows, rather than by text, because it's naturally "spatial" and connected.

...

Regarding the example from the article:


It can be traversed and processed recursively, but it could be done also iteratively, up to particular "depth", length, and since in human case that length is quite limited anyway, and it doesn't grow infinitely, unlike the claims in the artificial syntactic examples, most people very quickly get lost in the relations betweeen words - probably like in this sentence.

That has a simple general interpretation, though and exposure to nested input can't help it: too limited working memory (resources) do not allow to spawn and to hold enough objects/patterns and relations in order to understand their relations or to combine them into new; while just activation/recall is cheaper, so a system may be able to just recall memories and reconstruct already visited patterns, which may require less activations in the higher levels (PFC), but couldn't have an even higher level which use many of these into new imagined entities, combined by other parts. The relations are patterns and they seem to be expensive ones for humans.

Even if you had that skill of "PFS" within the cognitive repertoir and you can generate sequences in principle, if you had that "modern imagination", if the available resources allow just to manage one level of certain complexity, the result will be the same.

This is exemplified by the complexity of the used sentences and language, which is supposed to grow during language acquisition, but it has a limit. That goes also for the complexity of programming code expressions for developers' skills.

The math "word problems" given to students are also tests for the working memory capacity:

John has 5 apples, Mary has 3 apples more, but she gave 2 to Kate who had 1. How many apples Mary has now?

These problems require correspondingly big enough working memory for such patterns in order to hold the elements and not just in principle to be able to do "recursion" (or nesting or chaining).

...

Mapping to vocalisation

I agree about with Boris that a mapping to vocalisation exists ( to recording spoken words in working memory as well), I've measured sometimes the lenght of text which I can remember and write correctly while listening to a talk, a show etc, see also my old compressed definition of natural language, given for example in the "What's Wrong With the NLP" series:

Todor: Natural language is a hierarchical redirection/abstraction/generalization/compression of sequences of multi-modal sensory inputs and motor outputs, and records and predictions for both.

When we learn the language in a multi-modal net which includes motor commands to the vocal tract and sensory from the records of other people talking, our own sounds and utterances, expectations of what sound would be produced when we do this or that sequence of motor commands to the vocal system, starting with current state etc., then it would recall memories of "vocalization" etc.

https://en.wikipedia.org/wiki/Vocal_tract


Declarative Memory, Hippocampus, Consciousness, Creativity, Sequences

I also agree that there is a correlation/relation between declarative memory capacity/clarity/skills and general intelligence and creativity (and have at least myself as an example for that phenomenon), so if Hippocampus is required and important for the former, then it would be for the general hierarchical sequence generations and analysis/re-syntehsies as well, which is in short what producing and cognition of creative works is, either "hard" as code or scientific theories, or more artistic as creative writing, music, films. Also "declarative memory" is required for navigation as "memory" - you must be able to record and compare what have you visited and what's "history", in what sequence, what was prior and what follows etc.

It is known by research that the capacity of the working memory strongly correlates with the G-factor - general intelligene, - which suggests both/either the existence of general processing within "mind" and that this memory is somewhat distributed, and/or something that Schopenhauer has suggested in 19-th century if I remember correctly - that the difference between genius and average/mediocre is actually quantative, not qualitative, the latter just runs "out of memory" too quickly and can't reach to the required complexity and length to understand, discover or produce something which is "more meaningful" or original than the expected*.


* If I'm not mistaken that was in "Parerga and Paralipomena"

Prepositions

That reminds me of insights I had about "prepositions" in language and how they related to the Cognitive Algorithm theory of Boris as I knew and understood it back in the end of 2015.

I may revisit the discussion:

To be continued...


* Regarding the "prepositions", mentioned in the article, as material syntactical elements, they could be virtual, implied in the word forms, but also, technically, as someone mentions in the comments section, too, many ancient languages usually are with cases (падежи), the prepositions are scarce or auxilliary or still require cases - such as the European languages: Latin, Greek, German (thus Proto-English), Slavic - except for Bulgarian which gradually lost the cases (except a few informal and in some expressions, and we generally understand some of the cases in kin languages because they use morphemes/suffixes which we use and understand, such as - "у", "ов", "му" - etc. and there are archaisms and Old Bulgarian ("Old Church-Slavonic") which are known such as "Православному българскому народу", "Моли се Богу", и съвременното: "У нас" (At my home, at our home) и пр.

Read More