Artificial General Intelligence

Kommentare · 30 Ansichten

Artificial basic intelligence (AGI) is a kind of synthetic intelligence (AI) that matches or surpasses human cognitive capabilities across a wide variety of cognitive jobs.

Artificial general intelligence (AGI) is a kind of synthetic intelligence (AI) that matches or exceeds human cognitive capabilities across a large range of cognitive jobs. This contrasts with narrow AI, which is restricted to particular jobs. [1] Artificial superintelligence (ASI), on the other hand, describes AGI that significantly surpasses human cognitive capabilities. AGI is considered one of the meanings of strong AI.


Creating AGI is a primary objective of AI research study and of business such as OpenAI [2] and Meta. [3] A 2020 study determined 72 active AGI research and development jobs across 37 nations. [4]

The timeline for achieving AGI remains a topic of continuous dispute among researchers and specialists. Since 2023, some argue that it may be possible in years or decades; others keep it may take a century or longer; a minority think it may never be accomplished; and another minority claims that it is currently here. [5] [6] Notable AI scientist Geoffrey Hinton has expressed concerns about the rapid progress towards AGI, suggesting it might be attained sooner than many expect. [7]

There is argument on the exact definition of AGI and concerning whether modern-day large language designs (LLMs) such as GPT-4 are early types of AGI. [8] AGI is a common subject in sci-fi and futures research studies. [9] [10]

Contention exists over whether AGI represents an existential threat. [11] [12] [13] Many experts on AI have mentioned that alleviating the danger of human termination posed by AGI must be an international top priority. [14] [15] Others find the advancement of AGI to be too remote to provide such a threat. [16] [17]

Terminology


AGI is also called strong AI, [18] [19] complete AI, [20] human-level AI, [5] human-level intelligent AI, or basic intelligent action. [21]

Some academic sources reserve the term "strong AI" for computer system programs that experience life or awareness. [a] On the other hand, weak AI (or narrow AI) has the ability to solve one specific issue but does not have basic cognitive capabilities. [22] [19] Some academic sources utilize "weak AI" to refer more broadly to any programs that neither experience awareness nor have a mind in the exact same sense as people. [a]

Related principles consist of synthetic superintelligence and transformative AI. An artificial superintelligence (ASI) is a theoretical type of AGI that is a lot more generally smart than people, [23] while the idea of transformative AI connects to AI having a big effect on society, for example, comparable to the farming or industrial transformation. [24]

A structure for classifying AGI in levels was proposed in 2023 by Google DeepMind researchers. They define five levels of AGI: emerging, skilled, expert, virtuoso, and superhuman. For instance, a qualified AGI is specified as an AI that outshines 50% of competent adults in a wide variety of non-physical jobs, and a superhuman AGI (i.e. an artificial superintelligence) is likewise defined but with a threshold of 100%. They think about big language models like ChatGPT or LLaMA 2 to be instances of emerging AGI. [25]

Characteristics


Various popular meanings of intelligence have been proposed. Among the leading propositions is the Turing test. However, there are other widely known meanings, and some researchers disagree with the more popular techniques. [b]

Intelligence traits


Researchers generally hold that intelligence is required to do all of the following: [27]

factor, use technique, fix puzzles, and make judgments under uncertainty
represent understanding, consisting of good sense understanding
strategy
find out
- communicate in natural language
- if essential, incorporate these abilities in completion of any provided objective


Many interdisciplinary techniques (e.g. cognitive science, computational intelligence, and decision making) consider additional qualities such as creativity (the ability to form unique mental images and ideas) [28] and autonomy. [29]

Computer-based systems that display much of these abilities exist (e.g. see computational creativity, automated reasoning, choice support group, robotic, evolutionary calculation, intelligent agent). There is debate about whether contemporary AI systems possess them to a sufficient degree.


Physical qualities


Other abilities are considered preferable in intelligent systems, as they might affect intelligence or aid in its expression. These consist of: [30]

- the capability to sense (e.g. see, hear, etc), and
- the capability to act (e.g. move and control things, change place to check out, and so on).


This consists of the ability to discover and react to threat. [31]

Although the capability to sense (e.g. see, hear, etc) and the capability to act (e.g. move and control objects, modification place to explore, and so on) can be preferable for some smart systems, [30] these physical abilities are not strictly required for an entity to certify as AGI-particularly under the thesis that big language designs (LLMs) may currently be or end up being AGI. Even from a less positive perspective on LLMs, there is no company requirement for an AGI to have a human-like type; being a silicon-based computational system suffices, offered it can process input (language) from the external world in location of human senses. This interpretation lines up with the understanding that AGI has actually never ever been proscribed a specific physical embodiment and thus does not require a capacity for mobility or conventional "eyes and ears". [32]

Tests for human-level AGI


Several tests indicated to confirm human-level AGI have been considered, including: [33] [34]

The idea of the test is that the machine needs to attempt and pretend to be a male, by addressing questions put to it, and it will just pass if the pretence is reasonably persuading. A substantial portion of a jury, who should not be skilled about machines, must be taken in by the pretence. [37]

AI-complete issues


A problem is informally called "AI-complete" or "AI-hard" if it is believed that in order to fix it, one would require to execute AGI, because the option is beyond the capabilities of a purpose-specific algorithm. [47]

There are lots of problems that have actually been conjectured to need basic intelligence to fix in addition to humans. Examples consist of computer system vision, natural language understanding, and dealing with unexpected scenarios while solving any real-world issue. [48] Even a particular task like translation needs a machine to read and compose in both languages, follow the author's argument (reason), understand the context (understanding), and faithfully replicate the author's original intent (social intelligence). All of these problems require to be resolved at the same time in order to reach human-level maker performance.


However, a number of these tasks can now be carried out by modern big language designs. According to Stanford University's 2024 AI index, AI has reached human-level efficiency on lots of criteria for checking out understanding and visual thinking. [49]

History


Classical AI


Modern AI research study began in the mid-1950s. [50] The very first generation of AI scientists were convinced that artificial general intelligence was possible which it would exist in just a few years. [51] AI leader Herbert A. Simon wrote in 1965: "machines will be capable, within twenty years, of doing any work a guy can do." [52]

Their forecasts were the inspiration for Stanley Kubrick and Arthur C. Clarke's character HAL 9000, who embodied what AI researchers believed they might develop by the year 2001. AI leader Marvin Minsky was an expert [53] on the project of making HAL 9000 as reasonable as possible according to the agreement forecasts of the time. He said in 1967, "Within a generation ... the issue of producing 'synthetic intelligence' will significantly be solved". [54]

Several classical AI projects, such as Doug Lenat's Cyc project (that began in 1984), and Allen Newell's Soar job, were directed at AGI.


However, in the early 1970s, it ended up being obvious that researchers had actually grossly underestimated the problem of the task. Funding firms ended up being hesitant of AGI and put scientists under increasing pressure to produce useful "applied AI". [c] In the early 1980s, Japan's Fifth Generation Computer Project revived interest in AGI, setting out a ten-year timeline that included AGI objectives like "continue a casual discussion". [58] In reaction to this and the success of specialist systems, both industry and federal government pumped money into the field. [56] [59] However, confidence in AI marvelously collapsed in the late 1980s, and the objectives of the Fifth Generation Computer Project were never ever satisfied. [60] For the 2nd time in twenty years, AI researchers who predicted the imminent accomplishment of AGI had been misinterpreted. By the 1990s, AI scientists had a track record for making vain promises. They became reluctant to make predictions at all [d] and avoided mention of "human level" synthetic intelligence for fear of being labeled "wild-eyed dreamer [s]. [62]

Narrow AI research study


In the 1990s and early 21st century, mainstream AI accomplished commercial success and academic respectability by concentrating on particular sub-problems where AI can produce verifiable results and business applications, such as speech recognition and recommendation algorithms. [63] These "applied AI" systems are now used thoroughly throughout the innovation industry, and research study in this vein is greatly moneyed in both academic community and industry. As of 2018 [upgrade], advancement in this field was thought about an emerging pattern, and a mature phase was anticipated to be reached in more than ten years. [64]

At the millenium, lots of traditional AI scientists [65] hoped that strong AI could be established by combining programs that solve various sub-problems. Hans Moravec composed in 1988:


I am positive that this bottom-up path to expert system will one day fulfill the standard top-down route more than half method, prepared to supply the real-world competence and the commonsense knowledge that has been so frustratingly evasive in thinking programs. Fully intelligent makers will result when the metaphorical golden spike is driven unifying the 2 efforts. [65]

However, even at the time, this was contested. For example, Stevan Harnad of Princeton University concluded his 1990 paper on the symbol grounding hypothesis by specifying:


The expectation has typically been voiced that "top-down" (symbolic) approaches to modeling cognition will somehow meet "bottom-up" (sensory) approaches somewhere in between. If the grounding considerations in this paper stand, then this expectation is hopelessly modular and there is really just one viable route from sense to symbols: from the ground up. A free-floating symbolic level like the software level of a computer will never ever be reached by this route (or vice versa) - nor is it clear why we should even try to reach such a level, because it looks as if arriving would simply total up to uprooting our signs from their intrinsic meanings (thus simply decreasing ourselves to the practical equivalent of a programmable computer). [66]

Modern artificial basic intelligence research study


The term "artificial general intelligence" was utilized as early as 1997, by Mark Gubrud [67] in a discussion of the ramifications of fully automated military production and operations. A mathematical formalism of AGI was proposed by Marcus Hutter in 2000. Named AIXI, the proposed AGI representative maximises "the capability to please goals in a broad variety of environments". [68] This type of AGI, defined by the ability to maximise a mathematical definition of intelligence instead of display human-like behaviour, [69] was likewise called universal synthetic intelligence. [70]

The term AGI was re-introduced and promoted by Shane Legg and Ben Goertzel around 2002. [71] AGI research activity in 2006 was described by Pei Wang and Ben Goertzel [72] as "producing publications and preliminary results". The first summer school in AGI was arranged in Xiamen, China in 2009 [73] by the Xiamen university's Artificial Brain Laboratory and OpenCog. The first university course was given up 2010 [74] and 2011 [75] at Plovdiv University, Bulgaria by Todor Arnaudov. MIT presented a course on AGI in 2018, arranged by Lex Fridman and featuring a number of visitor lecturers.


As of 2023 [upgrade], a little number of computer system scientists are active in AGI research study, and many add to a series of AGI conferences. However, increasingly more researchers are interested in open-ended learning, [76] [77] which is the idea of enabling AI to continually find out and innovate like human beings do.


Feasibility


As of 2023, the advancement and possible achievement of AGI remains a subject of intense argument within the AI community. While traditional consensus held that AGI was a distant objective, recent improvements have actually led some researchers and market figures to declare that early types of AGI may currently exist. [78] AI leader Herbert A. Simon speculated in 1965 that "devices will be capable, within twenty years, of doing any work a guy can do". This prediction stopped working to come true. Microsoft co-founder Paul Allen thought that such intelligence is not likely in the 21st century due to the fact that it would require "unforeseeable and basically unpredictable developments" and a "scientifically deep understanding of cognition". [79] Writing in The Guardian, roboticist Alan Winfield claimed the gulf in between modern computing and human-level synthetic intelligence is as wide as the gulf in between existing area flight and practical faster-than-light spaceflight. [80]

A more difficulty is the lack of clarity in specifying what intelligence requires. Does it require consciousness? Must it display the capability to set goals as well as pursue them? Is it simply a matter of scale such that if model sizes increase sufficiently, intelligence will emerge? Are facilities such as planning, reasoning, and causal understanding needed? Does intelligence require explicitly replicating the brain and its specific faculties? Does it need emotions? [81]

Most AI scientists think strong AI can be attained in the future, however some thinkers, like Hubert Dreyfus and Roger Penrose, deny the possibility of achieving strong AI. [82] [83] John McCarthy is among those who believe human-level AI will be accomplished, but that today level of progress is such that a date can not properly be anticipated. [84] AI specialists' views on the feasibility of AGI wax and wane. Four surveys conducted in 2012 and 2013 recommended that the typical estimate amongst professionals for when they would be 50% confident AGI would arrive was 2040 to 2050, depending upon the poll, with the mean being 2081. Of the specialists, 16.5% answered with "never" when asked the exact same question however with a 90% confidence instead. [85] [86] Further current AGI development considerations can be found above Tests for validating human-level AGI.


A report by Stuart Armstrong and Kaj Sotala of the Machine Intelligence Research Institute found that "over [a] 60-year time frame there is a strong predisposition towards predicting the arrival of human-level AI as between 15 and 25 years from the time the forecast was made". They examined 95 forecasts made in between 1950 and 2012 on when human-level AI will come about. [87]

In 2023, Microsoft researchers released a detailed examination of GPT-4. They concluded: "Given the breadth and depth of GPT-4's abilities, we believe that it might reasonably be deemed an early (yet still incomplete) version of an artificial general intelligence (AGI) system." [88] Another study in 2023 reported that GPT-4 outperforms 99% of human beings on the Torrance tests of creative thinking. [89] [90]

Blaise Agüera y Arcas and Peter Norvig composed in 2023 that a significant level of general intelligence has already been accomplished with frontier models. They composed that unwillingness to this view comes from 4 main factors: a "healthy suspicion about metrics for AGI", an "ideological dedication to alternative AI theories or strategies", a "commitment to human (or biological) exceptionalism", or a "concern about the financial implications of AGI". [91]

2023 likewise marked the development of large multimodal designs (big language models efficient in processing or generating numerous methods such as text, audio, and images). [92]

In 2024, OpenAI launched o1-preview, the very first of a series of models that "spend more time thinking before they react". According to Mira Murati, this ability to think before responding represents a new, additional paradigm. It improves model outputs by spending more computing power when producing the response, whereas the design scaling paradigm enhances outputs by increasing the design size, training data and training calculate power. [93] [94]

An OpenAI worker, Vahid Kazemi, claimed in 2024 that the business had accomplished AGI, specifying, "In my opinion, we have actually already achieved AGI and it's much more clear with O1." Kazemi clarified that while the AI is not yet "much better than any human at any task", it is "much better than most humans at most tasks." He also resolved criticisms that big language models (LLMs) merely follow predefined patterns, comparing their knowing process to the scientific technique of observing, hypothesizing, and validating. These declarations have actually sparked dispute, as they rely on a broad and unconventional meaning of AGI-traditionally comprehended as AI that matches human intelligence throughout all domains. Critics argue that, while OpenAI's models show amazing adaptability, they might not totally fulfill this requirement. Notably, Kazemi's remarks came soon after OpenAI got rid of "AGI" from the regards to its collaboration with Microsoft, prompting speculation about the business's strategic intents. [95]

Timescales


Progress in expert system has actually historically gone through periods of rapid progress separated by durations when development appeared to stop. [82] Ending each hiatus were essential advances in hardware, software application or both to develop area for more progress. [82] [98] [99] For example, the hardware offered in the twentieth century was not sufficient to execute deep knowing, which needs large numbers of GPU-enabled CPUs. [100]

In the introduction to his 2006 book, [101] Goertzel states that price quotes of the time required before a genuinely versatile AGI is constructed vary from ten years to over a century. Since 2007 [upgrade], the consensus in the AGI research community seemed to be that the timeline discussed by Ray Kurzweil in 2005 in The Singularity is Near [102] (i.e. in between 2015 and 2045) was possible. [103] Mainstream AI scientists have given a large variety of viewpoints on whether development will be this quick. A 2012 meta-analysis of 95 such viewpoints discovered a predisposition towards anticipating that the start of AGI would happen within 16-26 years for contemporary and historic predictions alike. That paper has actually been criticized for how it classified viewpoints as specialist or non-expert. [104]

In 2012, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton established a neural network called AlexNet, which won the ImageNet competition with a top-5 test error rate of 15.3%, substantially better than the second-best entry's rate of 26.3% (the conventional technique used a weighted amount of scores from different pre-defined classifiers). [105] AlexNet was concerned as the initial ground-breaker of the current deep knowing wave. [105]

In 2017, researchers Feng Liu, Yong Shi, and Ying Liu performed intelligence tests on publicly offered and freely accessible weak AI such as Google AI, Apple's Siri, and others. At the optimum, these AIs reached an IQ value of about 47, which corresponds around to a six-year-old kid in very first grade. An adult comes to about 100 on average. Similar tests were performed in 2014, with the IQ score reaching an optimum worth of 27. [106] [107]

In 2020, OpenAI developed GPT-3, a language model capable of carrying out numerous diverse jobs without particular training. According to Gary Grossman in a VentureBeat article, while there is consensus that GPT-3 is not an example of AGI, it is considered by some to be too advanced to be classified as a narrow AI system. [108]

In the very same year, Jason Rohrer utilized his GPT-3 account to establish a chatbot, and provided a chatbot-developing platform called "Project December". OpenAI asked for changes to the chatbot to abide by their safety standards; Rohrer detached Project December from the GPT-3 API. [109]

In 2022, DeepMind established Gato, a "general-purpose" system efficient in carrying out more than 600 different tasks. [110]

In 2023, Microsoft Research released a research study on an early version of OpenAI's GPT-4, contending that it exhibited more basic intelligence than previous AI models and showed human-level efficiency in tasks spanning numerous domains, such as mathematics, coding, and law. This research study triggered an argument on whether GPT-4 could be considered an early, insufficient version of synthetic general intelligence, stressing the need for further exploration and examination of such systems. [111]

In 2023, the AI researcher Geoffrey Hinton mentioned that: [112]

The idea that this things could in fact get smarter than individuals - a couple of individuals thought that, [...] But the majority of people believed it was method off. And I believed it was method off. I believed it was 30 to 50 years or even longer away. Obviously, I no longer believe that.


In May 2023, Demis Hassabis likewise said that "The development in the last few years has been quite extraordinary", and that he sees no factor why it would slow down, anticipating AGI within a decade or perhaps a couple of years. [113] In March 2024, Nvidia's CEO, Jensen Huang, stated his expectation that within 5 years, AI would can passing any test at least as well as people. [114] In June 2024, the AI researcher Leopold Aschenbrenner, a former OpenAI employee, approximated AGI by 2027 to be "strikingly possible". [115]

Whole brain emulation


While the advancement of transformer models like in ChatGPT is thought about the most promising course to AGI, [116] [117] entire brain emulation can work as an alternative method. With entire brain simulation, a brain model is constructed by scanning and mapping a biological brain in detail, and then copying and simulating it on a computer system or another computational gadget. The simulation design must be sufficiently loyal to the original, so that it behaves in practically the exact same way as the original brain. [118] Whole brain emulation is a type of brain simulation that is talked about in computational neuroscience and neuroinformatics, and for medical research study functions. It has been talked about in artificial intelligence research [103] as an approach to strong AI. Neuroimaging innovations that could deliver the essential comprehensive understanding are improving rapidly, and futurist Ray Kurzweil in the book The Singularity Is Near [102] forecasts that a map of adequate quality will end up being offered on a comparable timescale to the computing power needed to replicate it.


Early approximates


For low-level brain simulation, a really effective cluster of computer systems or GPUs would be needed, provided the massive quantity of synapses within the human brain. Each of the 1011 (one hundred billion) neurons has on typical 7,000 synaptic connections (synapses) to other neurons. The brain of a three-year-old kid has about 1015 synapses (1 quadrillion). This number declines with age, stabilizing by the adult years. Estimates vary for an adult, ranging from 1014 to 5 × 1014 synapses (100 to 500 trillion). [120] A quote of the brain's processing power, based upon an easy switch design for neuron activity, is around 1014 (100 trillion) synaptic updates per second (SUPS). [121]

In 1997, Kurzweil took a look at various price quotes for the hardware needed to equate to the human brain and adopted a figure of 1016 calculations per second (cps). [e] (For comparison, if a "computation" was comparable to one "floating-point operation" - a procedure utilized to rate existing supercomputers - then 1016 "calculations" would be comparable to 10 petaFLOPS, achieved in 2011, while 1018 was achieved in 2022.) He utilized this figure to anticipate the required hardware would be readily available at some point in between 2015 and 2025, if the rapid development in computer power at the time of composing continued.


Current research study


The Human Brain Project, an EU-funded effort active from 2013 to 2023, has established an especially in-depth and openly accessible atlas of the human brain. [124] In 2023, scientists from Duke University performed a high-resolution scan of a mouse brain.


Criticisms of simulation-based methods


The synthetic nerve cell design presumed by Kurzweil and used in many present synthetic neural network executions is basic compared with biological nerve cells. A brain simulation would likely need to record the in-depth cellular behaviour of biological nerve cells, currently understood just in broad summary. The overhead introduced by complete modeling of the biological, chemical, and physical information of neural behaviour (especially on a molecular scale) would need computational powers several orders of magnitude bigger than Kurzweil's estimate. In addition, the quotes do not represent glial cells, which are understood to play a function in cognitive processes. [125]

A basic criticism of the simulated brain method derives from embodied cognition theory which asserts that human embodiment is an important aspect of human intelligence and is essential to ground significance. [126] [127] If this theory is proper, any fully practical brain model will need to encompass more than simply the neurons (e.g., a robotic body). Goertzel [103] proposes virtual personification (like in metaverses like Second Life) as a choice, but it is unknown whether this would be adequate.


Philosophical point of view


"Strong AI" as defined in approach


In 1980, thinker John Searle coined the term "strong AI" as part of his Chinese room argument. [128] He proposed a distinction in between 2 hypotheses about synthetic intelligence: [f]

Strong AI hypothesis: An expert system system can have "a mind" and "consciousness".
Weak AI hypothesis: An artificial intelligence system can (just) act like it thinks and has a mind and awareness.


The first one he called "strong" because it makes a more powerful declaration: it assumes something special has actually taken place to the maker that exceeds those capabilities that we can evaluate. The behaviour of a "weak AI" machine would be precisely identical to a "strong AI" machine, however the latter would likewise have subjective mindful experience. This use is also typical in academic AI research study and books. [129]

In contrast to Searle and traditional AI, some futurists such as Ray Kurzweil use the term "strong AI" to mean "human level synthetic general intelligence". [102] This is not the like Searle's strong AI, unless it is presumed that awareness is necessary for human-level AGI. Academic thinkers such as Searle do not think that holds true, and to most expert system scientists the concern is out-of-scope. [130]

Mainstream AI is most thinking about how a program acts. [131] According to Russell and Norvig, "as long as the program works, they do not care if you call it genuine or a simulation." [130] If the program can act as if it has a mind, then there is no requirement to know if it really has mind - certainly, there would be no method to tell. For AI research study, Searle's "weak AI hypothesis" is equivalent to the statement "artificial general intelligence is possible". Thus, according to Russell and Norvig, "most AI researchers take the weak AI hypothesis for approved, and do not care about the strong AI hypothesis." [130] Thus, for academic AI research study, "Strong AI" and "AGI" are two different things.


Consciousness


Consciousness can have different significances, and some aspects play substantial roles in sci-fi and the principles of expert system:


Sentience (or "phenomenal consciousness"): The capability to "feel" understandings or emotions subjectively, as opposed to the ability to factor about understandings. Some theorists, such as David Chalmers, utilize the term "awareness" to refer solely to phenomenal awareness, which is approximately equivalent to sentience. [132] Determining why and how subjective experience occurs is called the hard issue of awareness. [133] Thomas Nagel discussed in 1974 that it "feels like" something to be conscious. If we are not mindful, then it does not feel like anything. Nagel uses the example of a bat: we can smartly ask "what does it seem like to be a bat?" However, we are not likely to ask "what does it feel like to be a toaster?" Nagel concludes that a bat seems conscious (i.e., has consciousness) but a toaster does not. [134] In 2022, a Google engineer claimed that the business's AI chatbot, LaMDA, had achieved life, though this claim was extensively challenged by other specialists. [135]

Self-awareness: To have conscious awareness of oneself as a different person, especially to be consciously familiar with one's own ideas. This is opposed to just being the "subject of one's thought"-an os or debugger is able to be "familiar with itself" (that is, to represent itself in the very same method it represents everything else)-however this is not what individuals typically indicate when they use the term "self-awareness". [g]

These qualities have an ethical measurement. AI life would generate concerns of welfare and legal security, similarly to animals. [136] Other elements of awareness associated to cognitive capabilities are likewise appropriate to the concept of AI rights. [137] Determining how to integrate sophisticated AI with existing legal and social frameworks is an emergent issue. [138]

Benefits


AGI might have a wide range of applications. If oriented towards such goals, AGI could help reduce different problems in the world such as hunger, poverty and health issue. [139]

AGI might improve productivity and efficiency in the majority of tasks. For example, in public health, AGI might speed up medical research study, notably against cancer. [140] It might look after the senior, [141] and democratize access to rapid, high-quality medical diagnostics. It could provide fun, cheap and tailored education. [141] The need to work to subsist could become outdated if the wealth produced is correctly redistributed. [141] [142] This also raises the concern of the location of humans in a radically automated society.


AGI might also help to make reasonable decisions, and to prepare for and prevent disasters. It could likewise assist to reap the benefits of potentially catastrophic technologies such as nanotechnology or climate engineering, while avoiding the associated risks. [143] If an AGI's main goal is to prevent existential disasters such as human extinction (which might be difficult if the Vulnerable World Hypothesis ends up being true), [144] it might take steps to considerably lower the dangers [143] while minimizing the impact of these procedures on our quality of life.


Risks


Existential threats


AGI might represent numerous kinds of existential danger, which are risks that threaten "the early termination of Earth-originating intelligent life or the irreversible and extreme damage of its capacity for oke.zone preferable future advancement". [145] The danger of human extinction from AGI has been the topic of numerous debates, but there is also the possibility that the development of AGI would lead to a permanently flawed future. Notably, it could be used to spread and maintain the set of worths of whoever establishes it. If mankind still has ethical blind spots similar to slavery in the past, AGI may irreversibly entrench it, avoiding moral progress. [146] Furthermore, AGI might help with mass surveillance and indoctrination, which could be utilized to develop a steady repressive around the world totalitarian routine. [147] [148] There is also a risk for the machines themselves. If makers that are sentient or otherwise deserving of ethical consideration are mass created in the future, taking part in a civilizational course that indefinitely overlooks their well-being and interests might be an existential catastrophe. [149] [150] Considering how much AGI could enhance humankind's future and assistance minimize other existential dangers, Toby Ord calls these existential dangers "an argument for continuing with due caution", not for "deserting AI". [147]

Risk of loss of control and human extinction


The thesis that AI poses an existential danger for human beings, which this risk needs more attention, is questionable but has actually been backed in 2023 by lots of public figures, AI scientists and CEOs of AI business such as Elon Musk, Bill Gates, Geoffrey Hinton, Yoshua Bengio, Demis Hassabis and Sam Altman. [151] [152]

In 2014, Stephen Hawking slammed widespread indifference:


So, facing possible futures of incalculable advantages and dangers, the specialists are certainly doing whatever possible to make sure the finest outcome, right? Wrong. If a remarkable alien civilisation sent us a message saying, 'We'll show up in a few years,' would we just respond, 'OK, call us when you get here-we'll leave the lights on?' Probably not-but this is basically what is occurring with AI. [153]

The potential fate of mankind has in some cases been compared to the fate of gorillas threatened by human activities. The contrast specifies that higher intelligence allowed mankind to control gorillas, which are now susceptible in ways that they could not have actually anticipated. As a result, the gorilla has become an endangered species, not out of malice, but just as a civilian casualties from human activities. [154]

The skeptic Yann LeCun considers that AGIs will have no desire to control humanity which we ought to beware not to anthropomorphize them and analyze their intents as we would for humans. He stated that people won't be "clever enough to create super-intelligent makers, yet ridiculously foolish to the point of giving it moronic objectives without any safeguards". [155] On the other side, the idea of critical convergence suggests that almost whatever their goals, intelligent agents will have factors to attempt to survive and obtain more power as intermediary actions to attaining these goals. And that this does not require having emotions. [156]

Many scholars who are concerned about existential danger supporter for more research into solving the "control problem" to answer the question: what kinds of safeguards, algorithms, or architectures can programmers carry out to increase the likelihood that their recursively-improving AI would continue to behave in a friendly, rather than harmful, way after it reaches superintelligence? [157] [158] Solving the control issue is complicated by the AI arms race (which might result in a race to the bottom of safety preventative measures in order to launch items before competitors), [159] and making use of AI in weapon systems. [160]

The thesis that AI can posture existential risk likewise has critics. Skeptics normally state that AGI is unlikely in the short-term, or that concerns about AGI distract from other problems connected to current AI. [161] Former Google fraud czar Shuman Ghosemajumder considers that for lots of people outside of the technology industry, existing chatbots and LLMs are already perceived as though they were AGI, resulting in additional misconception and worry. [162]

Skeptics often charge that the thesis is crypto-religious, with an illogical belief in the possibility of superintelligence changing an illogical belief in a supreme God. [163] Some researchers think that the communication campaigns on AI existential threat by particular AI groups (such as OpenAI, Anthropic, DeepMind, and Conjecture) might be an at attempt at regulatory capture and to inflate interest in their products. [164] [165]

In 2023, the CEOs of Google DeepMind, OpenAI and Anthropic, together with other market leaders and researchers, provided a joint statement asserting that "Mitigating the threat of extinction from AI need to be an international priority alongside other societal-scale threats such as pandemics and nuclear war." [152]

Mass unemployment


Researchers from OpenAI approximated that "80% of the U.S. labor force could have at least 10% of their work tasks impacted by the intro of LLMs, while around 19% of employees may see a minimum of 50% of their jobs impacted". [166] [167] They consider office employees to be the most exposed, for example mathematicians, accountants or web designers. [167] AGI could have a better autonomy, ability to make choices, to user interface with other computer system tools, however likewise to control robotized bodies.


According to Stephen Hawking, the outcome of automation on the lifestyle will depend on how the wealth will be redistributed: [142]

Everyone can delight in a life of glamorous leisure if the machine-produced wealth is shared, or many people can wind up badly poor if the machine-owners successfully lobby against wealth redistribution. Up until now, the trend appears to be towards the second choice, with technology driving ever-increasing inequality


Elon Musk thinks about that the automation of society will need federal governments to embrace a universal fundamental income. [168]

See also


Artificial brain - Software and hardware with cognitive abilities comparable to those of the animal or human brain
AI effect
AI security - Research location on making AI safe and helpful
AI positioning - AI conformance to the desired goal
A.I. Rising - 2018 movie directed by Lazar Bodroža
Artificial intelligence
Automated artificial intelligence - Process of automating the application of maker knowing
BRAIN Initiative - Collaborative public-private research initiative announced by the Obama administration
China Brain Project
Future of Humanity Institute - Defunct Oxford interdisciplinary research study centre
General video game playing - Ability of expert system to play various games
Generative synthetic intelligence - AI system capable of producing material in action to triggers
Human Brain Project - Scientific research study task
Intelligence amplification - Use of info innovation to enhance human intelligence (IA).
Machine principles - Moral behaviours of manufactured machines.
Moravec's paradox.
Multi-task knowing - Solving multiple device learning tasks at the exact same time.
Neural scaling law - Statistical law in artificial intelligence.
Outline of expert system - Overview of and topical guide to artificial intelligence.
Transhumanism - Philosophical movement.
Synthetic intelligence - Alternate term for or form of expert system.
Transfer knowing - Artificial intelligence method.
Loebner Prize - Annual AI competition.
Hardware for artificial intelligence - Hardware specially designed and enhanced for artificial intelligence.
Weak expert system - Form of expert system.


Notes


^ a b See below for the origin of the term "strong AI", and see the academic definition of "strong AI" and weak AI in the post Chinese room.
^ AI founder John McCarthy writes: "we can not yet characterize in general what kinds of computational procedures we wish to call intelligent. " [26] (For a conversation of some definitions of intelligence utilized by artificial intelligence researchers, see philosophy of synthetic intelligence.).
^ The Lighthill report specifically slammed AI's "grandiose objectives" and led the dismantling of AI research study in England. [55] In the U.S., DARPA became determined to money just "mission-oriented direct research study, instead of standard undirected research study". [56] [57] ^ As AI founder John McCarthy composes "it would be a terrific relief to the rest of the employees in AI if the creators of brand-new basic formalisms would reveal their hopes in a more safeguarded form than has often been the case." [61] ^ In "Mind Children" [122] 1015 cps is used. More just recently, in 1997, [123] Moravec argued for 108 MIPS which would approximately represent 1014 cps. Moravec talks in regards to MIPS, not "cps", which is a non-standard term Kurzweil introduced.
^ As specified in a standard AI book: "The assertion that devices might potentially act smartly (or, possibly better, act as if they were smart) is called the 'weak AI' hypothesis by theorists, and the assertion that machines that do so are actually believing (instead of simulating thinking) is called the 'strong AI' hypothesis." [121] ^ Alan Turing made this point in 1950. [36] References


^ Krishna, Sri (9 February 2023). "What is synthetic narrow intelligence (ANI)?". VentureBeat. Retrieved 1 March 2024. ANI is created to perform a single task.
^ "OpenAI Charter". OpenAI. Retrieved 6 April 2023. Our mission is to guarantee that synthetic basic intelligence advantages all of humankind.
^ Heath, Alex (18 January 2024). "Mark Zuckerberg's brand-new objective is producing artificial basic intelligence". The Verge. Retrieved 13 June 2024. Our vision is to construct AI that is much better than human-level at all of the human senses.
^ Baum, Seth D. (2020 ). A Study of Artificial General Intelligence Projects for Ethics, Risk, and Policy (PDF) (Report). Global Catastrophic Risk Institute. Retrieved 28 November 2024. 72 AGI R&D projects were determined as being active in 2020.
^ a b c "AI timelines: What do professionals in synthetic intelligence anticipate for the future?". Our World in Data. Retrieved 6 April 2023.
^ Metz, Cade (15 May 2023). "Some Researchers Say A.I. Is Already Here, Stirring Debate in Tech Circles". The New York Times. Retrieved 18 May 2023.
^ "AI leader Geoffrey Hinton stops Google and cautions of threat ahead". The New York Times. 1 May 2023. Retrieved 2 May 2023. It is tough to see how you can prevent the bad actors from utilizing it for bad things.
^ Bubeck, Sébastien; Chandrasekaran, Varun; Eldan, Ronen; Gehrke, Johannes; Horvitz, Eric (2023 ). "Sparks of Artificial General Intelligence: Early try outs GPT-4". arXiv preprint. arXiv:2303.12712. GPT-4 reveals sparks of AGI.
^ Butler, Octavia E. (1993 ). Parable of the Sower. Grand Central Publishing. ISBN 978-0-4466-7550-5. All that you touch you change. All that you alter modifications you.
^ Vinge, Vernor (1992 ). A Fire Upon the Deep. Tor Books. ISBN 978-0-8125-1528-2. The Singularity is coming.
^ Morozov, Evgeny (30 June 2023). "The True Threat of Expert System". The New York Times. The genuine risk is not AI itself but the method we release it.
^ "Impressed by artificial intelligence? Experts state AGI is following, and it has 'existential' dangers". ABC News. 23 March 2023. Retrieved 6 April 2023. AGI could present existential dangers to mankind.
^ Bostrom, Nick (2014 ). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. ISBN 978-0-1996-7811-2. The first superintelligence will be the last creation that mankind needs to make.
^ Roose, Kevin (30 May 2023). "A.I. Poses 'Risk of Extinction,' Industry Leaders Warn". The New York City Times. Mitigating the threat of termination from AI must be a worldwide concern.
^ "Statement on AI Risk". Center for AI Safety. Retrieved 1 March 2024. AI specialists warn of danger of extinction from AI.
^ Mitchell, Melanie (30 May 2023). "Are AI's Doomsday Scenarios Worth Taking Seriously?". The New York Times. We are far from producing makers that can outthink us in general ways.
^ LeCun, Yann (June 2023). "AGI does not provide an existential risk". Medium. There is no reason to fear AI as an existential risk.
^ Kurzweil 2005, p. 260.
^ a b Kurzweil, Ray (5 August 2005), "Long Live AI", Forbes, archived from the initial on 14 August 2005: Kurzweil describes strong AI as "machine intelligence with the complete range of human intelligence.".
^ "The Age of Artificial Intelligence: George John at TEDxLondonBusinessSchool 2013". Archived from the initial on 26 February 2014. Retrieved 22 February 2014.
^ Newell & Simon 1976, This is the term they use for "human-level" intelligence in the physical symbol system hypothesis.
^ "The Open University on Strong and Weak AI". Archived from the initial on 25 September 2009. Retrieved 8 October 2007.
^ "What is artificial superintelligence (ASI)?|Definition from TechTarget". Enterprise AI. Retrieved 8 October 2023.
^ "Expert system is changing our world - it is on everybody to make sure that it goes well". Our World in Data. Retrieved 8 October 2023.
^ Dickson, Ben (16 November 2023). "Here is how far we are to achieving AGI, according to DeepMind". VentureBeat.
^ McCarthy, John (2007a). "Basic Questions". Stanford University. Archived from the original on 26 October 2007. Retrieved 6 December 2007.
^ This list of smart traits is based on the subjects covered by significant AI textbooks, consisting of: Russell & Norvig 2003, Luger & Stubblefield 2004, Poole, Mackworth & Goebel 1998 and Nilsson 1998.
^ Johnson 1987.
^ de Charms, R. (1968 ). Personal causation. New York City: Academic Press.
^ a b Pfeifer, R. and Bongard J. C., How the body shapes the method we believe: a brand-new view of intelligence (The MIT Press, 2007). ISBN 0-2621-6239-3.
^ White, R. W. (1959 ). "Motivation reconsidered: The principle of competence". Psychological Review. 66 (5 ): 297-333. doi:10.1037/ h0040934. PMID 13844397. S2CID 37385966.
^ White, R. W. (1959 ). "Motivation reconsidered: The principle of competence". Psychological Review. 66 (5 ): 297-333. doi:10.1037/ h0040934. PMID 13844397. S2CID 37385966.
^ Muehlhauser, Luke (11 August 2013). "What is AGI?". Machine Intelligence Research Institute. Archived from the original on 25 April 2014. Retrieved 1 May 2014.
^ "What is Artificial General Intelligence (AGI)?|4 Tests For Ensuring Artificial General Intelligence". Talky Blog. 13 July 2019. Archived from the original on 17 July 2019. Retrieved 17 July 2019.
^ Kirk-Giannini, Cameron Domenico; Goldstein, Simon (16 October 2023). "AI is closer than ever to passing the Turing test for 'intelligence'. What takes place when it does?". The Conversation. Retrieved 22 September 2024.
^ a b Turing 1950.
^ Turing, Alan (1952 ). B. Jack Copeland (ed.). Can Automatic Calculating Machines Be Said To Think?. Oxford: Oxford University Press. pp. 487-506. ISBN 978-0-1982-5079-1.
^ "Eugene Goostman is a genuine kid - the Turing Test states so". The Guardian. 9 June 2014. ISSN 0261-3077. Retrieved 3 March 2024.
^ "Scientists dispute whether computer 'Eugene Goostman' passed Turing test". BBC News. 9 June 2014. Retrieved 3 March 2024.
^ Jones, Cameron R.; Bergen, Benjamin K. (9 May 2024). "People can not distinguish GPT-4 from a human in a Turing test". arXiv:2405.08007 [cs.HC]
^ Varanasi, Lakshmi (21 March 2023). "AI models like ChatGPT and GPT-4 are acing whatever from the bar exam to AP Biology. Here's a list of difficult examinations both AI versions have actually passed". Business Insider. Retrieved 30 May 2023.
^ Naysmith, Caleb (7 February 2023). "6 Jobs Artificial Intelligence Is Already Replacing and How Investors Can Take Advantage Of It". Retrieved 30 May 2023.
^ Turk, Victoria (28 January 2015). "The Plan to Replace the Turing Test with a 'Turing Olympics'". Vice. Retrieved 3 March 2024.
^ Gopani, Avi (25 May 2022). "Turing Test is unreliable. The Winograd Schema is outdated. Coffee is the answer". Analytics India Magazine. Retrieved 3 March 2024.
^ Bhaimiya, Sawdah (20 June 2023). "DeepMind's co-founder recommended evaluating an AI chatbot's capability to turn $100,000 into $1 million to determine human-like intelligence". Business Insider. Retrieved 3 March 2024.
^ Suleyman, Mustafa (14 July 2023). "Mustafa Suleyman: My brand-new Turing test would see if AI can make $1 million". MIT Technology Review. Retrieved 3 March 2024.
^ Shapiro, Stuart C. (1992 ). "Artificial Intelligence" (PDF). In Stuart C. Shapiro (ed.). Encyclopedia of Artificial Intelligence (Second ed.). New York City: John Wiley. pp. 54-57. Archived (PDF) from the original on 1 February 2016. (Section 4 is on "AI-Complete Tasks".).
^ Yampolskiy, Roman V. (2012 ). Xin-She Yang (ed.). "Turing Test as a Defining Feature of AI-Completeness" (PDF). Artificial Intelligence, Evolutionary Computation and Metaheuristics (AIECM): 3-17. Archived (PDF) from the original on 22 May 2013.
^ "AI Index: State of AI in 13 Charts". Stanford University Human-Centered Artificial Intelligence. 15 April 2024. Retrieved 27 May 2024.
^ Crevier 1993, pp. 48-50.
^ Kaplan, Andreas (2022 ). "Expert System, Business and Civilization - Our Fate Made in Machines". Archived from the initial on 6 May 2022. Retrieved 12 March 2022.
^ Simon 1965, p. 96 priced estimate in Crevier 1993, p. 109.
^ "Scientist on the Set: An Interview with Marvin Minsky". Archived from the initial on 16 July 2012. Retrieved 5 April 2008.
^ Marvin Minsky to Darrach (1970 ), estimated in Crevier (1993, p. 109).
^ Lighthill 1973; Howe 1994.
^ a b NRC 1999, "Shift to Applied Research Increases Investment".
^ Crevier 1993, pp. 115-117; Russell & Norvig 2003, pp. 21-22.
^ Crevier 1993, p. 211, Russell & Norvig 2003, p. 24 and see also Feigenbaum & McCorduck 1983.
^ Crevier 1993, pp. 161-162, 197-203, 240; Russell & Norvig 2003, p. 25.
^ Crevier 1993, pp. 209-212.
^ McCarthy, John (2000 ). "Respond to Lighthill". Stanford University. Archived from the initial on 30 September 2008. Retrieved 29 September 2007.
^ Markoff, John (14 October 2005). "Behind Artificial Intelligence, a Squadron of Bright Real People". The New York Times. Archived from the initial on 2 February 2023. Retrieved 18 February 2017. At its low point, some computer system researchers and software application engineers avoided the term synthetic intelligence for worry of being considered as wild-eyed dreamers.
^ Russell & Norvig 2003, pp. 25-26
^ "Trends in the Emerging Tech Hype Cycle". Gartner Reports. Archived from the original on 22 May 2019. Retrieved 7 May 2019.
^ a b Moravec 1988, p. 20
^ Harnad, S. (1990 ). "The Symbol Grounding Problem". Physica D. 42 (1-3): 335-346. arXiv: cs/9906002. Bibcode:1990 PhyD ... 42..335 H. doi:10.1016/ 0167-2789( 90 )90087-6. S2CID 3204300.
^ Gubrud 1997
^ Hutter, Marcus (2005 ). Universal Expert System: Sequential Decisions Based on Algorithmic Probability. Texts in Theoretical Computer Science an EATCS Series. Springer. doi:10.1007/ b138233. ISBN 978-3-5402-6877-2. S2CID 33352850. Archived from the initial on 19 July 2022. Retrieved 19 July 2022.
^ Legg, Shane (2008 ). Machine Super Intelligence (PDF) (Thesis). University of Lugano. Archived (PDF) from the original on 15 June 2022. Retrieved 19 July 2022.
^ Goertzel, Ben (2014 ). Artificial General Intelligence. Lecture Notes in Computer Science. Vol. 8598. Journal of Artificial General Intelligence. doi:10.1007/ 978-3-319-09274-4. ISBN 978-3-3190-9273-7. S2CID 8387410.
^ "Who coined the term "AGI"?". goertzel.org. Archived from the initial on 28 December 2018. Retrieved 28 December 2018., via Life 3.0: 'The term "AGI" was popularized by ... Shane Legg, Mark Gubrud and Ben Goertzel'
^ Wang & Goertzel 2007
^ "First International Summer School in Artificial General Intelligence, Main summer school: June 22 - July 3, 2009, OpenCog Lab: July 6-9, 2009". Archived from the original on 28 September 2020. Retrieved 11 May 2020.
^ "Избираеми дисциплини 2009/2010 - пролетен триместър" [Elective courses 2009/2010 - spring trimester] Факултет по математика и информатика [Faculty of Mathematics and Informatics] (in Bulgarian). Archived from the original on 26 July 2020. Retrieved 11 May 2020.
^ "Избираеми дисциплини 2010/2011 - зимен триместър" [Elective courses 2010/2011 - winter trimester] Факултет по математика и информатика [Faculty of Mathematics and Informatics] (in Bulgarian). Archived from the initial on 26 July 2020. Retrieved 11 May 2020.
^ Shevlin, Henry; Vold, Karina; Crosby, Matthew; Halina, Marta (4 October 2019). "The limitations of device intelligence: Despite development in device intelligence, synthetic general intelligence is still a significant obstacle". EMBO Reports. 20 (10 ): e49177. doi:10.15252/ embr.201949177. ISSN 1469-221X. PMC 6776890. PMID 31531926.
^ Bubeck, Sébastien; Chandrasekaran, Varun; Eldan, Ronen; Gehrke, Johannes; Horvitz, Eric; Kamar, Ece; Lee, Peter; Lee, Yin Tat; Li, Yuanzhi; Lundberg, Scott; Nori, Harsha; Palangi, Hamid; Ribeiro, Marco Tulio; Zhang, Yi (27 March 2023). "Sparks of Artificial General Intelligence: Early try outs GPT-4". arXiv:2303.12712 [cs.CL]
^ "Microsoft Researchers Claim GPT-4 Is Showing "Sparks" of AGI". Futurism. 23 March 2023. Retrieved 13 December 2023.
^ Allen, Paul; Greaves, Mark (12 October 2011). "The Singularity Isn't Near". MIT Technology Review. Retrieved 17 September 2014.
^ Winfield, Alan. "Expert system will not turn into a Frankenstein's monster". The Guardian. Archived from the initial on 17 September 2014. Retrieved 17 September 2014.
^ Deane, George (2022 ). "Machines That Feel and Think: The Role of Affective Feelings and Mental Action in (Artificial) General Intelligence". Artificial Life. 28 (3 ): 289-309. doi:10.1162/ artl_a_00368. ISSN 1064-5462. PMID 35881678. S2CID 251069071.
^ a b c Clocksin 2003.
^ Fjelland, Ragnar (17 June 2020). "Why basic synthetic intelligence will not be recognized". Humanities and Social Sciences Communications. 7 (1 ): 1-9. doi:10.1057/ s41599-020-0494-4. hdl:11250/ 2726984. ISSN 2662-9992. S2CID 219710554.
^ McCarthy 2007b.
^ Khatchadourian, Raffi (23 November 2015). "The Doomsday Invention: Will synthetic intelligence bring us utopia or damage?". The New Yorker. Archived from the initial on 28 January 2016. Retrieved 7 February 2016.
^ Müller, V. C., & Bostrom, N. (2016 ). Future development in expert system: A survey of expert opinion. In Fundamental concerns of expert system (pp. 555-572). Springer, Cham.
^ Armstrong, Stuart, and Kaj Sotala. 2012. "How We're Predicting AI-or Failing To." In Beyond AI: Artificial Dreams, edited by Jan Romportl, Pavel Ircing, Eva Žáčková, Michal Polák and Radek Schuster, 52-75. Plzeň: University of West Bohemia
^ "Microsoft Now Claims GPT-4 Shows 'Sparks' of General Intelligence". 24 March 2023.
^ Shimek, Cary (6 July 2023). "AI Outperforms Humans in Creativity Test". Neuroscience News. Retrieved 20 October 2023.
^ Guzik, Erik E.; Byrge, Christian; Gilde, Christian (1 December 2023). "The originality of machines: AI takes the Torrance Test". Journal of Creativity. 33 (3 ): 100065. doi:10.1016/ j.yjoc.2023.100065. ISSN 2713-3745. S2CID 261087185.
^ Arcas, Blaise Agüera y (10 October 2023). "Artificial General Intelligence Is Already Here". Noema.
^ Zia, Tehseen (8 January 2024). "Unveiling of Large Multimodal Models: Shaping the Landscape of Language Models in 2024". Unite.ai. Retrieved 26 May 2024.
^ "Introducing OpenAI o1-preview". OpenAI. 12 September 2024.
^ Knight, Will. "OpenAI Announces a Brand-new AI Model, Code-Named Strawberry, That Solves Difficult Problems Step by Step". Wired. ISSN 1059-1028. Retrieved 17 September 2024.
^ "OpenAI Employee Claims AGI Has Been Achieved". Orbital Today. 13 December 2024. Retrieved 27 December 2024.
^ "AI Index: State of AI in 13 Charts". hai.stanford.edu. 15 April 2024. Retrieved 7 June 2024.
^ "Next-Gen AI: OpenAI and Meta's Leap Towards Reasoning Machines". Unite.ai. 19 April 2024. Retrieved 7 June 2024.
^ James, Alex P. (2022 ). "The Why, What, and How of Artificial General Intelligence Chip Development". IEEE Transactions on Cognitive and Developmental Systems. 14 (2 ): 333-347. arXiv:2012.06338. doi:10.1109/ TCDS.2021.3069871. ISSN 2379-8920. S2CID 228376556. Archived from the initial on 28 August 2022. Retrieved 28 August 2022.
^

Kommentare