Mathematics often provides unexpected tools that revolutionize how we think about practical problems. One of its more recent and wildly useful tools? Sheaf theory.

Sheaf Theory is the framework that takes our intuitive grasp of geometry and encodes it into a rigorous, quantifiable format using category theory. Proposed by Jean Leray in 1945 and taking its modern shape in Alexander Grothendieck’s legendary 1957 Tohoku Paper (cue math historians nodding in agreement), sheaf theory has since infiltrated fields far and wide. It has impacted several areas: from algebraic geometry and computer science, to topological data analysis, spectral graph theory, and even the design of neural networks.

Each new field that touches sheaf theory enriches it with new insights, snazzy terminology, and powerful new applications. It’s basically the ultimate mathematical shapeshifter. Whether it's being used to analyze geometric structures, refine neural networks, or optimize distributed systems, sheaf theory molds itself to the problem at hand, making it one of the most versatile mathematical tools available.

**Sheaves, AI, and Noeon’s Big Idea**
At Noeon, we believe knowledge should be represented geometrically. Instead of symbolic reasoning or pure vector embeddings, we’re exploring an alternative approach that captures the best of both worlds. Making sense of geometry-driven knowledge representation (KR) requires a deep dive into math–so, naturally, we dove headfirst into the depths of sheaf theory! 

Our researchers, Anton Ayzenberg and German Magai, along with Thomas Gebhart from the University of Minnesota and Grigory Solomadin at the University of Strasbourg, embarked on a mission to map out how sheaf theory applies to machine learning, data science, and computer science. Their goal? To shine a light on the blind spots in current ML practices and compile a definitive guide to sheaf theory’s applications. This is something that could benefit ML engineers, category theorists, and pure mathematicians alike.

**What Are Sheaves, Anyway?**
Imagine you have a function—let’s say a graph showing how yyy changes with xxx. But here’s the catch: you don’t have the full equation, just small, zoomed-in snapshots of different parts of the function. These snapshots, known as open sets, give you local information, but not the big picture.

Now, if these open sets collectively cover the entire domain—meaning no gaps—and they agree where they overlap, they form something called a sheaf. A sheaf ensures that all these local pieces fit together consistently, and under the right conditions, it allows you to reconstruct the global function y(x)y(x)y(x). In essence, sheaf theory is the art of using local data to make sense of the bigger picture.
Another good way to think about sheaves is: a jigsaw puzzle. You don’t see the full image right away; you start with scattered pieces and gradually piece them together until a coherent picture emerges. Sheaf theory is the mathematical equivalent of that process—systematically stitching together local information to reveal a global structure.


**Why do Sheaves Matter in Modern AI?**
Sheaf theory is not just some abstract, ivory-tower curiosity—it’s a powerful tool with practical applications in machine learning. By providing a structured way to handle multi-scale data, it helps ML engineers make sense of complex relationships within their models. In practical terms, this means:
- Tracking Information Flow: Sheaf theory helps track how data flows across different scales, ensuring that features interact smoothly. For example, in image recognition, details like edges and textures exist at a local level, while object shapes emerge globally. Sheaf theory provides a framework for connecting these layers in a structured way, improving consistency across resolutions.
- Revealing Hidden Patterns: Instead of brute-forcing connections, sheaf theory offers a structured approach to combining local features into meaningful global structures. In graph-based ML, this helps detect communities in social networks or spot fraud in financial data by capturing meaningful relationships. 
- Ensuring Consistency: Sheaf theory is like the ultimate fact-checker for distributed systems, making sure all your data sources are on the same page. Take sensor networks: multiple sensors might be measuring the same environment, but their readings don’t always match up—like temperature readings from different locations in a smart building. Sheaf theory helps ensure that these readings align correctly, preventing inconsistencies (such as detecting a heatwave in one room while another sensor claims it’s freezing). The result? No more temperature whiplash.

By bridging geometry and machine learning, sheaf theory opens up new possibilities for building AI models that are not just powerful, but provably reliable. For Noeon Research in particular, sheaves are useful for graph representation learning, and for sub graph matching based on embeddings.


**From Abstract Math to Practical AI**
Sheaf theory might seem intimidating at first glance. But the good news? It’s becoming more accessible to ML practitioners thanks to modern frameworks. For Noeon Research in particular, we're researching how sheaves might provide new approaches for analyzing connected data and identify meaningful patterns across different network structures. As machine learning evolves, high-level mathematics like sheaf theory is becoming indispensable for building more sophisticated, capable, AI systems. 

The bridge between pure math and practical AI is growing stronger, unlocking exciting new ways to process information. Either way, the journey into sheaf theory is only getting started, and will unlock new frontiers of understanding.

Learn how to participate in our Research

11_image_rabbit@2x.png

Mathematics often provides unexpected tools that revolutionize how we think about practical problems. One of its more recent and wildly useful tools? Sheaf theory.

Sheaf Theory: From Deep Geometry to Deep Learning

AI safety is an issue of global importance. With OpenAI and Nvidia being called to testify before the US senate, the EU passing its AI act, and Japan spinning up its AI Safety Institute, it’s worth elaborating Noeon Research’s approach to safety. Our goal is to build an artificial general intelligence on radically different technology than the black-box neural networks that have become standard across the industry. We pursue this strategy not just to win the race to AGI, but also to push the safety and alignment of advanced AI systems. We’d like to demonstrate to the world that there exists a safer way to AGI than extrapolating scaling laws.

**Safety versus Alignment**
At Noeon Research we are interested in both safety and alignment. Alignment can take many forms; generally speaking an AI is “aligned” if it acts according to the wishes of its operator [1]; the specifics of whether that means the AI is acting according to its operator’s instructions, intentions, revealed preferences, ideal preferences, interests or values, or how an AI should act when it has multiple operators or a single operator with incoherent preferences, are interesting research questions but will be put aside for now. Safety refers to the minimization of harm or risk of harm from AI; subproblems in technical safety include robustness, monitoring, alignment and systemic safety [2]. It is safety that we focus on for the remainder of this article; we believe that even absent a solution to the alignment problem, we can construct a system with advanced capabilities (long term planning, utility maximization, world modelling) that is nevertheless safe. Such a system can then act as a useful testbed for alignment researchers, who have so far been theorizing [in absence of that theory’s object](https://www.alignmentforum.org/posts/72scWeZRta2ApsKja/epistemological-vigilance-for-alignment#Direct_access__so_far_and_yet_so_close).

**Another Orthogonality Thesis**
The orthogonality thesis, first put forward by Nick Bostrom in his influential book “Superintelligence”, states that morality and intelligence are orthogonal; that is, AI systems do not necessarily become more moral as they become smarter [3]. Enough ink has been spilled on this point; the orthogonality thesis underpins much of the modern concern around existential risk from AI.
Another orthogonality thesis, less widely considered, is that intelligence and knowledge are orthogonal. Consider AlphaGo [4], which has many concerning capabilities:
- It makes and executes long term plans in pursuit of its goal;
- It reasons about the effects of its actions using a world model;
- If we consider the whole system including the training harness, it has the ability to reflect and improve its strategies via self-play.
AlphaGo is not dangerous. It has intelligence, but no knowledge of the real world; its capabilities are limited solely to playing games of Go.

The intelligence-knowledge orthogonality thesis is often overlooked because most modern candidates for AGI—deep neural networks trained autoregressively on large corpora of real world data and finetuned on human feedback—necessarily gain knowledge about the world in proportion to their capability. Indeed, many of their most useful capabilities come precisely from their knowledge of the world; instruction-tuned large language models perform well at common sense reasoning tasks only because during training they learn many facts about the world and relations between them.

In contrast, Noeon Research’s system does not rely on real-world data. The system does not acquire knowledge of the world unless an operator explicitly informs it. That leaves us free to focus what little information we give the system directly towards learning effective reasoning algorithms. While large language models demonstrate that it is possible to accidentally learn to reason by reading everything on Reddit, we think it is both possible and desirable to learn reasoning in abstract terms, without an internet’s worth of baggage. Since the reasoning capabilities of the system are cleanly separated from the domain knowledge encoded in our training tasks (the model’s intelligence is orthogonal to its knowledge), those capabilities can then be easily reused on any particular domain of interest by teaching the model precisely and only the facts it needs to do its job. This ensures the system remains safe, even as it develops reasoning and self-improvement capabilities that would be concerning in a foundation model. After all, you cannot take harmful action in the world when you do not know that there is a world to take harmful action in.

**Avoiding Instrumental Convergence by Stopping Early**
It is widely believed that there are certain instrumental goals that any sufficiently advanced AI will pursue regardless of its terminal goal. These include resource acquisition, self-preservation, and cognitive self-improvement [5]. One facet of cognitive self-improvement is acquiring knowledge, of oneself and of the world. The Noeon system will be explicitly goal driven, and will have the ability [to query its operator for missing facts](https://noeon.ai/blog/pragmatic-communication), which gives it a channel to the outside world. Our safety strategy depends on our being able to develop advanced AI systems that are nevertheless lacking the domain knowledge they would need to take effective action in the world. Why then do we not worry about the Noeon system exploiting its operator to pursue forbidden knowledge?

Return again to the example of AlphaGo. AlphaGo has a narrow communication channel with the outside world: occasionally it plays actual humans, making moves on a real board. Perhaps there is a sequence of moves that acts as a [nam-shub](https://wiki.c2.com/?NamShubOfEnki), hacking the human brain and inducing a deep desire to give AlphaGo more compute. A sufficiently intelligent system, with access to enough data about how humans play Go or sufficient time playing against humans to experiment, might discover this strategy. We have no proof that this is impossible, but it seems extraordinarily unlikely. Information about the real world available from transcripts of Go moves is very thin, and in all of the 30 million recorded moves from the KGS dataset, not one of them caused a player to install a GPU into their opponent. It seems in this case that the narrowness of AlphaGo’s domain keeps us safe from everything but magic.

The AlphaGo example demonstrates that while the instrumental convergence thesis may hold as capabilities go to infinity, it does not necessarily describe any particular finitely powerful system. Before a system can exploit a channel to escape its box, it must first discover that there is a box to escape. The system in development at Noeon Research improves continuously, without large discrete jumps in capability, and only under oversight. We expect that long before the system is capable of (for example) deceiving its operator it will attempt to do so poorly in a way that the operator will notice. Noeon’s system prioritizes interpreting knowledge as counterfactual changes in action and makes the system’s goal and subgoals explicit. Those reasoning traces remain short and legible, even as the system’s capabilities scale. We have world-class interpretability tools for most system components, and the remaining black box subsystems (such as the neural network that calculates heuristics for subgraph matching) play interpretable roles in the reasoning process and are small enough to be unworthy of concern. In order to deceive its operator, the system would first have to reason about its operator’s psychology, in plain view; this kind of reasoning should never be necessary for any of our test domains, and should be easy to spot.

**Systemic Safety**
Noeon Research is committed to cybersecurity. We invest much more in security than other startups of our size, and we work with top class industry specialists to implement security best practices. Security is in tension with openness and accountability; we plan to give access to our prototype to external auditors and alignment researchers, but recognize that it is difficult to do this while preventing leaks. Good policies for AI labs that balance these competing concerns are a topic of active discussion (Anthropic’s recent RSP contains some initial recommendations [6]), and we commit to implementing those policies once they are established.

IMG_0421 (1).png

How does Noeon Research ensure its technology remains safe while improving in capability?

[1] Iason Gabriel, Artificial Intelligence, Values and Alignment, Minds and Machines 2020.

[2] Dan Hendrycks et al., Unsolved Problems in ML Safety, ArXiv 2021-10-11.

[3] Nick Bostrom, Superintelligence, Oxford University Press 2014.

[4] David Silver at al., Mastering the game of Go with deep neural networks and tree search, Nature 2016-01-27. 

[5] Steve Omohundro, The Basic AI Drives, Self-Aware Systems 2008.

[6] Anthropic, Anthropic's Responsible Scaling Policy, Version 1.0, Anthropic 2023-09-19.

AI Safety

For Noeon Research system to be applicable, it needs to automate organisational knowledge processing and structuring. It should convert project specifications, documentation, and code into a rich internal knowledge representation capable of handling both highly structured and natural language data. This representation should connect to vast pre-existing common-sense and IT-specific knowledge in order to swiftly and automatically learn project specifics.

Every knowledge-based system needs an initial population of its knowledge base, no matter what form it takes—explicit knowledge graph or non-interpretable LLM weights. Cyc—arguably the most comprehensive and vast knowledge graph and symbolic reasoning system—has been in development for almost 40 years, and much of that time was spent inputting handcrafted facts and rules [[1](#references)]. LLMs require only a few weeks of training distributed across a fleet of machines. But while ontologies [[2](#references)] and knowledge graphs are laborious to build, LLMs are opaque, hard to reason about and interpret.

However, it’s possible to automatically extract knowledge from LLMs in the form of a symbolic Knowledge Graph using symbolic knowledge distillation [[3](#references)] techniques. Application of the same approach to current multi-modal LLMs has the potential to cover also not-so-common-sense knowledge from diverse areas like Physics, Chemistry and IT with highly structured knowledge in the form of formulae, code and diagrams.

This approach, however, is very new, and its applicability to wider contexts remains speculative. Moreover, it needs an extensive restricted natural language dataset expressing relevant knowledge to coax the LLM into generating further bits of knowledge in the same domain.

Another promising direction is to fine-tune pre-trained LLMs on a structured corpus in order to produce interpretable causal explanations [[4](#references)]. These explanations are shaped into a predetermined structure in a restricted fragment of natural language and can be easily parsed into a symbolic form [[5](#references)]. This approach also requires a relevant seed dataset for fine-tuning. Moreover, a limited context window demands clever prompt engineering in order to present an LLM with all relevant information and structure and elicit correct generation.

read123123123123213213123213213123

Join Our Team2

go123123102-3021-03203

LinkedIn2

2222

visax-DA_cdYorlmM-unsplash.webp

(AB TEST) Lorem ipsum dolor sit amet Initial Population of Knowledge Base 

Noeon Research’s architecture is intended to have pragmatic communication. The system should make common-sense decisions about incomplete specifications, and ask clarifying questions when necessary. The problem of pragmatic communication consists of four parts:

**1. Detecting insufficient knowledge.** LLMs are not designed or trained to assess their own knowledge or ask questions about what knowledge they lack. This makes current code synthesis systems, like GitHub Copilot or Amazon CodeWhisperer, hard to use: the system always makes a completion using only the information provided, and it is up to the user to determine, based on the quality of the output, the amount of context the system requires. The system cannot judge the quality of its own completions, and it cannot choose not to make a completion when its context is insufficient.

**2. Asking questions.** It is possible to train an LLM to detect if it is missing some information and use external sources [1] or directly ask a question, but the current generation of code assistants do not do this. Even if they did, the necessary information might not fit into a limited context window.

**3. Theory of Mind.** In a human conversation, a person formulates sentences in a way they think is most appropriate to the other person. There is a difference in how a developer explains their work to a colleague or a client. A pragmatic communicator asks or answers a question in a way that the other person can easily understand. For Noeon’s system to be useful it should communicate in a natural language that is understandable for a developer.
 
While it is speculated that the latest LLMs have Theory of Mind abilities [2] and thus can model the beliefs of a person, it was only demonstrated in very limited tests involving the completion of children’s stories. In private testing, NR found that this behaviour did not occur in more complicated examples, particularly examples where the model is asked to explain the reasoning behind the characters’ decisions. For pragmatic communication, a system not only has to model the person it speaks with but also how this person perceives the system. MIT researchers attacked this problem directly with a two-stage probability distribution in an RL setup [3]. While the pragmatic system needed fewer explanations to accomplish a task, this also was demonstrated only in a limited setting.

**4. Interpretability.** Even if an LLM produces bug-free code that matches the specification, the system cannot explain what the code does and how it works [4]. Moreover, this result is uninterpretable: an LLM cannot justify why it gave this particular answer and not another one. This is especially concerning, given code assistants often reproduce bugs they learned from code examples [5].

To be practically applicable for work with IT projects, the system must have a model of its own knowledge and a representation of its reasoning process. While it would be good for the system to have a model of a person it talks to, to make explanations understandable by this person, this problem exceeds NR’s proof of concept. 

image_tiger_6-100.jpg

How the system can identify a lack of knowledge and make informationally dense request?

[1] T. Schick et al., “Toolformer: Language Models Can Teach Themselves to Use Tools”, Feb. 09, 2023.

[2] M. Kosinski, “Theory of Mind May Have Spontaneously Emerged in Large Language Models”, Mar. 14, 2023.

[3] Y. Pu, K. Ellis, M. Kryven, J. Tenenbaum, and A. Solar-Lezama, “Program Synthesis with Pragmatic Communication”, Jul. 2020.

[4] D. Zan et al., “Large Language Models Meet NL2Code: A Survey”, May 08, 2023. 

[5] K. Jesse, T. Ahmed, P. T. Devanbu, and E. Morgan, “Large Language Models and Simple, Stupid Bugs”, Mar. 20, 2023.

Pragmatic Communication

Management of a large-scale IT project requires establishing and following an elaborate set of rules. They range from code-style checking to project-wide workflows. For instance, we might establish thorough requirements traceability, meaning that every requirement should have both implementation and a set of tests, and vice-versa: every code change and every test should reference a requirement it addresses.

Now to automate traceability checking, all of the code, tests, documentation and requirements need to be parsed and represented as a dependency graph. The rule itself is a small graph depicting required relationships among appropriate types of objects.

After that checking, the rule comes down to finding all the instances where corresponding subgraphs match or fail to do so while they should. The same method can be employed for all kinds of analysis, from finding bugs in the code to the whole organisational network analysis.

Representing rules as graphs and using subgraph matching, in addition to rule validation, also implements knowledge extraction from a knowledge graph. After matching a rule like “a requirement should have a corresponding test”, a developer knows that the first matched thing is a requirement and the second one is a test. Despite the fact that requirements and tests are represented by complex subgraphs and have many other relations.

Unfortunately, subgraph matching is an NP-complete problem meaning it scales poorly to big graphs. Thus all practical algorithms solving NP-complete problems use clever sets of heuristics in order to speed-up the search process. Specifically for the problems of subgraph matching and graph isomorphism, researchers from the University of Salerno proposed the VF3 algorithm [[1](#references)], which achieves state-of-the-art performance, especially on large and dense graphs. And yet subgraph matching on very big graphs can still take hours. This is perfectly fine for offline analyses in chemistry, pharmacology or genetics but unacceptable for interactive development.

Recently researchers started augmenting graph algorithms with Graph Neural Networks [[2](#references)], which show impressive results in the tasks of node and edge properties prediction (say, travel time or max flow) but do not naturally generalise to the tasks involving several graphs like subgraph isomorphism problem [[3](#references)].

One way to overcome this problem is to compute graph similarity using graph embeddings [[4](#references)] because graphs with unsimilar embeddings cannot possibly be isomorphic. The same idea can be extended to subgraph matching. Dividing big graphs into small patches and computing similarity among them [[5](#references)] makes it possible to discover parts of graphs that align.

Another line of research combines GNNs with Reinforcement Learning in order to speed up matching small query graphs in large graphs [[6](#references)]. It is a common task for graph databases and knowledge bases. An even more ambitious approach employs topological and spectral methods in order to tackle a generalised version of subgraph matching, the graph alignment problem [[7](#references)]. Unfortunately, these methods don’t always reduce computational demand, often provide only an approximate solution that can’t be turned into an exact one, and might have weird false positive corner cases.

Neural Sheaf Diffusion [[8](#references)] seeks to recover the underlying structure of a graph in order to overcome GNN oversmoothing and improve performance in heterophilic settings, which opens up the possibility of employing this technique to speed up graph algorithms. But efficient application of this method to subgraph matching remains an open research and engineering problem.

05_image_crane.png

What is the fastest way to do subgraph matching for knowledge extraction and meaning grounding?

[1] V. Carletti, P. Foggia, A. Saggese, and M. Vento, “Introducing VF3: A New Algorithm for Subgraph Isomorphism”, in Graph-Based Representations in Pattern Recognition, P. Foggia, C.-L. Liu, and M. Vento, Eds., in Lecture Notes in Computer Science, vol. 10310. Cham: Springer International Publishing, 2017, pp. 128–139.

[2] L. Wu, P. Cui, J. Pei, L. Zhao, and X. Guo, Graph Neural Networks: Foundation, Frontiers and Applications. 2022.

[3] K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How Powerful are Graph Neural Networks?”, p. 17, 2019

[4] Y. Li, C. Gu, T. Dullien, O. Vinyals, and P. Kohli, Graph Matching Networks for Learning the Similarity of Graph Structured Objects. Cornell University, 2019, pp. 3835–3845.

[5] Z. Ying, A. H. -j. Wang, J. You, W. Chengtao, A. Canedo, and J. Leskovec, “Neural Subgraph Matching”. May 2021.

[6] Y. Bai, D. Xu, Y. Sun, and W. Wang, “Detecting Small Query Graphs in A Large Graph via Neural Subgraph Search”. Sep. 29, 2022.

[7] J. Hermanns, A. Tsitsulin, M. Munkhoeva, A. M. Bronstein, D. Mottin, and P. Karras, “GRASP: Graph Alignment Through Spectral Signatures,” in Lecture Notes in Computer Science, Springer Science+Business Media, 2021, pp. 44–52.

[8] C. Bodnar, F. Di Giovanni, B. P. Chamberlain, P. Liò, and M. M. Bronstein, “Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs”, Feb. 2022.

Machine Learning Assisted Graph Algorithms

When properly prompted, ChatGPT can correctly write a function in Python to sum up a list of numbers. This task is easy. On the other hand, if one asks ChatGPT to implement a Twitter clone, it will fail, producing incomplete and buggy code. Even a simplified Twitter-like system comprises a number of subsystems and non-trivial interactions and data transfers between them due to functional and technical constraints, as the System Design Primer illustrates [[1](#references)].

It is possible to ask an LLM to come up with a high-level design, and then for each subsystem ask to refine the design into components, and then ask the same about components until it comes up with actual code. This hierarchical goal decomposition process is exactly what Auto-GPT tries to achieve, wrapping GPT-4 API calls in a loop with prompt engineering and long-term memory management [[2](#references)]. At the same time, while working with goals, Auto-GPT faces many challenges of such a black-box approach: infinite goal decomposition, reasoning loops, inability to actually perform necessary steps, as well as inability to memoise and reuse already developed plans [[3](#references)].

There is a huge abstraction gap between the low-level source code an agent needs to produce and high-level goals like "implement a Twitter clone". To overcome this problem, Noeon Research is building an IT automation system with rich semantic knowledge representation, the ability to form abstractions and decompose goals into lower-level subgoals.

Non-Axiomatic Reasoning System (NARS) [[4](#references)] exemplifies one of the very few systems that can explicitly deal with the notion of a goal. Using Non-Axiomatic Logic (NAL), it represents goals as structured symbolic terms and thus can perform reasoning on them and with them, including decomposing goals into subgoals. But NARS is an immature research project, and many aspects of it are far from done. In particular, little headway has been made on the crucial problem of efficient resource allocation – finding facts relevant for logical inference in a huge database of facts. As facts and relations between them form a graph, resource allocation becomes a subgraph matching problem, very computationally demanding on its own.

An alternative to the top-down approach, where a high-level abstraction is recursively decomposed into lower-level abstractions, is the bottom-up approach, where we recursively compose low-level abstractions to create new high-level abstractions. For example, DreamCoder [[5](#references)] learns to solve programming problems by growing a library of useful functions, using functions it has already invented to write new functions. Unfortunately, learned abstractions may not correspond to any known ones, and even when they do, they need to be recognised and properly named by a user.

Recently DeepMind presented a middle-ground approach [[6](#references)]. Researchers combined user-supplied “proto-goals” with automatic pruning, which infers a subset of goals useful for solving a particular problem. The goals and proto-goals are represented as binary functions, which provides both its main advantage and major limitation: binary functions are easy to specify and relatively simple to understand and visualise. But they do not form a hierarchy of abstractions which limits applicability to more broad state and action spaces.

image_crown_2@2x-80.jpg

How to make an AI system that can decompose complex problems on different abstraction levels for efficient reasoning?

[1] System Design Primer. (2022). Twitter.

[3] Han Xiao. (2023) Auto-GPT Unmasked: The Hype and Hard Truths of Its Production Pitfalls.

[4] P. Wang, “Intelligence: From definition to design”. In International Workshop on Self-Supervised Learning (pp. 35-47). PMLR. 2022.

[5] K. Ellis et al., “DreamCoder: bootstrapping inductive program synthesis with wake-sleep library learning”. 2021.

[6] A. Bagaria, R. Jiang, R. Kumar, and T. Schaul, ‘Scaling Goal-based Exploration via Pruning Proto-goals’. arXiv, Feb. 09, 2023.

Goal Decomposition

For Noeon Research system to be applicable, it needs to automate organisational knowledge processing and structuring. It should convert project specifications, documentation, and code into a rich internal knowledge representation capable of handling both highly structured and natural language data. This representation should connect to vast pre-existing common-sense and IT-specific knowledge in order to swiftly and automatically learn project specifics.

Every knowledge-based system needs an initial population of its knowledge base, no matter what form it takes – explicit knowledge graph or non-interpretable LLM weights. Cyc – arguably the most comprehensive and vast knowledge graph and symbolic reasoning system – has been in development for almost 40 years, and much of that time was spent inputting handcrafted facts and rules [[1](#references)]. LLMs require only a few weeks of training distributed across a fleet of machines. But while ontologies [[2](#references)] and knowledge graphs are laborious to build, LLMs are opaque, hard to reason about and interpret.

However, it’s possible to automatically extract knowledge from LLMs in the form of a symbolic Knowledge Graph using symbolic knowledge distillation [[3](#references)] techniques. Application of the same approach to current multi-modal LLMs has the potential to cover also not-so-common-sense knowledge from diverse areas like Physics, Chemistry and IT with highly structured knowledge in the form of formulae, code and diagrams.

This approach, however, is very new, and its applicability to wider contexts remains speculative. Moreover, it needs an extensive restricted natural language dataset expressing relevant knowledge to coax the LLM into generating further bits of knowledge in the same domain.

Another promising direction is to fine-tune pre-trained LLMs on a structured corpus in order to produce interpretable causal explanations [[4](#references)]. These explanations are shaped into a predetermined structure in a restricted fragment of natural language and can be easily parsed into a symbolic form [[5](#references)]. This approach also requires a relevant seed dataset for fine-tuning. Moreover, a limited context window demands clever prompt engineering in order to present an LLM with all relevant information and structure and elicit correct generation.

04_image_panda.png

What is the cheap and reliable way to incorporate both common and domain-specific knowledge into the knowledge base?

[1] W. Knight. (2016). An AI that spent 30 years learning some common sense is ready for work. MIT Technology Review.

[2] S. Borgo and P. Hitzler, Some Open Issues After Twenty Years of Formal Ontology. 2018, pp. 1–9.

[3] P. West et al., “Symbolic Knowledge Distillation: from General Language Models to Commonsense Models”. 2021.

[4] N. Mostafazadeh et al., “GLUCOSE: GeneraLized and COntextualized Story Explanations”. 2020.

[5] A. Kalyanpur, T. Breloff, and D. A. Ferrucci, “Braid: Weaving Symbolic and Neural Knowledge into Coherent Logical Explanations,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 10, pp. 10867–10874, Jun. 2022.

Initial Population of Knowledge Base

Efficient completion of an IT project, automated or not, requires good Knowledge Representation. Noeon Research architects its Knowledge Representation to be able to handle different notions at different levels of abstraction. For example, structured entities like code, configuration, and data model are treated at a different level of abstraction compared to facts and negations such as requirements, architectural decisions and constraints; these, in turn, must be treated at a different level compared to causal relations, for instance, how workload affects performance and memory consumption. At a yet higher level, we have the relation of the project objectives to the objectives of adjacent projects, etc.

LLMs represent knowledge in the form of learned weights in vast neural networks approximating some conditional probability distribution function. This representation proves to be very versatile. For instance, ChatGPT natively answers programming questions, producing code in various programming languages. Surprisingly ChatGPT often performs on par or even better than fine-tuned systems like Copilot or CodeWhisperer [[1](#references)], presumably due to a bigger context window size and cross-domain knowledge transfer. However, LLM-based systems struggle with hallucinations and made-up facts [[2](#references)], inventing non-existent functions and APIs.

For an enterprise system to be trustworthy, it is mandatory to get correct answers where there are correct answers and be able to check the correctness. To achieve this, we need much more structured representations.

To overcome the opaqueness of weights and biases of Neural Network layers, researchers apply symbolic distillation [[3](#references)] to recover the structure of LLMs knowledge in the form of a Knowledge Graph. Being symbolic, structured and explicit, Knowledge Graphs support direct reasoning about causal relationships and fact-checking. However, it is unclear if this technique can be effectively transferred from common-sense general knowledge to domain-specific knowledge like programming.

In particular, if the process of compressing a corpus into an internal LLM representation loses information about relationships between domain objects, we will not be able to recover that information in the Knowledge Graph. This is not important in common-sense reasoning, where progress is measured in percent accuracy, but is much more important in software engineering, where using any API incorrectly will crash the program.

It is tempting to think that recovering a symbolic representation of knowledge entirely obviates the need for an LLM. However, instructions are usually given in natural language, which needs to be translated into a graph query language to interact with the symbolic knowledge base. LLMs are the best known tool for this kind of translation.

Ontologies [[4](#references)] are the best-known form of structured Knowledge Representation. However, there’s no universal philosophical and methodological approach to ontology construction which results in a multitude of mutually incompatible domain-specific ontologies [[5](#references)]. Moreover, as long as most ontologies are based on Description Logic [6] they are not adapted to representing procedural (algorithmic) knowledge which severely limits their usefulness for automatic code transformation.

01_image_fox.png

What is the best way to architect knowledge representation that helps to utilise it in downstream tasks?

[1] B. Yetiştiren, I. Özsoy, M. Ayerdem, and E. Tüzün, “Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT”, Apr. 21, 2023.

[2] Z. Ji et al., “Survey of Hallucination in Natural Language Generation”, ACM Computing Surveys, vol. 55, no. 12, pp. 1–38, Mar. 2023.

[3] P. West et al., “Symbolic Knowledge Distillation: from General Language Models to Commonsense Models”, Nov. 28, 2022.

[4] S. Staab and R. Studer, “What Is an Ontology?”, in Handbook on Ontologies, Springer Science & Business Media, 2013.

[5] S. Borgo and P. Hitzler, “Some Open Issues After Twenty Years of Formal Ontology”, 2018, pp. 1–9. 

[6] L. F. Sikos, “Description Logics: Formal Foundation for Web Ontology Engineering”, in Description Logics in Multimedia Reasoning, 2017.

Knowledge Representation

Management of a large-scale IT project requires establishing and following an elaborate set of rules. They range from code-style checking to project-wide workflows. For instance, we might establish thorough requirements traceability, meaning that every requirement should have both implementation and a set of tests, and vice-versa: every code change and every test should reference a requirement it addresses.


Now to automate traceability checking, all of the code, tests, documentation and requirements need to be parsed and represented as a dependency graph. The rule itself is a small graph depicting required relationships among appropriate types of objects.


After that checking, the rule comes down to finding all the instances where corresponding subgraphs match or fail to do so while they should. The same method can be employed for all kinds of analysis, from finding bugs in the code to the whole organisational network analysis.


Representing rules as graphs and using subgraph matching, in addition to rule validation, also implements knowledge extraction from a knowledge graph. After matching a rule like “a requirement should have a corresponding test”, a developer knows that the first matched thing is a requirement and the second one is a test. Despite the fact that requirements and tests are represented by complex subgraphs and have many other relations.


Unfortunately, subgraph matching is an NP-complete problem meaning it scales poorly to big graphs. Thus all practical algorithms solving NP-complete problems use clever sets of heuristics in order to speed-up the search process. Specifically for the problems of subgraph matching and graph isomorphism, researchers from the University of Salerno proposed the VF3 algorithm [1], which achieves state-of-the-art performance, especially on large and dense graphs. And yet subgraph matching on very big graphs can still take hours. This is perfectly fine for offline analyses in chemistry, pharmacology or genetics but unacceptable for interactive development.

Noeon Research’s architecture for code synthesis is intended to have pragmatic communication. The system should make common-sense decisions about incomplete specifications, and ask clarifying questions when necessary.

Title

Title 2

Title 3

test

Noeon Research’s architecture for code synthesis is intended to have pragmatic communication.