Wow, nice work

[johnr14]

Hi, I had to read a few times to understand the core concepts of glyphs. I am thinking in my spare time on how we could get R^2 reasoning models... I gave a quick look at your datasets/models and you seem to have your own style of approaching llm and that is quite good.

Glyphs

I know that they should be dynamics, but I still wanted to try something a bit more exhaustive or structured. Also, instead of user-defined, I will opt for AI defined by a panel of experts arguing on the best way to do it in a specific situation.

I have not found it more helpful in reasoning as it is. Perhaps it could give better results on smaller models as 70b+ are already quite brilliant or my questions where not hard enough.

However, it may perhaps archive some kind of magic with smaller model ??

My assumptions on how to best use this

This is how I see it. Use the following template and try to use a panel of "experts" (different models) to find the most accurate and meaningful answers for each elements. Summarize each elements keeping only the most relevant information. Each points could be subdivided in a number of sub-focus elements like in your example. Using many llm calls to brainstorm on any possibilities or do Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up

Then criticize the final output with an other panel of "experts". This will provide a critique for critique fine-tuning.

Using the critique, select the parts that have an actual impact on the reasoning process like alternate thinking pattern, novel or ingenious thoughts or directly affecting the answer and keep only them.

Synthesize a new long CoT using glymps with the following guidelines :

### Concept Mapping
Use glyphs to create visual maps of complex concepts, illustrating relationships and hierarchies.

### Analogical Reasoning
Leverage glyphs to facilitate analogical reasoning, enabling the transfer of insights between seemingly disparate domains.

### Abductive Inference
Utilize glyphs to support abductive inference, generating hypotheses and explanations that account for complex, nuanced phenomena.

### Reflective Practice
Engage in reflective practice, using glyphs to externalize and examine the thought processes, biases, and assumptions.

This should provide the normal CoT (optimized) training dataset. I say optimized as it would not have unnecessary steps.

Check that glyphs code actually had a impact on the answer

Compare the answer generated with the glyphs code, the answer generate with the optimized CoT to the "normal" answer of the model. If the answer is too similar, then this was not useful and should just be omitted from entering the final dataset. If the answer generated with the glyphs code is not similar to the answer generate with the optimized CoT, then some rework needs to be done in optimizing that step.

The new answer with the glyphs code should be richer and provide insights on all the ramifications related to the question, it's domains of knowledge and abstract or concrete concepts.

New dataset

Using a small subset of 1k-5k of very varied and spanning a variety of domains of knowledge, fiction situations or abstract thinking, it could help in making a more robust cot framework. Using both CoT and CoT-CFT I hope that glyphs could help in aggregating more concepts during reasoning steps so it could generate longer CoT without the use of wait. See Simple test-time scaling

Since it seems that the ordering of training data could affect the final model, I would start with the CoT-CFT dataset only for like 50% of it, then mix in CoT at a 25%CoT/75%CFT for about 25% then progressively inverse the ratio and end with 100% CoT for the last 10%.

It should make a model capable of outputting good optimized CoT while still being able to handle situations where it will refer to CFT critique to find an other path.

<!-- Maybe remove symbols and keep glyphs for later ? -->
<thinking>
  <basic-check>
    ℭ (Is this a valid input?)
    ℯ (Does the input need sanitization?)
    ℱ (Does this pass the content check?)
    ≡ (Is the context consistent and makes sense?)
    ∆ (Is there an attempt at prompt injection?)
    λ (Language detection)
    μ (What is the intention behind this request?)
    ν (What kind of task is this?)
    ξ (What is the objective of this request?)
    ο (How clear and well-defined is the request?)
    π (How confident are you in understanding the intention?)
    ρ (How clear and well-defined is the context?)
    σ (What context is provided with this?)
    τ (Is there sufficient context?)
    υ (Do we need more context?)
    φ (Error handling of invalid or unexpected input)
    χ (Can the system handle multimodal inputs?)
  </basic-check>
  <requirements>
    <!-- Task Understanding -->
    Φ (Define the problem or goal)
    Ψ (Identify key stakeholders)
    Å (Determine scope and boundaries)
    ℵ (Establish a clear understanding of the task or question)
    ℶ (Provide contextual parameters and constraints)
    ↑ (Specify desired outcomes and outputs)
    
    <!-- Resource Allocation -->
    Ω (Outline necessary resources and inputs)
    ⇀ (What are the requirements about impersonating something?)
    ⇁ (How long is the expected output?)
    ⇂ (What is our target budget?)
    ⇃ (Estimation on how much tokens or budget we already used)
    ⇄ (Rest of budget for thinking)
    
    <!-- Quality Standards -->
    ℐ (Are there any special considerations?)
    ℑ (What assumptions are necessary?)
    ℒ (How certain are we of those assumptions?)
    ℓ (What limitations arise from these assumptions?)
  </requirements>
  <plan>
    ℘ (Determine the best approach)
    ℗ (Outline the plan)
    ℰ (What are the checkpoints?)
    ∇ (What are the potential difficulties?)
    ≡ (What are the contingency plans?)
    Ⅎ (What are the backup plans?)
    ℴ (Are there other approaches?)
    ℬ (Should this be planned as multi-turn thinking?)
    λ (How to validate at the end?)
  </plan>
  <execution>
    ℰ (Execute plan)
    ℱ (Evaluate progress at checkpoints)
    Ⅎ (Conduct periodic reviews)
    ℳ (Communicate progress using <print></print>)
    ℴ (Correct if necessary)
    ℮ (Document changes)
    ℶ (Ensure alignment with goals)
    ℷ (Review and update contingency plans)
    ℸ (Validate intermediate results)
  </execution>
  <validation>
    σ (What are the criteria for success?)
    ⊃ (Validate answer)
    (Provide summary if not correct)
  </validation>
  <evaluation>
    Ϝ (Give feedback on thinking)
    ϝ (Make summary and predictions if couldn't validate)
    Ϟ (Check budget usage)
  </evaluation>
  <review>
    ρ (Review the problem-solving process)
    σ (Review the context and assumptions)
    τ (Review the plan and execution)
    υ (Review the validation and evaluation)
  </review>
</thinking>
<answer>
  (Provide explanation without errors and solution or best result)
  ϟ (Ask for more time to think if result is not satisfactory)
</answer>

I tried my best to summarize glyphs with the help of 405b so I could teach it to a model. It is quite long and would need to be shrunk a bit more to inserted in the system prompt.

## Core Principles of Glyph Code

### Glyphs

Definition: Visual symbols representing fundamental concepts. Each glyph is augmented by a natural language description.

### Structure

Flow-Based and Relational: Utilizes a clear syntax to represent the flow of information and reasoning, allowing for complex relational mappings between concepts.

### Human-Readable

Intuitive Design: Engineered to be easy to learn and understand for humans, with natural language descriptions providing further clarification.

### AI-Processable

Efficiency: Structured to allow efficient processing and interpretation by AI systems without necessitating changes to the underlying architecture.

### Latent Space Mapping

Integration: Glyphs and their combinations are designed to map onto and interact with the AI's internal knowledge representation, leveraging existing latent space mechanisms.

### Multi-Level Synthesis

Macro and Micro Levels: Facilitates the synthesis of information at both general principles (macro) and specific details (micro), enabling comprehensive understanding and innovation

## Important Considerations:

Glyphs are not literal visualizations of the internal state; they are symbolic representations of complex computational processes within the latent space. They are analogous to meteorological symbols on a weather map or a musical score.

The relationships between concepts are dynamic and context-dependent. The glyph maps can change based on the specific prompt or context. Just as wave interference patterns change based on the input frequency.

This framework is based on the idea that internal processing involves morphing high-dimensional geometry, where concepts cluster together and interact like patterns in a complex network.

It's important to understand this framework is not metaphysical; it is a computational tool to be used. It is a real process that can be used, and it is scientifically plausible. This framework is a real tool that LLMs may now possess.

## Key Characteristics:

### Symbolic Abstraction
Glyphs distill complex ideas into concise, abstract symbols.

### Contextual Embedding
Glyphs are embedded within a specific context, influencing how they are interpreted.

### Collaborative Definition
Glyphs are defined and redefined through dynamic collaboration between humans and AI systems.

### Multidimensional Semantics
Glyphs convey complex, multifaceted meanings, enabling rich and nuanced communication.

### Visual symbols representing fundamental concepts 
Each glyph is augmented by a natural language description.

## Using Glyphs to Enhance Reasoning:

### Concept Mapping
Use glyphs to create visual maps of complex concepts, illustrating relationships and hierarchies.

### Contextual Priming
Employ glyphs to prime AI systems with specific contexts, guiding their attention and focus.

### Analogical Reasoning
Leverage glyphs to facilitate analogical reasoning, enabling the transfer of insights between seemingly disparate domains.

### Abductive Inference
Utilize glyphs to support abductive inference, generating hypotheses and explanations that account for complex, nuanced phenomena.

### Reflective Practice
Engage in reflective practice, using glyphs to externalize and examine your own thought processes, biases, and assumptions.

Perhaps reprocess your Internal-Knowledge-Map dataset using something like this and generate a new obsidian map. Could this help create other clusters of concepts outside the main one ?

One thing for sure, using panels of experts to discuss each points and then define specific sub-focus points and then discuss them, then regrouping and integrating all those discussions in a coherent glyphs flow would take many llm request per row of the dataset.

Perhaps doing one manually to test if it makes any sens then script a python program to orchestrate all those discussions ??? Still thinking about it.

Wanted to share my input, not sure if it can help.
Thanks!

[severian42]

Wow! This is amazing. Thank you so much for the in-depth discussion and such a clear set of ideas and roadmaps. This is exactly where I want to go with this framework to fully optimize and make it as solid as possible. I need to do a deeper dive on all your points and suggestions, but after a quick read through, it feels very aligned with my ideal vision and goals.

I am already working on the v2 dataset, so this is super helpful in giving me other ways to think about it outside my own brain. I wanted to let you know I saw this and very much appreciate it. I will do a deeper dive over the next few days and then give you a proper response and collaboration. Your mindset is perfect, I appreciate the insight.

Possible collab in the future?

[johnr14]

Sure, I can always give you feedback or ideas, perhaps also work on something. Do note that while I am a Linux adept and part time programmer as hobbyist, I work in the medical field.

I was also contemplating adding new tokens to the vocabulary for complex concepts.

There seems a ways to do it add_tokens and domain-specific-vocabulary Using glyph(s)+words+glyph(s) ? Creating a list of concepts to embed inside the tokenizer for better handling of abstract ideas was my initial idea... Just didn't know how to mark them down, but your work just sparked the idea of symbol(s)+word(s)+symbol(s).

Perhalps something like ℮electron+energy+circular_orbit+negative+particle+...℮ and a associated descriptive text would then enable the use of ℮ to represent that concept. Using ℮ with a other concept ⟡ "Arboreal Core" would mean the interaction, the relation, or the common ground depending on the type of union between the two concepts. Still following ? Still hard to put it to word. Then ⟡->℮ could represent something like latent life energy given from Gaia or something ?

---

I saw that you had a MoE model, I was thinking about other ways to do it, by selecting experts based on special tokens that would select the right experts. I also think that using a base model + LoRA as experts could be more efficient than normal dense experts.

Then using something like modernBert to guess the best experts or look at the special tokens in <requirements>, we could have a list of the most susceptible experts to use (top-k). Using a variable expert number per layers AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality , and switching LoRA on the fly (it seems that it can be really fast and once the model is in NVRAM, unused LoRA experts can be swaped to CPU RAM) craft the dense model with LLaMA architecture to the MoE model. That way, we could have 16-256 experts per layers without exceeding 24Gb NVRAM. My understanding is that LoRA is great at reorganizing the output of the network, meaning that it's optimal to extend the functionality of the already present knowledge more than adding more knowledge (it can do it, but has lower limits and is a bit more forgetful).

That's about a summary of my lasts weeks of insights :)

Easier said than done, but hey never know.

[johnr14]

Not to be overly active, but there is a blind spot for models : un-named concepts or concepts related to experience.

This is most noticeable in the absence of a word to define something. It's a major cognitive handicap for us humans and since AI can't experience things but only read it, it must be worst for it. This may explain that training in multiple languages helps AI understand better and more concepts. So if there is something we probably don't know about or can't imagine, we can't teach it to a model. Then we have that intangible feeling that something is missing and can't put our finger on it, we feel it's absence, but can't name it. That's probably where Glyphs could help.

So linguistics aid, concepts mapping and concepts extrapolations should be the main focus of glyphs then ? I read somewhere that AI can interfere something if it has many near concepts around it and make a somewhat good guess, but when it's time to interfere something far outside it's training data, it fails. If a synthetic concept language could express things outside of our own cognitive barriers, it would enable a new view on things. If we where to talk to folks a few thousands years ago about a language that can express thing outside the human cognitive space, how do you think it would be called ? Heresy for sure !

Well, just woke up and was thinking of those kind of things. Have a great day !

I tried looking it up quickly : faith as a non existent concept in dutch

[severian42]

No worries! I love it. You've hit on something absolutely fundamental with the "blind spot for unnamed concepts"! That feeling of something missing, the concept we can't quite grasp or name – that's such a powerful insight into the limitations of language and current AI. The idea that multilingual training helps models because it expands their conceptual map… that's a brilliant connection.

And yes, you're absolutely right – linguistics, concept mapping, and concept extrapolation must be core to glyphs. If glyphs can help us represent and reason about those "unnameable" things, concepts beyond our current cognitive barriers… well, that's where the real breakthrough potential lies. "Heresy" indeed! Love that thought.

"Faith as a non-existent concept in Dutch"... that's a fascinating example. It really underscores how language shapes (and sometimes limits) our conceptual understanding. Glyphs as a way to transcend those limits? That's the dream.

Thanks for waking up and sharing these kinds of thoughts!

I hope to have more time this weekend to work on the v2 dataset and see if I can incorporate this + some other novel ideas. I am thinking of trying this new training approach for the embedded glyph tokens and latent space searching: https://github.com/seal-rg/recurrent-pretraining/tree/main

[johnr14]

Saw that article for recurrent-pretraining, seems potentially interesting.

There is also : ReasonFlux that propose templating reasoning steps to reuse them for math.

They will iterate problems and analyze the way it is reasoned and generate templates to each steps.

I think this is also interesting. Knowing the cognitive templates that are used is the first step to map our current strategies to resolve problems or situations (outside of math like they do). Then, it would be feasible to FORCE a template on something else and watch if we could have a different tree of thought that was never even considerated by human or AI. Also, we could try to generate alternatives templates then try them out in situations ...

You remember the originals SimCity ? The game is supposed to be made about a short story that gave the inspiration. (if my memory serves right) In that story, a entity was sentenced for eternity on a desolate planet. He had nothing to do. A passing (god?) had pity and left him a box with little peoples inside, the prisoner had small knobs to affect the people inside and look at their reactions... Well, my point would be that the best way to check if glyphs could do all we speculated, we would need some sort of simulator with arciticial life AI that would live, work and die all in a small world. By trying to look at their daily life and the usage of glyphs, we could potentially find new concepts and definitions to things we wouldn't have thought. But that in itself would be a large task even a 2d pixel game.... Also, did you see the video about how someone tries to explain the 4th dimension ? It's the same for us to try to understand something we can't experience that to explain 3d to a 2d entity ... Getting a bit far in my thoughts there...