Share:

Through the past decade we have seen a flood in the development and innovation of new AI models across the seven seas. With cutting edge Transformer based LMs being just the tip of the ice berg, AI refining and deployment has been revolutionised by complex but smart algorithms that climb the mountain which before was only dreamt of-in stories older than the industrial revolution. Furthermore, in today's rapidly evolving era, two Ai algorithms have proven their might from the mob, named as Dense architecture and Moe architecture-both being the sharpest blade of their batch.

Dense architecture is the traditional and more popular path for most pioneers of ai empowering OpenAi, Google Gemini, Perplexity and much more to be named. Though aging more than a decade, this architecture was made popular by the release of ChatGPT in the year 2022 crushing billions of dollars in market cap within just a few days of release.The LMs made up of this architecture are heavy in size and consist of billions if not trillions of parameters in them requiring expensive compute and billions of dollars in hardware resources not easily obtained. In the process of training these models, a huge amount of local resources are used to cool off the hardware and throw the wastes which cause heavy environmental pollution.

Though being computationally heavy, dense models are the go-to option for many AI enthusiasts in building code that reads human language, interprets it and produces a result. Moreover, these models understand a text well and are smarter currently because of their genius architecture of transformers which fail even the mightiest of human brains on the planet. Dense models are easier to train, reducing human efforts and thus not being reliant on much skilled manpower with most dense models being trained by even first year computer science students. These models require simple layering making them easier to comprehend theoretically by even high schoolers and practically by undergraduates.

Being the prodigy of LLM architecture, MOE which stands for Mixture of Experts architecture is the more efficient and smart LLM architecture which saves heavy compute and engraves itself as the example of smartness kills the strongest being the limelight of attention over the past year. Though being the once failed idea that died before it even took shape in OpenAi labs, was reborn on January 20th of 2025 shaping the great 600+ billion parameter LLM of DeepSeek R1, crushing the market cap of most American companies including Nvidia and OpenAi. Rather than training the whole model in one go, the mixture of experts in architecture trains many smaller models (usually 10-50 Billion parameters) known as “Experts” who specialize in a specific field, the examples being math, english, image generation etc. The Mixture of Experts LLM architecture consists of a ‘Gate’ or often called Router model which classifies the user query into categories which are handled by the model’s “Experts”. The router does the job of picking the right experts for a specific query from which the algorithm only uses the experts needed for the specific query instead of using the whole model, thus saving compute and money. Some great examples of the mixture of experts architecture LLMs are X ai’s Grok, Mistral and the most popular DeepSeek which sunk the market cap of many companies on the day of its release. Though being controversially full of Chinese propaganda, it achieved a great milestone of training huge LLMs in limited hardware. Due to the political instability between the American and Chinese governments, Deepseek was forbidden from using American world class GPUs and was forced to think smart.

Both architectures have shaped the foundation of AI proving their might through shaping actual non living matter i.e machinery to think which is a great feat in itself. Even just 50 years ago, no human thought of the silly rocks they found everyday to one day start thinking as humans. “AI has not replaced us and never will, AI is the chariot that will let us get closer to the human brain itself.” With more architectures yet to be discovered, the human mind will most certainly foster greater inventions that will amaze even the most brilliant minds. We shouldn’t be afraid of AI as it can't replace the curious nature of humans which becomes greater exponentially as more things get discovered.