There is however a different philosophical alternative that has been dubbed JEPA (Joint-Embedding Predictive Architectures).
The primary discussion surrounding these two separate architectures is what the current limitations are with LLM's are and the bounds in which they're not really able to escape.
For example, the biggest LLM's (think OpenAI's ChatGPT, Anthropics Claude, Googles Gemini) on the market at the moment have really only made improvements by their physical growth. The larger the transformers are behind these LLM's, the more sophisticated they become but the real bottleneck now seems to be the fact that they've all essentially consumed all of the internets data so the only way to continue having them grow and learn is to inseminate them with synthetic data / private data.
The other issue is how an LLM in general operates. They work very well in the sense of text generation fluency and analysing patterns but one can argue that this acts as a constraint where LLM's aren't able to achieve a means of general intelligence (however you may define it).
Whereas with JEPA architecture, you're looking at world models being built for physical reasoning, planning and overall aiming to be more efficient and in turn have 'common sense'.
So what do I mean by this? Lets say I go into a room and turn off a light switch. If I asked a human what would happen as a result of flicking this switch, they could say "the light would turn off" or "the room would go dark" or "the light would stop working", etc and all of these answers would be correct because given the context, these are all common sense responses these are all completely different answers for a large language model. They use different tokens, different words that are separate sequences entirely so an LLM needs to exert more energy to understand word styling, responses and so on when all it really needs to do is understand the concept of turning a light switch off.
JEPA architecture doesn't problem solve with text generation like an LLM does. It instead predicts what the answer to the question should actually mean via embedding space.
It's really interesting stuff. I've included a video where Yann LeCun is being interviewed about this very concept. It's really worth a watch.
In the meantime, I'm going to continue going down the JEPA rabbit hole.
It's really interesting stuff. I've included a video where Yann LeCun is being interviewed about this very concept. It's really worth a watch.
In the meantime, I'm going to continue going down the JEPA rabbit hole.