AI llm models are growing and improving at break neck speed. Google has announced Gemini 1.5 Pro. https://developers.googleblog.com/2024/02/gemini-15-available-for-private-preview-in-google-ai-studio.html
Following technique used by Mistral, Google’s latest Gemini 1.5 Pro will use a mix of experts approach to LLM. A query is routed to a smaller expert LLM. This results in faster and more precise response.
Gemini 1.5 Pro will also feature a new word record for the context window size. The prior record is 200k tokens. Gemini 1.5 Pro will allow context window of up to 1 million tokens. A large context window enables more types of use cases. For instance instead of limited to asking a question about one pdf, you can now ask about 100 pdfs. In the blog post, Google had gif showing a question asking about some funny moments based on the transcript from NASA’s Apollo 11 moon mission. The pdf it used has 326k tokens. The transcript is available at https://www.nasa.gov/history/alsj/a11/a11trans.html. Another example is sending the entire code base of a project with 800k tokens and use LLM to learn about the code. It also allows for about 1 hour of video analysis.
Google published it technical paper on Gemini 1.5 Pro at https://arxiv.org/pdf/2403.05530.pdf. In this paper there was an example of ingesting the code base of JAX which is numpy for CPU/GPU/TPU. It is 746k tokens and 116 files and Gemini 1.5 Pro can take that and answer questions about it.