What LLMs Can and Cannot Do?

Large Language Models (LLMs) like ChatGPT have taken the world by storm, promising a revolution in how we interact with information and technology. But amidst the hype, it’s crucial to understand both their capabilities and limitations. Let’s break down what LLMs can and cannot do, drawing from insights shared by AI expert Andrew Ng.

The “Fresh College Grad” Mental Model

To understand the capabilities of LLMs, Andrew Ng introduces a useful mental framework: asking yourself, “Can a fresh college grad, following only the instructions in the prompts, complete the tasks you want?” This simple question can help you gauge whether an LLM is likely to perform well on a given task.

What LLMs Can Do

1. Read and Analyze Text: Just like a fresh college grad, LLMs can read emails to determine if they’re complaints or analyze restaurant reviews for sentiment. These tasks involve understanding context and nuance in written language, which LLMs excel at.
2. Generate Content with Given Context: If provided with relevant information, LLMs can create content such as press releases. However, the quality of the output depends on the quality and specificity of the input.
3. Apply General Knowledge: LLMs have a vast repository of general knowledge, similar to a well-educated college graduate. They can draw upon this knowledge to complete various tasks or answer questions across a wide range of topics.

When applying this mental model, keep in mind:

• LLMs, like our hypothetical fresh grad, don’t have access to external search engines during the task.
• They don’t have specific knowledge about you or your business unless explicitly provided.
• Each prompt is essentially a new interaction, so the LLM doesn’t “learn” or remember information from previous prompts.

Key Limitations of LLMs 🚧

1. Knowledge Cutoffs: LLMs are frozen in time, unaware of events or discoveries after their training data’s final update.
2. Hallucinations: They can confidently generate false or misleading information, even fabricating quotes or legal cases.
3. Input/Output Length Limitations: Prompts and responses have size restrictions, limiting the amount of context they can handle.
4. Challenges with Structured Data: LLMs excel with unstructured data (text, images, audio) but struggle with organized data (spreadsheets, databases).
5. Bias and Harmful Speech: LLMs can reflect biases present in their training data and may occasionally produce toxic or harmful content (though major providers are actively working to mitigate this).

What LLMs Excel At 🚀

Despite these limitations, LLMs have impressive capabilities that make them valuable tools in many scenarios:

1. Natural Language Understanding

LLMs demonstrate remarkable ability in comprehending and analyzing human language:

• They can interpret context, detect nuances, and understand implied meanings in text.
• This makes them excellent for tasks like sentiment analysis, content categorization, and extracting key information from documents.

2. Language Generation

One of the most powerful features of LLMs is their ability to generate human-like text:

• They can produce a wide variety of content, from creative writing to technical documentation.
• LLMs can adapt their writing style based on prompts, making them versatile for different audiences and purposes.

3. Question Answering

LLMs can serve as powerful knowledge bases:

• They can provide detailed answers to a wide range of questions across various domains.
• This makes them useful for customer support, research assistance, and general information retrieval.

4. Text Summarization

LLMs are adept at distilling large amounts of text into concise summaries:

• They can extract key points and main ideas from lengthy documents.
• This is valuable for quickly grasping the essence of reports, articles, or any long-form content.

5. Language Translation

While not their primary function, many LLMs show impressive capabilities in translating between languages:

• They can often capture nuances and idiomatic expressions better than traditional translation tools.
• This makes them useful for cross-lingual communication and content localization.

Best Practices for Using LLMs

To make the most of LLMs while mitigating their limitations, consider the following best practices:

1. Provide Clear Context: The more specific and relevant information you provide in your prompts, the better the LLM’s output will be. Think about what a person would need to know to complete the task effectively.
2. Verify Critical Information: Given the potential for hallucinations, always fact-check important information generated by LLMs, especially for professional or high-stakes applications.
3. Break Down Complex Tasks: For tasks that exceed input limitations or require multiple steps, break them down into smaller, manageable prompts.
4. Use Appropriate Tools for Data Analysis: For tasks involving structured data or complex numerical analysis, rely on traditional data analysis tools and techniques rather than LLMs.
5. Be Aware of Biases: When using LLMs for applications that could impact people’s lives or perpetuate stereotypes, carefully review and potentially adjust the output to ensure fairness and accuracy.
6. Stay Updated on Model Capabilities: As LLM technology rapidly evolves, stay informed about the latest developments, including improvements in handling limitations and new capabilities.

The Future of LLMs

Despite their current limitations, LLMs are evolving rapidly. As they become more sophisticated, their ability to reason, understand context, and generate accurate information will improve significantly.

By understanding what LLMs can and cannot do, we can harness their potential while mitigating their shortcomings. Whether you’re a content creator, researcher, or simply curious about AI, this knowledge will help you navigate the exciting world of large language models.