LLM工程实验室

📘 [Guide & Theory (index.ipynb)](Agent/index.ipynb)

A comprehensive interactive notebook that documents the design philosophy: - Agent Design Paradigms: Explores core concepts like Reflection, Tool Use, and Planning (referenced from Andrew Ng's framework). - System Architecture: Visualizes the Core Module Flowchart and data flow between Memory, Tools, and the Planning engine. - Use Case Demonstrations: Step-by-step walkthroughs of real-world scenarios (e.g., Sales Analysis, Automated Reporting).

Function Calling Demos

- POI(Point of Interest): This demo uses Amap's (Gaode Map) public API to retrieve information about hotels, restaurants, attractions, and other points of interest (POIs) near a specific location. It allows querying nearby POIs relative to a given point.

- SQL: This demo demonstrates how Function Calling handles sophisticated database tasks and generates SQL queries.

- Stream: This demo showcases examples of Function Calling in Stream mode.

Key Modules

• Transformer Source Code Analysis: A deep dive into the vanilla Transformer architecture, focusing on a line-by-line implementation analysis of Self-Attention mechanisms, Multi-Head Attention, and Layer Normalization to understand the foundational building blocks.

• Llama Series: QLoRA & Quantization: Implementation of QLoRA (Quantized Low-Rank Adapters) to fine-tune Llama 2 (7B/13B) on consumer hardware.
• Analyzed 4-bit NormalFloat (NF4) quantization and Double Quantization for memory optimization.
• Covered the full pipeline: Data Prep → Fine-tuning → Merging Adapters → Quantization → Deployment.

• Mistral 7B: Architecture & Domain Adaptation: Fine-tuned Mistral 7B for a Generative Recommendation task.
• Explored Mistral's specific architectural advantages, including Sliding Window Attention (SWA) for long contexts and Grouped-Query Attention (GQA) for inference speedup.

• RLHF & TRL (Transformer Reinforcement Learning): Leveraged Hugging Face's TRL library to implement the RLHF (Reinforcement Learning from Human Feedback) alignment pipeline:
• SFT (Supervised Fine-Tuning): Instruction tuning with formatted datasets.
• DPO (Direct Preference Optimization): Implemented DPO as a more stable and efficient alternative to PPO (Proximal Policy Optimization) for aligning models with human preferences.

Core Components of LangChain

• #### Model I/O Wrapping

LLMs: Large Language Models Chat Models: Generally based on LLMs but restructured for conversational purposes PromptTemplate: Templates for prompt creation OutputParser: Parses the output from models

• #### Data Connection Wrapping

Document Loaders: Loaders for various file formats Document Transformers: Common operations on documents such as splitting, filtering, translating, and extracting metadata Text Embedding Models: Convert text into vector representations, useful for tasks like retrieval Vectorstores: Stores for vectors (used in retrieval tasks) Retrievers: Tools for retrieving vectors from storage RAG Pipline with Langchain

• #### Memory Wrapping

Memory: Not physical memory; it manages "context", "history", or "memory" from a text perspective

• #### Architecture Wrapping

Chain: Implements a single function or a series of sequential functions, LangChain Expression Language (LCEL) Agent: Automatically plans and executes steps based on user input, selecting the necessary tools for each step to achieve the desired task Tools: Functions for calling external functionalities, such as Google search, file I/O, Linux shell, etc. Toolkits: A set of tools designed to operate specific software, such as a toolkit for managing databases or Gmail

• #### Callbacks

Pipelines

- run_RAG_vector_database_pipeline: RAG Pipeline based on ChromaDB Vector Database.

The Offline Steps are as follows:

| Document Loading | Document Splitting | Vectorization | Insert into Vector Database | |-----------------------|---------------------|---------------|------------------------------| | → | → | → | → | The Online Steps are as follows: | Receive User Query | Vectorize User Query | Retrieve from Vector Database | Populate Prompt Template | Call LLM with Final Prompt | Generate Response | |-----------------------|----------------------|-------------------------------|---------------------------|----------------------------|---------------------| | → | → | → | → | → | → |

- run_RAG_ES_pipeline: RAG Pipeline based on Elasticsearch (ES).

- RAG_pipeline_pdf_table_processing: In this RAG Pipline, use data from tables in PDFs to implement RAG. Offline: Convert PDF to images and extract tables from the images → Use GPT-4 to generate textual descriptions of the table images → Store the textual descriptions (documents), their embeddings (embeddings), and image URLs (metadatas) into the vector database. Online: Receive a query and search the vector database → Retrieve table image URLs from search results (based on similarity between textual descriptions and the query) → Use GPT-4 to query and retrieve information from the table images. The pipeline flowchart is as follows: