org-ai — AI 图像生成工具中文文档

Features and Usage

Demos

Setting up speech input / output

Whisper

These setup steps are optional. If you don't want to use speech input / output, you can skip this section.

_Note: My personal config for org-ai can be found in this gist. It contains a working whisper setup.

This has been tested on macOS and Linux. Someone with a Windows computer, please test this and let me know what needs to be done to make it work (Thank You!).

The speech input uses whisper.el and ffmpeg. You need to clone the repo directly or use straight.el to install it.

1. install ffmpeg (e.g. brew install ffmpeg on macOS) or sudo apt install ffmpeg on Linux.
2. Clone whisper.el: git clone https://github.com/natrys/whisper.el path/to/whisper.el

You should now be able to load it inside Emacs:

(use-package whisper
  :load-path "path/to/whisper.el"
  :bind ("M-s-r" . whisper-run))

Now also load:

(use-package greader :ensure)
(require 'whisper)
(require 'org-ai-talk)

;; macOS speech settings, optional
(setq org-ai-talk-say-words-per-minute 210)
(setq org-ai-talk-say-voice "Karen")

macOS specific steps

On macOS you will need to do two more things: 1. Allow Emacs to record audio 2. Tell whisper.el what microphone to use

###### 1. Allow Emacs to record audio You can use the tccutil helper:

git clone https://github.com/DocSystem/tccutil
cd tccutil
sudo python ./tccutil.py -p /Applications/Emacs.app -e --microphone

When you now run ffmpeg -f avfoundation -i :0 output.mp3 from within an Emacs shell, there should be no abort trap: 6 error.

(As an alternative to tccutil.py see the method mentioned in this issue.)

###### 2. Tell whisper.el what microphone to use

You can use the output of ffmpeg -f avfoundation -list_devices true -i "" to list the audio input devices and then tell whisper.el about it: (setq whisper--ffmpeg-input-device ":0"). :0 is the microphone index, see the output of the command above to use another one.

I've created an emacs helper that let's you select the microphone interactively. See this gist.

My full speech enabled config then looks like:

(use-package whisper
  :load-path (lambda () (expand-file-name "lisp/other-libs/whisper.el" user-emacs-directory))
  :config
  (setq whisper-model "base"
        whisper-language "en"
        whisper-translate nil)
  (when *is-a-mac*
    (rk/select-default-audio-device "Macbook Pro Microphone")
    (when rk/default-audio-device)
    (setq whisper--ffmpeg-input-device (format ":%s" rk/default-audio-device))))

###### macOS alternative: Siri dictation

On macOS, instead of whisper, you can also use the built-in Siri dictation. To enable that, go to Preferences -> Keyboard -> Dictation, enable it and set up a shortcut. The default is ctrl-ctrl.

Windows specific steps

The way (defun whisper--check-install-and-run) is implemented does not work on Win10 (see https://github.com/rksm/org-ai/issues/66).

A workaround is to install whisper.cpp and model manually and patch:

 elisp
(defun whisper--check-install-and-run (buffer status)
  (whisper--record-audio))

espeak / greader

Speech output on non-macOS systems defaults to using the greader package which uses espeak underneath to synthesize speech. You will need to install greader manually (e.g. via M-x package-install). From that point on it should "just work". You can test it by selecting some text and calling M-x org-ai-talk-read-region.

Setting up Stable Diffusion

An API for Stable Diffusion can be hosted with the stable-diffusion-webui project. Go through the install steps for your platform, then start an API-only server:

cd path/to/stable-diffusion-webui
./webui.sh --nowebui

This will start a server on http://127.0.0.1:7861 by default. In order to use it with org-ai, you need to set org-ai-sd-endpoint-base:

(setq org-ai-sd-endpoint-base "http://localhost:7861/sdapi/v1/")

If you use a server hosted elsewhere, change that URL accordingly.

OpenAI API key

You can either directly set your api token in your config:

(setq org-ai-openai-api-token "<ENTER YOUR API TOKEN HERE>")

Alternatively, org-ai supports auth-source for retrieving your API key. You can store a secret in the format

machine api.openai.com login org-ai password <your-api-key>

in your ~/authinfo.gpg file. If this is present, org-ai will use this mechanism to retrieve the token when a request is made. If you do not want org-ai to try to retrieve the key from auth-source, you can set org-ai-use-auth-source to nil before loading org-ai.

Using other services than OpenAI

Azure

You can switch to Azure by customizing these variables, either interactively with M-x customize-variable or by adding them to your config:

(setq org-ai-service 'azure-openai
      org-ai-azure-openai-api-base "https://your-instance.openai.azure.com"
      org-ai-azure-openai-deployment "azure-openai-deployment-name"
      org-ai-azure-openai-api-version "2023-07-01-preview")

To store the API credentials, follow the authinfo instructions above but use org-ai-azure-openai-api-base as the machine name.

perplexity.ai

For a list of available models see the perplexity.ai documentation.

Either switch the default service in your config:

(setq org-ai-service 'perplexity.ai)
(setq org-ai-default-chat-model "llama-3-sonar-large-32k-online")

or per block:

#+begin_ai :service perplexity.ai :model llama-3-sonar-large-32k-online
[ME]: Tell me fun facts about Emacs.
#+end_ai

For the authentication have an entry like machine api.perplexity.ai login org-ai password pplx-*** in your authinfo.gpg or set org-ai-openai-api-token.

Note: Currently the perplexity.ai does not give access to references/links via the API so Emacs will not be able to display references. They have a beta program for that running and I sure hope that this will be available generally soon.

Anthropic / Claude

Similar to the above. E.g.

#+begin_ai :service anthropic :model claude-3-opus-20240229
[ME]: Tell me fun facts about Emacs.
#+end_ai

Anthropic models are here. There is currently only one API version that is set via org-ai-anthropic-api-version. If other version come out you can find them here.

For the API token use machine api.anthropic.com login org-ai password sk-ant-*** in your authinfo.gpg.

org-ai [](https://melpa.org/#/org-ai)

Minor mode for Emacs org-mode that provides access to generative AI models. Currently supported are - OpenAI API (ChatGPT, DALL-E, other text models), optionally run against Azure API instead of OpenAI - Stable Diffusion through stable-diffusion-webui

Inside an org-mode buffer you can - use ChatGPT to generate text, having full control over system and user prompts (demo) - Speech input and output! Talk with your AI! - generate images and image variations with a text prompt using Stable Diffusion or DALL-E (demo 1, demo 2) - org-ai everywhere: Various commands usable outside org-mode for prompting using the selected text or multiple files.

_Note: In order to use the OpenAI API you'll need an OpenAI account and you need to get an API token. As far as I can tell, the current usage limits for the free tier get you pretty far.

------------------------------

FAQ