Nuclia RAG-as-a-Service

Prerequisites

First you need to have NVM, NODE and YARN installed.

To install nvm, run:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash

or

wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash

To check if it was installed properly, close and reopen the terminal and run command -v nvm and should return nvm. In case there is something else going on, troubleshoot with this documentation. To see all the commands, simply run nvm.

To install the latest stable version of node, run:

nvm install --lts

To check if node and npm is properly installed, run: node --version and npm --version. _{Any problems should be resolved with the nvm documentation.}

To install yarn, run:

npm install --global yarn

Check if Yarn is installed by running: yarn --version.

Note

In the rest of this documentation, we use commands like nx and missdev. Those can be find in node_modules/.bin folder. You can also install nx globally:

npm install -g nx

Dependencies installation

yarn

Pastanaga-angular installation must be done through missdev so sistema-demo can run:

yarn missdev

If it fails for any reason, you can try to clone Pastanaga manually:

cd libs
git clone git@github.com:plone/pastanaga-angular.git

Dependencies Updates

It's important to keep all our dependencies up-to-date, and we use nx migration tool in order to update our nx and angular dependencies. Now we have an npm harness in place in order to protect our repo from supply chain attacks, with a minimal age gate of two weeks for new packages.

When running the migration command, you may encounter the following error:

nx migrate 23.2 --verbose

 NX   An error occurred while checking the provenance of nx@latest. This might be due to a custom registry configuration (https://pkg.harness.io/pkg/ct8onj8YTdaXtKaFsYCRLg/org-nuclia-npm/npm/). Please check whether provenance is correctly configured for your registry. To disable this check at your own risk, you can set the NX_SKIP_PROVENANCE_CHECK environment variable to true.

 Error: No attestation URL found
Error: An error occurred while checking the provenance of nx@latest. This might be due to a custom registry configuration (https://pkg.harness.io/pkg/ct8onj8YTdaXtKaFsYCRLg/org-nuclia-npm/npm/). Please check whether provenance is correctly configured for your registry. To disable this check at your own risk, you can set the NX_SKIP_PROVENANCE_CHECK environment variable to true.
 Error: No attestation URL found
    at ensurePackageHasProvenance (/Users/pellerin/Workspace/nuclia/frontend/node_modules/nx/dist/src/utils/provenance.js:40:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async nxCliPath (/Users/pellerin/Workspace/nuclia/frontend/node_modules/nx/dist/src/command-line/migrate/migrate.js:2415:5)
    at async /Users/pellerin/Workspace/nuclia/frontend/node_modules/nx/dist/src/command-line/migrate/migrate.js:2333:23
    at async handleErrors (/Users/pellerin/Workspace/nuclia/frontend/node_modules/nx/dist/src/utils/handle-errors.js:9:24)
    at async Object.handler (/Users/pellerin/Workspace/nuclia/frontend/node_modules/nx/dist/src/command-line/migrate/command-object.js:15:39)

In which case, simply add the env variable at the beginning of the command:

NX_SKIP_PROVENANCE_CHECK=true nx migrate 23.2 --verbose

You may still have an error message in case some packages are not old enough:

NX_SKIP_PROVENANCE_CHECK=true nx migrate 23.2 --verbose

 NX   All versions satisfying "23.2" are quarantined

Wait until a matching version is older than the configured window, lower yarn npmMinimalAgeGate (20160 min), or add nx to npmPreapprovedPackages in .yarnrc.yml.

in which case you just have to check the release date of the targetted packages and try again once they're more than 2 weeks old.

External dependencies

We used to load some external libs from cdn.jsdelivr.net or cd./dashjs.net, but it was sometimes conflicting with some customers security policy.

So the following files have been manually uploaded in the Nuclia CDN:

https://cdn.jsdelivr.net/npm/marked/marked.min.js https://cdn.jsdelivr.net/npm/pdfjs-dist@2.16.105/build/pdf.min.js https://cdn.jsdelivr.net/npm/pdfjs-dist@2.16.105/build/pdf.worker.js\n https://cdn.jsdelivr.net/npm/pdfjs-dist@2.16.105/web/pdf_viewer.css https://cdn.dashjs.org/v4.7.1/dash.all.min.js

Setup certificates and nginx server for the apps

Install brew first if you don't already have it https://brew.sh/. Then you need to install a few dependencies:

brew install nginx mkcert nss

Create the certificates needed for both auth and dashboard apps. The following commands are for working with dev cluster as a backend, if you want to work with stage, replace auth.gcp-global-dev-1.nuclia.io by auth.stashify.cloud in all the commands and in the nginx template.

Note:

• We created new local-stage-proxy and local-dev-proxy configurations for dashboard app, ask a supervisor to give it to you.
• Make sure you have a siteKey property in your app-config.json on local-dev and local-stage of all your apps in order to have the OAuth working well with local proxy. If you don't have siteKey, ask a supervisor to provide them to you.

Commands to create the certificates:

cd ./local_server
mkcert -install
mkcert auth.gcp-global-dev-1.nuclia.io
mkcert rag.gcp-global-dev-1.nuclia.io
mkcert admin.gcp-global-dev-1.nuclia.io

Then copy local_server/nginx.template.conf to a nginx.conf and use your local certificates in it.

Finally, using sudo, in /etc/hosts, add commented proxies for stage and dev clusters:

```bash

Setup a proxy to work locally with auth and dashboard apps

Finally, you should run the apps specifying the host. When working with dev cluster as a backend:

• dashboard: nx run dashboard:serve:local-dev-proxy --host 127.0.0.1
• admin: nx run admin:serve:local-dev --host 127.0.0.1
• auth: nx run auth:serve:local-dev --host 127.0.0.1

When working with stage as a backend:

• dashboard: nx run dashboard:serve:local-stage-proxy --host 127.0.0.1
• admin: nx run admin:serve:local-stage --host 127.0.0.1
• auth: nx run auth:serve:local-stage --host 127.0.0.1

Finally, enable your proxy and launch the nginx local server from local_server folder:

cd local_server
sudo nano /etc/hosts  # uncomment the 3 lines corresponding to the environment you're using then save
sudo nginx -p $(pwd) -c nginx.conf -g 'daemon off;'

Notes: Do not forget to comment the lines added in /etc/hosts once finished, else you will not be able to access the real cluster. In case of trouble accessing the dashboard after the proxy has been removed from /etc/hosts, reboot your browser (working on private window is a good way to avoid troubles).

CI/CD Deployment

Deploying to stage

When merging a PR, if it impacts the manager app, it is built and our deploy_manager job (in our deploy GitHub Action) will update Helm and then trigger a Repository Dispatch event to the frontend_deploy repo.

That's how the manager is deployed to stage.

SDK

Documentation