overleaf/services/clsi
Jakob Ackermann 5d2549fe52 [misc] add/update CI pipelines for monorepo
GitOrigin-RevId: 8946b8be124e6977d2cafc6c3bf065412d765583
2021-08-05 13:48:05 +00:00
..
.github [misc] upgrade build scripts to version 3.11.0 and cleanup packages 2021-07-13 11:55:19 +01:00
app add option for apparmor profile 2021-07-21 14:53:35 +01:00
bin remove ./bin/install_texlive_gce.sh which shouldn't be needed 2020-03-11 10:06:55 +00:00
config add option for apparmor profile 2021-07-21 14:53:35 +01:00
db [misc] move the sqlite database into a db/ directory 2020-04-03 12:36:31 +02:00
scripts [ContentCacheManager] use PDF.js Xref table instead of stream detection (#242) 2021-05-31 09:20:25 +01:00
seccomp add epoll_pwait to secomp profile 2019-01-24 12:30:37 +00:00
src Add in a synctex end point 2014-04-08 15:18:56 +01:00
test [misc] make build scripts happy again 2021-07-26 12:20:04 +01:00
.dockerignore Upgrade to Node 12 2021-03-12 16:27:33 -05:00
.eslintrc [misc] temporary override a few new/changed eslint rules 2021-07-13 12:24:43 +01:00
.gitignore add git ignore for output directory 2021-01-22 11:05:52 +00:00
.mocharc.json Add a global test setup file 2021-03-12 16:27:33 -05:00
.nvmrc [misc] upgrade node version to latest v12 LTS version 12.22.3 2021-07-13 12:26:36 +01:00
.prettierrc [misc] upgrade build scripts to version 3.11.0 and cleanup packages 2021-07-13 11:55:19 +01:00
.viminfo update build script and add load balancer agent 2018-03-29 12:12:29 +01:00
app.js [misc] run format_fix and lint:fix 2021-07-13 12:04:48 +01:00
buildscript.txt [misc] add/update CI pipelines for monorepo 2021-08-05 13:48:05 +00:00
debug update build script and add load balancer agent 2018-03-29 12:12:29 +01:00
docker-compose-config.yml Update references to .coffee files 2021-05-19 18:57:35 +01:00
docker-compose.ci.yml [misc] merge pdf caching into main (#226) 2021-05-13 14:07:54 +01:00
docker-compose.yml [misc] merge pdf caching into main (#226) 2021-05-13 14:07:54 +01:00
Dockerfile [misc] upgrade node version to latest v12 LTS version 12.22.3 2021-07-13 12:26:36 +01:00
entrypoint.sh Add /app/output 2021-02-18 15:33:16 +00:00
install_deps.sh [perf] trim down install_deps.sh -- install docker cli only 2021-07-20 14:14:00 +01:00
kube.yaml updateded build scripts 2018-03-02 17:14:23 +00:00
LICENSE Create LICENSE 2014-02-18 17:09:54 +00:00
Makefile Upgrade to Node 12 2021-03-12 16:27:33 -05:00
nodemon.json [misc] bump the dev-env to 3.3.2 2020-08-10 17:01:11 +01:00
package-lock.json [misc] upgrade build scripts to version 3.11.0 and cleanup packages 2021-07-13 11:55:19 +01:00
package.json [misc] make build scripts happy again 2021-07-26 12:20:04 +01:00
patch-texlive-dockerfile update build script and add load balancer agent 2018-03-29 12:12:29 +01:00
README.md Explain the situation with permissions on Linux 2021-03-19 14:32:54 +00:00
synctex.profile update build script and add load balancer agent 2018-03-29 12:12:29 +01:00

overleaf/clsi

A web api for compiling LaTeX documents in the cloud

The Common LaTeX Service Interface (CLSI) provides a RESTful interface to traditional LaTeX tools (or, more generally, any command line tool for composing marked-up documents into a display format such as PDF or HTML). The CLSI listens on the following ports by default:

  • TCP/3013 - the RESTful interface
  • TCP/3048 - reports load information
  • TCP/3049 - HTTP interface to control the CLSI service

These defaults can be modified in config/settings.defaults.js.

The provided Dockerfile builds a Docker image which has the Docker command line tools installed. The configuration in docker-compose-config.yml mounts the Docker socket, in order that the CLSI container can talk to the Docker host it is running in. This allows it to spin up sibling containers running an image with a TeX distribution installed to perform the actual compiles.

The CLSI can be configured through the following environment variables:

  • ALLOWED_COMPILE_GROUPS - Space separated list of allowed compile groups
  • ALLOWED_IMAGES - Space separated list of allowed Docker TeX Live images
  • CATCH_ERRORS - Set to true to log uncaught exceptions
  • COMPILE_GROUP_DOCKER_CONFIGS - JSON string of Docker configs for compile groups
  • COMPILES_HOST_DIR - Working directory for LaTeX compiles
  • COMPILE_SIZE_LIMIT - Sets the body-parser limit
  • DOCKER_RUNNER - Set to true to use sibling containers
  • DOCKER_RUNTIME -
  • FILESTORE_DOMAIN_OVERRIDE - The url for the filestore service e.g.http://$FILESTORE_HOST:3009
  • FILESTORE_PARALLEL_FILE_DOWNLOADS - Number of parallel file downloads
  • FILESTORE_PARALLEL_SQL_QUERY_LIMIT - Number of parallel SQL queries
  • LISTEN_ADDRESS - The address for the RESTful service to listen on. Set to 0.0.0.0 to listen on all network interfaces
  • PROCESS_LIFE_SPAN_LIMIT_MS - Process life span limit in milliseconds
  • SENTRY_DSN - Sentry Data Source Name
  • SMOKE_TEST - Whether to run smoke tests
  • SQLITE_PATH - Path to SQLite database
  • SYNCTEX_BIN_HOST_PATH - Path to SyncTeX binary
  • TEXLIVE_IMAGE - The TeX Live Docker image to use for sibling containers, e.g. gcr.io/overleaf-ops/texlive-full:2017.1
  • TEX_LIVE_IMAGE_NAME_OVERRIDE - The name of the registry for the Docker image e.g. gcr.io/overleaf-ops
  • TEXLIVE_IMAGE_USER - When using sibling containers, the user to run as in the TeX Live image. Defaults to tex
  • TEXLIVE_OPENOUT_ANY - Sets the openout_any environment variable for TeX Live (see the \openout primitive documentation)

Further environment variables configure the metrics module

Installation

The CLSI can be installed and set up as part of the entire Overleaf stack (complete with front end editor and document storage), or it can be run as a standalone service. To run is as a standalone service, first checkout this repository:

$ git clone git@github.com:overleaf/clsi.git

Then build the Docker image:

$ docker build . -t overleaf/clsi

Then pull the TeX Live image:

$ docker pull texlive/texlive

Then start the Docker container:

$ docker run --rm \
    -p 127.0.0.1:3013:3013 \
    -e LISTEN_ADDRESS=0.0.0.0 \
    -e DOCKER_RUNNER=true \
    -e TEXLIVE_IMAGE=texlive/texlive \
    -e TEXLIVE_IMAGE_USER=root \
    -e COMPILES_HOST_DIR="$PWD/compiles" \
    -v "$PWD/compiles:/app/compiles" \
    -v "$PWD/cache:/app/cache" \
    -v /var/run/docker.sock:/var/run/docker.sock \
    --name clsi \
    overleaf/clsi

Note: if you're running the CLSI in macOS you may need to use -v /var/run/docker.sock.raw:/var/run/docker.sock instead.

The CLSI should then be running at http://localhost:3013

Important note for Linux users

The Node application runs as user node in the CLSI, which has uid 1000. As a consequence of this, the compiles folder gets created on your host with uid and gid set to 1000.

ls -lnd compiles
drwxr-xr-x 2 1000 1000 4096 Mar 19 12:41 compiles

If there is a user/group on your host which also happens to have uid / gid 1000 then that user/group will have ownership of the compiles folder on your host.

LaTeX runs in the sibling containers as the user specified in the TEXLIVE_IMAGE_USER environment variable. In the example above this is set to root, which has uid 0. This creates a problem with the above permissions, as the root user does not have permission to write to subfolders of compiles.

A quick fix is to give the root group ownership and read write permissions to compiles, with setgid set so that new subfolders also inherit this ownership:

sudo chown -R 1000:root compiles
sudo chmod -R g+w compiles
sudo chmod g+s compiles

Another solution is to create a sharelatex group and add both root and the user with uid 1000 to it. If the host does not have a user with that uid, you will need to create one first.

sudo useradd --uid 1000 host-node-user # If required
sudo groupadd sharelatex
sudo usermod -a -G sharelatex root
sudo usermod -a -G sharelatex $(id -nu 1000)
sudo chown -R 1000:sharelatex compiles
sudo chmod -R g+w compiles
sudo chmod g+s compiles

This is a facet of the way docker works on Linux. See this upstream issue

Config

The CLSI will use a SQLite database by default, but you can optionally set up a MySQL database and then fill in the database name, username and password in the config file at config/settings.development.js.

API

The CLSI is based on a JSON API.

Example Request

(Note that valid JSON should not contain any comments like the example below).

POST /project/<project-id>/compile
{
    "compile": {
        "options": {
            // Which compiler to use. Can be latex, pdflatex, xelatex or lualatex
            "compiler": "lualatex",
            // How many seconds to wait before killing the process. Default is 60.
            "timeout": 40
        },
        // The main file to run LaTeX on
        "rootResourcePath": "main.tex",
        // An array of files to include in the compilation. May have either the content
        // passed directly, or a URL where it can be downloaded.
        "resources": [
          {
            "path": "main.tex",
            "content": "\\documentclass{article}\n\\begin{document}\nHello World\n\\end{document}"
          }
          // ,{
          //     "path": "image.png",
          //     "url": "www.example.com/image.png",
          //     "modified": 123456789 // Unix time since epoch
          // }
        ]
    }
}

With curl, if you place the above JSON in a file called data.json, the request would look like this:

$ curl -X POST -H 'Content-Type: application/json' -d @data.json http://localhost:3013/project/<id>/compile

You can specify any project-id in the URL, and the files and LaTeX environment will be persisted between requests. URLs will be downloaded and cached until provided with a more recent modified date.

Example Response

{
    "compile": {
        "status": "success",
        "outputFiles": [{
            "type": "pdf",
            "url": "http://localhost:3013/project/<project-id>/output/output.pdf"
        }, {
            "type": "log",
            "url": "http://localhost:3013/project/<project-id>/output/output.log"
        }]
    }
}

License

The code in this repository is released under the GNU AFFERO GENERAL PUBLIC LICENSE, version 3. A copy can be found in the LICENSE file.

Copyright (c) Overleaf, 2014-2021.