github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-30 02:15:13 -05:00
Code Issues Projects Releases Packages Wiki Activity
22035 commits 3 branches 7 tags 206 MiB
Commit graph

451 commits

Author SHA1 Message Date
Jakob Ackermann
b456ea726d [misc] merge pdf caching into main (#226) 
* wip generate directory for hash content

* cleanup, remove console logging

* add content caching module

* Return PDF stream ranges with compile response

* Return the PDF file size in the compile response

* PDF range endpoint

* [misc] WIP: pdf caching: preserve the m-time on static content files

* [misc] WIP: pdf caching: improve browser caching, emit caching headers

* [misc] WIP: pdf caching: do not emit very small chunks <1kB

* [misc] keep up with moving output files into a separate directory

* [OutputCacheManager] add global feature flag for enabling pdf caching

* [misc] add contentId into the URL for protecting PDF stream contents

* [misc] support PDF stream caching for anonymous users

* [misc] add per-request feature flag for enabling PDF stream caching

* [misc] enable pdf caching in CI and emit metrics at the end of run

* [misc] expose compile stats and timings to the frontend

* [misc] log an error in case saving output files fails

* [misc] add metrics for pdf bandwidth and pdf caching performance

* [misc] add a dark mode to the pdf caching for computing ranges only

* [misc] move pdf caching metrics into ContentCacheMetrics

* [misc] add a config option for the min chunk size of pdf ranges

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
 2021-05-13 14:07:54 +01:00
Jakob Ackermann
cfa8127c1b [misc] add linting for missing explicit dependencies and fix any errors 2021-04-29 15:30:54 +01:00
Jakob Ackermann
b3ed820444 Merge pull request #219 from overleaf/jpa-read-image-name-for-synctex 
[misc] consume and validate a custom imageName for synctex requests
 2021-03-31 10:33:53 +02:00
Jakob Ackermann
6cb4246eb1 [misc] consume and validate a custom imageName for synctex requests 2021-03-30 13:57:40 +01:00
Shane Kilkelly
072d0a9dab Add flag to qpdf, to preserve PDF/A compliance 2021-03-19 11:35:59 +00:00
Brian Gough
865e68051e include fallback for missing state file 2021-01-26 13:59:28 +00:00
Brian Gough
a8e47da9e9 Revert "Merge pull request #205 from overleaf/revert-200-bg-decaff-cleanup" 
This reverts commit 76d8d3181b9464d1e1bbc713a2729ca269d9c047, reversing
changes made to 31a8dc3a98d73c2707d633712f0ef7207013e78b.
 2021-01-26 11:04:33 +00:00
Brian Gough
9dc55729ac Revert "decaff cleanup ResourceStateManager" 2021-01-25 15:26:53 +00:00
Brian Gough
f41f3aba8c Merge pull request #204 from overleaf/bg-use-output-dir 
use separate output dir for generated files
 2021-01-25 14:10:09 +00:00
Brian Gough
90c0d931ad Merge pull request #201 from overleaf/bg-more-cleanup 
decaff cleanup of OutputFileManager
 2021-01-25 14:09:46 +00:00
Brian Gough
c67c8a6c80 Merge pull request #200 from overleaf/bg-decaff-cleanup 
decaff cleanup ResourceStateManager
 2021-01-25 14:09:24 +00:00
Brian Gough
b5346658b0 clear output directory when clearing project 2021-01-22 11:05:52 +00:00
Brian Gough
692dbc8d6b add output directory 2021-01-22 11:05:52 +00:00
Jakob Ackermann
49b764a308 [misc] CompileController: exact match for output.pdf 
The previous regex could mistake user provided pdf files, like
 `fake_output.pdf`, as the final output file.
The frontend expects to find a `output.pdf` file on success.
 2020-12-28 13:16:31 +00:00
Brian Gough
e183fffa4b format fix 2020-12-18 16:15:06 +00:00
Brian Gough
dafe69b5e6 remove unused module var 2020-12-18 16:15:06 +00:00
Brian Gough
36b646bb84 remove comments 2020-12-18 16:15:06 +00:00
Brian Gough
f20a0d1a69 remove unused var 2020-12-18 16:15:03 +00:00
Brian Gough
df1caa14b8 remove unused require 2020-12-18 16:14:36 +00:00
Brian Gough
ca98ee5cff use a Set instead of an Object 2020-12-18 16:14:09 +00:00
Brian Gough
43b0429c28 clean up relative path checking 2020-12-18 16:14:05 +00:00
Brian Gough
73e09ff99f remove unnecessary requires 2020-12-18 15:28:06 +00:00
Brian Gough
cee93a0cd9 clean up unnecessary var 2020-12-18 15:28:06 +00:00
Brian Gough
4169f7fc73 use once for callback 2020-12-18 15:28:06 +00:00
Brian Gough
e84d6305f3 remove unnecessary return 2020-12-18 15:16:07 +00:00
Brian Gough
ae064e8997 lint fix 2020-12-18 15:11:29 +00:00
Brian Gough
323890cedb remove Array.from 2020-12-18 15:10:53 +00:00
Brian Gough
c044db4897 remove guard helper 2020-12-18 15:09:48 +00:00
Brian Gough
0bae53c7f3 remove unnecessary returns 2020-12-18 15:02:40 +00:00
Brian Gough
eaec57cd01 simplify null check 2020-12-18 15:01:25 +00:00
Brian Gough
411d69e36b remove unnecessary callback code 2020-12-18 14:59:48 +00:00
Brian Gough
11ef3c27ed use Set instead of object 2020-12-18 14:56:53 +00:00
Brian Gough
64ea22d259 remove unnecessary null checks 2020-12-17 12:05:17 +00:00
Brian Gough
299d7cc5fe remove unnecessary returns 2020-12-17 12:01:14 +00:00
Brian Gough
5a539b7f90 remove guard function 2020-12-17 11:59:15 +00:00
Brian Gough
4d8e4d54e5 remove Array.from 2020-12-17 11:49:02 +00:00
Jakob Ackermann
7b7cd8cc8c [misc] bump metrics module to 3.4.1 
- renamed package from `metrics-sharelatex` to `@overleaf/metrics`
- drop support for statsd backend
- decaffeinate
- compress `/metrics` response using gzip
- bump debugging agents to latest versions
- expose prometheus interfaces for custom metrics (custom tags)
- cleanup of open sockets metrics
- fix deprecation warnings for header access
 2020-11-25 11:57:25 +00:00
Eric Mc Sween
1c13f6fe94 Mount /home/tex in an anonymous volume 
When we mount the container's root filesystem as read-only, mount
an anonymous volume in /home/tex so that it's writable. Our TeX Live
images have cached content in /home/tex. This content will automatically
get copied by Docker into this anonymous volume.
 2020-09-04 08:49:56 -04:00
Eric Mc Sween
67f4a6eeeb Decaf cleanup: normalize functions 
Use function keyword for declarations and arrow functions for callbacks.
 2020-09-03 15:58:16 -04:00
Eric Mc Sween
5cd889038e Use _.once() instead of ad hoc implementation 2020-09-03 15:52:09 -04:00
Eric Mc Sween
a853950a99 Fix container monitor cleanup function 
The intent here is clearly to clear both the timeout and the interval.
 2020-09-03 15:50:45 -04:00
Eric Mc Sween
30a44edded Decaf cleanup: simplify variable declarations 2020-09-03 15:50:12 -04:00
Eric Mc Sween
a282bccd48 Do not instantiate errors at module load time 
This prevents the right stack trace from being captured.
 2020-09-03 15:34:19 -04:00
Eric Mc Sween
4905e7db2c Decaf cleanup: unused vars 2020-09-03 15:30:07 -04:00
Eric Mc Sween
44bf38d6db Decaf cleanup: convert async function to sync 
The examineOldContainer() function doesn't need to use callbacks since
it only does synchronous work.
 2020-09-03 15:13:04 -04:00
Eric Mc Sween
2584586ba2 Decaf cleanup: camel-case variables 2020-09-03 15:13:04 -04:00
Eric Mc Sween
c52d7d8f02 Decaf cleanup: simplify null checks 2020-09-03 15:13:04 -04:00
Eric Mc Sween
ee4c08868c Decaf cleanup: remove default callbacks 2020-09-03 15:13:04 -04:00
Eric Mc Sween
32f0bbe266 Decaf cleanup: remove IIFEs 2020-09-03 15:13:04 -04:00
Eric Mc Sween
f650da8675 Decaf cleanup: remove __guard__ 2020-09-02 17:06:35 -04:00
Eric Mc Sween
99648341e2 Decaf cleanup: remove unnecessary returns 2020-09-02 16:58:41 -04:00
Eric Mc Sween
08be54a43e Decaf cleanup: remove unnecessary Array.from() 2020-09-02 16:45:49 -04:00
Jakob Ackermann
1d1b9ebebc [misc] bump the dev-env to 3.3.2 2020-08-10 17:01:11 +01:00
Jakob Ackermann
5846ebb367 Merge pull request #182 from overleaf/msm-fix-npe-community-edition 
Fixed NPE when Settings.clsi is defined but Settings.clsi.docker is not
 2020-07-15 11:01:08 +02:00
Jakob Ackermann
0f9d0d9790 Merge pull request #183 from overleaf/jpa-clsi-allowed-image-names 
[misc] RequestParser: restrict imageName to an allow list and add tests
 2020-07-15 10:58:36 +02:00
Jakob Ackermann
c3a3ed341b Merge pull request #132 from das7pad/hotfix-double-call 
[LocalCommandRunner] run: block a double call of the callback
 2020-07-03 12:58:25 +02:00
Jakob Ackermann
f703b2ca41 [misc] move the image check prior to the base image override 2020-07-01 10:01:25 +01:00
Jakob Ackermann
aa02df7b81 [misc] apply review feedback 
- move setting into clsi.docker namespace
- rename the variable for images to allowedImages / ALLOWED_IMAGES
- add an additional check for the image name into the DockerRunner

Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
 2020-06-30 12:01:21 +01:00
Jakob Ackermann
c857371fed [misc] wordcount: restrict image to an allow list and add tests 2020-06-26 13:28:12 +01:00
Jakob Ackermann
8846efe7ce [misc] RequestParser: restrict imageName to an allow list and add tests 2020-06-26 13:28:09 +01:00
Miguel Serrano
f8ca906393 Fixed NPE when Settings.clsi is defined but Settings.clsi.docker is not 2020-06-25 12:31:10 +02:00
Brian Gough
2b342c6e53 Merge pull request #181 from overleaf/bg-fix-503-response 
handle EPIPE errors in CompileController
 2020-06-22 09:30:35 +01:00
Brian Gough
22480f1a52 handle EPIPE errors in CompileController 2020-06-18 09:54:18 +01:00
Brian Gough
bdebef91eb Merge branch 'master' into bg-add-compile-groups 2020-06-17 11:58:26 +01:00
Brian Gough
f6fddf2944 Merge pull request #179 from overleaf/bg-fix-synctex-error 
fix synctex error
 2020-06-16 08:57:54 +01:00
Brian Gough
2eb4327256 fix format 2020-06-16 08:45:53 +01:00
Brian Gough
f41b4daca2 Merge branch 'master' into bg-use-lodash 2020-06-16 08:35:17 +01:00
Brian Gough
2ce03f0554 add initial compileGroup support 2020-06-15 15:28:53 +01:00
Brian Gough
ede70b6f99 migrate from underscore to lodash 2020-06-15 09:52:21 +01:00
Brian Gough
5368630754 check output file exists before running synctex 2020-06-12 15:15:27 +01:00
Tim Alby
d3ff214b88 partially revert "[DockerRunner] fix metric incrementing and error logging" 
This reverts commits:
- 2b2fcca39ce8dee0fdc0c342aa0d6c822592bcec
- 9e82ab0890c5cc8c7fb95362c3f7edbcaad0cf29
- e3da458b376871c3ce72d6984d14bf1ee668b04b
 2020-06-11 12:51:26 +02:00
Brian Gough
3d6b911295 add setting TEXLIVE_OPENOUT_ANY 2020-06-04 11:47:22 +01:00
Brian Gough
aa44bae222 fix eslint errors 2020-06-02 09:51:34 +01:00
Brian Gough
a3d27b63f1 fix formatting with make format_fix 2020-06-02 09:18:38 +01:00
Brian Gough
d9265b779f Merge pull request #170 from overleaf/jpa-import-141 
[DockerRunner] destroyOldContainers: fix a race confition
 2020-06-02 09:04:59 +01:00
Brian Gough
59cd14e160 Merge pull request #169 from overleaf/bg-record-latexmk-output 
record latexmk output
 2020-06-02 09:03:43 +01:00
Jakob Ackermann
563a973388 [DockerRunner] destroyOldContainers: normalize the container name 
The docker api returns each name with a `/` prefix.

In order to not interfere with pending compiles, the deletion process
 has to acquire an internal lock on the container. The LockManager uses
 the plain container name without the slash: `project-xxx`.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-05-29 11:28:26 +01:00
Brian Gough
6d5dfb7758 clean up log file deletion and add unit test 2020-05-20 14:12:08 +01:00
Brian Gough
63770bf390 clean up the stdout/stderr recording 2020-05-20 11:45:29 +01:00
Henry Oswald
5c3da905ba Merge pull request #162 from overleaf/ta-jpa-epipe-retry 
[DockerRunner] retry container inspect on EPIPE
 2020-05-19 11:15:25 +01:00
Henry Oswald
2547725f35 Merge pull request #164 from overleaf/bg-fix-buffer-deprecations 
fix deprecated usage of Buffer constructor
 2020-05-19 10:26:56 +01:00
Henry Oswald
e9c6f3b074 Merge pull request #163 from overleaf/bg-use-encoding-on-process-output 
set encoding when reading from streams
 2020-05-19 10:26:26 +01:00
Henry Oswald
457b7d6657 Merge pull request #165 from overleaf/ho-retry-url-downloads 
add pipeUrlToFileWithRetry for file downloads
 2020-05-19 10:25:19 +01:00
Henry Oswald
3925839c7f add refreshExpiryTimeout function 
on clsi all data lives inside of / dir
dynamically reduce size of EXPIRY_TIMEOUT if disk starts to get full
 2020-05-18 15:17:19 +01:00
Brian Gough
3db513cfc9 record latexmk output 2020-05-15 16:08:10 +01:00
Henry Oswald
77e8ba74a7 add pipeUrlToFileWithRetry function to retry file downloads 3 times 2020-05-14 13:24:58 +01:00
Brian Gough
4947abe88b fix deprecated usage of Buffer constructor 2020-05-07 10:42:05 +01:00
Brian Gough
8e86f02c43 set encoding when reading from streams 
using .toString() works most of the time but can lead to utf8 characters being
broken across chunk boundaries.

https://nodejs.org/api/stream.html#stream_readable_setencoding_encoding
 2020-05-07 10:30:14 +01:00
Henry Oswald
e1c90ec234 Merge branch 'master' into bg-support-other-runtimes 
port coffee changes to js
 2020-04-23 13:59:05 +01:00
Jakob Ackermann
cf8533bee1 [LocalCommandRunner] run: block a double call of the callback 
The subprocess event handler fires the "error" and "close" event in case
 of a failure.
Both events would call the given callback, resulting in double
 processing of the subprocess result downstream.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-04-16 15:55:55 +02:00
Jakob Ackermann
cb9e9321f0 [DockerRunner] fix metric incrementing and error logging 
- do not log on first EPIPE
- inc 'container-inspect-epipe-error' on permanent error only

Co-Authored-By: Tim Alby <timothee.alby@gmail.com>
 2020-04-10 14:44:57 +02:00
Tim Alby
ecaa7035f5 add metrics for EPIPE errors 
Co-Authored-By: Jakob Ackermann <jakob.ackermann@overleaf.com>
 2020-04-10 12:28:48 +02:00
Tim Alby
e3b1472c7f retry once on EPIPE errors 
Co-Authored-By: Jakob Ackermann <jakob.ackermann@overleaf.com>
 2020-04-10 12:28:11 +02:00
Jakob Ackermann
c700a0ab9d Merge pull request #159 from overleaf/jpa-fix-docker-modem-error-scheme 
[misc] keep up with the error signature of dockerode/docker-modem
 2020-04-03 14:04:06 +02:00
Jakob Ackermann
6777432569 [misc] add a metric for failing downloads 2020-04-03 12:35:40 +02:00
Jakob Ackermann
4a19f022fc [misc] keep up with the error signature of dockerode/docker-modem 
https://github.com/apocas/docker-modem/blob/v2.1.1/lib/modem.js#L296
 2020-03-27 10:39:45 +01:00
mserranom
d5bd1790a8 updated sequelize 2020-03-16 16:31:02 +01:00
mserranom
cf6c8ab496 removed mkdirp dependency and replaced with fs.mkdir 2020-03-12 10:22:08 +01:00
mserranom
fa37ed865a added container monitor cleanup to fix hanging tests 2020-02-20 17:24:28 +01:00
mserranom
8729acd48c prettier: convert app/js decaffeinated files to Prettier format 2020-02-19 12:14:37 +01:00
mserranom
3af6bdd588 decaffeinate: rename app/coffee dir to app/js 2020-02-19 12:14:34 +01:00
decaffeinate
beb6100e25 decaffeinate: Run post-processing cleanups on CommandRunner.coffee and 25 other files 2020-02-19 12:14:28 +01:00
decaffeinate
4eab36e3d5 decaffeinate: Convert CommandRunner.coffee and 25 other files to JS 2020-02-19 12:14:14 +01:00
decaffeinate
2d07bab23d decaffeinate: Rename CommandRunner.coffee and 25 other files from .coffee to .js 2020-02-19 12:14:01 +01:00
mserranom
e37c261bb0 removed unneeded default function arg preventing from decaffeination 2020-02-19 12:06:09 +01:00
Brian Gough
659cd44377 support other runtimes 2020-02-10 16:10:18 +00:00
Eric Mc Sween
58bc71a0d2 Show output files in subfolders 
This fixes a tiny regexp bug that prevents output files in subfolders
from being shown in the "Other logs & files" panel.

We also downgrade the corresponding log because it's very noisy and does
not indicate a problem.
 2019-11-07 08:27:24 -05:00
Eric Mc Sween
dab6e9aa8e Send output files on timeout 
The unconventional use of callbacks to return both an error and data
after compilation created a subtle bug where the output files were
dropped by the LockManager in case of an error such as a timeout.

This prevented the frontend to show error logs when a timeout occurs,
creating confusion among users.

We now attach the output files to the error so that they reach the
controller and are sent back to the web service.
 2019-10-22 16:31:37 -04:00
Tailing Yuan
68e842b12a fix CompileManager and LocalCommandRunner 2019-10-04 23:02:03 +08:00
Shane Kilkelly
880ec16827 Increase the hard-timeout to 10 minutes. 
In practice most projects will still be limited to five minutes,
but this allows us to bump up the limit for some projects,
especially legacy v1 projects that have been imported to v2
 2019-06-06 16:39:16 +01:00
Michael Mazour
1ddf9283f2 Add flags option to request JSON 
Adds a `flags` parameter to the request JSON, appearing under the `compile.options` key (alongside such stalwarts as `compiler`, `timeout`, etc.).

This is primarily to support `-file-line-error` as an option, but could have other uses as well.

`flags` should be an array of strings, or absent. If supplied, the listed arguments are added to the base latexmk command.
 2019-05-14 16:24:34 +01:00
Henry Oswald
900e2f70b4 change console.log for logger.log 2019-03-22 20:42:26 +00:00
Brian Gough
038c81f868 use explicit json content-type to avoid security issues with text/html 2019-02-12 16:54:59 +00:00
Christopher Hoskin
3829732494 Fix broken spacing 2019-01-16 15:12:23 +00:00
Henry Oswald
c67e88fcb3 have failed compiles warn rather than be an error 2018-11-23 15:10:35 +00:00
Brian Gough
55fa22caa9 fix exception when content undefined in TikzManager 2018-10-15 10:01:52 +01:00
Brian Gough
187786b4e4 improve log message 2018-10-12 10:49:54 +01:00
Brian Gough
34acce8bda use TikzManager to create main file for pstool package 2018-10-10 16:13:20 +01:00
Brian Gough
74dd560979 extend caching for tikz, minted and markdown files 2018-10-04 16:56:48 +01:00
Brian Gough
9870683ae8 Merge pull request #96 from sharelatex/bg-cache-eps-to-pdf-converted-files 
cache pdf files generated by epstopdf
 2018-09-11 13:31:26 +01:00
Henry Oswald
3fc3cd11f4 fix unit tests 2018-09-11 10:21:37 +01:00
Henry Oswald
4a7cbef3e5 Merge branch 'master' into ho-dockerise 2018-09-11 10:02:24 +01:00
Henry Oswald
ce44fa3390 bump wordcount timeouts, taken from 82b996b145196711e439d7d7045f53498c1afa1a 2018-09-11 09:55:10 +01:00
Henry Oswald
bec46504eb improve synctex logging 2018-09-11 09:51:20 +01:00
Brian Gough
5a635e5465 cache pdf files generated by epstopdf 2018-09-11 09:44:22 +01:00
Henry Oswald
905a761544 don't error on a bad synctex call 2018-08-23 11:32:50 +01:00
Henry Oswald
250f4a42f0 reduce log level 2018-08-23 11:16:28 +01:00
Henry Oswald
59638c261d fse.ensureDir when running synctex and wordcount 2018-08-23 08:34:18 +01:00
Henry Oswald
2aaadc6124 fix unit tests 2018-08-23 00:21:05 +01:00
Henry Oswald
a80700f3e4 improve error reporting 2018-08-23 00:00:43 +01:00
Henry Oswald
6c0665bb49 change sync to async for lockfile debugging 2018-08-22 22:17:02 +01:00
Henry Oswald
40f4357cd6 fix sql query checking last access time 2018-08-22 18:21:15 +01:00
Henry Oswald
9f5fbefb8a add log on exited error code 2018-08-21 12:02:12 +01:00
Henry Oswald
f0b927e8e2 improve error reporting 2018-08-20 10:12:32 +01:00
Henry Oswald
e61907bf48 added loads of debugging 2018-08-19 11:38:27 +01:00
Henry Oswald
361eaf9217 comment out erroring log for moment 2018-08-01 14:32:17 +01:00
Henry Oswald
6f926007d2 null check host options 2018-08-01 14:10:22 +01:00
Henry Oswald
44c0922a5b reduce logging 2018-08-01 13:59:09 +01:00
Henry Oswald
ac4d07352f make Settings.parallelSqlQueryLimit a config setting 2018-07-31 14:38:24 +01:00
Henry Oswald
cd5bcdd7ca fix expired projects command 2018-07-30 17:37:30 +01:00
Henry Oswald
f9754c4b95 fix missing cb’s 2018-07-30 17:01:59 +01:00
Henry Oswald
759988401b add db queue file for global db query queues 2018-07-30 16:46:47 +01:00
Henry Oswald
1080c2c428 added a queue with 1 concurency to db queries 2018-07-30 16:22:04 +01:00
Henry Oswald
daf40fc757 added some debugging 2018-07-30 15:18:25 +01:00
Henry Oswald
f2dfb718c6 add sync= off and read_uncommited=true to improve perf 2018-07-30 15:16:06 +01:00
Henry Oswald
dc225f6aa3 remove some console.logs 2018-07-30 15:11:41 +01:00
Henry Oswald
1ef8ea328c remove password from clsi for sql 
sequalise fails when it is set to null
 2018-07-30 14:04:33 +01:00
Henry Oswald
65679af0cb dd wal logging 2018-07-26 16:12:26 +01:00
Henry Oswald
e8c735e794 only set wal for sqlite 2018-07-17 12:53:07 +01:00
Henry Oswald
d840246d29 add PRAGMA journal_mode=WAL; 2018-07-17 12:50:33 +01:00