github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-07 20:31:06 -05:00
Code Issues Projects Releases Packages Wiki Activity
19029 commits 3 branches 7 tags 201 MiB
Commit graph

293 commits

Author SHA1 Message Date
Jakob Ackermann
224ae0c254 [ContentCacheManager] add support for stream detection across chunks 
Retain a small part (6 or 9 bytes) of each chunk in memory for providing
 the next iteration with enough context for finding the start/end marker
 of a range.
 2021-05-17 17:36:05 +01:00
Brian Gough
8c80ddd27a remove unnecessary console.log from hash benchmark 2021-05-17 14:14:00 +01:00
Brian Gough
939ad3af31 include UV_THREADPOOL_SIZE in benchmark logs 2021-05-17 14:14:00 +01:00
Brian Gough
cbeba4af00 run the hash benchmark 10 times 2021-05-17 14:14:00 +01:00
Brian Gough
7b6434ef81 add benchmark script for hashing 2021-05-17 14:14:00 +01:00
Brian Gough
371de76a4a use fs.copyFile instead of fse.copy in UrlCache module 2021-05-17 10:54:11 +01:00
Brian Gough
44e0742aa3 use fse.copy for performance 
it uses the native fs.copyFile method
 2021-05-14 15:49:20 +01:00
Jakob Ackermann
b456ea726d [misc] merge pdf caching into main (#226) 
* wip generate directory for hash content

* cleanup, remove console logging

* add content caching module

* Return PDF stream ranges with compile response

* Return the PDF file size in the compile response

* PDF range endpoint

* [misc] WIP: pdf caching: preserve the m-time on static content files

* [misc] WIP: pdf caching: improve browser caching, emit caching headers

* [misc] WIP: pdf caching: do not emit very small chunks <1kB

* [misc] keep up with moving output files into a separate directory

* [OutputCacheManager] add global feature flag for enabling pdf caching

* [misc] add contentId into the URL for protecting PDF stream contents

* [misc] support PDF stream caching for anonymous users

* [misc] add per-request feature flag for enabling PDF stream caching

* [misc] enable pdf caching in CI and emit metrics at the end of run

* [misc] expose compile stats and timings to the frontend

* [misc] log an error in case saving output files fails

* [misc] add metrics for pdf bandwidth and pdf caching performance

* [misc] add a dark mode to the pdf caching for computing ranges only

* [misc] move pdf caching metrics into ContentCacheMetrics

* [misc] add a config option for the min chunk size of pdf ranges

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
 2021-05-13 14:07:54 +01:00
Jakob Ackermann
6cb4246eb1 [misc] consume and validate a custom imageName for synctex requests 2021-03-30 13:57:40 +01:00
Eric Mc Sween
9f2d219102 Add a global test setup file 
Set up generally useful stuff:

* chai.should()
* logger stubs
* globals in SandboxedModule, including Buffer and process, which are
  now required in Node 12
 2021-03-12 16:27:33 -05:00
Brian Gough
a86e521ac0 add unit test for non-existent state file 2021-01-26 11:03:18 +00:00
Brian Gough
f41f3aba8c Merge pull request #204 from overleaf/bg-use-output-dir 
use separate output dir for generated files
 2021-01-25 14:10:09 +00:00
Brian Gough
90c0d931ad Merge pull request #201 from overleaf/bg-more-cleanup 
decaff cleanup of OutputFileManager
 2021-01-25 14:09:46 +00:00
Brian Gough
b5346658b0 clear output directory when clearing project 2021-01-22 11:05:52 +00:00
Jakob Ackermann
49b764a308 [misc] CompileController: exact match for output.pdf 
The previous regex could mistake user provided pdf files, like
 `fake_output.pdf`, as the final output file.
The frontend expects to find a `output.pdf` file on success.
 2020-12-28 13:16:31 +00:00
Brian Gough
4169f7fc73 use once for callback 2020-12-18 15:28:06 +00:00
Eric Mc Sween
14e1e02a68 Test anonymous volumes are removed with containers 2020-09-04 11:34:08 -04:00
Eric Mc Sween
1c13f6fe94 Mount /home/tex in an anonymous volume 
When we mount the container's root filesystem as read-only, mount
an anonymous volume in /home/tex so that it's writable. Our TeX Live
images have cached content in /home/tex. This content will automatically
get copied by Docker into this anonymous volume.
 2020-09-04 08:49:56 -04:00
Jakob Ackermann
1d1b9ebebc [misc] bump the dev-env to 3.3.2 2020-08-10 17:01:11 +01:00
Jakob Ackermann
0f9d0d9790 Merge pull request #183 from overleaf/jpa-clsi-allowed-image-names 
[misc] RequestParser: restrict imageName to an allow list and add tests
 2020-07-15 10:58:36 +02:00
Jakob Ackermann
cd87d3018e [misc] fix formatting 2020-07-03 11:47:53 +01:00
Jakob Ackermann
a5c3bad7f1 [ExampleDocumentTests] drop out in case of an error during compilation 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-07-03 11:38:12 +01:00
Jakob Ackermann
aa02df7b81 [misc] apply review feedback 
- move setting into clsi.docker namespace
- rename the variable for images to allowedImages / ALLOWED_IMAGES
- add an additional check for the image name into the DockerRunner

Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
 2020-06-30 12:01:21 +01:00
Jakob Ackermann
c857371fed [misc] wordcount: restrict image to an allow list and add tests 2020-06-26 13:28:12 +01:00
Jakob Ackermann
8846efe7ce [misc] RequestParser: restrict imageName to an allow list and add tests 2020-06-26 13:28:09 +01:00
Brian Gough
bdebef91eb Merge branch 'master' into bg-add-compile-groups 2020-06-17 11:58:26 +01:00
Brian Gough
f6fddf2944 Merge pull request #179 from overleaf/bg-fix-synctex-error 
fix synctex error
 2020-06-16 08:57:54 +01:00
Brian Gough
f41b4daca2 Merge branch 'master' into bg-use-lodash 2020-06-16 08:35:17 +01:00
Brian Gough
2ce03f0554 add initial compileGroup support 2020-06-15 15:28:53 +01:00
Brian Gough
262ea01911 add acceptance test for synctex when project/file does not exist 2020-06-15 10:55:01 +01:00
Brian Gough
ede70b6f99 migrate from underscore to lodash 2020-06-15 09:52:21 +01:00
Brian Gough
f99023320d use json parsing in request 2020-06-12 15:15:51 +01:00
Tim Alby
d3ff214b88 partially revert "[DockerRunner] fix metric incrementing and error logging" 
This reverts commits:
- 2b2fcca39ce8dee0fdc0c342aa0d6c822592bcec
- 9e82ab0890c5cc8c7fb95362c3f7edbcaad0cf29
- e3da458b376871c3ce72d6984d14bf1ee668b04b
 2020-06-11 12:51:26 +02:00
Brian Gough
ce1f2e2bd4 fix broken unit test 2020-06-02 11:12:57 +01:00
Brian Gough
aa44bae222 fix eslint errors 2020-06-02 09:51:34 +01:00
Brian Gough
0de2933812 fix unreachable code lint error 2020-06-02 09:28:04 +01:00
Brian Gough
a3d27b63f1 fix formatting with make format_fix 2020-06-02 09:18:38 +01:00
Brian Gough
d9265b779f Merge pull request #170 from overleaf/jpa-import-141 
[DockerRunner] destroyOldContainers: fix a race confition
 2020-06-02 09:04:59 +01:00
Brian Gough
59cd14e160 Merge pull request #169 from overleaf/bg-record-latexmk-output 
record latexmk output
 2020-06-02 09:03:43 +01:00
Jakob Ackermann
563a973388 [DockerRunner] destroyOldContainers: normalize the container name 
The docker api returns each name with a `/` prefix.

In order to not interfere with pending compiles, the deletion process
 has to acquire an internal lock on the container. The LockManager uses
 the plain container name without the slash: `project-xxx`.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-05-29 11:28:26 +01:00
Brian Gough
6d5dfb7758 clean up log file deletion and add unit test 2020-05-20 14:12:08 +01:00
Brian Gough
a684619bce add unit tests 2020-05-20 11:52:53 +01:00
Henry Oswald
5c3da905ba Merge pull request #162 from overleaf/ta-jpa-epipe-retry 
[DockerRunner] retry container inspect on EPIPE
 2020-05-19 11:15:25 +01:00
Jakob Ackermann
2a3c2dd3d5 [misc] simplify the smoke test and process shutdown 2020-05-19 10:30:59 +01:00
Henry Oswald
2547725f35 Merge pull request #164 from overleaf/bg-fix-buffer-deprecations 
fix deprecated usage of Buffer constructor
 2020-05-19 10:26:56 +01:00
Henry Oswald
e9c6f3b074 Merge pull request #163 from overleaf/bg-use-encoding-on-process-output 
set encoding when reading from streams
 2020-05-19 10:26:26 +01:00
Henry Oswald
457b7d6657 Merge pull request #165 from overleaf/ho-retry-url-downloads 
add pipeUrlToFileWithRetry for file downloads
 2020-05-19 10:25:19 +01:00
Henry Oswald
3925839c7f add refreshExpiryTimeout function 
on clsi all data lives inside of / dir
dynamically reduce size of EXPIRY_TIMEOUT if disk starts to get full
 2020-05-18 15:17:19 +01:00
Henry Oswald
77e8ba74a7 add pipeUrlToFileWithRetry function to retry file downloads 3 times 2020-05-14 13:24:58 +01:00
Brian Gough
4947abe88b fix deprecated usage of Buffer constructor 2020-05-07 10:42:05 +01:00
Brian Gough
8e86f02c43 set encoding when reading from streams 
using .toString() works most of the time but can lead to utf8 characters being
broken across chunk boundaries.

https://nodejs.org/api/stream.html#stream_readable_setencoding_encoding
 2020-05-07 10:30:14 +01:00
Tim Alby
e3b1472c7f retry once on EPIPE errors 
Co-Authored-By: Jakob Ackermann <jakob.ackermann@overleaf.com>
 2020-04-10 12:28:11 +02:00
Tim Alby
66447211d0 fix arguments order 
Co-Authored-By: Jakob Ackermann <jakob.ackermann@overleaf.com>
 2020-04-10 12:27:15 +02:00
Jakob Ackermann
6a615d50d3 Merge remote-tracking branch 'origin/master' into jpa-entry-point-cleanup 2020-04-03 14:05:36 +02:00
Jakob Ackermann
c700a0ab9d Merge pull request #159 from overleaf/jpa-fix-docker-modem-error-scheme 
[misc] keep up with the error signature of dockerode/docker-modem
 2020-04-03 14:04:06 +02:00
Jakob Ackermann
101c4ea4dc [misc] use a directory in /tmp for temporary data 2020-04-03 12:36:32 +02:00
Jakob Ackermann
6777432569 [misc] add a metric for failing downloads 2020-04-03 12:35:40 +02:00
Jakob Ackermann
4a19f022fc [misc] keep up with the error signature of dockerode/docker-modem 
https://github.com/apocas/docker-modem/blob/v2.1.1/lib/modem.js#L296
 2020-03-27 10:39:45 +01:00
mserranom
8641d0ef0f updated sandboxed-module, chai and metrics-sharelatex 2020-03-16 17:14:04 +01:00
mserranom
7e7d234350 updated mocha and sinon, fixed tests 2020-03-12 10:35:11 +01:00
mserranom
cf6c8ab496 removed mkdirp dependency and replaced with fs.mkdir 2020-03-12 10:22:08 +01:00
mserranom
41533d8888 moved decaffeinated files to js folder 2020-02-25 17:15:31 +01:00
decaffeinate
252f4c704b decaffeinate: Run post-processing cleanups on loadTest.coffee 2020-02-25 17:03:58 +01:00
decaffeinate
3901fe9cd2 decaffeinate: Convert loadTest.coffee to JS 2020-02-25 17:03:57 +01:00
decaffeinate
e20da38e09 decaffeinate: Rename loadTest.coffee from .coffee to .js 2020-02-25 17:03:57 +01:00
decaffeinate
2057a565ec decaffeinate: Run post-processing cleanups on SmokeTests.coffee 2020-02-25 17:03:39 +01:00
decaffeinate
b6991f5ff9 decaffeinate: Convert SmokeTests.coffee to JS 2020-02-25 17:03:37 +01:00
decaffeinate
92ed86bb33 decaffeinate: Rename SmokeTests.coffee from .coffee to .js 2020-02-25 17:03:36 +01:00
mserranom
fa37ed865a added container monitor cleanup to fix hanging tests 2020-02-20 17:24:28 +01:00
mserranom
4c0fd2b6f4 fixed test paths 2020-02-19 12:38:54 +01:00
mserranom
778f2a8420 prettier: convert test/acceptance decaffeinated files to Prettier format 2020-02-19 12:16:14 +01:00
mserranom
c5a6496b71 decaffeinate: rename test/acceptance/coffee to test/acceptance/js 2020-02-19 12:16:11 +01:00
decaffeinate
d1da978e39 decaffeinate: Run post-processing cleanups on BrokenLatexFileTests.coffee and 9 other files 2020-02-19 12:16:07 +01:00
decaffeinate
3d3861cb24 decaffeinate: Convert BrokenLatexFileTests.coffee and 9 other files to JS 2020-02-19 12:16:00 +01:00
decaffeinate
035786b204 decaffeinate: Rename BrokenLatexFileTests.coffee and 9 other files from .coffee to .js 2020-02-19 12:15:54 +01:00
mserranom
a62d8186e9 prettier: convert test/unit decaffeinated files to Prettier format 2020-02-19 12:15:37 +01:00
mserranom
89360bfe77 decaffeinate: rename test/unit/coffee to test/unit/js 2020-02-19 12:15:30 +01:00
decaffeinate
4c79f22389 decaffeinate: Run post-processing cleanups on CompileControllerTests.coffee and 17 other files 2020-02-19 12:15:25 +01:00
decaffeinate
37b4e96de4 decaffeinate: Convert CompileControllerTests.coffee and 17 other files to JS 2020-02-19 12:15:08 +01:00
decaffeinate
e57097afb4 decaffeinate: Rename CompileControllerTests.coffee and 17 other files from .coffee to .js 2020-02-19 12:14:56 +01:00
Eric Mc Sween
dab6e9aa8e Send output files on timeout 
The unconventional use of callbacks to return both an error and data
after compilation created a subtle bug where the output files were
dropped by the LockManager in case of an error such as a timeout.

This prevented the frontend to show error logs when a timeout occurs,
creating confusion among users.

We now attach the output files to the error so that they reach the
controller and are sent back to the web service.
 2019-10-22 16:31:37 -04:00
Brian Gough
418bda28e1 Merge pull request #119 from overleaf/bg-increase-acceptance-test-timeout 
increase timeout for long-running acceptance tests
 2019-05-16 09:17:26 +01:00
Michael Mazour
1ddf9283f2 Add flags option to request JSON 
Adds a `flags` parameter to the request JSON, appearing under the `compile.options` key (alongside such stalwarts as `compiler`, `timeout`, etc.).

This is primarily to support `-file-line-error` as an option, but could have other uses as well.

`flags` should be an array of strings, or absent. If supplied, the listed arguments are added to the base latexmk command.
 2019-05-14 16:24:34 +01:00
Brian Gough
bc6e560448 increase timeout for long-running acceptance tests 2019-04-01 09:42:54 +01:00
Brian Gough
038c81f868 use explicit json content-type to avoid security issues with text/html 2019-02-12 16:54:59 +00:00
Christopher Hoskin
7a3f1c81bd Switch to node 10 2019-01-16 15:11:49 +00:00
Henry Oswald
c67e88fcb3 have failed compiles warn rather than be an error 2018-11-23 15:10:35 +00:00
Brian Gough
34acce8bda use TikzManager to create main file for pstool package 2018-10-10 16:13:20 +01:00
Brian Gough
74dd560979 extend caching for tikz, minted and markdown files 2018-10-04 16:56:48 +01:00
Brian Gough
9870683ae8 Merge pull request #96 from sharelatex/bg-cache-eps-to-pdf-converted-files 
cache pdf files generated by epstopdf
 2018-09-11 13:31:26 +01:00
Henry Oswald
3c12f60eeb change timeout test latex code 2018-09-11 11:34:25 +01:00
Henry Oswald
3fc3cd11f4 fix unit tests 2018-09-11 10:21:37 +01:00
Henry Oswald
ce44fa3390 bump wordcount timeouts, taken from 82b996b145196711e439d7d7045f53498c1afa1a 2018-09-11 09:55:10 +01:00
Brian Gough
5a635e5465 cache pdf files generated by epstopdf 2018-09-11 09:44:22 +01:00
Henry Oswald
2aaadc6124 fix unit tests 2018-08-23 00:21:05 +01:00
Henry Oswald
6c0665bb49 change sync to async for lockfile debugging 2018-08-22 22:17:02 +01:00
Henry Oswald
e46b6563c0 change override to leave image name so it works for wl_texlive 2018-07-16 17:25:14 +01:00
Henry Oswald
846c2b1e28 move texliveImageNameOveride further down request so it works for 
compile tests
 2018-07-13 11:52:49 +01:00
Henry Oswald
4ec8a423cb added texliveImageNameOveride 2018-07-13 10:37:22 +01:00
Henry Oswald
eb89ca1d3b added filestoreDomainOveride 2018-07-03 16:41:34 +01:00