Commit graph

267 commits

Author SHA1 Message Date
Jakob Ackermann
76e749777d [misc] make build scripts happy again
- move pdf.js tests into test/unit/js
- add env override to build script config file
- update build scripts
2021-07-26 12:20:04 +01:00
Jakob Ackermann
0e5f5afe79 [misc] temporary override a few new/changed eslint rules 2021-07-13 12:24:43 +01:00
Jakob Ackermann
f285e503b4 [misc] run format_fix and lint:fix 2021-07-13 12:04:48 +01:00
Jakob Ackermann
64551f0198 [misc] switch from settings-sharelatex to @overleaf/settings 2021-07-12 17:47:21 +01:00
Jakob Ackermann
516525126b [UrlFetcher] do not override domain for clsi-perf requests 2021-07-02 09:17:29 +01:00
Jakob Ackermann
b09e52510f [misc] bail out from pdf caching processing after 10s or earlier
...for fast compiles.
2021-06-23 14:20:04 +01:00
Jakob Ackermann
ffaff1bd72 [CompileController] emit status=failure for an empty output.pdf file 2021-06-22 12:15:12 +01:00
Jakob Ackermann
fb3966ef35 [misc] CompileController: simplify composing of outputFiles 2021-06-22 12:15:12 +01:00
Jakob Ackermann
7380b54900 [ProjectPersistenceManager] check all user content dirs for full disk 2021-06-01 16:13:48 +01:00
Jakob Ackermann
294088fb27 [ContentCacheManager] use PDF.js Xref table instead of stream detection (#242)
* make the content cache manager tests configurable

* extend stream content in unit tests

* [ContentCacheManagerTests] prepare for full object caching

* filesystem stream for pdfjs

* working??

* cleaning up

* handle overflow

* [misc] install pdfjs-dist

* [misc] move pdfjs code into app/lib/ and scripts/, also use CamelCase

* [misc] abstract the file loading and parsing of xRef tables into helper

* [misc] pdfjsTests: add snapshot based tests for the Xref table parser

* [misc] FSStream: throw proper error and drop commented code

* [misc] FSStream: integrate throwing of MissingDataException into getter

* [misc] pdfjs: fix eslint errors

* [misc] pdfjs: run format_fix

* [misc] pdfjs: allocate very small non empty dummy buffers explicitly

* [misc] install @overleaf/o-error

* [ContentCacheManager] use PDF.js Xref table instead of stream detection

Co-Authored-By: Brian Gough <brian.gough@overleaf.com>

* [pdfjs] parseXrefTable: handle empty PDF files gracefully

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
2021-05-31 09:20:25 +01:00
Jakob Ackermann
d56a490121 [misc] AllowedImageNamesTests: add missing done callbacks 2021-05-25 10:24:49 +01:00
Jakob Ackermann
f820d32221 Merge pull request #240 from overleaf/bg-jpa-hash-tracker
[ContentCacheManager] track hashes and expire unused ones
2021-05-21 11:35:05 +02:00
Jakob Ackermann
fdcfbcd6ea Merge pull request #238 from overleaf/jpa-drop-broken-timings
[LatexRunner] do not emit empty cpu/sys timings
2021-05-21 11:34:54 +02:00
Alf Eaton
6507d668ae Update references to .coffee files 2021-05-19 18:57:35 +01:00
Alf Eaton
1b1c4aa4a0 Fix formatting 2021-05-19 18:47:51 +01:00
Alf Eaton
f31abe7cb3 Tidy and format 2021-05-19 18:36:27 +01:00
decaffeinate
85920b3120 decaffeinate: Run post-processing cleanups on settings.test.coffee 2021-05-19 16:50:46 +01:00
decaffeinate
8c984a77c4 decaffeinate: Convert settings.test.coffee to JS 2021-05-19 16:50:45 +01:00
decaffeinate
5af8f57dd0 decaffeinate: Rename settings.test.coffee from .coffee to .js 2021-05-19 16:50:45 +01:00
Jakob Ackermann
b53a105681 [LatexRunner] do not emit empty cpu/sys timings 2021-05-19 11:17:08 +01:00
Jakob Ackermann
7aeeb5a5a9 [ContentCacheManager] finish tracking of ranges across builds 2021-05-18 18:06:15 +01:00
Jakob Ackermann
567d02881d [misc] fix unit tests following the merge of atomic writes 2021-05-18 11:09:30 +01:00
Jakob Ackermann
9b8763aed4 Merge pull request #234 from overleaf/jpa-stream-detection-across-chunks
[ContentCacheManager] add support for stream detection across chunks
2021-05-18 11:44:09 +02:00
Jakob Ackermann
7cf019ab78 Merge pull request #231 from overleaf/bg-use-fse-copy
use fs.copyFile for performance
2021-05-18 11:43:43 +02:00
Jakob Ackermann
224ae0c254 [ContentCacheManager] add support for stream detection across chunks
Retain a small part (6 or 9 bytes) of each chunk in memory for providing
 the next iteration with enough context for finding the start/end marker
 of a range.
2021-05-17 17:36:05 +01:00
Brian Gough
8c80ddd27a remove unnecessary console.log from hash benchmark 2021-05-17 14:14:00 +01:00
Brian Gough
939ad3af31 include UV_THREADPOOL_SIZE in benchmark logs 2021-05-17 14:14:00 +01:00
Brian Gough
cbeba4af00 run the hash benchmark 10 times 2021-05-17 14:14:00 +01:00
Brian Gough
7b6434ef81 add benchmark script for hashing 2021-05-17 14:14:00 +01:00
Brian Gough
371de76a4a use fs.copyFile instead of fse.copy in UrlCache module 2021-05-17 10:54:11 +01:00
Brian Gough
44e0742aa3 use fse.copy for performance
it uses the native fs.copyFile method
2021-05-14 15:49:20 +01:00
Jakob Ackermann
b456ea726d [misc] merge pdf caching into main (#226)
* wip generate directory for hash content

* cleanup, remove console logging

* add content caching module

* Return PDF stream ranges with compile response

* Return the PDF file size in the compile response

* PDF range endpoint

* [misc] WIP: pdf caching: preserve the m-time on static content files

* [misc] WIP: pdf caching: improve browser caching, emit caching headers

* [misc] WIP: pdf caching: do not emit very small chunks <1kB

* [misc] keep up with moving output files into a separate directory

* [OutputCacheManager] add global feature flag for enabling pdf caching

* [misc] add contentId into the URL for protecting PDF stream contents

* [misc] support PDF stream caching for anonymous users

* [misc] add per-request feature flag for enabling PDF stream caching

* [misc] enable pdf caching in CI and emit metrics at the end of run

* [misc] expose compile stats and timings to the frontend

* [misc] log an error in case saving output files fails

* [misc] add metrics for pdf bandwidth and pdf caching performance

* [misc] add a dark mode to the pdf caching for computing ranges only

* [misc] move pdf caching metrics into ContentCacheMetrics

* [misc] add a config option for the min chunk size of pdf ranges

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
2021-05-13 14:07:54 +01:00
Jakob Ackermann
6cb4246eb1 [misc] consume and validate a custom imageName for synctex requests 2021-03-30 13:57:40 +01:00
Eric Mc Sween
9f2d219102 Add a global test setup file
Set up generally useful stuff:

* chai.should()
* logger stubs
* globals in SandboxedModule, including Buffer and process, which are
  now required in Node 12
2021-03-12 16:27:33 -05:00
Brian Gough
a86e521ac0 add unit test for non-existent state file 2021-01-26 11:03:18 +00:00
Brian Gough
f41f3aba8c Merge pull request #204 from overleaf/bg-use-output-dir
use separate output dir for generated files
2021-01-25 14:10:09 +00:00
Brian Gough
90c0d931ad Merge pull request #201 from overleaf/bg-more-cleanup
decaff cleanup of OutputFileManager
2021-01-25 14:09:46 +00:00
Brian Gough
b5346658b0 clear output directory when clearing project 2021-01-22 11:05:52 +00:00
Jakob Ackermann
49b764a308 [misc] CompileController: exact match for output.pdf
The previous regex could mistake user provided pdf files, like
 `fake_output.pdf`, as the final output file.
The frontend expects to find a `output.pdf` file on success.
2020-12-28 13:16:31 +00:00
Brian Gough
4169f7fc73 use once for callback 2020-12-18 15:28:06 +00:00
Eric Mc Sween
14e1e02a68 Test anonymous volumes are removed with containers 2020-09-04 11:34:08 -04:00
Eric Mc Sween
1c13f6fe94 Mount /home/tex in an anonymous volume
When we mount the container's root filesystem as read-only, mount
an anonymous volume in /home/tex so that it's writable. Our TeX Live
images have cached content in /home/tex. This content will automatically
get copied by Docker into this anonymous volume.
2020-09-04 08:49:56 -04:00
Jakob Ackermann
1d1b9ebebc [misc] bump the dev-env to 3.3.2 2020-08-10 17:01:11 +01:00
Jakob Ackermann
0f9d0d9790 Merge pull request #183 from overleaf/jpa-clsi-allowed-image-names
[misc] RequestParser: restrict imageName to an allow list and add tests
2020-07-15 10:58:36 +02:00
Jakob Ackermann
cd87d3018e [misc] fix formatting 2020-07-03 11:47:53 +01:00
Jakob Ackermann
a5c3bad7f1 [ExampleDocumentTests] drop out in case of an error during compilation
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-07-03 11:38:12 +01:00
Jakob Ackermann
aa02df7b81 [misc] apply review feedback
- move setting into clsi.docker namespace
- rename the variable for images to allowedImages / ALLOWED_IMAGES
- add an additional check for the image name into the DockerRunner

Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
2020-06-30 12:01:21 +01:00
Jakob Ackermann
c857371fed [misc] wordcount: restrict image to an allow list and add tests 2020-06-26 13:28:12 +01:00
Jakob Ackermann
8846efe7ce [misc] RequestParser: restrict imageName to an allow list and add tests 2020-06-26 13:28:09 +01:00
Brian Gough
bdebef91eb Merge branch 'master' into bg-add-compile-groups 2020-06-17 11:58:26 +01:00