Jakob Ackermann
9c9beef305
Merge pull request #5737 from overleaf/jpa-clsi-cleanup-in-memory
...
[clsi] separate clearing cache and clearing output cache
GitOrigin-RevId: 9cd903eaf8d291d4821ba8c9ac6c1a9f8d3d9632
2021-11-16 09:02:36 +00:00
Jakob Ackermann
f6795a324a
Merge pull request #5572 from overleaf/jpa-no-path-concat
...
[misc] fix eslint violations for node/no-path-concat
GitOrigin-RevId: 5122826fb9ae23c373e8c5a6802ebb35eb20314f
2021-10-28 08:03:37 +00:00
Jakob Ackermann
669606f797
Merge pull request #5568 from overleaf/jpa-node-no-callback-literal
...
[misc] fix eslint violations for node/no-callback-literal
GitOrigin-RevId: af5aace52e1476b1b0ee48cc8be2aabbe04efac6
2021-10-28 08:03:33 +00:00
Jakob Ackermann
7f9fd00bda
Merge pull request #5367 from overleaf/jpa-node-handle-callback-err
...
[misc] fix eslint violations for node/handle-callback-err
GitOrigin-RevId: 83a4900e8861010df1917bff49382bd9c93375bd
2021-10-28 08:03:26 +00:00
Jakob Ackermann
a6f05109a3
Merge pull request #5352 from overleaf/jpa-no-var
...
[misc] fix eslint violations for `no-var`
GitOrigin-RevId: c52e82f3a8a993b8662cc5aa56e7b95ca3c55832
2021-10-27 08:03:00 +00:00
Jakob Ackermann
94c208311c
Merge pull request #5351 from overleaf/jpa-clsi-drop-sqlite
...
[clsi] goodbye sqlite db
GitOrigin-RevId: e7f16920c44e74a425b92884b48a60272dfb015b
2021-10-07 08:03:50 +00:00
Jakob Ackermann
02918e7483
Merge pull request #4649 from overleaf/jpa-fs-based-caching
...
[perf] UrlCache: pure fs based cache state for downloads
GitOrigin-RevId: d19afc396324d4c3318b31620c8ad0c04e0544ce
2021-10-07 08:03:18 +00:00
Brian Gough
cc1b73336a
Merge pull request #4944 from overleaf/bg-use-file-line-errors
...
use file line errors in log output
GitOrigin-RevId: 6732b19552fe15431a70fbefbc572253c389c64e
2021-09-22 08:03:25 +00:00
Eric Mc Sween
9ee92daea3
Merge pull request #4893 from overleaf/em-synctex
...
Use the synctex distributed with TeX Live
GitOrigin-RevId: 5a133f21f48fd1e217ab463e8cb2a5cdec8be1af
2021-09-07 16:26:17 +00:00
Eric Mc Sween
2feea2592d
Merge pull request #4887 from overleaf/em-decaf
...
Decaf cleanup for the CLSI CompileManager
GitOrigin-RevId: 06bba5c8af8808d0fa04187b10c8f31e08cd8754
2021-08-31 08:03:32 +00:00
Jakob Ackermann
5122558469
Merge pull request #4842 from overleaf/jpa-drop-unused-packages
...
[misc] drop unused debug packages
GitOrigin-RevId: 83b1d96ee5a62c14c4a2dc51cefd179306a75a84
2021-08-27 08:03:17 +00:00
Jakob Ackermann
76e749777d
[misc] make build scripts happy again
...
- move pdf.js tests into test/unit/js
- add env override to build script config file
- update build scripts
2021-07-26 12:20:04 +01:00
Jakob Ackermann
0e5f5afe79
[misc] temporary override a few new/changed eslint rules
2021-07-13 12:24:43 +01:00
Jakob Ackermann
f285e503b4
[misc] run format_fix and lint:fix
2021-07-13 12:04:48 +01:00
Jakob Ackermann
64551f0198
[misc] switch from settings-sharelatex to @overleaf/settings
2021-07-12 17:47:21 +01:00
Jakob Ackermann
516525126b
[UrlFetcher] do not override domain for clsi-perf requests
2021-07-02 09:17:29 +01:00
Jakob Ackermann
b09e52510f
[misc] bail out from pdf caching processing after 10s or earlier
...
...for fast compiles.
2021-06-23 14:20:04 +01:00
Jakob Ackermann
ffaff1bd72
[CompileController] emit status=failure for an empty output.pdf file
2021-06-22 12:15:12 +01:00
Jakob Ackermann
fb3966ef35
[misc] CompileController: simplify composing of outputFiles
2021-06-22 12:15:12 +01:00
Jakob Ackermann
7380b54900
[ProjectPersistenceManager] check all user content dirs for full disk
2021-06-01 16:13:48 +01:00
Jakob Ackermann
294088fb27
[ContentCacheManager] use PDF.js Xref table instead of stream detection ( #242 )
...
* make the content cache manager tests configurable
* extend stream content in unit tests
* [ContentCacheManagerTests] prepare for full object caching
* filesystem stream for pdfjs
* working??
* cleaning up
* handle overflow
* [misc] install pdfjs-dist
* [misc] move pdfjs code into app/lib/ and scripts/, also use CamelCase
* [misc] abstract the file loading and parsing of xRef tables into helper
* [misc] pdfjsTests: add snapshot based tests for the Xref table parser
* [misc] FSStream: throw proper error and drop commented code
* [misc] FSStream: integrate throwing of MissingDataException into getter
* [misc] pdfjs: fix eslint errors
* [misc] pdfjs: run format_fix
* [misc] pdfjs: allocate very small non empty dummy buffers explicitly
* [misc] install @overleaf/o-error
* [ContentCacheManager] use PDF.js Xref table instead of stream detection
Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
* [pdfjs] parseXrefTable: handle empty PDF files gracefully
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
2021-05-31 09:20:25 +01:00
Jakob Ackermann
d56a490121
[misc] AllowedImageNamesTests: add missing done callbacks
2021-05-25 10:24:49 +01:00
Jakob Ackermann
f820d32221
Merge pull request #240 from overleaf/bg-jpa-hash-tracker
...
[ContentCacheManager] track hashes and expire unused ones
2021-05-21 11:35:05 +02:00
Jakob Ackermann
fdcfbcd6ea
Merge pull request #238 from overleaf/jpa-drop-broken-timings
...
[LatexRunner] do not emit empty cpu/sys timings
2021-05-21 11:34:54 +02:00
Alf Eaton
6507d668ae
Update references to .coffee files
2021-05-19 18:57:35 +01:00
Alf Eaton
1b1c4aa4a0
Fix formatting
2021-05-19 18:47:51 +01:00
Alf Eaton
f31abe7cb3
Tidy and format
2021-05-19 18:36:27 +01:00
decaffeinate
85920b3120
decaffeinate: Run post-processing cleanups on settings.test.coffee
2021-05-19 16:50:46 +01:00
decaffeinate
8c984a77c4
decaffeinate: Convert settings.test.coffee to JS
2021-05-19 16:50:45 +01:00
decaffeinate
5af8f57dd0
decaffeinate: Rename settings.test.coffee from .coffee to .js
2021-05-19 16:50:45 +01:00
Jakob Ackermann
b53a105681
[LatexRunner] do not emit empty cpu/sys timings
2021-05-19 11:17:08 +01:00
Jakob Ackermann
7aeeb5a5a9
[ContentCacheManager] finish tracking of ranges across builds
2021-05-18 18:06:15 +01:00
Jakob Ackermann
567d02881d
[misc] fix unit tests following the merge of atomic writes
2021-05-18 11:09:30 +01:00
Jakob Ackermann
9b8763aed4
Merge pull request #234 from overleaf/jpa-stream-detection-across-chunks
...
[ContentCacheManager] add support for stream detection across chunks
2021-05-18 11:44:09 +02:00
Jakob Ackermann
7cf019ab78
Merge pull request #231 from overleaf/bg-use-fse-copy
...
use fs.copyFile for performance
2021-05-18 11:43:43 +02:00
Jakob Ackermann
224ae0c254
[ContentCacheManager] add support for stream detection across chunks
...
Retain a small part (6 or 9 bytes) of each chunk in memory for providing
the next iteration with enough context for finding the start/end marker
of a range.
2021-05-17 17:36:05 +01:00
Brian Gough
8c80ddd27a
remove unnecessary console.log from hash benchmark
2021-05-17 14:14:00 +01:00
Brian Gough
939ad3af31
include UV_THREADPOOL_SIZE in benchmark logs
2021-05-17 14:14:00 +01:00
Brian Gough
cbeba4af00
run the hash benchmark 10 times
2021-05-17 14:14:00 +01:00
Brian Gough
7b6434ef81
add benchmark script for hashing
2021-05-17 14:14:00 +01:00
Brian Gough
371de76a4a
use fs.copyFile instead of fse.copy in UrlCache module
2021-05-17 10:54:11 +01:00
Brian Gough
44e0742aa3
use fse.copy for performance
...
it uses the native fs.copyFile method
2021-05-14 15:49:20 +01:00
Jakob Ackermann
b456ea726d
[misc] merge pdf caching into main ( #226 )
...
* wip generate directory for hash content
* cleanup, remove console logging
* add content caching module
* Return PDF stream ranges with compile response
* Return the PDF file size in the compile response
* PDF range endpoint
* [misc] WIP: pdf caching: preserve the m-time on static content files
* [misc] WIP: pdf caching: improve browser caching, emit caching headers
* [misc] WIP: pdf caching: do not emit very small chunks <1kB
* [misc] keep up with moving output files into a separate directory
* [OutputCacheManager] add global feature flag for enabling pdf caching
* [misc] add contentId into the URL for protecting PDF stream contents
* [misc] support PDF stream caching for anonymous users
* [misc] add per-request feature flag for enabling PDF stream caching
* [misc] enable pdf caching in CI and emit metrics at the end of run
* [misc] expose compile stats and timings to the frontend
* [misc] log an error in case saving output files fails
* [misc] add metrics for pdf bandwidth and pdf caching performance
* [misc] add a dark mode to the pdf caching for computing ranges only
* [misc] move pdf caching metrics into ContentCacheMetrics
* [misc] add a config option for the min chunk size of pdf ranges
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
2021-05-13 14:07:54 +01:00
Jakob Ackermann
6cb4246eb1
[misc] consume and validate a custom imageName for synctex requests
2021-03-30 13:57:40 +01:00
Eric Mc Sween
9f2d219102
Add a global test setup file
...
Set up generally useful stuff:
* chai.should()
* logger stubs
* globals in SandboxedModule, including Buffer and process, which are
now required in Node 12
2021-03-12 16:27:33 -05:00
Brian Gough
a86e521ac0
add unit test for non-existent state file
2021-01-26 11:03:18 +00:00
Brian Gough
f41f3aba8c
Merge pull request #204 from overleaf/bg-use-output-dir
...
use separate output dir for generated files
2021-01-25 14:10:09 +00:00
Brian Gough
90c0d931ad
Merge pull request #201 from overleaf/bg-more-cleanup
...
decaff cleanup of OutputFileManager
2021-01-25 14:09:46 +00:00
Brian Gough
b5346658b0
clear output directory when clearing project
2021-01-22 11:05:52 +00:00
Jakob Ackermann
49b764a308
[misc] CompileController: exact match for output.pdf
...
The previous regex could mistake user provided pdf files, like
`fake_output.pdf`, as the final output file.
The frontend expects to find a `output.pdf` file on success.
2020-12-28 13:16:31 +00:00
Brian Gough
4169f7fc73
use once for callback
2020-12-18 15:28:06 +00:00
Eric Mc Sween
14e1e02a68
Test anonymous volumes are removed with containers
2020-09-04 11:34:08 -04:00
Eric Mc Sween
1c13f6fe94
Mount /home/tex in an anonymous volume
...
When we mount the container's root filesystem as read-only, mount
an anonymous volume in /home/tex so that it's writable. Our TeX Live
images have cached content in /home/tex. This content will automatically
get copied by Docker into this anonymous volume.
2020-09-04 08:49:56 -04:00
Jakob Ackermann
1d1b9ebebc
[misc] bump the dev-env to 3.3.2
2020-08-10 17:01:11 +01:00
Jakob Ackermann
0f9d0d9790
Merge pull request #183 from overleaf/jpa-clsi-allowed-image-names
...
[misc] RequestParser: restrict imageName to an allow list and add tests
2020-07-15 10:58:36 +02:00
Jakob Ackermann
cd87d3018e
[misc] fix formatting
2020-07-03 11:47:53 +01:00
Jakob Ackermann
a5c3bad7f1
[ExampleDocumentTests] drop out in case of an error during compilation
...
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-07-03 11:38:12 +01:00
Jakob Ackermann
aa02df7b81
[misc] apply review feedback
...
- move setting into clsi.docker namespace
- rename the variable for images to allowedImages / ALLOWED_IMAGES
- add an additional check for the image name into the DockerRunner
Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
2020-06-30 12:01:21 +01:00
Jakob Ackermann
c857371fed
[misc] wordcount: restrict image to an allow list and add tests
2020-06-26 13:28:12 +01:00
Jakob Ackermann
8846efe7ce
[misc] RequestParser: restrict imageName to an allow list and add tests
2020-06-26 13:28:09 +01:00
Brian Gough
bdebef91eb
Merge branch 'master' into bg-add-compile-groups
2020-06-17 11:58:26 +01:00
Brian Gough
f6fddf2944
Merge pull request #179 from overleaf/bg-fix-synctex-error
...
fix synctex error
2020-06-16 08:57:54 +01:00
Brian Gough
f41b4daca2
Merge branch 'master' into bg-use-lodash
2020-06-16 08:35:17 +01:00
Brian Gough
2ce03f0554
add initial compileGroup support
2020-06-15 15:28:53 +01:00
Brian Gough
262ea01911
add acceptance test for synctex when project/file does not exist
2020-06-15 10:55:01 +01:00
Brian Gough
ede70b6f99
migrate from underscore to lodash
2020-06-15 09:52:21 +01:00
Brian Gough
f99023320d
use json parsing in request
2020-06-12 15:15:51 +01:00
Tim Alby
d3ff214b88
partially revert "[DockerRunner] fix metric incrementing and error logging"
...
This reverts commits:
- 2b2fcca39ce8dee0fdc0c342aa0d6c822592bcec
- 9e82ab0890c5cc8c7fb95362c3f7edbcaad0cf29
- e3da458b376871c3ce72d6984d14bf1ee668b04b
2020-06-11 12:51:26 +02:00
Brian Gough
ce1f2e2bd4
fix broken unit test
2020-06-02 11:12:57 +01:00
Brian Gough
aa44bae222
fix eslint errors
2020-06-02 09:51:34 +01:00
Brian Gough
0de2933812
fix unreachable code lint error
2020-06-02 09:28:04 +01:00
Brian Gough
a3d27b63f1
fix formatting with make format_fix
2020-06-02 09:18:38 +01:00
Brian Gough
d9265b779f
Merge pull request #170 from overleaf/jpa-import-141
...
[DockerRunner] destroyOldContainers: fix a race confition
2020-06-02 09:04:59 +01:00
Brian Gough
59cd14e160
Merge pull request #169 from overleaf/bg-record-latexmk-output
...
record latexmk output
2020-06-02 09:03:43 +01:00
Jakob Ackermann
563a973388
[DockerRunner] destroyOldContainers: normalize the container name
...
The docker api returns each name with a `/` prefix.
In order to not interfere with pending compiles, the deletion process
has to acquire an internal lock on the container. The LockManager uses
the plain container name without the slash: `project-xxx`.
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-05-29 11:28:26 +01:00
Brian Gough
6d5dfb7758
clean up log file deletion and add unit test
2020-05-20 14:12:08 +01:00
Brian Gough
a684619bce
add unit tests
2020-05-20 11:52:53 +01:00
Henry Oswald
5c3da905ba
Merge pull request #162 from overleaf/ta-jpa-epipe-retry
...
[DockerRunner] retry container inspect on EPIPE
2020-05-19 11:15:25 +01:00
Jakob Ackermann
2a3c2dd3d5
[misc] simplify the smoke test and process shutdown
2020-05-19 10:30:59 +01:00
Henry Oswald
2547725f35
Merge pull request #164 from overleaf/bg-fix-buffer-deprecations
...
fix deprecated usage of Buffer constructor
2020-05-19 10:26:56 +01:00
Henry Oswald
e9c6f3b074
Merge pull request #163 from overleaf/bg-use-encoding-on-process-output
...
set encoding when reading from streams
2020-05-19 10:26:26 +01:00
Henry Oswald
457b7d6657
Merge pull request #165 from overleaf/ho-retry-url-downloads
...
add pipeUrlToFileWithRetry for file downloads
2020-05-19 10:25:19 +01:00
Henry Oswald
3925839c7f
add refreshExpiryTimeout function
...
on clsi all data lives inside of / dir
dynamically reduce size of EXPIRY_TIMEOUT if disk starts to get full
2020-05-18 15:17:19 +01:00
Henry Oswald
77e8ba74a7
add pipeUrlToFileWithRetry function to retry file downloads 3 times
2020-05-14 13:24:58 +01:00
Brian Gough
4947abe88b
fix deprecated usage of Buffer constructor
2020-05-07 10:42:05 +01:00
Brian Gough
8e86f02c43
set encoding when reading from streams
...
using .toString() works most of the time but can lead to utf8 characters being
broken across chunk boundaries.
https://nodejs.org/api/stream.html#stream_readable_setencoding_encoding
2020-05-07 10:30:14 +01:00
Tim Alby
e3b1472c7f
retry once on EPIPE errors
...
Co-Authored-By: Jakob Ackermann <jakob.ackermann@overleaf.com>
2020-04-10 12:28:11 +02:00
Tim Alby
66447211d0
fix arguments order
...
Co-Authored-By: Jakob Ackermann <jakob.ackermann@overleaf.com>
2020-04-10 12:27:15 +02:00
Jakob Ackermann
6a615d50d3
Merge remote-tracking branch 'origin/master' into jpa-entry-point-cleanup
2020-04-03 14:05:36 +02:00
Jakob Ackermann
c700a0ab9d
Merge pull request #159 from overleaf/jpa-fix-docker-modem-error-scheme
...
[misc] keep up with the error signature of dockerode/docker-modem
2020-04-03 14:04:06 +02:00
Jakob Ackermann
101c4ea4dc
[misc] use a directory in /tmp for temporary data
2020-04-03 12:36:32 +02:00
Jakob Ackermann
6777432569
[misc] add a metric for failing downloads
2020-04-03 12:35:40 +02:00
Jakob Ackermann
4a19f022fc
[misc] keep up with the error signature of dockerode/docker-modem
...
https://github.com/apocas/docker-modem/blob/v2.1.1/lib/modem.js#L296
2020-03-27 10:39:45 +01:00
mserranom
8641d0ef0f
updated sandboxed-module, chai and metrics-sharelatex
2020-03-16 17:14:04 +01:00
mserranom
7e7d234350
updated mocha and sinon, fixed tests
2020-03-12 10:35:11 +01:00
mserranom
cf6c8ab496
removed mkdirp dependency and replaced with fs.mkdir
2020-03-12 10:22:08 +01:00
mserranom
41533d8888
moved decaffeinated files to js folder
2020-02-25 17:15:31 +01:00
decaffeinate
252f4c704b
decaffeinate: Run post-processing cleanups on loadTest.coffee
2020-02-25 17:03:58 +01:00
decaffeinate
3901fe9cd2
decaffeinate: Convert loadTest.coffee to JS
2020-02-25 17:03:57 +01:00
decaffeinate
e20da38e09
decaffeinate: Rename loadTest.coffee from .coffee to .js
2020-02-25 17:03:57 +01:00