Commit graph

360 commits

Author SHA1 Message Date
Jakob Ackermann
9b8763aed4 Merge pull request #234 from overleaf/jpa-stream-detection-across-chunks
[ContentCacheManager] add support for stream detection across chunks
2021-05-18 11:44:09 +02:00
Jakob Ackermann
74e45c60f4 Merge pull request #233 from overleaf/jpa-drop-sync-syscall
[perf] drop useless synchronous syscall on hot path for writing docs
2021-05-18 11:44:00 +02:00
Jakob Ackermann
7cf019ab78 Merge pull request #231 from overleaf/bg-use-fse-copy
use fs.copyFile for performance
2021-05-18 11:43:43 +02:00
Jakob Ackermann
b254e4285b Merge pull request #229 from overleaf/bg-log-expiry-timeout
log the expiry timeout value when disk space is low
2021-05-18 11:43:15 +02:00
Jakob Ackermann
224ae0c254 [ContentCacheManager] add support for stream detection across chunks
Retain a small part (6 or 9 bytes) of each chunk in memory for providing
 the next iteration with enough context for finding the start/end marker
 of a range.
2021-05-17 17:36:05 +01:00
Brian Gough
371de76a4a use fs.copyFile instead of fse.copy in UrlCache module 2021-05-17 10:54:11 +01:00
Jakob Ackermann
18e943742e [perf] drop useless synchronous syscall on hot path for writing docs 2021-05-17 09:25:29 +01:00
Brian Gough
44e0742aa3 use fse.copy for performance
it uses the native fs.copyFile method
2021-05-14 15:49:20 +01:00
Brian Gough
ff2175e727 add validation for express :content_id parameter 2021-05-13 15:48:17 +01:00
Brian Gough
013d38552a log the expiry timeout value when disk is low 2021-05-13 15:13:29 +01:00
Jakob Ackermann
b456ea726d [misc] merge pdf caching into main (#226)
* wip generate directory for hash content

* cleanup, remove console logging

* add content caching module

* Return PDF stream ranges with compile response

* Return the PDF file size in the compile response

* PDF range endpoint

* [misc] WIP: pdf caching: preserve the m-time on static content files

* [misc] WIP: pdf caching: improve browser caching, emit caching headers

* [misc] WIP: pdf caching: do not emit very small chunks <1kB

* [misc] keep up with moving output files into a separate directory

* [OutputCacheManager] add global feature flag for enabling pdf caching

* [misc] add contentId into the URL for protecting PDF stream contents

* [misc] support PDF stream caching for anonymous users

* [misc] add per-request feature flag for enabling PDF stream caching

* [misc] enable pdf caching in CI and emit metrics at the end of run

* [misc] expose compile stats and timings to the frontend

* [misc] log an error in case saving output files fails

* [misc] add metrics for pdf bandwidth and pdf caching performance

* [misc] add a dark mode to the pdf caching for computing ranges only

* [misc] move pdf caching metrics into ContentCacheMetrics

* [misc] add a config option for the min chunk size of pdf ranges

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
2021-05-13 14:07:54 +01:00
Jakob Ackermann
b3ed820444 Merge pull request #219 from overleaf/jpa-read-image-name-for-synctex
[misc] consume and validate a custom imageName for synctex requests
2021-03-31 10:33:53 +02:00
Jakob Ackermann
6cb4246eb1 [misc] consume and validate a custom imageName for synctex requests 2021-03-30 13:57:40 +01:00
Shane Kilkelly
072d0a9dab Add flag to qpdf, to preserve PDF/A compliance 2021-03-19 11:35:59 +00:00
Brian Gough
865e68051e include fallback for missing state file 2021-01-26 13:59:28 +00:00
Brian Gough
a8e47da9e9 Revert "Merge pull request #205 from overleaf/revert-200-bg-decaff-cleanup"
This reverts commit 76d8d3181b9464d1e1bbc713a2729ca269d9c047, reversing
changes made to 31a8dc3a98d73c2707d633712f0ef7207013e78b.
2021-01-26 11:04:33 +00:00
Brian Gough
9dc55729ac Revert "decaff cleanup ResourceStateManager" 2021-01-25 15:26:53 +00:00
Brian Gough
f41f3aba8c Merge pull request #204 from overleaf/bg-use-output-dir
use separate output dir for generated files
2021-01-25 14:10:09 +00:00
Brian Gough
90c0d931ad Merge pull request #201 from overleaf/bg-more-cleanup
decaff cleanup of OutputFileManager
2021-01-25 14:09:46 +00:00
Brian Gough
c67c8a6c80 Merge pull request #200 from overleaf/bg-decaff-cleanup
decaff cleanup ResourceStateManager
2021-01-25 14:09:24 +00:00
Brian Gough
b5346658b0 clear output directory when clearing project 2021-01-22 11:05:52 +00:00
Brian Gough
692dbc8d6b add output directory 2021-01-22 11:05:52 +00:00
Jakob Ackermann
49b764a308 [misc] CompileController: exact match for output.pdf
The previous regex could mistake user provided pdf files, like
 `fake_output.pdf`, as the final output file.
The frontend expects to find a `output.pdf` file on success.
2020-12-28 13:16:31 +00:00
Brian Gough
e183fffa4b format fix 2020-12-18 16:15:06 +00:00
Brian Gough
dafe69b5e6 remove unused module var 2020-12-18 16:15:06 +00:00
Brian Gough
36b646bb84 remove comments 2020-12-18 16:15:06 +00:00
Brian Gough
f20a0d1a69 remove unused var 2020-12-18 16:15:03 +00:00
Brian Gough
df1caa14b8 remove unused require 2020-12-18 16:14:36 +00:00
Brian Gough
ca98ee5cff use a Set instead of an Object 2020-12-18 16:14:09 +00:00
Brian Gough
43b0429c28 clean up relative path checking 2020-12-18 16:14:05 +00:00
Brian Gough
73e09ff99f remove unnecessary requires 2020-12-18 15:28:06 +00:00
Brian Gough
cee93a0cd9 clean up unnecessary var 2020-12-18 15:28:06 +00:00
Brian Gough
4169f7fc73 use once for callback 2020-12-18 15:28:06 +00:00
Brian Gough
e84d6305f3 remove unnecessary return 2020-12-18 15:16:07 +00:00
Brian Gough
ae064e8997 lint fix 2020-12-18 15:11:29 +00:00
Brian Gough
323890cedb remove Array.from 2020-12-18 15:10:53 +00:00
Brian Gough
c044db4897 remove guard helper 2020-12-18 15:09:48 +00:00
Brian Gough
0bae53c7f3 remove unnecessary returns 2020-12-18 15:02:40 +00:00
Brian Gough
eaec57cd01 simplify null check 2020-12-18 15:01:25 +00:00
Brian Gough
411d69e36b remove unnecessary callback code 2020-12-18 14:59:48 +00:00
Brian Gough
11ef3c27ed use Set instead of object 2020-12-18 14:56:53 +00:00
Brian Gough
64ea22d259 remove unnecessary null checks 2020-12-17 12:05:17 +00:00
Brian Gough
299d7cc5fe remove unnecessary returns 2020-12-17 12:01:14 +00:00
Brian Gough
5a539b7f90 remove guard function 2020-12-17 11:59:15 +00:00
Brian Gough
4d8e4d54e5 remove Array.from 2020-12-17 11:49:02 +00:00
Jakob Ackermann
7b7cd8cc8c [misc] bump metrics module to 3.4.1
- renamed package from `metrics-sharelatex` to `@overleaf/metrics`
- drop support for statsd backend
- decaffeinate
- compress `/metrics` response using gzip
- bump debugging agents to latest versions
- expose prometheus interfaces for custom metrics (custom tags)
- cleanup of open sockets metrics
- fix deprecation warnings for header access
2020-11-25 11:57:25 +00:00
Eric Mc Sween
1c13f6fe94 Mount /home/tex in an anonymous volume
When we mount the container's root filesystem as read-only, mount
an anonymous volume in /home/tex so that it's writable. Our TeX Live
images have cached content in /home/tex. This content will automatically
get copied by Docker into this anonymous volume.
2020-09-04 08:49:56 -04:00
Eric Mc Sween
67f4a6eeeb Decaf cleanup: normalize functions
Use function keyword for declarations and arrow functions for callbacks.
2020-09-03 15:58:16 -04:00
Eric Mc Sween
5cd889038e Use _.once() instead of ad hoc implementation 2020-09-03 15:52:09 -04:00
Eric Mc Sween
a853950a99 Fix container monitor cleanup function
The intent here is clearly to clear both the timeout and the interval.
2020-09-03 15:50:45 -04:00