Jakob Ackermann
f361cdfaca
[ContentCacheManager] deeply integrate the HashFileTracker
...
- Update the tracker contents as we hash ranges
- Let the tracker be authoritative for already written ranges
This is saving a stat call per newly written range.
2021-05-18 18:15:58 +01:00
Jakob Ackermann
7aeeb5a5a9
[ContentCacheManager] finish tracking of ranges across builds
2021-05-18 18:06:15 +01:00
Brian Gough
d493238eaf
wip expire old hash files
2021-05-18 16:25:24 +01:00
Jakob Ackermann
23dd93ae50
Merge pull request #236 from overleaf/jpa-skip-duplicate-writes
...
[ContentCacheManager] skip writing of duplicate streams
2021-05-18 11:44:23 +02:00
Jakob Ackermann
e787106eed
Merge pull request #235 from overleaf/jpa-atomic-writes
...
[ContentCacheManager] write streams to disk atomically
2021-05-18 11:44:16 +02:00
Jakob Ackermann
9b8763aed4
Merge pull request #234 from overleaf/jpa-stream-detection-across-chunks
...
[ContentCacheManager] add support for stream detection across chunks
2021-05-18 11:44:09 +02:00
Jakob Ackermann
74e45c60f4
Merge pull request #233 from overleaf/jpa-drop-sync-syscall
...
[perf] drop useless synchronous syscall on hot path for writing docs
2021-05-18 11:44:00 +02:00
Jakob Ackermann
7cf019ab78
Merge pull request #231 from overleaf/bg-use-fse-copy
...
use fs.copyFile for performance
2021-05-18 11:43:43 +02:00
Jakob Ackermann
b254e4285b
Merge pull request #229 from overleaf/bg-log-expiry-timeout
...
log the expiry timeout value when disk space is low
2021-05-18 11:43:15 +02:00
Jakob Ackermann
bc1ed82c6c
[ContentCacheManager] skip writing of duplicate streams
2021-05-18 10:22:05 +01:00
Jakob Ackermann
224ae0c254
[ContentCacheManager] add support for stream detection across chunks
...
Retain a small part (6 or 9 bytes) of each chunk in memory for providing
the next iteration with enough context for finding the start/end marker
of a range.
2021-05-17 17:36:05 +01:00
Jakob Ackermann
6b9c8bced6
[ContentCacheManager] write streams to disk atomically
...
Use an intermediate file for writing to disk, then rename to the target.
2021-05-17 14:22:37 +01:00
Brian Gough
371de76a4a
use fs.copyFile instead of fse.copy in UrlCache module
2021-05-17 10:54:11 +01:00
Jakob Ackermann
18e943742e
[perf] drop useless synchronous syscall on hot path for writing docs
2021-05-17 09:25:29 +01:00
Brian Gough
44e0742aa3
use fse.copy for performance
...
it uses the native fs.copyFile method
2021-05-14 15:49:20 +01:00
Brian Gough
ff2175e727
add validation for express :content_id parameter
2021-05-13 15:48:17 +01:00
Brian Gough
013d38552a
log the expiry timeout value when disk is low
2021-05-13 15:13:29 +01:00
Jakob Ackermann
b456ea726d
[misc] merge pdf caching into main ( #226 )
...
* wip generate directory for hash content
* cleanup, remove console logging
* add content caching module
* Return PDF stream ranges with compile response
* Return the PDF file size in the compile response
* PDF range endpoint
* [misc] WIP: pdf caching: preserve the m-time on static content files
* [misc] WIP: pdf caching: improve browser caching, emit caching headers
* [misc] WIP: pdf caching: do not emit very small chunks <1kB
* [misc] keep up with moving output files into a separate directory
* [OutputCacheManager] add global feature flag for enabling pdf caching
* [misc] add contentId into the URL for protecting PDF stream contents
* [misc] support PDF stream caching for anonymous users
* [misc] add per-request feature flag for enabling PDF stream caching
* [misc] enable pdf caching in CI and emit metrics at the end of run
* [misc] expose compile stats and timings to the frontend
* [misc] log an error in case saving output files fails
* [misc] add metrics for pdf bandwidth and pdf caching performance
* [misc] add a dark mode to the pdf caching for computing ranges only
* [misc] move pdf caching metrics into ContentCacheMetrics
* [misc] add a config option for the min chunk size of pdf ranges
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
2021-05-13 14:07:54 +01:00
Jakob Ackermann
b3ed820444
Merge pull request #219 from overleaf/jpa-read-image-name-for-synctex
...
[misc] consume and validate a custom imageName for synctex requests
2021-03-31 10:33:53 +02:00
Jakob Ackermann
6cb4246eb1
[misc] consume and validate a custom imageName for synctex requests
2021-03-30 13:57:40 +01:00
Shane Kilkelly
072d0a9dab
Add flag to qpdf, to preserve PDF/A compliance
2021-03-19 11:35:59 +00:00
Brian Gough
865e68051e
include fallback for missing state file
2021-01-26 13:59:28 +00:00
Brian Gough
a8e47da9e9
Revert "Merge pull request #205 from overleaf/revert-200-bg-decaff-cleanup"
...
This reverts commit 76d8d3181b9464d1e1bbc713a2729ca269d9c047, reversing
changes made to 31a8dc3a98d73c2707d633712f0ef7207013e78b.
2021-01-26 11:04:33 +00:00
Brian Gough
9dc55729ac
Revert "decaff cleanup ResourceStateManager"
2021-01-25 15:26:53 +00:00
Brian Gough
f41f3aba8c
Merge pull request #204 from overleaf/bg-use-output-dir
...
use separate output dir for generated files
2021-01-25 14:10:09 +00:00
Brian Gough
90c0d931ad
Merge pull request #201 from overleaf/bg-more-cleanup
...
decaff cleanup of OutputFileManager
2021-01-25 14:09:46 +00:00
Brian Gough
c67c8a6c80
Merge pull request #200 from overleaf/bg-decaff-cleanup
...
decaff cleanup ResourceStateManager
2021-01-25 14:09:24 +00:00
Brian Gough
b5346658b0
clear output directory when clearing project
2021-01-22 11:05:52 +00:00
Brian Gough
692dbc8d6b
add output directory
2021-01-22 11:05:52 +00:00
Jakob Ackermann
49b764a308
[misc] CompileController: exact match for output.pdf
...
The previous regex could mistake user provided pdf files, like
`fake_output.pdf`, as the final output file.
The frontend expects to find a `output.pdf` file on success.
2020-12-28 13:16:31 +00:00
Brian Gough
e183fffa4b
format fix
2020-12-18 16:15:06 +00:00
Brian Gough
dafe69b5e6
remove unused module var
2020-12-18 16:15:06 +00:00
Brian Gough
36b646bb84
remove comments
2020-12-18 16:15:06 +00:00
Brian Gough
f20a0d1a69
remove unused var
2020-12-18 16:15:03 +00:00
Brian Gough
df1caa14b8
remove unused require
2020-12-18 16:14:36 +00:00
Brian Gough
ca98ee5cff
use a Set instead of an Object
2020-12-18 16:14:09 +00:00
Brian Gough
43b0429c28
clean up relative path checking
2020-12-18 16:14:05 +00:00
Brian Gough
73e09ff99f
remove unnecessary requires
2020-12-18 15:28:06 +00:00
Brian Gough
cee93a0cd9
clean up unnecessary var
2020-12-18 15:28:06 +00:00
Brian Gough
4169f7fc73
use once for callback
2020-12-18 15:28:06 +00:00
Brian Gough
e84d6305f3
remove unnecessary return
2020-12-18 15:16:07 +00:00
Brian Gough
ae064e8997
lint fix
2020-12-18 15:11:29 +00:00
Brian Gough
323890cedb
remove Array.from
2020-12-18 15:10:53 +00:00
Brian Gough
c044db4897
remove guard helper
2020-12-18 15:09:48 +00:00
Brian Gough
0bae53c7f3
remove unnecessary returns
2020-12-18 15:02:40 +00:00
Brian Gough
eaec57cd01
simplify null check
2020-12-18 15:01:25 +00:00
Brian Gough
411d69e36b
remove unnecessary callback code
2020-12-18 14:59:48 +00:00
Brian Gough
11ef3c27ed
use Set instead of object
2020-12-18 14:56:53 +00:00
Brian Gough
64ea22d259
remove unnecessary null checks
2020-12-17 12:05:17 +00:00
Brian Gough
299d7cc5fe
remove unnecessary returns
2020-12-17 12:01:14 +00:00