Eric Mc Sween
7f6e2c971e
Merge pull request #8757 from overleaf/em-clear-cache-on-timeout
...
Clear project after a compile has been interrupted
GitOrigin-RevId: 1b8e85c39843de3f63b7119f3a8a433567f57bcb
2022-07-19 08:03:56 +00:00
Jakob Ackermann
97624d0c6c
Merge pull request #8847 from overleaf/jpa-emit-start-of-xref-table
...
[clsi] try to emit the start of the xref table
GitOrigin-RevId: 6d8348a349572cc997ac5924664428228c00fed1
2022-07-18 08:04:17 +00:00
Eric Mc Sween
77aa2baa9d
Merge pull request #8714 from overleaf/em-promisify-compile-manager
...
Promisify CompileManager
GitOrigin-RevId: 644ed061ae139d6196b24f8ead38579de6b844a3
2022-07-08 08:04:22 +00:00
Jakob Ackermann
338eaf9a07
Merge pull request #8591 from overleaf/jpa-fix-pdf-caching-metrics
...
[clsi] fix options forwarding to pdf caching summary metrics
GitOrigin-RevId: 255fca6a1e980f08deec97ede5bc1d27e83dbda1
2022-07-01 08:04:24 +00:00
June Kelly
a450a74351
Upgrade async
package to 3.2.2 ( #8447 )
...
* Revert "Revert "Bump async to 3.2.2 (#7618 )""
This reverts commit 75153a555211d654744c2e61e27fe21085826c22.
* [web] fix usage of async.queue.drain in script
* [clsi] fix usage of async.queue.drain
* [spelling] fix usage of async.queue.drain
* [redis-wrapper] fix usage of async.queue.drain
* [web] Test that LockManager queue is cleared
This protects against a regression found when upgrading the
async package. Here we test that the `queue.drain` callback
is really getting called, and the lock is being removed from
the LOCK_QUEUES map.
* [redis-wrapper] Upgrade async to 3.2.2
GitOrigin-RevId: df921e6d7f1d505bd467f22e58600ba1aff48869
2022-06-22 08:03:35 +00:00
Eric Mc Sween
cb657d1f1c
Merge pull request #8297 from overleaf/em-halt-on-error-logs
...
Stop on first error info box in logs pane
GitOrigin-RevId: cf11f65d582d98bea93c6506393940d9a6144c0d
2022-06-08 08:03:12 +00:00
Eric Mc Sween
baaf4d4240
Merge pull request #8286 from overleaf/em-halt-on-error-backend
...
Stop on first error backend implementation
GitOrigin-RevId: 497b1ed2c13f544760d8ad8d029359db75275389
2022-06-07 08:02:21 +00:00
Eric Mc Sween
bda307fbb5
Merge pull request #8267 from overleaf/em-decaf-clsi
...
Decaf cleanup some CLSI files
GitOrigin-RevId: afb4cd6e9eb9e95703efa955b93d4fedada10e3c
2022-06-07 08:02:18 +00:00
Jakob Ackermann
a78bcee15f
Merge pull request #8135 from overleaf/jpa-refactor-zonal-download
...
[misc] refactor handling of zone prefix in compile response
GitOrigin-RevId: f1f33d7d257854176f383bb5d786710f6b09f737
2022-05-26 08:03:53 +00:00
Jakob Ackermann
9a71372c36
Merge pull request #8071 from overleaf/jpa-clsi-zonal
...
[misc] prefix output file downloads with /zone/X
GitOrigin-RevId: ba59e97ae0284c68ba551dd49dc5d3daa4d61aa9
2022-05-25 08:08:39 +00:00
Jakob Ackermann
f0bd6dda23
Merge pull request #7986 from overleaf/jpa-eslint-8
...
[misc] upgrade eslint packages to the latest version everywhere
GitOrigin-RevId: f1480d4a171acef82fb26c4aa54be3a6088b0ab3
2022-05-17 08:05:59 +00:00
Eric Mc Sween
e0d91eaa26
Merge pull request #7906 from overleaf/em-downgrade-logs
...
Downgrade all INFO logs to DEBUG
GitOrigin-RevId: 05ed582ef0721fcada059f0ad158565f50feca27
2022-05-17 08:05:26 +00:00
Brian Gough
406f8b145a
fix formatting
...
GitOrigin-RevId: 60bd152226db705250f49eb892c6cf75b1b265fc
2022-05-12 10:41:05 +00:00
Christopher Schenk
d6d664312c
Ignore stderr output when running commands
...
Fixes #1006
2022-04-01 16:03:19 +02:00
Jakob Ackermann
b25ff076b8
Merge pull request #6962 from overleaf/jpa-clsi-tweak-health-check
...
[clsi] tweak health check
GitOrigin-RevId: 2caad90d55bb441c7008e3084b34d683b072dfbd
2022-03-03 09:03:19 +00:00
Jakob Ackermann
a3e85f5357
Merge pull request #6959 from overleaf/jpa-clsi-zero-bucket
...
[clsi] add zero bucket and collect histogram data in timing as well
GitOrigin-RevId: 6390f1c213e4850ceb629dae1e6406f6077759aa
2022-03-03 09:03:16 +00:00
Jakob Ackermann
264b107e39
Merge pull request #6927 from overleaf/jpa-split-clsi-metrics
...
[misc] split clsi metrics into multiple categories
GitOrigin-RevId: 964ab57b344b92383a2937beff50139bae273a0e
2022-03-02 09:02:52 +00:00
Jakob Ackermann
958216d578
Merge pull request #6926 from overleaf/jpa-clsi-bump-logger-metrics
...
[clsi] upgrade logger and metrics module
GitOrigin-RevId: 85c346b5eed683672a77d86d6a434dc313b7824b
2022-03-02 09:02:50 +00:00
Jakob Ackermann
9c9beef305
Merge pull request #5737 from overleaf/jpa-clsi-cleanup-in-memory
...
[clsi] separate clearing cache and clearing output cache
GitOrigin-RevId: 9cd903eaf8d291d4821ba8c9ac6c1a9f8d3d9632
2021-11-16 09:02:36 +00:00
Jakob Ackermann
669606f797
Merge pull request #5568 from overleaf/jpa-node-no-callback-literal
...
[misc] fix eslint violations for node/no-callback-literal
GitOrigin-RevId: af5aace52e1476b1b0ee48cc8be2aabbe04efac6
2021-10-28 08:03:33 +00:00
Jakob Ackermann
7f9fd00bda
Merge pull request #5367 from overleaf/jpa-node-handle-callback-err
...
[misc] fix eslint violations for node/handle-callback-err
GitOrigin-RevId: 83a4900e8861010df1917bff49382bd9c93375bd
2021-10-28 08:03:26 +00:00
Jakob Ackermann
a6f05109a3
Merge pull request #5352 from overleaf/jpa-no-var
...
[misc] fix eslint violations for `no-var`
GitOrigin-RevId: c52e82f3a8a993b8662cc5aa56e7b95ca3c55832
2021-10-27 08:03:00 +00:00
Jakob Ackermann
358e8b7424
Merge pull request #5349 from overleaf/jpa-no-depreacted-api
...
[misc] fix eslint violations for node/no-depreacted-api
GitOrigin-RevId: 0f7d64984da9e789c4ab95381db34afb89fa1a94
2021-10-21 08:03:18 +00:00
Jakob Ackermann
94c208311c
Merge pull request #5351 from overleaf/jpa-clsi-drop-sqlite
...
[clsi] goodbye sqlite db
GitOrigin-RevId: e7f16920c44e74a425b92884b48a60272dfb015b
2021-10-07 08:03:50 +00:00
Jakob Ackermann
44d0a8d162
Merge pull request #4650 from overleaf/jpa-in-memory-last-access-time
...
[perf] store the last access time of a project in memory
GitOrigin-RevId: 16e2bee28f58eced18f4c3ec5571ea9d10805cbb
2021-10-07 08:03:21 +00:00
Jakob Ackermann
02918e7483
Merge pull request #4649 from overleaf/jpa-fs-based-caching
...
[perf] UrlCache: pure fs based cache state for downloads
GitOrigin-RevId: d19afc396324d4c3318b31620c8ad0c04e0544ce
2021-10-07 08:03:18 +00:00
Brian Gough
dc576f3b6f
Merge pull request #4943 from overleaf/bg-increase-max-print-line
...
allow setting texlive max_print_line variable
GitOrigin-RevId: 7588fed6aa5868a6ed6b6121cbd6f9c008c2aa0f
2021-09-22 08:03:21 +00:00
Eric Mc Sween
9ee92daea3
Merge pull request #4893 from overleaf/em-synctex
...
Use the synctex distributed with TeX Live
GitOrigin-RevId: 5a133f21f48fd1e217ab463e8cb2a5cdec8be1af
2021-09-07 16:26:17 +00:00
Eric Mc Sween
2feea2592d
Merge pull request #4887 from overleaf/em-decaf
...
Decaf cleanup for the CLSI CompileManager
GitOrigin-RevId: 06bba5c8af8808d0fa04187b10c8f31e08cd8754
2021-08-31 08:03:32 +00:00
Jakob Ackermann
56a3b0dcde
Merge pull request #4819 from overleaf/jpa-pdf-caching-new-event-loop
...
[clsi] pdf-caching: move cpu intensive work onto a new event loop
GitOrigin-RevId: 4cb5cd4528fa1c5df6a8e91f9caa38cb64d94463
2021-08-25 08:03:38 +00:00
Brian Gough
262793c04f
add option for apparmor profile
2021-07-21 14:53:35 +01:00
Jakob Ackermann
0e5f5afe79
[misc] temporary override a few new/changed eslint rules
2021-07-13 12:24:43 +01:00
Jakob Ackermann
f285e503b4
[misc] run format_fix and lint:fix
2021-07-13 12:04:48 +01:00
Jakob Ackermann
64551f0198
[misc] switch from settings-sharelatex to @overleaf/settings
2021-07-12 17:47:21 +01:00
Jakob Ackermann
516525126b
[UrlFetcher] do not override domain for clsi-perf requests
2021-07-02 09:17:29 +01:00
Jakob Ackermann
efa0c54ca5
[misc] add timings for the sync stage and output stage
2021-06-30 11:45:34 +01:00
Jakob Ackermann
743bfced64
[misc] ContentCacheManager: apply review feedback
...
- count stages
- lower bound is 1s
Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
2021-06-23 14:42:39 +01:00
Jakob Ackermann
b09e52510f
[misc] bail out from pdf caching processing after 10s or earlier
...
...for fast compiles.
2021-06-23 14:20:04 +01:00
Jakob Ackermann
c3dba7f74d
Merge pull request #253 from overleaf/jpa-debug-slow-pdf-caching
...
[misc] ContentCacheMetrics: log slow pdf caching performance
2021-06-23 12:39:13 +02:00
Jakob Ackermann
97693b49c2
[ContentCacheMetrics] add new metric for absolute time spent in PDF.js
2021-06-23 11:28:31 +01:00
Jakob Ackermann
11a44ff07c
[misc] ContentCacheMetrics: apply review feedback: ignore fast compiles
...
Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
2021-06-23 11:27:19 +01:00
Jakob Ackermann
a4389fb761
[misc] ContentCacheMetrics: log slow pdf caching performance
2021-06-23 09:25:16 +01:00
Jakob Ackermann
ffaff1bd72
[CompileController] emit status=failure for an empty output.pdf file
2021-06-22 12:15:12 +01:00
Jakob Ackermann
fb3966ef35
[misc] CompileController: simplify composing of outputFiles
2021-06-22 12:15:12 +01:00
Jakob Ackermann
8caa19afed
Merge pull request #247 from overleaf/jpa-track-db-times
...
[misc] track delays of using sqlite in a few places
2021-06-15 10:11:46 +02:00
Jakob Ackermann
878cea8087
Merge pull request #221 from overleaf/jpa-explicit-dependencies
...
[misc] add linting for missing explicit dependencies and fix any errors
2021-06-15 10:09:43 +02:00
Jakob Ackermann
8cc4517d4c
[misc] move import to avoid git conflict
2021-06-15 09:08:32 +01:00
Jakob Ackermann
8f1ea7f0d1
[misc] track delays of using sqlite for url caching details
2021-06-14 13:03:02 +01:00
Jakob Ackermann
d40065fd8e
[misc] track delay of using sqlite for last access time of a project
2021-06-14 12:41:16 +01:00
Jakob Ackermann
7380b54900
[ProjectPersistenceManager] check all user content dirs for full disk
2021-06-01 16:13:48 +01:00
Jakob Ackermann
294088fb27
[ContentCacheManager] use PDF.js Xref table instead of stream detection ( #242 )
...
* make the content cache manager tests configurable
* extend stream content in unit tests
* [ContentCacheManagerTests] prepare for full object caching
* filesystem stream for pdfjs
* working??
* cleaning up
* handle overflow
* [misc] install pdfjs-dist
* [misc] move pdfjs code into app/lib/ and scripts/, also use CamelCase
* [misc] abstract the file loading and parsing of xRef tables into helper
* [misc] pdfjsTests: add snapshot based tests for the Xref table parser
* [misc] FSStream: throw proper error and drop commented code
* [misc] FSStream: integrate throwing of MissingDataException into getter
* [misc] pdfjs: fix eslint errors
* [misc] pdfjs: run format_fix
* [misc] pdfjs: allocate very small non empty dummy buffers explicitly
* [misc] install @overleaf/o-error
* [ContentCacheManager] use PDF.js Xref table instead of stream detection
Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
* [pdfjs] parseXrefTable: handle empty PDF files gracefully
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
2021-05-31 09:20:25 +01:00
Jakob Ackermann
f820d32221
Merge pull request #240 from overleaf/bg-jpa-hash-tracker
...
[ContentCacheManager] track hashes and expire unused ones
2021-05-21 11:35:05 +02:00
Jakob Ackermann
b53a105681
[LatexRunner] do not emit empty cpu/sys timings
2021-05-19 11:17:08 +01:00
Jakob Ackermann
f361cdfaca
[ContentCacheManager] deeply integrate the HashFileTracker
...
- Update the tracker contents as we hash ranges
- Let the tracker be authoritative for already written ranges
This is saving a stat call per newly written range.
2021-05-18 18:15:58 +01:00
Jakob Ackermann
7aeeb5a5a9
[ContentCacheManager] finish tracking of ranges across builds
2021-05-18 18:06:15 +01:00
Brian Gough
d493238eaf
wip expire old hash files
2021-05-18 16:25:24 +01:00
Jakob Ackermann
23dd93ae50
Merge pull request #236 from overleaf/jpa-skip-duplicate-writes
...
[ContentCacheManager] skip writing of duplicate streams
2021-05-18 11:44:23 +02:00
Jakob Ackermann
e787106eed
Merge pull request #235 from overleaf/jpa-atomic-writes
...
[ContentCacheManager] write streams to disk atomically
2021-05-18 11:44:16 +02:00
Jakob Ackermann
9b8763aed4
Merge pull request #234 from overleaf/jpa-stream-detection-across-chunks
...
[ContentCacheManager] add support for stream detection across chunks
2021-05-18 11:44:09 +02:00
Jakob Ackermann
74e45c60f4
Merge pull request #233 from overleaf/jpa-drop-sync-syscall
...
[perf] drop useless synchronous syscall on hot path for writing docs
2021-05-18 11:44:00 +02:00
Jakob Ackermann
7cf019ab78
Merge pull request #231 from overleaf/bg-use-fse-copy
...
use fs.copyFile for performance
2021-05-18 11:43:43 +02:00
Jakob Ackermann
b254e4285b
Merge pull request #229 from overleaf/bg-log-expiry-timeout
...
log the expiry timeout value when disk space is low
2021-05-18 11:43:15 +02:00
Jakob Ackermann
bc1ed82c6c
[ContentCacheManager] skip writing of duplicate streams
2021-05-18 10:22:05 +01:00
Jakob Ackermann
224ae0c254
[ContentCacheManager] add support for stream detection across chunks
...
Retain a small part (6 or 9 bytes) of each chunk in memory for providing
the next iteration with enough context for finding the start/end marker
of a range.
2021-05-17 17:36:05 +01:00
Jakob Ackermann
6b9c8bced6
[ContentCacheManager] write streams to disk atomically
...
Use an intermediate file for writing to disk, then rename to the target.
2021-05-17 14:22:37 +01:00
Brian Gough
371de76a4a
use fs.copyFile instead of fse.copy in UrlCache module
2021-05-17 10:54:11 +01:00
Jakob Ackermann
18e943742e
[perf] drop useless synchronous syscall on hot path for writing docs
2021-05-17 09:25:29 +01:00
Brian Gough
44e0742aa3
use fse.copy for performance
...
it uses the native fs.copyFile method
2021-05-14 15:49:20 +01:00
Brian Gough
ff2175e727
add validation for express :content_id parameter
2021-05-13 15:48:17 +01:00
Brian Gough
013d38552a
log the expiry timeout value when disk is low
2021-05-13 15:13:29 +01:00
Jakob Ackermann
b456ea726d
[misc] merge pdf caching into main ( #226 )
...
* wip generate directory for hash content
* cleanup, remove console logging
* add content caching module
* Return PDF stream ranges with compile response
* Return the PDF file size in the compile response
* PDF range endpoint
* [misc] WIP: pdf caching: preserve the m-time on static content files
* [misc] WIP: pdf caching: improve browser caching, emit caching headers
* [misc] WIP: pdf caching: do not emit very small chunks <1kB
* [misc] keep up with moving output files into a separate directory
* [OutputCacheManager] add global feature flag for enabling pdf caching
* [misc] add contentId into the URL for protecting PDF stream contents
* [misc] support PDF stream caching for anonymous users
* [misc] add per-request feature flag for enabling PDF stream caching
* [misc] enable pdf caching in CI and emit metrics at the end of run
* [misc] expose compile stats and timings to the frontend
* [misc] log an error in case saving output files fails
* [misc] add metrics for pdf bandwidth and pdf caching performance
* [misc] add a dark mode to the pdf caching for computing ranges only
* [misc] move pdf caching metrics into ContentCacheMetrics
* [misc] add a config option for the min chunk size of pdf ranges
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
2021-05-13 14:07:54 +01:00
Jakob Ackermann
cfa8127c1b
[misc] add linting for missing explicit dependencies and fix any errors
2021-04-29 15:30:54 +01:00
Jakob Ackermann
b3ed820444
Merge pull request #219 from overleaf/jpa-read-image-name-for-synctex
...
[misc] consume and validate a custom imageName for synctex requests
2021-03-31 10:33:53 +02:00
Jakob Ackermann
6cb4246eb1
[misc] consume and validate a custom imageName for synctex requests
2021-03-30 13:57:40 +01:00
Shane Kilkelly
072d0a9dab
Add flag to qpdf, to preserve PDF/A compliance
2021-03-19 11:35:59 +00:00
Brian Gough
865e68051e
include fallback for missing state file
2021-01-26 13:59:28 +00:00
Brian Gough
a8e47da9e9
Revert "Merge pull request #205 from overleaf/revert-200-bg-decaff-cleanup"
...
This reverts commit 76d8d3181b9464d1e1bbc713a2729ca269d9c047, reversing
changes made to 31a8dc3a98d73c2707d633712f0ef7207013e78b.
2021-01-26 11:04:33 +00:00
Brian Gough
9dc55729ac
Revert "decaff cleanup ResourceStateManager"
2021-01-25 15:26:53 +00:00
Brian Gough
f41f3aba8c
Merge pull request #204 from overleaf/bg-use-output-dir
...
use separate output dir for generated files
2021-01-25 14:10:09 +00:00
Brian Gough
90c0d931ad
Merge pull request #201 from overleaf/bg-more-cleanup
...
decaff cleanup of OutputFileManager
2021-01-25 14:09:46 +00:00
Brian Gough
c67c8a6c80
Merge pull request #200 from overleaf/bg-decaff-cleanup
...
decaff cleanup ResourceStateManager
2021-01-25 14:09:24 +00:00
Brian Gough
b5346658b0
clear output directory when clearing project
2021-01-22 11:05:52 +00:00
Brian Gough
692dbc8d6b
add output directory
2021-01-22 11:05:52 +00:00
Jakob Ackermann
49b764a308
[misc] CompileController: exact match for output.pdf
...
The previous regex could mistake user provided pdf files, like
`fake_output.pdf`, as the final output file.
The frontend expects to find a `output.pdf` file on success.
2020-12-28 13:16:31 +00:00
Brian Gough
e183fffa4b
format fix
2020-12-18 16:15:06 +00:00
Brian Gough
dafe69b5e6
remove unused module var
2020-12-18 16:15:06 +00:00
Brian Gough
36b646bb84
remove comments
2020-12-18 16:15:06 +00:00
Brian Gough
f20a0d1a69
remove unused var
2020-12-18 16:15:03 +00:00
Brian Gough
df1caa14b8
remove unused require
2020-12-18 16:14:36 +00:00
Brian Gough
ca98ee5cff
use a Set instead of an Object
2020-12-18 16:14:09 +00:00
Brian Gough
43b0429c28
clean up relative path checking
2020-12-18 16:14:05 +00:00
Brian Gough
73e09ff99f
remove unnecessary requires
2020-12-18 15:28:06 +00:00
Brian Gough
cee93a0cd9
clean up unnecessary var
2020-12-18 15:28:06 +00:00
Brian Gough
4169f7fc73
use once for callback
2020-12-18 15:28:06 +00:00
Brian Gough
e84d6305f3
remove unnecessary return
2020-12-18 15:16:07 +00:00
Brian Gough
ae064e8997
lint fix
2020-12-18 15:11:29 +00:00
Brian Gough
323890cedb
remove Array.from
2020-12-18 15:10:53 +00:00
Brian Gough
c044db4897
remove guard helper
2020-12-18 15:09:48 +00:00
Brian Gough
0bae53c7f3
remove unnecessary returns
2020-12-18 15:02:40 +00:00
Brian Gough
eaec57cd01
simplify null check
2020-12-18 15:01:25 +00:00