github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-21 20:47:08 -05:00
Code Issues Projects Releases Packages Wiki Activity
23880 commits 3 branches 7 tags 206 MiB
Commit graph

227 commits

Author SHA1 Message Date
Jakob Ackermann
8caa19afed Merge pull request #247 from overleaf/jpa-track-db-times 
[misc] track delays of using sqlite in a few places
 2021-06-15 10:11:46 +02:00
Jakob Ackermann
878cea8087 Merge pull request #221 from overleaf/jpa-explicit-dependencies 
[misc] add linting for missing explicit dependencies and fix any errors
 2021-06-15 10:09:43 +02:00
Jakob Ackermann
8cc4517d4c [misc] move import to avoid git conflict 2021-06-15 09:08:32 +01:00
Jakob Ackermann
8f1ea7f0d1 [misc] track delays of using sqlite for url caching details 2021-06-14 13:03:02 +01:00
Jakob Ackermann
d40065fd8e [misc] track delay of using sqlite for last access time of a project 2021-06-14 12:41:16 +01:00
Jakob Ackermann
7380b54900 [ProjectPersistenceManager] check all user content dirs for full disk 2021-06-01 16:13:48 +01:00
Jakob Ackermann
294088fb27 [ContentCacheManager] use PDF.js Xref table instead of stream detection (#242) 
* make the content cache manager tests configurable

* extend stream content in unit tests

* [ContentCacheManagerTests] prepare for full object caching

* filesystem stream for pdfjs

* working??

* cleaning up

* handle overflow

* [misc] install pdfjs-dist

* [misc] move pdfjs code into app/lib/ and scripts/, also use CamelCase

* [misc] abstract the file loading and parsing of xRef tables into helper

* [misc] pdfjsTests: add snapshot based tests for the Xref table parser

* [misc] FSStream: throw proper error and drop commented code

* [misc] FSStream: integrate throwing of MissingDataException into getter

* [misc] pdfjs: fix eslint errors

* [misc] pdfjs: run format_fix

* [misc] pdfjs: allocate very small non empty dummy buffers explicitly

* [misc] install @overleaf/o-error

* [ContentCacheManager] use PDF.js Xref table instead of stream detection

Co-Authored-By: Brian Gough <brian.gough@overleaf.com>

* [pdfjs] parseXrefTable: handle empty PDF files gracefully

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
 2021-05-31 09:20:25 +01:00
Jakob Ackermann
f820d32221 Merge pull request #240 from overleaf/bg-jpa-hash-tracker 
[ContentCacheManager] track hashes and expire unused ones
 2021-05-21 11:35:05 +02:00
Jakob Ackermann
b53a105681 [LatexRunner] do not emit empty cpu/sys timings 2021-05-19 11:17:08 +01:00
Jakob Ackermann
f361cdfaca [ContentCacheManager] deeply integrate the HashFileTracker 
- Update the tracker contents as we hash ranges
- Let the tracker be authoritative for already written ranges
  This is saving a stat call per newly written range.
 2021-05-18 18:15:58 +01:00
Jakob Ackermann
7aeeb5a5a9 [ContentCacheManager] finish tracking of ranges across builds 2021-05-18 18:06:15 +01:00
Brian Gough
d493238eaf wip expire old hash files 2021-05-18 16:25:24 +01:00
Jakob Ackermann
23dd93ae50 Merge pull request #236 from overleaf/jpa-skip-duplicate-writes 
[ContentCacheManager] skip writing of duplicate streams
 2021-05-18 11:44:23 +02:00
Jakob Ackermann
e787106eed Merge pull request #235 from overleaf/jpa-atomic-writes 
[ContentCacheManager] write streams to disk atomically
 2021-05-18 11:44:16 +02:00
Jakob Ackermann
9b8763aed4 Merge pull request #234 from overleaf/jpa-stream-detection-across-chunks 
[ContentCacheManager] add support for stream detection across chunks
 2021-05-18 11:44:09 +02:00
Jakob Ackermann
74e45c60f4 Merge pull request #233 from overleaf/jpa-drop-sync-syscall 
[perf] drop useless synchronous syscall on hot path for writing docs
 2021-05-18 11:44:00 +02:00
Jakob Ackermann
7cf019ab78 Merge pull request #231 from overleaf/bg-use-fse-copy 
use fs.copyFile for performance
 2021-05-18 11:43:43 +02:00
Jakob Ackermann
b254e4285b Merge pull request #229 from overleaf/bg-log-expiry-timeout 
log the expiry timeout value when disk space is low
 2021-05-18 11:43:15 +02:00
Jakob Ackermann
bc1ed82c6c [ContentCacheManager] skip writing of duplicate streams 2021-05-18 10:22:05 +01:00
Jakob Ackermann
224ae0c254 [ContentCacheManager] add support for stream detection across chunks 
Retain a small part (6 or 9 bytes) of each chunk in memory for providing
 the next iteration with enough context for finding the start/end marker
 of a range.
 2021-05-17 17:36:05 +01:00
Jakob Ackermann
6b9c8bced6 [ContentCacheManager] write streams to disk atomically 
Use an intermediate file for writing to disk, then rename to the target.
 2021-05-17 14:22:37 +01:00
Brian Gough
371de76a4a use fs.copyFile instead of fse.copy in UrlCache module 2021-05-17 10:54:11 +01:00
Jakob Ackermann
18e943742e [perf] drop useless synchronous syscall on hot path for writing docs 2021-05-17 09:25:29 +01:00
Brian Gough
44e0742aa3 use fse.copy for performance 
it uses the native fs.copyFile method
 2021-05-14 15:49:20 +01:00
Brian Gough
ff2175e727 add validation for express :content_id parameter 2021-05-13 15:48:17 +01:00
Brian Gough
013d38552a log the expiry timeout value when disk is low 2021-05-13 15:13:29 +01:00
Jakob Ackermann
b456ea726d [misc] merge pdf caching into main (#226) 
* wip generate directory for hash content

* cleanup, remove console logging

* add content caching module

* Return PDF stream ranges with compile response

* Return the PDF file size in the compile response

* PDF range endpoint

* [misc] WIP: pdf caching: preserve the m-time on static content files

* [misc] WIP: pdf caching: improve browser caching, emit caching headers

* [misc] WIP: pdf caching: do not emit very small chunks <1kB

* [misc] keep up with moving output files into a separate directory

* [OutputCacheManager] add global feature flag for enabling pdf caching

* [misc] add contentId into the URL for protecting PDF stream contents

* [misc] support PDF stream caching for anonymous users

* [misc] add per-request feature flag for enabling PDF stream caching

* [misc] enable pdf caching in CI and emit metrics at the end of run

* [misc] expose compile stats and timings to the frontend

* [misc] log an error in case saving output files fails

* [misc] add metrics for pdf bandwidth and pdf caching performance

* [misc] add a dark mode to the pdf caching for computing ranges only

* [misc] move pdf caching metrics into ContentCacheMetrics

* [misc] add a config option for the min chunk size of pdf ranges

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
 2021-05-13 14:07:54 +01:00
Jakob Ackermann
cfa8127c1b [misc] add linting for missing explicit dependencies and fix any errors 2021-04-29 15:30:54 +01:00
Jakob Ackermann
b3ed820444 Merge pull request #219 from overleaf/jpa-read-image-name-for-synctex 
[misc] consume and validate a custom imageName for synctex requests
 2021-03-31 10:33:53 +02:00
Jakob Ackermann
6cb4246eb1 [misc] consume and validate a custom imageName for synctex requests 2021-03-30 13:57:40 +01:00
Shane Kilkelly
072d0a9dab Add flag to qpdf, to preserve PDF/A compliance 2021-03-19 11:35:59 +00:00
Brian Gough
865e68051e include fallback for missing state file 2021-01-26 13:59:28 +00:00
Brian Gough
a8e47da9e9 Revert "Merge pull request #205 from overleaf/revert-200-bg-decaff-cleanup" 
This reverts commit 76d8d3181b9464d1e1bbc713a2729ca269d9c047, reversing
changes made to 31a8dc3a98d73c2707d633712f0ef7207013e78b.
 2021-01-26 11:04:33 +00:00
Brian Gough
9dc55729ac Revert "decaff cleanup ResourceStateManager" 2021-01-25 15:26:53 +00:00
Brian Gough
f41f3aba8c Merge pull request #204 from overleaf/bg-use-output-dir 
use separate output dir for generated files
 2021-01-25 14:10:09 +00:00
Brian Gough
90c0d931ad Merge pull request #201 from overleaf/bg-more-cleanup 
decaff cleanup of OutputFileManager
 2021-01-25 14:09:46 +00:00
Brian Gough
c67c8a6c80 Merge pull request #200 from overleaf/bg-decaff-cleanup 
decaff cleanup ResourceStateManager
 2021-01-25 14:09:24 +00:00
Brian Gough
b5346658b0 clear output directory when clearing project 2021-01-22 11:05:52 +00:00
Brian Gough
692dbc8d6b add output directory 2021-01-22 11:05:52 +00:00
Jakob Ackermann
49b764a308 [misc] CompileController: exact match for output.pdf 
The previous regex could mistake user provided pdf files, like
 `fake_output.pdf`, as the final output file.
The frontend expects to find a `output.pdf` file on success.
 2020-12-28 13:16:31 +00:00
Brian Gough
e183fffa4b format fix 2020-12-18 16:15:06 +00:00
Brian Gough
dafe69b5e6 remove unused module var 2020-12-18 16:15:06 +00:00
Brian Gough
36b646bb84 remove comments 2020-12-18 16:15:06 +00:00
Brian Gough
f20a0d1a69 remove unused var 2020-12-18 16:15:03 +00:00
Brian Gough
df1caa14b8 remove unused require 2020-12-18 16:14:36 +00:00
Brian Gough
ca98ee5cff use a Set instead of an Object 2020-12-18 16:14:09 +00:00
Brian Gough
43b0429c28 clean up relative path checking 2020-12-18 16:14:05 +00:00
Brian Gough
73e09ff99f remove unnecessary requires 2020-12-18 15:28:06 +00:00
Brian Gough
cee93a0cd9 clean up unnecessary var 2020-12-18 15:28:06 +00:00
Brian Gough
4169f7fc73 use once for callback 2020-12-18 15:28:06 +00:00
Brian Gough
e84d6305f3 remove unnecessary return 2020-12-18 15:16:07 +00:00
Brian Gough
ae064e8997 lint fix 2020-12-18 15:11:29 +00:00
Brian Gough
323890cedb remove Array.from 2020-12-18 15:10:53 +00:00
Brian Gough
c044db4897 remove guard helper 2020-12-18 15:09:48 +00:00
Brian Gough
0bae53c7f3 remove unnecessary returns 2020-12-18 15:02:40 +00:00
Brian Gough
eaec57cd01 simplify null check 2020-12-18 15:01:25 +00:00
Brian Gough
411d69e36b remove unnecessary callback code 2020-12-18 14:59:48 +00:00
Brian Gough
11ef3c27ed use Set instead of object 2020-12-18 14:56:53 +00:00
Brian Gough
64ea22d259 remove unnecessary null checks 2020-12-17 12:05:17 +00:00
Brian Gough
299d7cc5fe remove unnecessary returns 2020-12-17 12:01:14 +00:00
Brian Gough
5a539b7f90 remove guard function 2020-12-17 11:59:15 +00:00
Brian Gough
4d8e4d54e5 remove Array.from 2020-12-17 11:49:02 +00:00
Jakob Ackermann
7b7cd8cc8c [misc] bump metrics module to 3.4.1 
- renamed package from `metrics-sharelatex` to `@overleaf/metrics`
- drop support for statsd backend
- decaffeinate
- compress `/metrics` response using gzip
- bump debugging agents to latest versions
- expose prometheus interfaces for custom metrics (custom tags)
- cleanup of open sockets metrics
- fix deprecation warnings for header access
 2020-11-25 11:57:25 +00:00
Eric Mc Sween
1c13f6fe94 Mount /home/tex in an anonymous volume 
When we mount the container's root filesystem as read-only, mount
an anonymous volume in /home/tex so that it's writable. Our TeX Live
images have cached content in /home/tex. This content will automatically
get copied by Docker into this anonymous volume.
 2020-09-04 08:49:56 -04:00
Eric Mc Sween
67f4a6eeeb Decaf cleanup: normalize functions 
Use function keyword for declarations and arrow functions for callbacks.
 2020-09-03 15:58:16 -04:00
Eric Mc Sween
5cd889038e Use _.once() instead of ad hoc implementation 2020-09-03 15:52:09 -04:00
Eric Mc Sween
a853950a99 Fix container monitor cleanup function 
The intent here is clearly to clear both the timeout and the interval.
 2020-09-03 15:50:45 -04:00
Eric Mc Sween
30a44edded Decaf cleanup: simplify variable declarations 2020-09-03 15:50:12 -04:00
Eric Mc Sween
a282bccd48 Do not instantiate errors at module load time 
This prevents the right stack trace from being captured.
 2020-09-03 15:34:19 -04:00
Eric Mc Sween
4905e7db2c Decaf cleanup: unused vars 2020-09-03 15:30:07 -04:00
Eric Mc Sween
44bf38d6db Decaf cleanup: convert async function to sync 
The examineOldContainer() function doesn't need to use callbacks since
it only does synchronous work.
 2020-09-03 15:13:04 -04:00
Eric Mc Sween
2584586ba2 Decaf cleanup: camel-case variables 2020-09-03 15:13:04 -04:00
Eric Mc Sween
c52d7d8f02 Decaf cleanup: simplify null checks 2020-09-03 15:13:04 -04:00
Eric Mc Sween
ee4c08868c Decaf cleanup: remove default callbacks 2020-09-03 15:13:04 -04:00
Eric Mc Sween
32f0bbe266 Decaf cleanup: remove IIFEs 2020-09-03 15:13:04 -04:00
Eric Mc Sween
f650da8675 Decaf cleanup: remove __guard__ 2020-09-02 17:06:35 -04:00
Eric Mc Sween
99648341e2 Decaf cleanup: remove unnecessary returns 2020-09-02 16:58:41 -04:00
Eric Mc Sween
08be54a43e Decaf cleanup: remove unnecessary Array.from() 2020-09-02 16:45:49 -04:00
Jakob Ackermann
1d1b9ebebc [misc] bump the dev-env to 3.3.2 2020-08-10 17:01:11 +01:00
Jakob Ackermann
5846ebb367 Merge pull request #182 from overleaf/msm-fix-npe-community-edition 
Fixed NPE when Settings.clsi is defined but Settings.clsi.docker is not
 2020-07-15 11:01:08 +02:00
Jakob Ackermann
0f9d0d9790 Merge pull request #183 from overleaf/jpa-clsi-allowed-image-names 
[misc] RequestParser: restrict imageName to an allow list and add tests
 2020-07-15 10:58:36 +02:00
Jakob Ackermann
c3a3ed341b Merge pull request #132 from das7pad/hotfix-double-call 
[LocalCommandRunner] run: block a double call of the callback
 2020-07-03 12:58:25 +02:00
Jakob Ackermann
f703b2ca41 [misc] move the image check prior to the base image override 2020-07-01 10:01:25 +01:00
Jakob Ackermann
aa02df7b81 [misc] apply review feedback 
- move setting into clsi.docker namespace
- rename the variable for images to allowedImages / ALLOWED_IMAGES
- add an additional check for the image name into the DockerRunner

Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
 2020-06-30 12:01:21 +01:00
Jakob Ackermann
c857371fed [misc] wordcount: restrict image to an allow list and add tests 2020-06-26 13:28:12 +01:00
Jakob Ackermann
8846efe7ce [misc] RequestParser: restrict imageName to an allow list and add tests 2020-06-26 13:28:09 +01:00
Miguel Serrano
f8ca906393 Fixed NPE when Settings.clsi is defined but Settings.clsi.docker is not 2020-06-25 12:31:10 +02:00
Brian Gough
2b342c6e53 Merge pull request #181 from overleaf/bg-fix-503-response 
handle EPIPE errors in CompileController
 2020-06-22 09:30:35 +01:00
Brian Gough
22480f1a52 handle EPIPE errors in CompileController 2020-06-18 09:54:18 +01:00
Brian Gough
bdebef91eb Merge branch 'master' into bg-add-compile-groups 2020-06-17 11:58:26 +01:00
Brian Gough
f6fddf2944 Merge pull request #179 from overleaf/bg-fix-synctex-error 
fix synctex error
 2020-06-16 08:57:54 +01:00
Brian Gough
2eb4327256 fix format 2020-06-16 08:45:53 +01:00
Brian Gough
f41b4daca2 Merge branch 'master' into bg-use-lodash 2020-06-16 08:35:17 +01:00
Brian Gough
2ce03f0554 add initial compileGroup support 2020-06-15 15:28:53 +01:00
Brian Gough
ede70b6f99 migrate from underscore to lodash 2020-06-15 09:52:21 +01:00
Brian Gough
5368630754 check output file exists before running synctex 2020-06-12 15:15:27 +01:00
Tim Alby
d3ff214b88 partially revert "[DockerRunner] fix metric incrementing and error logging" 
This reverts commits:
- 2b2fcca39ce8dee0fdc0c342aa0d6c822592bcec
- 9e82ab0890c5cc8c7fb95362c3f7edbcaad0cf29
- e3da458b376871c3ce72d6984d14bf1ee668b04b
 2020-06-11 12:51:26 +02:00
Brian Gough
3d6b911295 add setting TEXLIVE_OPENOUT_ANY 2020-06-04 11:47:22 +01:00
Brian Gough
aa44bae222 fix eslint errors 2020-06-02 09:51:34 +01:00
Brian Gough
a3d27b63f1 fix formatting with make format_fix 2020-06-02 09:18:38 +01:00
Brian Gough
d9265b779f Merge pull request #170 from overleaf/jpa-import-141 
[DockerRunner] destroyOldContainers: fix a race confition
 2020-06-02 09:04:59 +01:00
Brian Gough
59cd14e160 Merge pull request #169 from overleaf/bg-record-latexmk-output 
record latexmk output
 2020-06-02 09:03:43 +01:00
Jakob Ackermann
563a973388 [DockerRunner] destroyOldContainers: normalize the container name 
The docker api returns each name with a `/` prefix.

In order to not interfere with pending compiles, the deletion process
 has to acquire an internal lock on the container. The LockManager uses
 the plain container name without the slash: `project-xxx`.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-05-29 11:28:26 +01:00
Brian Gough
6d5dfb7758 clean up log file deletion and add unit test 2020-05-20 14:12:08 +01:00
Brian Gough
63770bf390 clean up the stdout/stderr recording 2020-05-20 11:45:29 +01:00
Henry Oswald
5c3da905ba Merge pull request #162 from overleaf/ta-jpa-epipe-retry 
[DockerRunner] retry container inspect on EPIPE
 2020-05-19 11:15:25 +01:00
Henry Oswald
2547725f35 Merge pull request #164 from overleaf/bg-fix-buffer-deprecations 
fix deprecated usage of Buffer constructor
 2020-05-19 10:26:56 +01:00
Henry Oswald
e9c6f3b074 Merge pull request #163 from overleaf/bg-use-encoding-on-process-output 
set encoding when reading from streams
 2020-05-19 10:26:26 +01:00
Henry Oswald
457b7d6657 Merge pull request #165 from overleaf/ho-retry-url-downloads 
add pipeUrlToFileWithRetry for file downloads
 2020-05-19 10:25:19 +01:00
Henry Oswald
3925839c7f add refreshExpiryTimeout function 
on clsi all data lives inside of / dir
dynamically reduce size of EXPIRY_TIMEOUT if disk starts to get full
 2020-05-18 15:17:19 +01:00
Brian Gough
3db513cfc9 record latexmk output 2020-05-15 16:08:10 +01:00
Henry Oswald
77e8ba74a7 add pipeUrlToFileWithRetry function to retry file downloads 3 times 2020-05-14 13:24:58 +01:00
Brian Gough
4947abe88b fix deprecated usage of Buffer constructor 2020-05-07 10:42:05 +01:00
Brian Gough
8e86f02c43 set encoding when reading from streams 
using .toString() works most of the time but can lead to utf8 characters being
broken across chunk boundaries.

https://nodejs.org/api/stream.html#stream_readable_setencoding_encoding
 2020-05-07 10:30:14 +01:00
Henry Oswald
e1c90ec234 Merge branch 'master' into bg-support-other-runtimes 
port coffee changes to js
 2020-04-23 13:59:05 +01:00
Jakob Ackermann
cf8533bee1 [LocalCommandRunner] run: block a double call of the callback 
The subprocess event handler fires the "error" and "close" event in case
 of a failure.
Both events would call the given callback, resulting in double
 processing of the subprocess result downstream.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-04-16 15:55:55 +02:00
Jakob Ackermann
cb9e9321f0 [DockerRunner] fix metric incrementing and error logging 
- do not log on first EPIPE
- inc 'container-inspect-epipe-error' on permanent error only

Co-Authored-By: Tim Alby <timothee.alby@gmail.com>
 2020-04-10 14:44:57 +02:00
Tim Alby
ecaa7035f5 add metrics for EPIPE errors 
Co-Authored-By: Jakob Ackermann <jakob.ackermann@overleaf.com>
 2020-04-10 12:28:48 +02:00
Tim Alby
e3b1472c7f retry once on EPIPE errors 
Co-Authored-By: Jakob Ackermann <jakob.ackermann@overleaf.com>
 2020-04-10 12:28:11 +02:00
Jakob Ackermann
c700a0ab9d Merge pull request #159 from overleaf/jpa-fix-docker-modem-error-scheme 
[misc] keep up with the error signature of dockerode/docker-modem
 2020-04-03 14:04:06 +02:00
Jakob Ackermann
6777432569 [misc] add a metric for failing downloads 2020-04-03 12:35:40 +02:00
Jakob Ackermann
4a19f022fc [misc] keep up with the error signature of dockerode/docker-modem 
https://github.com/apocas/docker-modem/blob/v2.1.1/lib/modem.js#L296
 2020-03-27 10:39:45 +01:00
mserranom
d5bd1790a8 updated sequelize 2020-03-16 16:31:02 +01:00
mserranom
cf6c8ab496 removed mkdirp dependency and replaced with fs.mkdir 2020-03-12 10:22:08 +01:00
mserranom
fa37ed865a added container monitor cleanup to fix hanging tests 2020-02-20 17:24:28 +01:00
mserranom
8729acd48c prettier: convert app/js decaffeinated files to Prettier format 2020-02-19 12:14:37 +01:00
mserranom
3af6bdd588 decaffeinate: rename app/coffee dir to app/js 2020-02-19 12:14:34 +01:00