github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-10-17 21:05:04 -04:00
Code Issues Projects Releases Packages Wiki Activity
4419 commits 3 branches 7 tags 201 MiB
Commit graph

3812 commits

Author SHA1 Message Date
Simon Detheridge
1eb22a7d6d Export prometheus metrics directly from handler 2021-06-22 16:40:18 +01:00
Simon Detheridge
63ca055637 Use ServletHolder lifecycle more carefully 
Fixes metrics with Jetty > 9.4.20
 2021-06-22 15:41:24 +01:00
Simon Detheridge
70f4e5801d Merge pull request #107 from overleaf/sk-upgrade-jetty-2106 
Upgrade jetty and jetty-servlet
 2021-06-22 13:15:03 +01:00
Simon Detheridge
eb9cf6f256 Export prometheus metrics 2021-06-22 13:06:40 +01:00
Jakob Ackermann
ffaff1bd72 [CompileController] emit status=failure for an empty output.pdf file 2021-06-22 12:15:12 +01:00
Jakob Ackermann
fb3966ef35 [misc] CompileController: simplify composing of outputFiles 2021-06-22 12:15:12 +01:00
Shane Kilkelly
c29795f043 Upgrade jetty-servlet to 9.4.38.v20210224 2021-06-22 11:33:14 +01:00
dependabot[bot]
c6897d3b42 Bump jetty-server from 9.4.8.v20171121 to 9.4.38.v20210224 
Bumps [jetty-server](https://github.com/eclipse/jetty.project) from 9.4.8.v20171121 to 9.4.38.v20210224.
- [Release notes](https://github.com/eclipse/jetty.project/releases)
- [Commits](https://github.com/eclipse/jetty.project/compare/jetty-9.4.8.v20171121...jetty-9.4.38.v20210224)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-06-22 10:36:28 +01:00
Simon Detheridge
7a934b8eba Update mockito 2021-06-21 11:22:44 +01:00
Simon Detheridge
6009393013 Add some core tools for debugging processes 2021-06-21 11:21:58 +01:00
Simon Detheridge
bce5af615f Add gc logging to the Git Bridge 2021-06-21 11:21:58 +01:00
Simon Detheridge
9ef4d1ee5c Update Java to 11 2021-06-21 11:21:53 +01:00
Shane Kilkelly
594009aeab Reduce memory limit to 50 percent 2021-06-17 14:24:07 +01:00
Shane Kilkelly
ed3507977b Switch to UseContainerSupport and MaxRAMPercentage 2021-06-16 17:23:38 +01:00
Shane Kilkelly
8a68c3aef0 Merge pull request #102 from overleaf/sk-add-sqlite-prod 
Add sqlite3 package to the production image
 2021-06-16 10:36:33 +01:00
Shane Kilkelly
cbad8bcfb6 Add sqlite3 package to the production image 2021-06-16 09:46:53 +01:00
Shane Kilkelly
d824cb8451 Add heap flags to production entry script 2021-06-15 14:48:45 +01:00
Jakob Ackermann
f32f5e58a8 Merge pull request #248 from overleaf/jpa-add-missing-dep 
[misc] add missing dependency entry for send
 2021-06-15 10:27:35 +02:00
Jakob Ackermann
a76a9a3ef8 [misc] add missing dependency entry for send 2021-06-15 09:24:23 +01:00
Jakob Ackermann
8caa19afed Merge pull request #247 from overleaf/jpa-track-db-times 
[misc] track delays of using sqlite in a few places
 2021-06-15 10:11:46 +02:00
Jakob Ackermann
7659f462da Merge pull request #246 from overleaf/dependabot-npm_and_yarn-normalize-url-4.5.1 
Bump normalize-url from 4.5.0 to 4.5.1
 2021-06-15 10:10:59 +02:00
Jakob Ackermann
f7bfc6bf5f Merge pull request #245 from overleaf/dependabot-npm_and_yarn-glob-parent-5.1.2 
Bump glob-parent from 5.1.0 to 5.1.2
 2021-06-15 10:10:53 +02:00
Jakob Ackermann
2fbc3d9127 Merge pull request #223 from overleaf/dependabot-npm_and_yarn-hosted-git-info-2.8.9 
Bump hosted-git-info from 2.8.5 to 2.8.9
 2021-06-15 10:10:45 +02:00
Jakob Ackermann
83e5405839 Merge pull request #222 from overleaf/dependabot-npm_and_yarn-lodash-4.17.21 
Bump lodash from 4.17.20 to 4.17.21
 2021-06-15 10:10:38 +02:00
Jakob Ackermann
8da9a5b7c8 Merge pull request #220 from overleaf/dependabot-npm_and_yarn-y18n-4.0.1 
Bump y18n from 4.0.0 to 4.0.1
 2021-06-15 10:10:20 +02:00
Jakob Ackermann
878cea8087 Merge pull request #221 from overleaf/jpa-explicit-dependencies 
[misc] add linting for missing explicit dependencies and fix any errors
 2021-06-15 10:09:43 +02:00
Jakob Ackermann
8cc4517d4c [misc] move import to avoid git conflict 2021-06-15 09:08:32 +01:00
Jakob Ackermann
8f1ea7f0d1 [misc] track delays of using sqlite for url caching details 2021-06-14 13:03:02 +01:00
Jakob Ackermann
d40065fd8e [misc] track delay of using sqlite for last access time of a project 2021-06-14 12:41:16 +01:00
dependabot[bot]
138119a063 Bump normalize-url from 4.5.0 to 4.5.1 
Bumps [normalize-url](https://github.com/sindresorhus/normalize-url) from 4.5.0 to 4.5.1.
- [Release notes](https://github.com/sindresorhus/normalize-url/releases)
- [Commits](https://github.com/sindresorhus/normalize-url/commits)

---
updated-dependencies:
- dependency-name: normalize-url
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-06-12 07:23:12 +00:00
dependabot[bot]
0090748379 Bump glob-parent from 5.1.1 to 5.1.2 
Bumps [glob-parent](https://github.com/gulpjs/glob-parent) from 5.1.1 to 5.1.2.
- [Release notes](https://github.com/gulpjs/glob-parent/releases)
- [Changelog](https://github.com/gulpjs/glob-parent/blob/main/CHANGELOG.md)
- [Commits](https://github.com/gulpjs/glob-parent/compare/v5.1.1...v5.1.2)

---
updated-dependencies:
- dependency-name: glob-parent
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-06-11 14:36:06 +00:00
dependabot[bot]
0c06d0daa6 Bump glob-parent from 5.1.0 to 5.1.2 
Bumps [glob-parent](https://github.com/gulpjs/glob-parent) from 5.1.0 to 5.1.2.
- [Release notes](https://github.com/gulpjs/glob-parent/releases)
- [Changelog](https://github.com/gulpjs/glob-parent/blob/main/CHANGELOG.md)
- [Commits](https://github.com/gulpjs/glob-parent/compare/v5.1.0...v5.1.2)

---
updated-dependencies:
- dependency-name: glob-parent
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-06-11 07:58:49 +00:00
dependabot[bot]
302576b386 Bump glob-parent from 5.1.1 to 5.1.2 
Bumps [glob-parent](https://github.com/gulpjs/glob-parent) from 5.1.1 to 5.1.2.
- [Release notes](https://github.com/gulpjs/glob-parent/releases)
- [Changelog](https://github.com/gulpjs/glob-parent/blob/main/CHANGELOG.md)
- [Commits](https://github.com/gulpjs/glob-parent/compare/v5.1.1...v5.1.2)

---
updated-dependencies:
- dependency-name: glob-parent
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-06-10 09:11:02 +00:00
Jakob Ackermann
d69c29e4dc [misc] apply review feedback on naming functions and parameters 
Co-Authored-By: Simon Detheridge <s@sd.ai>
 2021-06-08 19:29:10 +01:00
Jakob Ackermann
4996f8bbcd [DocArchiveManager] (un-)archive docs in batches and let db filter docs 
Also drop the broken 404 logic after switching to db-side filtering.
 2021-06-08 12:59:13 +01:00
Jakob Ackermann
7380b54900 [ProjectPersistenceManager] check all user content dirs for full disk 2021-06-01 16:13:48 +01:00
Jakob Ackermann
1127bf640e [misc] wait for refreshing of expiry timeout before triggering cleanup 2021-06-01 15:51:09 +01:00
Jakob Ackermann
294088fb27 [ContentCacheManager] use PDF.js Xref table instead of stream detection (#242) 
* make the content cache manager tests configurable

* extend stream content in unit tests

* [ContentCacheManagerTests] prepare for full object caching

* filesystem stream for pdfjs

* working??

* cleaning up

* handle overflow

* [misc] install pdfjs-dist

* [misc] move pdfjs code into app/lib/ and scripts/, also use CamelCase

* [misc] abstract the file loading and parsing of xRef tables into helper

* [misc] pdfjsTests: add snapshot based tests for the Xref table parser

* [misc] FSStream: throw proper error and drop commented code

* [misc] FSStream: integrate throwing of MissingDataException into getter

* [misc] pdfjs: fix eslint errors

* [misc] pdfjs: run format_fix

* [misc] pdfjs: allocate very small non empty dummy buffers explicitly

* [misc] install @overleaf/o-error

* [ContentCacheManager] use PDF.js Xref table instead of stream detection

Co-Authored-By: Brian Gough <brian.gough@overleaf.com>

* [pdfjs] parseXrefTable: handle empty PDF files gracefully

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
 2021-05-31 09:20:25 +01:00
nate stemen
cee6c0aad9 Merge pull request #105 from overleaf/dependabot-npm_and_yarn-underscore-1.13.1 
Bump underscore from 1.10.2 to 1.13.1
 2021-05-27 11:53:50 -04:00
nate stemen
081201f796 Merge pull request #103 from overleaf/jpa-retry-archive-deletion 
[DocArchiveManager] retry the deletion of doc archives three times
 2021-05-27 11:45:53 -04:00
nate stemen
3bbc9156cd Merge pull request #108 from overleaf/ae-decaffeinate-config 
Finish decaffeinating settings.default.js
 2021-05-27 11:45:13 -04:00
nate stemen
64e201283b Merge pull request #107 from overleaf/dependabot-npm_and_yarn-hosted-git-info-2.8.9 
Bump hosted-git-info from 2.8.5 to 2.8.9
 2021-05-27 11:45:00 -04:00
nate stemen
fcf0e026c3 Merge pull request #106 from overleaf/dependabot-npm_and_yarn-lodash-4.17.21 
Bump lodash from 4.17.19 to 4.17.21
 2021-05-27 11:44:52 -04:00
dependabot[bot]
91c19e1f88 Bump underscore from 1.10.2 to 1.13.1 
Bumps [underscore](https://github.com/jashkenas/underscore) from 1.10.2 to 1.13.1.
- [Release notes](https://github.com/jashkenas/underscore/releases)
- [Commits](https://github.com/jashkenas/underscore/compare/1.10.2...1.13.1)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-27 15:43:37 +00:00
nate stemen
d0d9cec7e2 Merge pull request #102 from overleaf/jpa-explicit-dependencies 
[misc] add linting for missing explicit dependencies and fix any errors
 2021-05-27 11:43:04 -04:00
nate stemen
20f70b120a Merge pull request #97 from overleaf/dependabot-npm_and_yarn-y18n-4.0.1 
Bump y18n from 4.0.0 to 4.0.1
 2021-05-27 11:42:43 -04:00
Alf Eaton
de3e7a82c3 Delete prepopulate.coffee 2021-05-26 13:21:14 +01:00
Alf Eaton
674ad4acf0 Rename folder from coffee to js 2021-05-25 16:57:24 +01:00
Alf Eaton
5961e85b10 Fix path in test script 2021-05-25 16:56:57 +01:00
Alf Eaton
0649773c15 Manual tidying 2021-05-25 15:54:50 +01:00
Alf Eaton
b5b2bb3d22 Run format:fix 2021-05-25 15:53:36 +01:00
Alf Eaton
24fd9449f2 Run format:fix 2021-05-25 15:52:19 +01:00
Alf Eaton
358786fccd Run format:fix 2021-05-25 15:45:42 +01:00
Alf Eaton
f4db3d8f45 Manual tidying 2021-05-25 14:32:03 +01:00
Alf Eaton
de50e59557 Run format:fix 2021-05-25 14:27:45 +01:00
Jakob Ackermann
d56a490121 [misc] AllowedImageNamesTests: add missing done callbacks 2021-05-25 10:24:49 +01:00
dependabot[bot]
4d578bf52d Bump lodash from 4.17.20 to 4.17.21 
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.20 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.20...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-21 09:35:58 +00:00
Jakob Ackermann
f820d32221 Merge pull request #240 from overleaf/bg-jpa-hash-tracker 
[ContentCacheManager] track hashes and expire unused ones
 2021-05-21 11:35:05 +02:00
Jakob Ackermann
fdcfbcd6ea Merge pull request #238 from overleaf/jpa-drop-broken-timings 
[LatexRunner] do not emit empty cpu/sys timings
 2021-05-21 11:34:54 +02:00
Shane Kilkelly
82ab89fab3 Remove exception from signature 2021-05-21 09:57:28 +01:00
Shane Kilkelly
2efc7f22a7 Remove old username/password settings 
These were once used to secure interactions with the web api, back when
we allowed anonymous access to git repositories. This feature was
dropped in the migration to Overleaf v2, and we use OAuth on those
interactions anyway, so these settings are not actually used for
anything, and keeping these around is just confusing.
 2021-05-20 15:59:26 +01:00
Shane Kilkelly
c043988027 Remove default user/pass in config template 2021-05-20 10:05:27 +01:00
Shane Kilkelly
ae08604417 Fix variable name in config template 2021-05-20 09:50:05 +01:00
decaffeinate
03cbad4e9f decaffeinate: Run post-processing cleanups on expire_docops.coffee and 2 other files 2021-05-19 20:25:07 +01:00
decaffeinate
5ad0b6eea2 decaffeinate: Convert expire_docops.coffee and 2 other files to JS 2021-05-19 20:25:07 +01:00
decaffeinate
e1e0d26b1f decaffeinate: Rename expire_docops.coffee and 2 other files from .coffee to .js 2021-05-19 20:25:06 +01:00
Alf Eaton
edacebb53c Remove unused parameter 2021-05-19 20:25:04 +01:00
Alf Eaton
acbf8b5898 Tidy and format 2021-05-19 20:22:02 +01:00
Alf Eaton
6507d668ae Update references to .coffee files 2021-05-19 18:57:35 +01:00
Alf Eaton
1b1c4aa4a0 Fix formatting 2021-05-19 18:47:51 +01:00
Alf Eaton
f31abe7cb3 Tidy and format 2021-05-19 18:36:27 +01:00
decaffeinate
0b62828041 decaffeinate: Run post-processing cleanups on settings.defaults.coffee 2021-05-19 17:24:23 +01:00
decaffeinate
3acf183240 decaffeinate: Convert settings.defaults.coffee to JS 2021-05-19 17:24:23 +01:00
decaffeinate
8786542ad6 decaffeinate: Rename settings.defaults.coffee from .coffee to .js 2021-05-19 17:24:22 +01:00
decaffeinate
85920b3120 decaffeinate: Run post-processing cleanups on settings.test.coffee 2021-05-19 16:50:46 +01:00
decaffeinate
8c984a77c4 decaffeinate: Convert settings.test.coffee to JS 2021-05-19 16:50:45 +01:00
decaffeinate
5af8f57dd0 decaffeinate: Rename settings.test.coffee from .coffee to .js 2021-05-19 16:50:45 +01:00
Jakob Ackermann
b53a105681 [LatexRunner] do not emit empty cpu/sys timings 2021-05-19 11:17:08 +01:00
Jakob Ackermann
f361cdfaca [ContentCacheManager] deeply integrate the HashFileTracker 
- Update the tracker contents as we hash ranges
- Let the tracker be authoritative for already written ranges
  This is saving a stat call per newly written range.
 2021-05-18 18:15:58 +01:00
Jakob Ackermann
7aeeb5a5a9 [ContentCacheManager] finish tracking of ranges across builds 2021-05-18 18:06:15 +01:00
Jakob Ackermann
011a228727 [misc] install p-limit and nodemon 2021-05-18 17:32:21 +01:00
Brian Gough
d493238eaf wip expire old hash files 2021-05-18 16:25:24 +01:00
Shane Kilkelly
bbc062f942 default user/pass for development 2021-05-18 16:08:59 +01:00
nate stemen
1f10b45ce7 Merge pull request #147 from overleaf/dependabot-npm_and_yarn-lodash-4.17.21 
Bump lodash from 4.17.20 to 4.17.21
 2021-05-18 10:20:48 -04:00
nate stemen
948b4929c7 Merge pull request #146 from overleaf/jpa-explicit-dependencies 
[misc] add linting for missing explicit dependencies and fix any errors
 2021-05-18 10:20:27 -04:00
Jakob Ackermann
567d02881d [misc] fix unit tests following the merge of atomic writes 2021-05-18 11:09:30 +01:00
Jakob Ackermann
23dd93ae50 Merge pull request #236 from overleaf/jpa-skip-duplicate-writes 
[ContentCacheManager] skip writing of duplicate streams
 2021-05-18 11:44:23 +02:00
Jakob Ackermann
e787106eed Merge pull request #235 from overleaf/jpa-atomic-writes 
[ContentCacheManager] write streams to disk atomically
 2021-05-18 11:44:16 +02:00
Jakob Ackermann
9b8763aed4 Merge pull request #234 from overleaf/jpa-stream-detection-across-chunks 
[ContentCacheManager] add support for stream detection across chunks
 2021-05-18 11:44:09 +02:00
Jakob Ackermann
74e45c60f4 Merge pull request #233 from overleaf/jpa-drop-sync-syscall 
[perf] drop useless synchronous syscall on hot path for writing docs
 2021-05-18 11:44:00 +02:00
Jakob Ackermann
8a3ec2bfda Merge pull request #232 from overleaf/bg-upgrade-fs-extra 
upgrade fs-extra
 2021-05-18 11:43:50 +02:00
Jakob Ackermann
7cf019ab78 Merge pull request #231 from overleaf/bg-use-fse-copy 
use fs.copyFile for performance
 2021-05-18 11:43:43 +02:00
Jakob Ackermann
b254e4285b Merge pull request #229 from overleaf/bg-log-expiry-timeout 
log the expiry timeout value when disk space is low
 2021-05-18 11:43:15 +02:00
Jakob Ackermann
59f18b667c Merge pull request #228 from overleaf/bg-add-content-id-validation 
add validation for express content routes
 2021-05-18 11:43:08 +02:00
Jakob Ackermann
bc1ed82c6c [ContentCacheManager] skip writing of duplicate streams 2021-05-18 10:22:05 +01:00
nate stemen
0592a6be74 Merge pull request #65 from overleaf/dependabot-npm_and_yarn-hosted-git-info-2.8.9 
build(deps): bump hosted-git-info from 2.8.5 to 2.8.9
 2021-05-17 17:07:51 -04:00
nate stemen
ee62d864e5 Merge pull request #64 from overleaf/dependabot-npm_and_yarn-lodash-4.17.21 
build(deps): bump lodash from 4.17.19 to 4.17.21
 2021-05-17 17:07:44 -04:00
nate stemen
fcaaef09b8 Merge pull request #63 from overleaf/dependabot-npm_and_yarn-underscore-1.13.1 
build(deps): bump underscore from 1.9.2 to 1.13.1
 2021-05-17 17:07:37 -04:00
nate stemen
bc99218596 Merge pull request #61 from overleaf/jpa-explicit-dependencies 
[misc] add linting for missing explicit dependencies and fix any errors
 2021-05-17 17:07:30 -04:00
Jakob Ackermann
224ae0c254 [ContentCacheManager] add support for stream detection across chunks 
Retain a small part (6 or 9 bytes) of each chunk in memory for providing
 the next iteration with enough context for finding the start/end marker
 of a range.
 2021-05-17 17:36:05 +01:00
nate stemen
290c0d90c8 Merge pull request #44 from overleaf/dependabot-npm_and_yarn-hosted-git-info-2.8.9 
Bump hosted-git-info from 2.8.8 to 2.8.9
 2021-05-17 11:02:06 -04:00
nate stemen
1de6c67910 Merge pull request #43 from overleaf/dependabot-npm_and_yarn-lodash-4.17.21 
Bump lodash from 4.17.20 to 4.17.21
 2021-05-17 11:01:50 -04:00
nate stemen
b076823e7a Merge pull request #42 from overleaf/dependabot-npm_and_yarn-underscore-1.13.1 
Bump underscore from 1.9.2 to 1.13.1
 2021-05-17 11:01:41 -04:00
nate stemen
c33d6e4188 Merge pull request #40 from overleaf/jpa-explicit-dependencies 
[misc] add linting for missing explicit dependencies and fix any errors
 2021-05-17 11:01:30 -04:00
Jakob Ackermann
6b9c8bced6 [ContentCacheManager] write streams to disk atomically 
Use an intermediate file for writing to disk, then rename to the target.
 2021-05-17 14:22:37 +01:00
Brian Gough
8c80ddd27a remove unnecessary console.log from hash benchmark 2021-05-17 14:14:00 +01:00
Brian Gough
939ad3af31 include UV_THREADPOOL_SIZE in benchmark logs 2021-05-17 14:14:00 +01:00
Brian Gough
cbeba4af00 run the hash benchmark 10 times 2021-05-17 14:14:00 +01:00
Brian Gough
7b6434ef81 add benchmark script for hashing 2021-05-17 14:14:00 +01:00
nate stemen
0b5b8f56cb Merge pull request #73 from overleaf/dependabot-npm_and_yarn-hosted-git-info-2.8.9 
Bump hosted-git-info from 2.8.8 to 2.8.9
 2021-05-17 08:10:57 -04:00
nate stemen
a7c31132fe Merge pull request #72 from overleaf/dependabot-npm_and_yarn-lodash-4.17.21 
Bump lodash from 4.17.20 to 4.17.21
 2021-05-17 08:10:50 -04:00
nate stemen
007141f545 Merge pull request #71 from overleaf/jpa-explicit-dependencies 
[misc] add linting for missing explicit dependencies and fix any errors
 2021-05-17 08:10:43 -04:00
nate stemen
d1c2dd6cf1 Merge pull request #70 from overleaf/dependabot-npm_and_yarn-y18n-4.0.1 
Bump y18n from 4.0.0 to 4.0.1
 2021-05-17 08:10:32 -04:00
nate stemen
227b9e9ed8 Merge pull request #66 from overleaf/dependabot-npm_and_yarn-ini-1.3.8 
Bump ini from 1.3.5 to 1.3.8
 2021-05-17 08:10:17 -04:00
nate stemen
9b234e5946 Merge pull request #65 from overleaf/dependabot-npm_and_yarn-dot-prop-4.2.1 
Bump dot-prop from 4.2.0 to 4.2.1
 2021-05-17 08:10:08 -04:00
Brian Gough
371de76a4a use fs.copyFile instead of fse.copy in UrlCache module 2021-05-17 10:54:11 +01:00
Jakob Ackermann
dde645e770 Merge pull request #176 from overleaf/dependabot-npm_and_yarn-hosted-git-info-2.8.9 
Bump hosted-git-info from 2.8.8 to 2.8.9
 2021-05-17 10:57:18 +02:00
Jakob Ackermann
e92b9fc754 Merge pull request #174 from overleaf/jpa-explicit-dependencies 
[misc] add linting for missing explicit dependencies and fix any errors
 2021-05-17 10:57:12 +02:00
Jakob Ackermann
f97117a6ba Merge pull request #175 from overleaf/jpa-double-down-doc-size-limit 
[misc] double down on enforcing doc size limits
 2021-05-17 10:56:55 +02:00
Jakob Ackermann
18e943742e [perf] drop useless synchronous syscall on hot path for writing docs 2021-05-17 09:25:29 +01:00
Brian Gough
24ef5786a0 upgrade fs-extra 2021-05-14 15:58:11 +01:00
Brian Gough
44e0742aa3 use fse.copy for performance 
it uses the native fs.copyFile method
 2021-05-14 15:49:20 +01:00
Brian Gough
ff2175e727 add validation for express :content_id parameter 2021-05-13 15:48:17 +01:00
Brian Gough
013d38552a log the expiry timeout value when disk is low 2021-05-13 15:13:29 +01:00
Jakob Ackermann
b456ea726d [misc] merge pdf caching into main (#226) 
* wip generate directory for hash content

* cleanup, remove console logging

* add content caching module

* Return PDF stream ranges with compile response

* Return the PDF file size in the compile response

* PDF range endpoint

* [misc] WIP: pdf caching: preserve the m-time on static content files

* [misc] WIP: pdf caching: improve browser caching, emit caching headers

* [misc] WIP: pdf caching: do not emit very small chunks <1kB

* [misc] keep up with moving output files into a separate directory

* [OutputCacheManager] add global feature flag for enabling pdf caching

* [misc] add contentId into the URL for protecting PDF stream contents

* [misc] support PDF stream caching for anonymous users

* [misc] add per-request feature flag for enabling PDF stream caching

* [misc] enable pdf caching in CI and emit metrics at the end of run

* [misc] expose compile stats and timings to the frontend

* [misc] log an error in case saving output files fails

* [misc] add metrics for pdf bandwidth and pdf caching performance

* [misc] add a dark mode to the pdf caching for computing ranges only

* [misc] move pdf caching metrics into ContentCacheMetrics

* [misc] add a config option for the min chunk size of pdf ranges

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
 2021-05-13 14:07:54 +01:00
Christopher Hoskin
725a9e0669 Use CMD as in other apps 2021-05-12 12:32:14 +01:00
Christopher Hoskin
0827795fc7 Consistently use GIT_BRIDGE 2021-05-12 08:42:45 +01:00
dependabot[bot]
da81a97b82 Bump hosted-git-info from 2.8.8 to 2.8.9 
Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases)
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md)
- [Commits](https://github.com/npm/hosted-git-info/compare/v2.8.8...v2.8.9)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-11 17:18:40 +00:00
dependabot[bot]
e66ff7a2ca Bump hosted-git-info from 2.8.5 to 2.8.9 
Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.5 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases)
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md)
- [Commits](https://github.com/npm/hosted-git-info/compare/v2.8.5...v2.8.9)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-11 14:32:37 +00:00
dependabot[bot]
eb8a3118d8 Bump hosted-git-info from 2.8.5 to 2.8.9 
Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.5 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases)
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md)
- [Commits](https://github.com/npm/hosted-git-info/compare/v2.8.5...v2.8.9)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-11 13:04:24 +00:00
dependabot[bot]
bc423fa0cd build(deps): bump hosted-git-info from 2.8.5 to 2.8.9 
Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.5 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases)
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md)
- [Commits](https://github.com/npm/hosted-git-info/compare/v2.8.5...v2.8.9)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-11 12:49:25 +00:00
dependabot[bot]
ce32ca9238 Bump hosted-git-info from 2.8.8 to 2.8.9 
Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases)
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md)
- [Commits](https://github.com/npm/hosted-git-info/compare/v2.8.8...v2.8.9)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-11 12:30:36 +00:00
dependabot[bot]
54bbb3dbbd Bump hosted-git-info from 2.8.8 to 2.8.9 
Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases)
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md)
- [Commits](https://github.com/npm/hosted-git-info/compare/v2.8.8...v2.8.9)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-11 10:38:18 +00:00
dependabot[bot]
fc48f8dd98 Bump hosted-git-info from 2.8.8 to 2.8.9 
Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases)
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md)
- [Commits](https://github.com/npm/hosted-git-info/compare/v2.8.8...v2.8.9)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-11 08:45:50 +00:00
dependabot[bot]
847ff87ce3 Bump lodash from 4.17.19 to 4.17.21 
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.19 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.19...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-11 04:33:05 +00:00
dependabot[bot]
c0fc4c329e build(deps): bump lodash from 4.17.19 to 4.17.21 
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.19 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.19...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-11 04:20:08 +00:00
dependabot[bot]
78c0b90eb7 Bump lodash from 4.17.20 to 4.17.21 
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.20 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.20...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-11 03:28:23 +00:00
dependabot[bot]
39249d0b7e Bump lodash from 4.17.20 to 4.17.21 
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.20 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.20...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-11 00:17:10 +00:00
dependabot[bot]
785a69bdb4 Bump lodash from 4.17.20 to 4.17.21 
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.20 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.20...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-10 17:09:32 +00:00
Shane Kilkelly
b47f21a44e Configure from env vars at run time 
In both local developent, and production.  Also, vendor the `envsubst`
binary from https://github.com/a8m/envsubst, as it supports default
values, which the gnu envsubst (from gettext-base) does not.
 2021-05-07 09:47:49 +01:00
dependabot[bot]
7d8f439b4c build(deps): bump underscore from 1.9.2 to 1.13.1 
Bumps [underscore](https://github.com/jashkenas/underscore) from 1.9.2 to 1.13.1.
- [Release notes](https://github.com/jashkenas/underscore/releases)
- [Commits](https://github.com/jashkenas/underscore/compare/1.9.2...1.13.1)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-07 05:21:44 +00:00
dependabot[bot]
dd88d03254 Bump underscore from 1.9.2 to 1.13.1 
Bumps [underscore](https://github.com/jashkenas/underscore) from 1.9.2 to 1.13.1.
- [Release notes](https://github.com/jashkenas/underscore/releases)
- [Commits](https://github.com/jashkenas/underscore/compare/1.9.2...1.13.1)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-07 05:10:51 +00:00
Jakob Ackermann
309ad818f6 [SizeCheckTests] add acceptance tests for size limit checks 2021-05-06 18:21:19 +01:00
Jakob Ackermann
757e1e98c5 [RedisManager] block updating of too large docs in redis 2021-05-06 18:21:19 +01:00
Jakob Ackermann
c707d0b345 [RedisManager] block inserting of too large docs into redis 2021-05-06 18:21:18 +01:00
Jakob Ackermann
a4ae0ea12f [ShareJsUpdateManager] double check doc size before flushing 2021-05-06 18:20:51 +01:00
Jakob Ackermann
9ba8237850 [DocArchiveManager] retry the deletion of doc archives three times 2021-05-05 10:58:33 +01:00
Christopher Hoskin
f1c9d6108e Generate a config file from env vars and template 2021-04-30 15:49:37 +01:00
Jakob Ackermann
7e583c1bcd [misc] drop request from devDependencies 
Co-Authored-By: Alf Eaton <alf.eaton@overleaf.com>
 2021-04-30 09:44:26 +01:00
Jakob Ackermann
f5274e52c1 [misc] add linting for missing explicit dependencies and fix any errors 2021-04-29 16:20:31 +01:00
Jakob Ackermann
cfa8127c1b [misc] add linting for missing explicit dependencies and fix any errors 2021-04-29 15:30:54 +01:00
Jakob Ackermann
6e123b873d [misc] add linting for missing explicit dependencies and fix any errors 2021-04-29 15:30:53 +01:00
Jakob Ackermann
fe272bf89a [misc] add linting for missing explicit dependencies and fix any errors 2021-04-29 15:30:52 +01:00
Jakob Ackermann
126f5bef11 [misc] add linting for missing explicit dependencies and fix any errors 2021-04-29 15:30:51 +01:00
Jakob Ackermann
ed8c38908e [misc] add linting for missing explicit dependencies and fix any errors 2021-04-29 15:30:49 +01:00
Jakob Ackermann
33fad08c47 [misc] add linting for missing explicit dependencies and fix any errors 2021-04-29 15:30:48 +01:00
Brian Gough
ed3ddacac6 Merge pull request #172 from overleaf/bg-upgrade-ioredis 
upgrade ioredis via @overleaf/redis-wrapper
 2021-04-29 14:15:45 +01:00
Brian Gough
01112e10ab upgrade ioredis via @overleaf/redis-wrapper 
also update dependencies
 2021-04-29 11:49:13 +01:00
Brian Gough
b1a1446054 remove the limit of 300 maxSockets 2021-04-29 10:29:34 +01:00
Jakob Ackermann
54c6cce902 Merge pull request #100 from overleaf/jpa-drop-name-filter 
[MongoManager] drop name filter from getProjectsDeletedDocs
 2021-04-21 14:38:40 +02:00
Jakob Ackermann
74aaa73ad2 Merge pull request #101 from overleaf/jpa-destroy-in-small-batches 
[DocArchiveManager] destroy in small batches
 2021-04-16 11:24:04 +02:00
Jakob Ackermann
3f94dd4dbf [DocArchiveManager] destroy in small batches 
Also add settings options for PARALLEL_JOBS and DESTROY_BATCH_SIZE.
 2021-04-16 09:51:41 +01:00
Shane Kilkelly
8006e122d5 Merge pull request #170 from overleaf/sk-logger-update-2 
upgrade logger to 2.2.0
 2021-04-14 09:58:14 +01:00
Jakob Ackermann
34fc349646 [benchmarks] add benchmark for multi vs mget/mset 2021-04-13 11:47:10 +01:00
Jakob Ackermann
6e551f9b34 [perf] use MGET/MSET/DEL for manipulating multiple keys in one operation 
In some cases we can get rid of MULTI/EXEC operations too.
- putDocInMemory: from 10 down to 2 operations
- removeDocFromMemory: from 14+4 down to 4+4 operations
- updateDoc: from 13 down to 8 operations
 2021-04-13 11:47:10 +01:00
Jakob Ackermann
178440395f [perf] switch write sequence for doc contents and doc tracking 
Doc contents are added only after the tracking has been setup.
All read paths on the tracking have been checked to gracefully handle
 the case of existing doc_id but missing doc contents.

- getDoc: -1 operation

REF: 0a2b47c660c60b95e360d8f3b3e30b862ceb6d79
 2021-04-13 11:46:44 +01:00
Shane Kilkelly
83bbdfa9d1 upgrade logger to 2.2.0 2021-04-13 10:58:05 +01:00
Jakob Ackermann
5bcd142b6f [MongoManager] drop name filter from getProjectsDeletedDocs 
All the doc names we had in the projects collection are back-filled now.
A few docs will get dummy values back-filled.
 2021-04-12 17:31:26 +01:00
Jakob Ackermann
e80d013268 [misc] drop the deleteDoc endpoint -- use patchDoc instead 2021-04-12 17:31:18 +01:00
Jakob Ackermann
50b24043b7 [perf] use MGET for fetching multiple keys in one operation 
- getDoc: from 13 down to 2 operations
 2021-04-09 08:42:35 +01:00
Jakob Ackermann
189897cb30 Merge pull request #145 from overleaf/dependabot-npm_and_yarn-y18n-4.0.1 
Bump y18n from 4.0.0 to 4.0.1
 2021-04-07 14:11:01 +02:00
Simon Detheridge
8ba799035b Revert "Update logger-sharelatex to 2.2.0" 2021-04-06 15:29:09 +01:00
Shane Kilkelly
927d401111 Merge branch 'jpa-silent-download' 2021-04-06 13:34:50 +01:00
Shane Kilkelly
1e4e4b5ec0 Update logger-sharelatex to 2.2.0 2021-04-06 11:47:22 +01:00
dependabot[bot]
acbbd88d8a Bump y18n from 4.0.0 to 4.0.1 
Bumps [y18n](https://github.com/yargs/y18n) from 4.0.0 to 4.0.1.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-04-06 10:40:56 +00:00
Eric Mc Sween
4d70bd664f Reintroduce Node 12 and metrics upgrades 
These changes were previously merged, not deployed, and reverted. This
reverts the revert.

This reverts commit a6b8c6c658b33b6eee78b8b99e43308f32211ae2, reversing
changes made to 93c98921372eed4244d22fce800716cb27eca299.
 2021-04-01 15:51:00 -04:00
dependabot[bot]
ab57729f98 Bump y18n from 4.0.0 to 4.0.1 
Bumps [y18n](https://github.com/yargs/y18n) from 4.0.0 to 4.0.1.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-03-31 17:52:26 +00:00
Eric Mc Sween
4dd1b26b2e Revert "Merge pull request #161 from overleaf/em-upgrade-node-12" 
This reverts commit d44102751b9436ad89c5b3b05e7abdff51fcc78a, reversing
changes made to 6c2f5b8d053b75c677da2b7ddd04f998d2be6fff.
 2021-03-31 12:07:11 -04:00
Eric Mc Sween
b233e6588c Revert "Merge pull request #158 from overleaf/jpa-metrics-module-3-5-1" 
This reverts commit 93c98921372eed4244d22fce800716cb27eca299, reversing
changes made to d44102751b9436ad89c5b3b05e7abdff51fcc78a.
 2021-03-31 12:06:41 -04:00
dependabot[bot]
a21a6a2e72 Bump y18n from 4.0.0 to 4.0.1 
Bumps [y18n](https://github.com/yargs/y18n) from 4.0.0 to 4.0.1.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-03-31 15:23:12 +00:00
dependabot[bot]
3202e64f23 build(deps): bump y18n from 4.0.0 to 4.0.1 
Bumps [y18n](https://github.com/yargs/y18n) from 4.0.0 to 4.0.1.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-03-31 14:56:55 +00:00
Jakob Ackermann
98a32833a9 [misc] bump the version of the metrics module to 3.5.1 2021-03-31 10:08:04 -04:00
dependabot[bot]
96697eb91b Bump y18n from 4.0.0 to 4.0.1 
Bumps [y18n](https://github.com/yargs/y18n) from 4.0.0 to 4.0.1.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-03-31 14:06:02 +00:00
Jakob Ackermann
b3ed820444 Merge pull request #219 from overleaf/jpa-read-image-name-for-synctex 
[misc] consume and validate a custom imageName for synctex requests
 2021-03-31 10:33:53 +02:00
Jakob Ackermann
c1fe1cf5b7 Merge pull request #218 from overleaf/jpa-metrics-module-3-5-1 
[misc] bump the version of the metrics module to 3.5.1
 2021-03-31 10:33:41 +02:00
Jakob Ackermann
1a41fa55ad Merge pull request #217 from overleaf/sk-qpdf-newline 
Add flag to qpdf, to preserve PDF/A compliance
 2021-03-31 10:33:16 +02:00
dependabot[bot]
28b2d0ad87 Bump y18n from 4.0.0 to 4.0.1 
Bumps [y18n](https://github.com/yargs/y18n) from 4.0.0 to 4.0.1.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-03-31 03:00:58 +00:00
Jakob Ackermann
6cb4246eb1 [misc] consume and validate a custom imageName for synctex requests 2021-03-30 13:57:40 +01:00
dependabot[bot]
b6cd7bca9c Bump y18n from 4.0.0 to 4.0.1 
Bumps [y18n](https://github.com/yargs/y18n) from 4.0.0 to 4.0.1.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-03-30 08:56:10 +00:00
Jakob Ackermann
01059af954 [misc] drop unused node-statsd package 2021-03-30 09:45:07 +01:00
Alf Eaton
a271d5ce10 Merge pull request #212 from overleaf/ae-chown-app-folders 
Ensure that app folders exist before running chown in entrypoint.sh
 2021-03-23 09:43:31 +00:00
Alf Eaton
db0cd53ed4 Merge pull request #213 from overleaf/ae-readme 
Update README
 2021-03-23 09:41:44 +00:00
Jakob Ackermann
b563deef14 [misc] bump the version of the metrics module to 3.5.1 2021-03-22 12:59:43 +00:00
Eric Mc Sween
af72b4224c Merge pull request #36 from overleaf/jpa-metrics-module-3-5-1 
[misc] bump the version of the metrics module to 3.5.1
 2021-03-22 08:52:09 -04:00
Eric Mc Sween
7a6b86185b Merge pull request #215 from overleaf/ho-clsi-cookie-post 
support post and get for getting status of clsi project
 2021-03-22 07:56:09 -04:00
Christopher Hoskin
6ca016bb05 Explain the situation with permissions on Linux 2021-03-19 14:32:54 +00:00
Alf Eaton
268a144e43 Merge remote-tracking branch 'origin/master' into ae-readme 2021-03-19 12:06:03 +00:00
Alf Eaton
7e3edcee2c Add instructions for Linux 2021-03-19 12:05:22 +00:00
Alf Eaton
d4b4cb399a Revert "Make TEXLIVE_IMAGE_USER instruction macOS only" 
This reverts commit ab6fe228cadd3329885fdc60aea867d601579759.
 2021-03-19 12:00:47 +00:00
Shane Kilkelly
072d0a9dab Add flag to qpdf, to preserve PDF/A compliance 2021-03-19 11:35:59 +00:00