Commit graph

1825 commits

Author SHA1 Message Date
Jakob Ackermann
d69c29e4dc [misc] apply review feedback on naming functions and parameters
Co-Authored-By: Simon Detheridge <s@sd.ai>
2021-06-08 19:29:10 +01:00
Jakob Ackermann
4996f8bbcd [DocArchiveManager] (un-)archive docs in batches and let db filter docs
Also drop the broken 404 logic after switching to db-side filtering.
2021-06-08 12:59:13 +01:00
Jakob Ackermann
7380b54900 [ProjectPersistenceManager] check all user content dirs for full disk 2021-06-01 16:13:48 +01:00
Jakob Ackermann
1127bf640e [misc] wait for refreshing of expiry timeout before triggering cleanup 2021-06-01 15:51:09 +01:00
Jakob Ackermann
294088fb27 [ContentCacheManager] use PDF.js Xref table instead of stream detection (#242)
* make the content cache manager tests configurable

* extend stream content in unit tests

* [ContentCacheManagerTests] prepare for full object caching

* filesystem stream for pdfjs

* working??

* cleaning up

* handle overflow

* [misc] install pdfjs-dist

* [misc] move pdfjs code into app/lib/ and scripts/, also use CamelCase

* [misc] abstract the file loading and parsing of xRef tables into helper

* [misc] pdfjsTests: add snapshot based tests for the Xref table parser

* [misc] FSStream: throw proper error and drop commented code

* [misc] FSStream: integrate throwing of MissingDataException into getter

* [misc] pdfjs: fix eslint errors

* [misc] pdfjs: run format_fix

* [misc] pdfjs: allocate very small non empty dummy buffers explicitly

* [misc] install @overleaf/o-error

* [ContentCacheManager] use PDF.js Xref table instead of stream detection

Co-Authored-By: Brian Gough <brian.gough@overleaf.com>

* [pdfjs] parseXrefTable: handle empty PDF files gracefully

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
2021-05-31 09:20:25 +01:00
Jakob Ackermann
d657007291 Merge pull request #241 from overleaf/jpa-wait-for-tests
[misc] AllowedImageNamesTests: add missing done callbacks
2021-05-31 10:20:07 +02:00
Alf Eaton
5ef647742a Merge pull request #67 from overleaf/ae-remove-populate
Delete prepopulate.coffee
2021-05-28 10:47:55 +01:00
nate stemen
cee6c0aad9 Merge pull request #105 from overleaf/dependabot-npm_and_yarn-underscore-1.13.1
Bump underscore from 1.10.2 to 1.13.1
2021-05-27 11:53:50 -04:00
nate stemen
081201f796 Merge pull request #103 from overleaf/jpa-retry-archive-deletion
[DocArchiveManager] retry the deletion of doc archives three times
2021-05-27 11:45:53 -04:00
nate stemen
3bbc9156cd Merge pull request #108 from overleaf/ae-decaffeinate-config
Finish decaffeinating settings.default.js
2021-05-27 11:45:13 -04:00
nate stemen
64e201283b Merge pull request #107 from overleaf/dependabot-npm_and_yarn-hosted-git-info-2.8.9
Bump hosted-git-info from 2.8.5 to 2.8.9
2021-05-27 11:45:00 -04:00
nate stemen
fcf0e026c3 Merge pull request #106 from overleaf/dependabot-npm_and_yarn-lodash-4.17.21
Bump lodash from 4.17.19 to 4.17.21
2021-05-27 11:44:52 -04:00
dependabot[bot]
91c19e1f88 Bump underscore from 1.10.2 to 1.13.1
Bumps [underscore](https://github.com/jashkenas/underscore) from 1.10.2 to 1.13.1.
- [Release notes](https://github.com/jashkenas/underscore/releases)
- [Commits](https://github.com/jashkenas/underscore/compare/1.10.2...1.13.1)

Signed-off-by: dependabot[bot] <support@github.com>
2021-05-27 15:43:37 +00:00
nate stemen
d0d9cec7e2 Merge pull request #102 from overleaf/jpa-explicit-dependencies
[misc] add linting for missing explicit dependencies and fix any errors
2021-05-27 11:43:04 -04:00
nate stemen
20f70b120a Merge pull request #97 from overleaf/dependabot-npm_and_yarn-y18n-4.0.1
Bump y18n from 4.0.0 to 4.0.1
2021-05-27 11:42:43 -04:00
Alf Eaton
de3e7a82c3 Delete prepopulate.coffee 2021-05-26 13:21:14 +01:00
Jakob Ackermann
d56a490121 [misc] AllowedImageNamesTests: add missing done callbacks 2021-05-25 10:24:49 +01:00
dependabot[bot]
4d578bf52d Bump lodash from 4.17.20 to 4.17.21
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.20 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.20...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>
2021-05-21 09:35:58 +00:00
Jakob Ackermann
f820d32221 Merge pull request #240 from overleaf/bg-jpa-hash-tracker
[ContentCacheManager] track hashes and expire unused ones
2021-05-21 11:35:05 +02:00
Jakob Ackermann
fdcfbcd6ea Merge pull request #238 from overleaf/jpa-drop-broken-timings
[LatexRunner] do not emit empty cpu/sys timings
2021-05-21 11:34:54 +02:00
Alf Eaton
acbf8b5898 Tidy and format 2021-05-19 20:22:02 +01:00
Alf Eaton
6507d668ae Update references to .coffee files 2021-05-19 18:57:35 +01:00
Alf Eaton
1b1c4aa4a0 Fix formatting 2021-05-19 18:47:51 +01:00
Alf Eaton
f31abe7cb3 Tidy and format 2021-05-19 18:36:27 +01:00
decaffeinate
85920b3120 decaffeinate: Run post-processing cleanups on settings.test.coffee 2021-05-19 16:50:46 +01:00
decaffeinate
8c984a77c4 decaffeinate: Convert settings.test.coffee to JS 2021-05-19 16:50:45 +01:00
decaffeinate
5af8f57dd0 decaffeinate: Rename settings.test.coffee from .coffee to .js 2021-05-19 16:50:45 +01:00
Jakob Ackermann
b53a105681 [LatexRunner] do not emit empty cpu/sys timings 2021-05-19 11:17:08 +01:00
Jakob Ackermann
f361cdfaca [ContentCacheManager] deeply integrate the HashFileTracker
- Update the tracker contents as we hash ranges
- Let the tracker be authoritative for already written ranges
  This is saving a stat call per newly written range.
2021-05-18 18:15:58 +01:00
Jakob Ackermann
7aeeb5a5a9 [ContentCacheManager] finish tracking of ranges across builds 2021-05-18 18:06:15 +01:00
Jakob Ackermann
011a228727 [misc] install p-limit and nodemon 2021-05-18 17:32:21 +01:00
Brian Gough
d493238eaf wip expire old hash files 2021-05-18 16:25:24 +01:00
Jakob Ackermann
31a29403a1 Merge pull request #237 from overleaf/jpa-fix-unit-tests
[misc] fix unit tests following the merge of atomic writes
2021-05-18 12:17:56 +02:00
Jakob Ackermann
567d02881d [misc] fix unit tests following the merge of atomic writes 2021-05-18 11:09:30 +01:00
Jakob Ackermann
23dd93ae50 Merge pull request #236 from overleaf/jpa-skip-duplicate-writes
[ContentCacheManager] skip writing of duplicate streams
2021-05-18 11:44:23 +02:00
Jakob Ackermann
e787106eed Merge pull request #235 from overleaf/jpa-atomic-writes
[ContentCacheManager] write streams to disk atomically
2021-05-18 11:44:16 +02:00
Jakob Ackermann
9b8763aed4 Merge pull request #234 from overleaf/jpa-stream-detection-across-chunks
[ContentCacheManager] add support for stream detection across chunks
2021-05-18 11:44:09 +02:00
Jakob Ackermann
74e45c60f4 Merge pull request #233 from overleaf/jpa-drop-sync-syscall
[perf] drop useless synchronous syscall on hot path for writing docs
2021-05-18 11:44:00 +02:00
Jakob Ackermann
8a3ec2bfda Merge pull request #232 from overleaf/bg-upgrade-fs-extra
upgrade fs-extra
2021-05-18 11:43:50 +02:00
Jakob Ackermann
7cf019ab78 Merge pull request #231 from overleaf/bg-use-fse-copy
use fs.copyFile for performance
2021-05-18 11:43:43 +02:00
Jakob Ackermann
b254e4285b Merge pull request #229 from overleaf/bg-log-expiry-timeout
log the expiry timeout value when disk space is low
2021-05-18 11:43:15 +02:00
Jakob Ackermann
59f18b667c Merge pull request #228 from overleaf/bg-add-content-id-validation
add validation for express content routes
2021-05-18 11:43:08 +02:00
Jakob Ackermann
a93c2c3942 Merge pull request #224 from overleaf/bg-add-hash-benchmark-script
add hash benchmark script
2021-05-18 11:42:59 +02:00
Jakob Ackermann
bc1ed82c6c [ContentCacheManager] skip writing of duplicate streams 2021-05-18 10:22:05 +01:00
nate stemen
0592a6be74 Merge pull request #65 from overleaf/dependabot-npm_and_yarn-hosted-git-info-2.8.9
build(deps): bump hosted-git-info from 2.8.5 to 2.8.9
2021-05-17 17:07:51 -04:00
nate stemen
ee62d864e5 Merge pull request #64 from overleaf/dependabot-npm_and_yarn-lodash-4.17.21
build(deps): bump lodash from 4.17.19 to 4.17.21
2021-05-17 17:07:44 -04:00
nate stemen
fcaaef09b8 Merge pull request #63 from overleaf/dependabot-npm_and_yarn-underscore-1.13.1
build(deps): bump underscore from 1.9.2 to 1.13.1
2021-05-17 17:07:37 -04:00
nate stemen
bc99218596 Merge pull request #61 from overleaf/jpa-explicit-dependencies
[misc] add linting for missing explicit dependencies and fix any errors
2021-05-17 17:07:30 -04:00
nate stemen
a019a3776d Merge pull request #60 from overleaf/dependabot-npm_and_yarn-y18n-4.0.1
build(deps): bump y18n from 4.0.0 to 4.0.1
2021-05-17 17:07:22 -04:00
Jakob Ackermann
224ae0c254 [ContentCacheManager] add support for stream detection across chunks
Retain a small part (6 or 9 bytes) of each chunk in memory for providing
 the next iteration with enough context for finding the start/end marker
 of a range.
2021-05-17 17:36:05 +01:00