Jakob Ackermann
02918e7483
Merge pull request #4649 from overleaf/jpa-fs-based-caching
...
[perf] UrlCache: pure fs based cache state for downloads
GitOrigin-RevId: d19afc396324d4c3318b31620c8ad0c04e0544ce
2021-10-07 08:03:18 +00:00
Brian Gough
cc1b73336a
Merge pull request #4944 from overleaf/bg-use-file-line-errors
...
use file line errors in log output
GitOrigin-RevId: 6732b19552fe15431a70fbefbc572253c389c64e
2021-09-22 08:03:25 +00:00
Brian Gough
dc576f3b6f
Merge pull request #4943 from overleaf/bg-increase-max-print-line
...
allow setting texlive max_print_line variable
GitOrigin-RevId: 7588fed6aa5868a6ed6b6121cbd6f9c008c2aa0f
2021-09-22 08:03:21 +00:00
Eric Mc Sween
9ee92daea3
Merge pull request #4893 from overleaf/em-synctex
...
Use the synctex distributed with TeX Live
GitOrigin-RevId: 5a133f21f48fd1e217ab463e8cb2a5cdec8be1af
2021-09-07 16:26:17 +00:00
Eric Mc Sween
2feea2592d
Merge pull request #4887 from overleaf/em-decaf
...
Decaf cleanup for the CLSI CompileManager
GitOrigin-RevId: 06bba5c8af8808d0fa04187b10c8f31e08cd8754
2021-08-31 08:03:32 +00:00
Jakob Ackermann
1dc9f428fa
Merge pull request #4890 from overleaf/jpa-clsi-maint
...
[clsi] put server into maintenance mode when reaching zero capacity
GitOrigin-RevId: d1af260a3bfba7852519b3bc369a6747e79f3b18
2021-08-27 08:03:39 +00:00
Jakob Ackermann
b5e1ceef17
Merge pull request #4865 from overleaf/jpa-ho-load-shed-mvp
...
[clsi] very basic mvp for shedding load
GitOrigin-RevId: 4b82934b395f8b9358e463c03cadda6a16c20b4f
2021-08-27 08:03:21 +00:00
Jakob Ackermann
5122558469
Merge pull request #4842 from overleaf/jpa-drop-unused-packages
...
[misc] drop unused debug packages
GitOrigin-RevId: 83b1d96ee5a62c14c4a2dc51cefd179306a75a84
2021-08-27 08:03:17 +00:00
Jakob Ackermann
56a3b0dcde
Merge pull request #4819 from overleaf/jpa-pdf-caching-new-event-loop
...
[clsi] pdf-caching: move cpu intensive work onto a new event loop
GitOrigin-RevId: 4cb5cd4528fa1c5df6a8e91f9caa38cb64d94463
2021-08-25 08:03:38 +00:00
Jakob Ackermann
5d2549fe52
[misc] add/update CI pipelines for monorepo
...
GitOrigin-RevId: 8946b8be124e6977d2cafc6c3bf065412d765583
2021-08-05 13:48:05 +00:00
Jakob Ackermann
5e6401a1e0
Merge pull request #264 from overleaf/jpa-align-with-dev-env
...
[misc] make build scripts happy again
2021-07-27 10:46:16 +02:00
Jakob Ackermann
abcc532751
Merge pull request #262 from overleaf/jpa-faster-install
...
[perf] trim down install_deps.sh -- install docker cli only
2021-07-27 10:46:05 +02:00
Jakob Ackermann
76e749777d
[misc] make build scripts happy again
...
- move pdf.js tests into test/unit/js
- add env override to build script config file
- update build scripts
2021-07-26 12:20:04 +01:00
Brian Gough
262793c04f
add option for apparmor profile
2021-07-21 14:53:35 +01:00
Jakob Ackermann
d1c06d1878
[perf] trim down install_deps.sh -- install docker cli only
2021-07-20 14:14:00 +01:00
Jakob Ackermann
bc401ac44e
[misc] upgrade node version to latest v12 LTS version 12.22.3
2021-07-13 12:26:36 +01:00
Jakob Ackermann
0e5f5afe79
[misc] temporary override a few new/changed eslint rules
2021-07-13 12:24:43 +01:00
Jakob Ackermann
f285e503b4
[misc] run format_fix and lint:fix
2021-07-13 12:04:48 +01:00
Jakob Ackermann
631873fc83
[misc] upgrade build scripts to version 3.11.0 and cleanup packages
...
```
npm uninstall prettier-eslint-cli eslint-plugin-standard eslint-plugin-jsx-a11y eslint-plugin-react eslint-config-standard-jsx eslint-config-standard-react babel-eslint
npm dedupe
```
2021-07-13 11:55:19 +01:00
Jakob Ackermann
72a19ad895
[misc] goodbye coffee-script
2021-07-13 10:07:04 +01:00
Jakob Ackermann
4b066f615f
[misc] run npm dedupe
2021-07-12 17:51:07 +01:00
Jakob Ackermann
64551f0198
[misc] switch from settings-sharelatex to @overleaf/settings
2021-07-12 17:47:21 +01:00
Jakob Ackermann
ec49c4c9a6
[misc] install bunyan as production dependency
...
```
Error: Cannot find module 'bunyan'
Require stack:
- .../node_modules/@google-cloud/logging-bunyan/build/src/middleware/express.js
- .../node_modules/@google-cloud/logging-bunyan/build/src/index.js
- .../node_modules/logger-sharelatex/logging-manager.js
- .../node_modules/logger-sharelatex/index.js
- .../app.js
```
2021-07-12 17:35:51 +01:00
Jakob Ackermann
516525126b
[UrlFetcher] do not override domain for clsi-perf requests
2021-07-02 09:17:29 +01:00
Jakob Ackermann
efa0c54ca5
[misc] add timings for the sync stage and output stage
2021-06-30 11:45:34 +01:00
Jakob Ackermann
743bfced64
[misc] ContentCacheManager: apply review feedback
...
- count stages
- lower bound is 1s
Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
2021-06-23 14:42:39 +01:00
Jakob Ackermann
b09e52510f
[misc] bail out from pdf caching processing after 10s or earlier
...
...for fast compiles.
2021-06-23 14:20:04 +01:00
Jakob Ackermann
c3dba7f74d
Merge pull request #253 from overleaf/jpa-debug-slow-pdf-caching
...
[misc] ContentCacheMetrics: log slow pdf caching performance
2021-06-23 12:39:13 +02:00
Jakob Ackermann
97693b49c2
[ContentCacheMetrics] add new metric for absolute time spent in PDF.js
2021-06-23 11:28:31 +01:00
Jakob Ackermann
11a44ff07c
[misc] ContentCacheMetrics: apply review feedback: ignore fast compiles
...
Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
2021-06-23 11:27:19 +01:00
Jakob Ackermann
a4389fb761
[misc] ContentCacheMetrics: log slow pdf caching performance
2021-06-23 09:25:16 +01:00
Jakob Ackermann
ffaff1bd72
[CompileController] emit status=failure for an empty output.pdf file
2021-06-22 12:15:12 +01:00
Jakob Ackermann
fb3966ef35
[misc] CompileController: simplify composing of outputFiles
2021-06-22 12:15:12 +01:00
Jakob Ackermann
f32f5e58a8
Merge pull request #248 from overleaf/jpa-add-missing-dep
...
[misc] add missing dependency entry for send
2021-06-15 10:27:35 +02:00
Jakob Ackermann
a76a9a3ef8
[misc] add missing dependency entry for send
2021-06-15 09:24:23 +01:00
Jakob Ackermann
8caa19afed
Merge pull request #247 from overleaf/jpa-track-db-times
...
[misc] track delays of using sqlite in a few places
2021-06-15 10:11:46 +02:00
Jakob Ackermann
7659f462da
Merge pull request #246 from overleaf/dependabot-npm_and_yarn-normalize-url-4.5.1
...
Bump normalize-url from 4.5.0 to 4.5.1
2021-06-15 10:10:59 +02:00
Jakob Ackermann
f7bfc6bf5f
Merge pull request #245 from overleaf/dependabot-npm_and_yarn-glob-parent-5.1.2
...
Bump glob-parent from 5.1.0 to 5.1.2
2021-06-15 10:10:53 +02:00
Jakob Ackermann
2fbc3d9127
Merge pull request #223 from overleaf/dependabot-npm_and_yarn-hosted-git-info-2.8.9
...
Bump hosted-git-info from 2.8.5 to 2.8.9
2021-06-15 10:10:45 +02:00
Jakob Ackermann
83e5405839
Merge pull request #222 from overleaf/dependabot-npm_and_yarn-lodash-4.17.21
...
Bump lodash from 4.17.20 to 4.17.21
2021-06-15 10:10:38 +02:00
Jakob Ackermann
8da9a5b7c8
Merge pull request #220 from overleaf/dependabot-npm_and_yarn-y18n-4.0.1
...
Bump y18n from 4.0.0 to 4.0.1
2021-06-15 10:10:20 +02:00
Jakob Ackermann
878cea8087
Merge pull request #221 from overleaf/jpa-explicit-dependencies
...
[misc] add linting for missing explicit dependencies and fix any errors
2021-06-15 10:09:43 +02:00
Jakob Ackermann
8cc4517d4c
[misc] move import to avoid git conflict
2021-06-15 09:08:32 +01:00
Jakob Ackermann
8f1ea7f0d1
[misc] track delays of using sqlite for url caching details
2021-06-14 13:03:02 +01:00
Jakob Ackermann
d40065fd8e
[misc] track delay of using sqlite for last access time of a project
2021-06-14 12:41:16 +01:00
dependabot[bot]
138119a063
Bump normalize-url from 4.5.0 to 4.5.1
...
Bumps [normalize-url](https://github.com/sindresorhus/normalize-url ) from 4.5.0 to 4.5.1.
- [Release notes](https://github.com/sindresorhus/normalize-url/releases )
- [Commits](https://github.com/sindresorhus/normalize-url/commits )
---
updated-dependencies:
- dependency-name: normalize-url
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-06-12 07:23:12 +00:00
dependabot[bot]
0c06d0daa6
Bump glob-parent from 5.1.0 to 5.1.2
...
Bumps [glob-parent](https://github.com/gulpjs/glob-parent ) from 5.1.0 to 5.1.2.
- [Release notes](https://github.com/gulpjs/glob-parent/releases )
- [Changelog](https://github.com/gulpjs/glob-parent/blob/main/CHANGELOG.md )
- [Commits](https://github.com/gulpjs/glob-parent/compare/v5.1.0...v5.1.2 )
---
updated-dependencies:
- dependency-name: glob-parent
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-06-11 07:58:49 +00:00
Jakob Ackermann
7380b54900
[ProjectPersistenceManager] check all user content dirs for full disk
2021-06-01 16:13:48 +01:00
Jakob Ackermann
1127bf640e
[misc] wait for refreshing of expiry timeout before triggering cleanup
2021-06-01 15:51:09 +01:00
Jakob Ackermann
294088fb27
[ContentCacheManager] use PDF.js Xref table instead of stream detection ( #242 )
...
* make the content cache manager tests configurable
* extend stream content in unit tests
* [ContentCacheManagerTests] prepare for full object caching
* filesystem stream for pdfjs
* working??
* cleaning up
* handle overflow
* [misc] install pdfjs-dist
* [misc] move pdfjs code into app/lib/ and scripts/, also use CamelCase
* [misc] abstract the file loading and parsing of xRef tables into helper
* [misc] pdfjsTests: add snapshot based tests for the Xref table parser
* [misc] FSStream: throw proper error and drop commented code
* [misc] FSStream: integrate throwing of MissingDataException into getter
* [misc] pdfjs: fix eslint errors
* [misc] pdfjs: run format_fix
* [misc] pdfjs: allocate very small non empty dummy buffers explicitly
* [misc] install @overleaf/o-error
* [ContentCacheManager] use PDF.js Xref table instead of stream detection
Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
* [pdfjs] parseXrefTable: handle empty PDF files gracefully
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
2021-05-31 09:20:25 +01:00
Jakob Ackermann
d56a490121
[misc] AllowedImageNamesTests: add missing done callbacks
2021-05-25 10:24:49 +01:00
dependabot[bot]
4d578bf52d
Bump lodash from 4.17.20 to 4.17.21
...
Bumps [lodash](https://github.com/lodash/lodash ) from 4.17.20 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases )
- [Commits](https://github.com/lodash/lodash/compare/4.17.20...4.17.21 )
Signed-off-by: dependabot[bot] <support@github.com>
2021-05-21 09:35:58 +00:00
Jakob Ackermann
f820d32221
Merge pull request #240 from overleaf/bg-jpa-hash-tracker
...
[ContentCacheManager] track hashes and expire unused ones
2021-05-21 11:35:05 +02:00
Jakob Ackermann
fdcfbcd6ea
Merge pull request #238 from overleaf/jpa-drop-broken-timings
...
[LatexRunner] do not emit empty cpu/sys timings
2021-05-21 11:34:54 +02:00
Alf Eaton
6507d668ae
Update references to .coffee files
2021-05-19 18:57:35 +01:00
Alf Eaton
1b1c4aa4a0
Fix formatting
2021-05-19 18:47:51 +01:00
Alf Eaton
f31abe7cb3
Tidy and format
2021-05-19 18:36:27 +01:00
decaffeinate
85920b3120
decaffeinate: Run post-processing cleanups on settings.test.coffee
2021-05-19 16:50:46 +01:00
decaffeinate
8c984a77c4
decaffeinate: Convert settings.test.coffee to JS
2021-05-19 16:50:45 +01:00
decaffeinate
5af8f57dd0
decaffeinate: Rename settings.test.coffee from .coffee to .js
2021-05-19 16:50:45 +01:00
Jakob Ackermann
b53a105681
[LatexRunner] do not emit empty cpu/sys timings
2021-05-19 11:17:08 +01:00
Jakob Ackermann
f361cdfaca
[ContentCacheManager] deeply integrate the HashFileTracker
...
- Update the tracker contents as we hash ranges
- Let the tracker be authoritative for already written ranges
This is saving a stat call per newly written range.
2021-05-18 18:15:58 +01:00
Jakob Ackermann
7aeeb5a5a9
[ContentCacheManager] finish tracking of ranges across builds
2021-05-18 18:06:15 +01:00
Jakob Ackermann
011a228727
[misc] install p-limit and nodemon
2021-05-18 17:32:21 +01:00
Brian Gough
d493238eaf
wip expire old hash files
2021-05-18 16:25:24 +01:00
Jakob Ackermann
567d02881d
[misc] fix unit tests following the merge of atomic writes
2021-05-18 11:09:30 +01:00
Jakob Ackermann
23dd93ae50
Merge pull request #236 from overleaf/jpa-skip-duplicate-writes
...
[ContentCacheManager] skip writing of duplicate streams
2021-05-18 11:44:23 +02:00
Jakob Ackermann
e787106eed
Merge pull request #235 from overleaf/jpa-atomic-writes
...
[ContentCacheManager] write streams to disk atomically
2021-05-18 11:44:16 +02:00
Jakob Ackermann
9b8763aed4
Merge pull request #234 from overleaf/jpa-stream-detection-across-chunks
...
[ContentCacheManager] add support for stream detection across chunks
2021-05-18 11:44:09 +02:00
Jakob Ackermann
74e45c60f4
Merge pull request #233 from overleaf/jpa-drop-sync-syscall
...
[perf] drop useless synchronous syscall on hot path for writing docs
2021-05-18 11:44:00 +02:00
Jakob Ackermann
8a3ec2bfda
Merge pull request #232 from overleaf/bg-upgrade-fs-extra
...
upgrade fs-extra
2021-05-18 11:43:50 +02:00
Jakob Ackermann
7cf019ab78
Merge pull request #231 from overleaf/bg-use-fse-copy
...
use fs.copyFile for performance
2021-05-18 11:43:43 +02:00
Jakob Ackermann
b254e4285b
Merge pull request #229 from overleaf/bg-log-expiry-timeout
...
log the expiry timeout value when disk space is low
2021-05-18 11:43:15 +02:00
Jakob Ackermann
59f18b667c
Merge pull request #228 from overleaf/bg-add-content-id-validation
...
add validation for express content routes
2021-05-18 11:43:08 +02:00
Jakob Ackermann
bc1ed82c6c
[ContentCacheManager] skip writing of duplicate streams
2021-05-18 10:22:05 +01:00
Jakob Ackermann
224ae0c254
[ContentCacheManager] add support for stream detection across chunks
...
Retain a small part (6 or 9 bytes) of each chunk in memory for providing
the next iteration with enough context for finding the start/end marker
of a range.
2021-05-17 17:36:05 +01:00
Jakob Ackermann
6b9c8bced6
[ContentCacheManager] write streams to disk atomically
...
Use an intermediate file for writing to disk, then rename to the target.
2021-05-17 14:22:37 +01:00
Brian Gough
8c80ddd27a
remove unnecessary console.log from hash benchmark
2021-05-17 14:14:00 +01:00
Brian Gough
939ad3af31
include UV_THREADPOOL_SIZE in benchmark logs
2021-05-17 14:14:00 +01:00
Brian Gough
cbeba4af00
run the hash benchmark 10 times
2021-05-17 14:14:00 +01:00
Brian Gough
7b6434ef81
add benchmark script for hashing
2021-05-17 14:14:00 +01:00
Brian Gough
371de76a4a
use fs.copyFile instead of fse.copy in UrlCache module
2021-05-17 10:54:11 +01:00
Jakob Ackermann
18e943742e
[perf] drop useless synchronous syscall on hot path for writing docs
2021-05-17 09:25:29 +01:00
Brian Gough
24ef5786a0
upgrade fs-extra
2021-05-14 15:58:11 +01:00
Brian Gough
44e0742aa3
use fse.copy for performance
...
it uses the native fs.copyFile method
2021-05-14 15:49:20 +01:00
Brian Gough
ff2175e727
add validation for express :content_id parameter
2021-05-13 15:48:17 +01:00
Brian Gough
013d38552a
log the expiry timeout value when disk is low
2021-05-13 15:13:29 +01:00
Jakob Ackermann
b456ea726d
[misc] merge pdf caching into main ( #226 )
...
* wip generate directory for hash content
* cleanup, remove console logging
* add content caching module
* Return PDF stream ranges with compile response
* Return the PDF file size in the compile response
* PDF range endpoint
* [misc] WIP: pdf caching: preserve the m-time on static content files
* [misc] WIP: pdf caching: improve browser caching, emit caching headers
* [misc] WIP: pdf caching: do not emit very small chunks <1kB
* [misc] keep up with moving output files into a separate directory
* [OutputCacheManager] add global feature flag for enabling pdf caching
* [misc] add contentId into the URL for protecting PDF stream contents
* [misc] support PDF stream caching for anonymous users
* [misc] add per-request feature flag for enabling PDF stream caching
* [misc] enable pdf caching in CI and emit metrics at the end of run
* [misc] expose compile stats and timings to the frontend
* [misc] log an error in case saving output files fails
* [misc] add metrics for pdf bandwidth and pdf caching performance
* [misc] add a dark mode to the pdf caching for computing ranges only
* [misc] move pdf caching metrics into ContentCacheMetrics
* [misc] add a config option for the min chunk size of pdf ranges
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
2021-05-13 14:07:54 +01:00
dependabot[bot]
e66ff7a2ca
Bump hosted-git-info from 2.8.5 to 2.8.9
...
Bumps [hosted-git-info](https://github.com/npm/hosted-git-info ) from 2.8.5 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases )
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md )
- [Commits](https://github.com/npm/hosted-git-info/compare/v2.8.5...v2.8.9 )
Signed-off-by: dependabot[bot] <support@github.com>
2021-05-11 14:32:37 +00:00
Jakob Ackermann
cfa8127c1b
[misc] add linting for missing explicit dependencies and fix any errors
2021-04-29 15:30:54 +01:00
dependabot[bot]
ab57729f98
Bump y18n from 4.0.0 to 4.0.1
...
Bumps [y18n](https://github.com/yargs/y18n ) from 4.0.0 to 4.0.1.
- [Release notes](https://github.com/yargs/y18n/releases )
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md )
- [Commits](https://github.com/yargs/y18n/commits )
Signed-off-by: dependabot[bot] <support@github.com>
2021-03-31 17:52:26 +00:00
Jakob Ackermann
b3ed820444
Merge pull request #219 from overleaf/jpa-read-image-name-for-synctex
...
[misc] consume and validate a custom imageName for synctex requests
2021-03-31 10:33:53 +02:00
Jakob Ackermann
c1fe1cf5b7
Merge pull request #218 from overleaf/jpa-metrics-module-3-5-1
...
[misc] bump the version of the metrics module to 3.5.1
2021-03-31 10:33:41 +02:00
Jakob Ackermann
1a41fa55ad
Merge pull request #217 from overleaf/sk-qpdf-newline
...
Add flag to qpdf, to preserve PDF/A compliance
2021-03-31 10:33:16 +02:00
Jakob Ackermann
6cb4246eb1
[misc] consume and validate a custom imageName for synctex requests
2021-03-30 13:57:40 +01:00
Alf Eaton
a271d5ce10
Merge pull request #212 from overleaf/ae-chown-app-folders
...
Ensure that app folders exist before running chown in entrypoint.sh
2021-03-23 09:43:31 +00:00
Alf Eaton
db0cd53ed4
Merge pull request #213 from overleaf/ae-readme
...
Update README
2021-03-23 09:41:44 +00:00
Jakob Ackermann
b563deef14
[misc] bump the version of the metrics module to 3.5.1
2021-03-22 12:59:43 +00:00
Eric Mc Sween
7a6b86185b
Merge pull request #215 from overleaf/ho-clsi-cookie-post
...
support post and get for getting status of clsi project
2021-03-22 07:56:09 -04:00
Christopher Hoskin
6ca016bb05
Explain the situation with permissions on Linux
2021-03-19 14:32:54 +00:00