github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-21 20:47:08 -05:00
Code Issues Projects Releases Packages Wiki Activity
23349 commits 3 branches 7 tags 206 MiB
Commit graph

1027 commits

Author SHA1 Message Date
Eric Mc Sween
a2a1914a53 Use max_doc_length setting to limit incoming doc size 2020-05-11 11:15:37 -04:00
Eric Mc Sween
cb959ddfc1 Decaf cleanup: use arrow functions for callbacks 2020-05-11 11:14:37 -04:00
Eric Mc Sween
e4ac63dd19 Decaf cleanup: move functions to top level 2020-05-11 11:12:15 -04:00
Eric Mc Sween
64a881461f Decaf cleanup: camel case variables 2020-05-11 11:07:15 -04:00
Eric Mc Sween
fc73bbe1a5 Decaf cleanup: simplify null checks 2020-05-11 10:52:06 -04:00
Eric Mc Sween
80ea49c69c Decaf cleanup: remove __guard__ 2020-05-11 10:47:27 -04:00
Eric Mc Sween
814ac40e07 Decaf cleanup: unnecessary returns 2020-05-11 10:45:39 -04:00
Eric Mc Sween
3385ec5f26 Decaf cleanup: unnecessary Array.from() 2020-05-11 10:43:22 -04:00
Eric Mc Sween
41c0899b0c Add a test for document size slightly over max doc length 2020-05-11 10:41:32 -04:00
Eric Mc Sween
f99125c65a Decaf cleanup: camel case variables 2020-05-08 15:01:34 -04:00
Eric Mc Sween
75f9b0ff10 Decaf cleanup: handle errors 2020-05-08 15:01:34 -04:00
Eric Mc Sween
150c4a88f1 Decaf cleanup: simplify null checks 2020-05-08 15:01:34 -04:00
Eric Mc Sween
dc5d77998c Decaf cleanup: remove unnecessary returns 2020-05-08 15:01:34 -04:00
Eric Mc Sween
ff2d31c066 Decaf cleanup: remove Array.from() 2020-05-08 15:01:34 -04:00
Eric Mc Sween
e8f935d046 Make max JSON request size configurable and default to 8 MB 
This is to allow multi-document updates, for example when creating a new
project from a zip file.
 2020-05-08 15:01:34 -04:00
Eric Mc Sween
e60d9237d0 Decaf cleanup: simplify null checks 2020-05-08 15:01:34 -04:00
Eric Mc Sween
18b92adcef Decaf cleanup: remove __guard__() 2020-05-08 15:01:34 -04:00
Eric Mc Sween
7b2420413c Decaf cleanup: unnecessary returns and arrow functions in callbacks 2020-05-08 15:01:33 -04:00
Eric Mc Sween
53d79d86a9 Decaf cleanup: remove Array.from() 2020-05-08 14:54:54 -04:00
Tim Alby
9f6ea07002 fix SyntaxError on export var 2020-05-06 12:17:08 +02:00
Tim Alby
cf87daa754 fix Metrics module stub 2020-05-06 12:17:03 +02:00
Tim Alby
8176cb3e8d remove requires for coffee-script package in tests 2020-05-06 12:16:59 +02:00
Tim Alby
ed7d277f98 lint & format app.js 2020-05-06 12:16:53 +02:00
Tim Alby
e089cfc93c format config/settings.defaults.js & lib/diff_match_patch.js 2020-05-06 12:16:48 +02:00
Tim Alby
4184976099 fix lint errors in config/settings.defaults.js 2020-05-06 12:16:10 +02:00
Tim Alby
96e7a668b7 disable linting for lib/diff_match_patch.js 2020-05-06 12:16:04 +02:00
Tim Alby
6fdc094b5c Uninstall coffee-script 2020-05-06 12:13:24 +02:00
Tim Alby
29d5e8f8b9 prettier: convert individual decaffeinated files to Prettier format 2020-05-06 12:13:11 +02:00
Tim Alby
877bae34b3 decaffeinate: convert individual files to js 2020-05-06 12:13:10 +02:00
Tim Alby
86c6c38a6e decaffeinate: rename individual coffee files to js files 2020-05-06 12:13:07 +02:00
Tim Alby
0bae747b3e prettier: convert test/stress decaffeinated files to Prettier format 2020-05-06 12:13:07 +02:00
Tim Alby
02b29bfaa1 decaffeinate: rename test/stress/coffee to test/stress/js 2020-05-06 12:13:03 +02:00
decaffeinate
86a3b0842a decaffeinate: Run post-processing cleanups on run.coffee 2020-05-06 12:12:57 +02:00
decaffeinate
3b6c0d8ca6 decaffeinate: Convert run.coffee to JS 2020-05-06 12:12:55 +02:00
decaffeinate
b6cc463a1e decaffeinate: Rename run.coffee from .coffee to .js 2020-05-06 12:12:53 +02:00
Tim Alby
d15738cb98 prettier: convert test/acceptance decaffeinated files to Prettier format 2020-05-06 12:12:47 +02:00
Tim Alby
9680e62bb1 decaffeinate: rename test/acceptance/coffee to test/acceptance/js 2020-05-06 12:12:41 +02:00
decaffeinate
24ac4d4935 decaffeinate: Run post-processing cleanups on ApplyingUpdatesToADocTests.coffee and 14 other files 2020-05-06 12:12:36 +02:00
decaffeinate
adffde3059 decaffeinate: Convert ApplyingUpdatesToADocTests.coffee and 14 other files to JS 2020-05-06 12:12:17 +02:00
decaffeinate
f46fe5be7f decaffeinate: Rename ApplyingUpdatesToADocTests.coffee and 14 other files from .coffee to .js 2020-05-06 12:12:02 +02:00
Tim Alby
9a2f8ecbd4 prettier: convert test/unit decaffeinated files to Prettier format 2020-05-06 12:11:36 +02:00
Tim Alby
bb20394243 decaffeinate: rename test/unit/coffee to test/unit/js 2020-05-06 12:11:30 +02:00
decaffeinate
21850f1e6f decaffeinate: Run post-processing cleanups on DiffCodecTests.coffee and 23 other files 2020-05-06 12:11:22 +02:00
decaffeinate
c781526af0 decaffeinate: Convert DiffCodecTests.coffee and 23 other files to JS 2020-05-06 12:10:51 +02:00
decaffeinate
6c4d7fb838 decaffeinate: Rename DiffCodecTests.coffee and 23 other files from .coffee to .js 2020-05-06 12:10:11 +02:00
Tim Alby
dbf9e88dc3 prettier: convert app/js decaffeinated files to Prettier format 2020-05-06 12:09:33 +02:00
Tim Alby
a519980c10 decaffeinate: rename app/coffee dir to app/js 2020-05-06 12:09:23 +02:00
decaffeinate
dad1d1212f decaffeinate: Run post-processing cleanups on DeleteQueueManager.coffee and 58 other files 2020-05-06 12:09:15 +02:00
decaffeinate
1fa8882674 decaffeinate: Convert DeleteQueueManager.coffee and 58 other files to JS 2020-05-06 12:08:21 +02:00
decaffeinate
249b7247b5 decaffeinate: Rename DeleteQueueManager.coffee and 58 other files from .coffee to .js 2020-05-06 12:07:29 +02:00
Tim Alby
e2e8593f57 decaffeinate: add eslint and prettier packages 2020-05-06 12:06:28 +02:00
Tim Alby
16c98b3d15 decaffeinate: update .gitignore 2020-05-06 12:04:26 +02:00
Tim Alby
569a1f5824 decaffeinate: update build scripts to es 2020-05-06 12:04:19 +02:00
Tim Alby
bb9a5340f4 update build scripts to 2.2.0 2020-05-06 12:03:41 +02:00
Miguel Serrano
93a5ee063a regenerated lockfile 2020-04-27 13:41:23 +02:00
Miguel Serrano
f935c392bc Merge branch 'master' into sk-upgrade-dependencies 2020-04-27 13:38:21 +02:00
Brian Gough
79c934759c add default redis settings for project history 2020-04-22 14:04:28 +01:00
Brian Gough
2e24d1670c remove old unused ioredis keepalive option 2020-04-22 13:51:14 +01:00
Brian Gough
61da130cf4 keep maxRetriesPerRequest for project_history redis config 2020-04-22 13:50:39 +01:00
Brian Gough
a51f61a555 remove redis migration code 2020-04-21 14:48:47 +01:00
Brian Gough
248edc03fa add comment about the two history clients 2020-04-21 14:44:19 +01:00
Brian Gough
af93193d6e remove new_project_history and use project_history instead 2020-04-21 14:43:48 +01:00
Brian Gough
8e210fe441 update unit tests 2020-04-21 14:41:30 +01:00
Brian Gough
ac68f59487 Merge branch 'master' into bg-use-separate-redis-for-project-history 2020-04-16 15:48:21 +01:00
Brian Gough
55cbfd9962 Merge branch 'master' into bg-add-queue-metrics 2020-04-06 12:12:12 +01:00
Brian Gough
c095feaa06 upgrade logger-sharelatex 2020-04-06 10:43:53 +01:00
Brian Gough
beb3691795 add metrics for redis get/update 2020-04-06 10:00:48 +01:00
Brian Gough
2b72ec49a1 add comments for redis metrics 2020-04-02 11:33:52 +01:00
Brian Gough
21824d49da Merge branch 'bg-add-queue-metrics' of github.com:overleaf/document-updater into bg-add-queue-metrics 2020-04-01 16:04:52 +01:00
Brian Gough
3a8c362fba add doclines set/del metric 2020-04-01 15:59:25 +01:00
Brian Gough
00b11bda96 use separate loop for pendingUpdates metric 2020-04-01 14:50:55 +01:00
Henry Oswald
c2b050e286 bump redis to 1.0.12 2020-03-31 10:21:50 +01:00
Jakob Ackermann
17c2add0cf [misc] track redis pub/sub payload sizes on publish 2020-03-30 11:31:43 +02:00
Shane Kilkelly
861ab5be80 update metrics and logger for latest bug fixes 2020-03-26 13:40:47 +00:00
Brian Gough
fcb72b9bf7 update tests 2020-03-25 14:27:41 +00:00
Brian Gough
1a0550364d add metric for getdoc bytes 2020-03-25 14:27:41 +00:00
Brian Gough
891fcc696b add metric for pending updates queue 2020-03-25 14:27:41 +00:00
Brian Gough
e293d86c14 add metric for project history queue 2020-03-25 12:15:16 +00:00
Jakob Ackermann
b2d1718a2e [misc] bump logger-sharelatex to 1.9.1 2020-03-23 16:18:05 +01:00
Shane Kilkelly
ada4fba3dc Fix express deprecations 2020-03-19 15:39:57 +00:00
Shane Kilkelly
21194e52ca more upgrades 2020-03-19 15:23:45 +00:00
Shane Kilkelly
f069a2515d upgrade express 2020-03-19 15:20:32 +00:00
Shane Kilkelly
aa324a1920 npm audit fix 2020-03-19 15:08:59 +00:00
Shane Kilkelly
b631a741ac update build scripts 2020-03-19 15:06:36 +00:00
Brian Gough
e2564c9cdd update metrics module to 2.5.1 
reduce time window for summary metrics to 60s
 2020-03-11 10:25:48 +00:00
Brian Gough
69125df44b Merge pull request #121 from overleaf/bg-clean-up-package-json 
clean up package.json
 2020-03-11 10:20:30 +00:00
Eric Mc Sween
ff32104fe6 Merge pull request #123 from overleaf/em-doc-hard-delete 
Add ignore_flush_errors option to the doc delete endpoint
 2020-03-10 10:11:00 -04:00
Eric Mc Sween
d9caced0d6 Change skip_flush option to ignore_flush_errors in delete doc 
Instead of skipping the flush, we'll still try to flush and proceed with
the doc deletion, even when the flush fails.
 2020-03-10 09:40:49 -04:00
Eric Mc Sween
9b70eb75b3 Rename flush param to skip_flush in delete doc 
Also move it to the query string instead of the body.
 2020-03-09 16:27:32 -04:00
Eric Mc Sween
c09bc0e868 Add a "flush: false" option to the doc delete endpoint 
This will delete the document from Redis without flushing to web,
docstore or history. To be used when something is broken.
 2020-03-07 08:59:15 -05:00
nate stemen
ffd8d0745d use empty object for ranges if it doesn't exist 2020-03-06 13:49:30 -05:00
Brian Gough
852ae5fbae move bunyan out of devDependencies 
(needed by @google-cloud/logging-bunyan)
 2020-02-26 09:47:38 +00:00
Brian Gough
0419039d4d Merge branch 'master' into bg-use-separate-redis-for-project-history 2020-02-21 14:13:33 +00:00
Brian Gough
2e178b0e2d resolve merge conflicts 2020-02-21 14:06:58 +00:00
Brian Gough
922f237c39 Revert "add redislabs ca cert to repository" 
This reverts commit eb7419b0f45992228393086caf7ed6f66296801f.
 2020-02-19 09:26:59 +00:00
Brian Gough
412eabc306 Revert "add tls settings" 
This reverts commit 72a4994cebab2731f99f0ada7a094c8a0acb3293.
 2020-02-19 09:26:42 +00:00
Brian Gough
4e7c5c9046 update requestretry 2020-02-18 09:29:53 +00:00
Brian Gough
3aa997a0bc update package-lock.json 2020-02-17 14:06:13 +00:00
Brian Gough
d9138c91f4 upgrade request to latest version 2020-02-17 14:05:59 +00:00
Brian Gough
583af0bfc4 update package-lock.json 2020-02-17 13:42:57 +00:00
Brian Gough
638688a854 remove unused lynx module 2020-02-17 13:41:53 +00:00
Brian Gough
908e916779 move sinon and sandboxed-module to devDependencies 2020-02-17 13:41:34 +00:00
Brian Gough
dec18e0773 Merge pull request #120 from overleaf/dependabot/npm_and_yarn/lodash-4.17.13 
Bump lodash from 4.17.4 to 4.17.13
 2020-02-17 11:09:49 +00:00
Brian Gough
0969a39291 Merge pull request #119 from overleaf/dependabot/npm_and_yarn/sshpk-1.16.1 
Bump sshpk from 1.13.1 to 1.16.1
 2020-02-17 11:02:45 +00:00
Brian Gough
30567ca2da Merge pull request #118 from overleaf/dependabot/npm_and_yarn/stringstream-0.0.6 
Bump stringstream from 0.0.5 to 0.0.6
 2020-02-17 10:59:59 +00:00
dependabot[bot]
9a92cd7b31 Bump lodash from 4.17.4 to 4.17.13 
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.4 to 4.17.13.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.4...4.17.13)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-02-14 09:56:26 +00:00
dependabot[bot]
3fb0ed414c Bump sshpk from 1.13.1 to 1.16.1 
Bumps [sshpk](https://github.com/joyent/node-sshpk) from 1.13.1 to 1.16.1.
- [Release notes](https://github.com/joyent/node-sshpk/releases)
- [Commits](https://github.com/joyent/node-sshpk/compare/v1.13.1...v1.16.1)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-02-14 09:56:25 +00:00
dependabot[bot]
2acd4c0f23 Bump stringstream from 0.0.5 to 0.0.6 
Bumps [stringstream](https://github.com/mhart/StringStream) from 0.0.5 to 0.0.6.
- [Release notes](https://github.com/mhart/StringStream/releases)
- [Commits](https://github.com/mhart/StringStream/compare/v0.0.5...v0.0.6)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-02-14 09:56:24 +00:00
dependabot[bot]
c4bec0dcc7 Bump extend from 3.0.1 to 3.0.2 
Bumps [extend](https://github.com/justmoon/node-extend) from 3.0.1 to 3.0.2.
- [Release notes](https://github.com/justmoon/node-extend/releases)
- [Changelog](https://github.com/justmoon/node-extend/blob/master/CHANGELOG.md)
- [Commits](https://github.com/justmoon/node-extend/compare/v3.0.1...v3.0.2)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-02-14 09:56:23 +00:00
Jakob Ackermann
5fb8c0a098 Merge pull request #116 from overleaf/jpa-dependencies-cleanup 
[misc] rename npm-shrinkwrap.json to package-lock.json and run npm i
 2020-02-14 10:55:54 +01:00
Jakob Ackermann
af4211d167 [misc] rename npm-shrinkwrap.json to package-lock.json and run npm i 2020-02-12 14:39:51 +01:00
Brian Gough
81e21c5e76 remove unused .travis.yml file 2020-02-12 12:37:00 +00:00
Jakob Ackermann
82ab2f7d14 Merge pull request #65 from das7pad/hotfix-redis-config 
[redis] use the correct instance for the real time updates
 2020-02-12 13:21:04 +01:00
Jakob Ackermann
e73978410c Merge pull request #67 from das7pad/hotfix-static-queue-check 
[misc] tests: fix a static test
 2020-02-12 13:20:55 +01:00
Jakob Ackermann
776d85b5b0 Merge pull request #114 from overleaf/jpa-build-scripts-1-3-5 
[misc] update the build scripts to 1.3.5
 2020-02-12 13:20:44 +01:00
Brian Gough
26d8d07e7a remove unused redis settings from rate limit unit test 2020-02-11 14:32:49 +00:00
Jakob Ackermann
9cfc59734a [misc] update the build scripts to 1.3.5 2020-02-11 12:27:53 +01:00
Brian Gough
2d2e3bc2b1 Merge pull request #111 from overleaf/bg-add-comment-about-null-bytes 
add comment about null byte check
 2020-02-10 10:26:24 +00:00
Brian Gough
afe43fa252 update to node 10.19.0 2020-02-07 14:16:31 +00:00
Jakob Ackermann
4070b0ef6f [misc] tests: fix a static test 
`ProjectHistoryRedisManager.queueOps` is masked and can not push
 changes into redis during the unittest.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-02-07 12:36:46 +01:00
Jakob Ackermann
10d177f41b [misc] test/unit: stub the correct redis settings 2020-02-07 12:24:57 +01:00
Brian Gough
338d3609f5 add comment about null byte check 2020-01-30 15:17:13 +00:00
Brian Gough
544ae05212 added note about rollback 2020-01-23 16:22:26 +00:00
Brian Gough
626e19ed1a add logging of migration phase at startup 2020-01-23 15:46:54 +00:00
Brian Gough
d5a2b96df9 add note about deleting the migration key entries 2020-01-23 14:36:59 +00:00
Jakob Ackermann
fcfa3ecc9d [misc] upgrade node to 10.18.1 2020-01-23 10:05:57 +01:00
Brian Gough
31324fb65a add environment variable for migration_phase setting 2020-01-22 15:36:06 +00:00
Brian Gough
ad58fe76b2 add tls settings 2020-01-21 15:36:33 +00:00
Brian Gough
531d9b77b9 add redislabs ca cert to repository 2020-01-21 09:49:41 +00:00
Brian Gough
7036803acf add missing argument to metrics.inc 
also track retries rather than attempts (which is always 1 for a successful request)
 2020-01-14 15:00:21 +00:00
Brian Gough
3caa0e7c05 add failure/retry metrics for web-api requests 2020-01-14 13:53:50 +00:00
Brian Gough
a638ef4251 add comment about locking in redis migration 2020-01-13 15:56:28 +00:00
Brian Gough
c2714f9ae9 add acceptance tests for RedisMigrationManager 2020-01-06 16:50:06 +00:00
Brian Gough
27044c2d02 allow migration phase to be modified at runtime for testing 2020-01-06 16:46:35 +00:00
Brian Gough
8ae95ebf60 fix rclient check in migration metrics 2020-01-06 16:45:36 +00:00
Brian Gough
97cbf46160 add metrics for migration 2019-12-16 11:46:35 +00:00
Brian Gough
a2e63d009e fix migration phase check 2019-12-16 09:55:26 +00:00
Brian Gough
a85dffbcef fix acceptance tests 2019-12-16 09:27:00 +00:00
Brian Gough
d0c5eb5698 support migration of project history keys to separate redis instance 2019-12-13 16:38:41 +00:00
Brian Gough
1bb12a5035 allow pending updates to clear in acceptance tests 2019-12-11 14:43:59 +00:00
Brian Gough
ad19fee667 add setting so that double flush is the default 
can be disabled to stop flushing to track-changes
 2019-11-25 13:36:25 +00:00
Brian Gough
4f6583bbf2 fix getDocVersion and add tests 2019-11-25 13:28:36 +00:00
Brian Gough
68e12f4d2d add metrics for queue operations 2019-11-25 10:51:10 +00:00
Brian Gough
8b73bb9f13 Merge branch 'master' into bg-filter-track-changes-updates 2019-11-22 10:41:33 +00:00
Brian Gough
b7055eecee add metrics for history flushes 2019-11-22 09:14:32 +00:00
Brian Gough
65cf4cf7c7 make flush to track-changes failsafe 2019-11-21 14:58:35 +00:00
Brian Gough
dcd7649bad filter track-changes updates for projects using project-history 2019-11-19 10:02:56 +00:00
Brian Gough
08ee58abb8 remove console.log from tests 2019-11-15 09:34:13 +00:00
Brian Gough
82a72c9b73 fix missing bodyParser limit 2019-11-14 16:32:59 +00:00
dependabot[bot]
6871889bfb Bump request from 2.25.0 to 2.47.0 
Bumps [request](https://github.com/request/request) from 2.25.0 to 2.47.0.
- [Release notes](https://github.com/request/request/releases)
- [Changelog](https://github.com/request/request/blob/master/CHANGELOG.md)
- [Commits](https://github.com/request/request/compare/v2.25.0...v2.47.0)

Signed-off-by: dependabot[bot] <support@github.com>
 2019-11-05 11:38:25 +00:00
dependabot[bot]
ccc072e9da Bump express from 3.3.4 to 3.11.0 
Bumps [express](https://github.com/expressjs/express) from 3.3.4 to 3.11.0.
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/master/History.md)
- [Commits](https://github.com/expressjs/express/compare/3.3.4...3.11.0)

Signed-off-by: dependabot[bot] <support@github.com>
 2019-10-30 13:38:50 +00:00
Nate Stemen
1d3a1d22be bump build script to 1.1.24 2019-10-25 12:50:45 -04:00
Nate Stemen
1620956e2e replace private link with public one 2019-10-25 12:50:33 -04:00
Brian Gough
a9428c9ece Merge pull request #96 from overleaf/bg-add-ioredis-maxretriesperrequest 
add ioredis maxRetriesPerRequest
 2019-10-16 08:55:36 +01:00
Brian Gough
21a9126847 set maxRetriesPerRequest from environment variable 2019-10-16 08:44:08 +01:00
Brian Gough
781e01b8b4 Merge pull request #95 from overleaf/bg-update-ioredis 
update to ioredis 4.14.1
 2019-10-14 09:21:39 +01:00
Brian Gough
5a62632cfd add ioredis option maxRetriesPerRequest: 0 
see https://github.com/luin/ioredis/issues/965
 2019-10-04 11:53:14 +01:00
Brian Gough
51a821c03c remove old unused ioredis keepalive option 2019-10-04 11:51:37 +01:00
Brian Gough
18ccd11224 update to ioredis 4.14.1 2019-10-04 11:32:08 +01:00
Henry Oswald
e9efd6b93b logs out when a redis event occurs 2019-10-03 11:10:31 +01:00
Brian Gough
d82b180b76 avoid project history queues building up with deferred flush 2019-10-03 04:05:24 +01:00
Brian Gough
c1454bc4ac Merge pull request #92 from overleaf/bg-flush-queue-prod-fixes 
add continuous background flush
 2019-10-02 13:11:00 +01:00
Brian Gough
0c14b7d2f8 add comment about background flush limit 2019-10-01 15:06:01 +01:00
Brian Gough
ae3ebf2db6 start background flush after http server has started 2019-10-01 15:02:37 +01:00
Brian Gough
2845b23b70 add smoothing of delete spikes 2019-10-01 15:01:53 +01:00
Brian Gough
2c22a60052 add random jitter to cutoff time 2019-10-01 15:01:20 +01:00
Brian Gough
a32495d2b4 make background flush more adaptive 2019-10-01 14:09:41 +01:00
Brian Gough
73b4262186 add continuous background flush 2019-09-30 16:05:53 +01:00
Brian Gough
33fadf51c1 fix getDocTimestamps for multiple docs 2019-09-30 13:50:25 +01:00
Brian Gough
89a90399fe Merge pull request #89 from overleaf/bg-queue-deletes 
queue deletes for deferred processing
 2019-09-30 11:14:08 +01:00
John Lees-Miller
c5a9105c33 Update config 2019-09-28 11:07:33 +01:00
Brian Gough
260923f291 keep flushQueuedProjects in the foreground 2019-09-27 10:46:24 +01:00
Brian Gough
7561e05660 check timestamps array length 2019-09-27 10:39:56 +01:00
Brian Gough
0f0682df43 allow flush to complete in acceptance test 2019-09-26 16:00:11 +01:00
Brian Gough
b7f3b848af remove unused dryRun option 
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
 2019-09-26 15:50:55 +01:00
Brian Gough
3bc176259b fix log line 2019-09-26 15:46:54 +01:00
Brian Gough
8cdc8c410a fix error logging 2019-09-26 15:46:45 +01:00
Brian Gough
fc62abfcfa run flush of queued projects in the background 2019-09-26 15:46:14 +01:00
Brian Gough
ba35c73cb6 add comment about ZPOPMIN 2019-09-26 15:18:10 +01:00
Brian Gough
a709a0adaa for simplicity keep the cutoff time the same while flushing the queue 2019-09-26 15:05:38 +01:00
Brian Gough
eae4b352ca remove unnecessary check 2019-09-26 14:59:03 +01:00
Brian Gough
9f358ead9f add an acceptance test for flush with queue processing 2019-09-26 10:55:05 +01:00
Brian Gough
b49621b3e9 add comments 2019-09-26 10:14:49 +01:00
Brian Gough
83dd43b809 add metric for queue length 2019-09-25 17:04:36 +01:00
Brian Gough
f6b2ac7360 queue deletes for deferred processing 2019-09-25 16:42:49 +01:00
Brian Gough
c32101f0fb upgrade redis-sharelatex to 1.0.10 2019-09-19 16:22:43 +01:00
Brian Gough
912a3a7753 remove redis server-side hashing for performance 
we still  compute the document hash in node, and check it on retrieval
but we don't check the hash at the point of writing it in redis which
was previously done with a redis Lua script.
 2019-09-09 15:27:58 +01:00
Henry Oswald
aa15a76059 added log lines for all project flushing 2019-08-30 07:38:53 +01:00
Henry Oswald
0ae838dd2d add logger into project flusher 2019-08-29 20:36:00 +01:00
Brian Gough
e75f2cc325 add unit tests for skipping history flush 2019-08-15 11:01:45 +01:00
Brian Gough
7493462154 add acceptance test for realtime shutdown 2019-08-15 10:38:07 +01:00
Brian Gough
a76e0dca88 skip history flush when project is cleared by realtime shutdown 
history is flushed by a background cron job anyway
 2019-08-15 09:51:16 +01:00
Simon Detheridge
f49f5cbf05 Merge pull request #79 from overleaf/ho-mongo-connection-string 
use MONGO_CONNECTION_STRING in settings.defaults.coffee if set
 2019-08-08 14:16:49 +01:00
Simon Detheridge
6721b904a7 Merge pull request #82 from overleaf/bg-mongo-health-check 
add a combined health check for mongo and redis
 2019-08-08 14:16:38 +01:00
Simon Detheridge
06444d2cc4 Improve/fix serializers for update logging (#80) 
* Improve/fix serializers for update logging
 2019-08-08 14:10:54 +01:00
Brian Gough
40f6494b19 add a combined health check for mongo and redis 2019-08-07 16:25:23 +01:00
Henry Oswald
aa1c74a979 use MONGO_CONNECTION_STRING in settings.defaults.coffee if set 2019-08-07 15:44:57 +01:00
Simon Detheridge
df9ca8b272 Add serializer to print only length of large fields in production 2019-07-31 16:42:28 +01:00
Brian Gough
618880f99d remove unnecessary check for doc_id 2019-07-24 16:57:43 +01:00
Brian Gough
c9ccf62d71 support per-doc pubsub channels 2019-07-22 12:20:06 +01:00
Brian Gough
97487a077e fix cluster/sentinel connection with real-time 2019-07-10 09:42:05 +01:00
Henry Oswald
06ad0f7acd Merge pull request #75 from overleaf/ho-pubsub-connection 
Remove real time redis connection and consolidate on pubsub
 2019-07-08 13:58:41 +01:00
Henry Oswald
374585f47d Merge pull request #72 from overleaf/sk-dep-upgrades-2019-06 
Update logger, metrics, and redis
 2019-07-08 13:58:28 +01:00
Henry Oswald
3b3b2da0f5 add pubsub redis connection and remove real time redis connection 2019-07-04 13:34:31 +01:00
Henry Oswald
eed8e215d7 bump redis-sharelatex (and io redis) to 1.0.9 2019-07-02 16:58:30 +01:00
Shane Kilkelly
e8a8c446ed Increase timeout on test? 2019-06-19 11:34:38 +01:00
Shane Kilkelly
080b482e51 Update logger, metrics, and redis 
Also fix acceptance tests, broken by a change in the redis driver
behaviour. It now returns promises from most operations, which confuses mocha.
 2019-06-19 10:25:54 +01:00
Henry Oswald
fdef197271 Merge branch 'master' into ho-detailed-flush-status 2019-06-13 14:33:22 +01:00
Henry Oswald
d9a737f97c return failed and succesfully flushed projects when flushing everything 2019-06-13 14:21:38 +01:00
Brian Gough
e8dd1aae9c Merge pull request #70 from overleaf/bg-metric-for-invalid-hash 
add metric for invalid hash and other sharejs errors
 2019-06-12 13:50:34 +01:00
Brian Gough
d50b93df2f add metric for invalid hash and other sharejs errors 2019-06-11 16:48:06 +01:00
Brian Gough
984b2d38e6 add unit tests 2019-06-11 16:33:14 +01:00
Brian Gough
966478cac4 fix hash check to use 'v' field instead of version 2019-06-11 14:11:46 +01:00
Brian Gough
e95059f98e handle non-urgent flushes in background 2019-06-03 10:01:10 +01:00
Brian Gough
0bbfa7de27 Merge branch 'master' into bg-downgrade-delete-component-error 2019-05-08 09:07:02 +01:00
Henry Oswald
d169cca425 Merge branch 'master' into ho-flush-all-projects 2019-05-07 17:02:14 +01:00
Brian Gough
27a8248196 convert "Delete component" errors into warnings 2019-05-07 16:55:17 +01:00
Tim Alby
8b40da701e update README 
- remove build status badge
- change app name
- update copyright notice
- update links
 2019-05-07 17:03:09 +02:00
Tim Alby
d316f172bf update repo URL for Jenkins 2019-05-07 17:03:06 +02:00
Henry Oswald
e57741cb80 stub out project flusher for unit tests 2019-05-07 15:46:30 +01:00
Henry Oswald
4b8a27a220 change github url 2019-05-07 14:01:12 +01:00
Tim Alby
ca6bfd8929 Update README 2019-05-06 17:28:52 +02:00
Henry Oswald
daca83a057 add dryRun option to flush all projects 2019-05-02 16:54:22 +01:00
Henry Oswald
d5d1736a5e adds /flush_all_projects project 2019-05-02 16:43:35 +01:00
Tim Alby
c1c23e4bee record last author id on document flush 
This is a multi-steps process:
* get a update's `user_id` from the metadata
* store the `user_id` (`lastUpdatedBy`) and current date (`lastUpdatedAt`) for
  the document in Redis on every updates
* fetch `lastUpdatedAt` and `lastUpdatedBy` from Redis on document flush
* send the data to web to be persisted in Mongo
 2019-05-02 11:10:02 +01:00
Brian Gough
68e7b9c4e9 Merge pull request #48 from sharelatex/bg-check-incoming-hash 
check incoming hash when present
 2019-04-29 10:15:44 +01:00
James Allen
52f3596e53 Review feedback 2019-04-16 11:05:17 +01:00
James Allen
33478f95fd Fix package.json versions 2019-04-11 16:32:31 +01:00
James Allen
3d76f4b9bf Record a snapshot to mongo when a doc's comments/changes get collapsed 2019-04-11 13:27:46 +01:00
Brian Gough
3c635c8d98 check version before it is modified by applyOp 2019-04-09 09:20:48 +01:00
Brian Gough
cc1f3fce5b check incoming hash when present 2019-04-08 14:12:18 +01:00
Brian Gough
fd1425d83f include a unique id in every message published to redis 2019-03-21 12:10:15 +00:00
Henry Oswald
4d7f9f3c1a add sentry into settings.defaults 2019-03-05 17:34:27 +00:00
Brian Gough
11f07d1f09 Merge pull request #45 from sharelatex/bg-avoid-text-html-content-type-in-responses 
use explicit json content-type to avoid security issues with text/html
 2019-02-22 09:24:30 +00:00
Henry Oswald
937e8fe071 use redis cluster key in defaults for pendingupdates 2019-02-15 13:58:29 +00:00
Brian Gough
8c5d74faef use explicit json content-type to avoid security issues with text/html 2019-02-12 16:45:11 +00:00
Henry Oswald
0a0fc91f28 Update app.coffee 2019-02-12 13:27:13 +00:00
Henry Oswald
3bc4cb492a added log line 2019-02-07 16:30:53 +00:00
Henry Oswald
08723f8972 revert health check redis types 2019-02-07 15:53:26 +00:00
Henry Oswald
ecaef6485b revert the removal of realtime keyspace 2019-02-07 15:27:51 +00:00
Henry Oswald
b5564095f3 add /health_check/redis route back in 2019-02-07 15:10:40 +00:00
Henry Oswald
73bd264401 remove realtime keys in settings, no longer used 2019-02-07 14:55:24 +00:00
Henry Oswald
2998750a33 fix redis version lock 2019-02-06 16:01:44 +00:00
Henry Oswald
4e1a2c787c Revert "turn down logging, use logger.info for less important data" 
This reverts commit c5f91428e3c7702fbbd3ffd1ef7a772d513f33f2.
 2019-02-06 15:29:22 +00:00
Henry Oswald
8d3fb729c5 point to web_api 2019-02-01 20:10:47 +00:00
Henry Oswald
e44a02391a standardise name of document-updater 2019-02-01 20:04:43 +00:00
Henry Oswald
98de529e42 make config look like chef config 2019-02-01 19:52:00 +00:00
Henry Oswald
3051d7db4b bulk upgrade to 1.1.12, moved to npm rather than git 2019-01-31 16:00:24 +00:00
Henry Oswald
d279a9149c add dispatcherCount to config file 2019-01-25 17:24:19 +00:00
Henry Oswald
76b054c7f4 bump node redis to 1.0.5 2019-01-25 17:01:59 +00:00
Christopher Hoskin
3eb2d27d4a Add **/*.map to .gitignore 2019-01-08 15:01:48 +00:00
Christopher Hoskin
5db1913cec Bump settings to v1.1.0 2019-01-08 15:00:26 +00:00
Christopher Hoskin
a783c6b3cc Bump logger to 1.5.9 2019-01-04 09:33:08 +00:00
Christopher Hoskin
05a80c7cba Add app.map to .gitignore 2019-01-04 09:31:00 +00:00
Christopher Hoskin
0aa7315c27 Bump buildscripts to 1.1.10 2019-01-04 09:24:09 +00:00
Christopher Hoskin
d9ec90f0e2 Move to metrics v2 and shrinkwrap 2019-01-04 09:22:09 +00:00
Christopher Hoskin
1217d8a80a Merge branch 'master' into csh-ho-docker-issue-1338-bulk-upgrade 2019-01-04 09:18:40 +00:00
Henry Oswald
2505be283a turn down logging, use logger.info for less important data 2018-12-13 12:46:10 +00:00
Henry Oswald
25e1ad7491 remove redis.realtime 
real time pushes data into the document updater redis instance
 2018-12-05 16:17:41 +00:00
Henry Oswald
9437cf8b02 explicity split redis config values 2018-12-05 15:16:55 +00:00
Henry Oswald
5f046ed329 install metrics route and bump metrics version 2018-12-05 12:35:46 +00:00
Brian Gough
aae1352519 ensure that project history is flushed when the project is deleted 2018-10-30 12:03:44 +00:00
Henry Oswald
a11d6dcc48 Merge branch 'master' into ho-docker 2018-10-15 15:17:40 +01:00
Alasdair Smith
c128e0ab1c Use setting instead of hard-coding port 2018-10-09 11:46:27 +01:00
Henry Oswald
ff673c71d5 set redis port via env var 2018-09-28 17:04:34 +01:00
Henry Oswald
93cf574251 upgrade build scripts 1.1.9 2018-09-28 15:13:47 +01:00
Henry Oswald
3db7377d74 add process.env["REDIS_PASSWORD"] 2018-09-28 15:11:17 +01:00
Brian Gough
537bd9a0c0 Merge pull request #36 from sharelatex/bg-limit-resync-requests-to-web 
limit resync requests to web (connects to overleaf/sharelatex#775)
 2018-08-16 12:25:16 +01:00
Brian Gough
910b27357d add more logging to resync operations 2018-08-16 11:14:11 +01:00
Brian Gough
aa013f0bee limit parallel resync doc requests to web 2018-08-16 11:13:11 +01:00
Brian Gough
fdcb806518 set a timestamp for the first entry in the projectHistory:Ops queue 2018-07-24 09:17:31 +01:00
Brian Gough
e471730efb Revert "Merge pull request #33 from sharelatex/bg-add-timestamp-marker-to-project-history-queue" 
This reverts commit 8ec398b133aa75c45384dd7ceb630cd743f4a15c, reversing
changes made to e3a6c934cf53fd245d7d0df737463cc90f954885.
 2018-07-23 16:05:04 +01:00
Brian Gough
33e898f671 update to 1.1.9 build scripts 2018-07-23 15:35:54 +01:00
Brian Gough
6237577f87 set a timestamp for the first entry in the projectHistory:Ops queue 2018-07-20 10:43:31 +01:00
Hayden Faulds
62a10f25f2 move tk call to before sandboxed module call 2018-06-05 13:54:30 +01:00
Henry Oswald
581963a4af Dockerised env. 1.1.3 build scripts 2018-05-23 13:52:20 +01:00
Brian Gough
7e526395fb fix check for "delete component" message 2018-04-30 09:56:01 +01:00
Brian Gough
37f431f148 also block "delete component" messages from sentry 2018-04-27 16:03:46 +01:00
Brian Gough
64670f272a stop OpRangeNotAvailable errors going to sentry 2018-04-27 15:45:28 +01:00
Hayden Faulds
af92ca70a1 coerce projectHistoryId to integer after reading from Redis 2018-04-23 15:19:06 +01:00
Hayden Faulds
1b63141e49 Revert "Revert "Add projectHistoryId to updates"" 2018-04-23 12:08:04 +01:00
Hayden Faulds
3cc20d6311 Revert "Add projectHistoryId to updates" 2018-04-23 11:41:29 +01:00
Hayden Faulds
9629d0a11e Merge pull request #26 from sharelatex/hof-deleted-project-history 
Add projectHistoryId to updates
 2018-04-23 11:01:53 +01:00
Alberto Fernández Capel
7d597fe59e Do not specify node version in .travis.yml 
So travis picks instead the version from .nvmrc
 2018-04-19 14:03:35 +01:00
Hayden Faulds
25c0be4eaa add projectHistryId to project structure updates 2018-04-13 15:22:05 +01:00
Hayden Faulds
a220794d32 add projectHistoryId to resync updates 2018-04-13 15:22:05 +01:00
Hayden Faulds
fb1852a593 cache projectHistoryId with doc in Redis 2018-04-13 15:22:05 +01:00
Brian Gough
dd0f8b880a change update to projectUpdate in project related methods 2018-03-16 10:54:12 +00:00
Brian Gough
779f00f912 add acceptance test for ordering of project structure changes 2018-03-15 16:27:14 +00:00
Brian Gough
3385d2640a fix structure ordering bug 2018-03-15 16:27:10 +00:00
Brian Gough
75a5428cbf update acceptance tests 2018-03-15 16:24:14 +00:00
Brian Gough
b3887fd984 update unit tests for incoming project versions 2018-03-15 16:24:11 +00:00
Brian Gough
0642e3c8c9 support project version on incoming requests 2018-03-15 16:15:10 +00:00
Hayden Faulds
4a5731440a Merge pull request #21 from sharelatex/hof-history-sync 
resync project history endpoint
 2018-03-14 10:40:59 +00:00
Hayden Faulds
1226f96fc3 extract ProjectHistoryRedisManager 2018-03-09 15:36:38 +00:00
Hayden Faulds
fc6ef6ea7a improve logging 2018-03-09 12:21:48 +00:00
Hayden Faulds
088ab6e7ba rename queueResyncDocContents -> queueResyncDocContent 2018-03-09 12:21:37 +00:00
Hayden Faulds
64d1fe13bc additional logging for project history resync 2018-03-07 16:24:22 +00:00
Hayden Faulds
a72d228354 add unit tests 2018-03-07 13:29:53 +00:00
Hayden Faulds
152a25e96d rename resyncProject -> resyncProjectHistory 2018-03-07 13:29:42 +00:00
Hayden Faulds
039f997e80 update project history resync url 2018-03-07 11:37:26 +00:00
Hayden Faulds
d46217be52 queue doc content sync updates 2018-03-02 15:02:27 +00:00
Hayden Faulds
52d6b710bd add ability to resync project structure 2018-03-02 15:01:53 +00:00
James Allen
939eaa2d4b Don't allow a document to be loaded without a pathname 2018-03-02 10:02:49 +00:00
Hayden Faulds
5a11332aa3 remove unecessary method wrapping 2018-02-22 10:17:59 +00:00
Hayden Faulds
0f87ae1f74 simplify comments 2018-02-22 10:16:29 +00:00
Hayden Faulds
ea0dd9700b reduce the number of times we flush project history 2018-02-22 10:01:05 +00:00
James Allen
3d5740fd7d Merge pull request #18 from sharelatex/msw-flush-project-ops 
Flush project structure changes every 500
 2018-02-16 10:30:00 +00:00
James Allen
5425fdef8a update to 1.0.1 build scripts 2018-02-15 16:54:21 +00:00
James Allen
657da70d45 Fix unit test namespacing from mocha upgrade 2018-02-15 16:28:58 +00:00
James Allen
5d1659457e Update to build 1.0.0 build scripts 2018-02-15 16:28:40 +00:00
James Allen
f6c79f3203 Fix and extend unit tests 2018-02-15 15:55:12 +00:00
Michael Walker
6c57317f8d Add a test that sending too few updates does not flush history 2018-01-31 11:41:08 +00:00
Michael Walker
241d1b27d5 Remove _shouldFlushHistoryOps wrapper 2018-01-31 11:27:40 +00:00
Michael Walker
772ee4083a Turn a unit test into an acceptance test 2018-01-31 11:17:56 +00:00
Michael Walker
bdaa2ffa77 Add an acceptance test for flushing project changes 2018-01-26 14:41:47 +00:00
Michael Walker
d2865f0650 Correctly count project_ops_length 2018-01-26 14:41:28 +00:00
Michael Walker
3028fb9c3d Only flush project updates when crossing the threshold 2018-01-24 11:57:03 +00:00
Michael Walker
24c74db0dc Flush project changes when queue hits limit 2018-01-23 11:18:20 +00:00
Michael Walker
3d313ebc53 Add method to get number of queued project updates 2018-01-23 11:10:54 +00:00
James Allen
ccf6cb1a18 Provide hosts as environment settings, add npm run start script and update coffeescript 2017-12-29 08:13:16 +00:00
Brian Gough
0b583dd4e6 fix log message 2017-12-20 13:27:10 +00:00
Brian Gough
a8b14552c0 allow flush to succeed when doc is already deleted 2017-12-19 12:27:57 +00:00
Brian Gough
41f15c4fa8 fix unit test 2017-12-14 14:46:27 +00:00
Brian Gough
947e18eccf Merge branch 'bg-limit-ops-in-lock' 2017-12-14 14:26:44 +00:00
James Allen
6abf9c227d Calculate and send doc length before each update 2017-12-11 17:26:32 +00:00
Hayden Faulds
bd45fcb5f8 Merge pull request #12 from sharelatex/hof-version-entity-creation 
version entity creation; Connects to sharelatex/web-sharelatex-internal#104
 2017-11-24 10:02:51 +00:00
Brian Gough
fb572670cd Merge pull request #11 from sharelatex/bg-use-separate-flush-count-for-projects 
use a separate parameter for doc and project flush
 2017-11-21 13:54:03 +00:00
Brian Gough
0bb9f147aa fix whitespace 2017-11-13 11:56:08 +00:00
Brian Gough
a4bbf5cf52 use a separate parameter for doc and project flush 2017-11-13 11:53:39 +00:00
Hayden Faulds
d0e56e5054 add acceptance tests for entity additions 2017-11-10 15:01:37 +00:00
Hayden Faulds
a3420b1236 version entity additions 2017-11-10 14:54:56 +00:00
Hayden Faulds
944e633bac add acceptance test for entity renaming 2017-11-08 09:20:38 +00:00
Hayden Faulds
7e86afe55e version file renames 2017-11-08 09:20:38 +00:00
Hayden Faulds
6d571e6d23 version document renames 2017-11-08 09:20:37 +00:00
Hayden Faulds
b8052e7612 allow settings.redis.project_history to be undefined 2017-10-25 13:56:38 +01:00
Hayden Faulds
5b7be93c79 Merge pull request #2 from sharelatex/hof-doc-pathname 
Add metadata to updates for history
 2017-10-25 13:13:00 +01:00
Brian Gough
e590543e42 Merge pull request #4 from sharelatex/bg-fix-listen-in-acceptance-tests 
exit if mock servers fail to start
 2017-10-25 09:13:48 +01:00
Brian Gough
f3098f7470 upgrade ioredis to 3.2.1 via redis-sharelatex 2017-10-24 12:20:14 +01:00
Brian Gough
367ed1e941 Merge pull request #8 from sharelatex/bg-upgrade-to-node-6x 
upgrade to node 6.9.5
 2017-10-24 11:11:23 +01:00
Brian Gough
56fd6e3d48 upgrade to node 6.9.5 2017-10-24 11:08:20 +01:00
Brian Gough
d9d58393d7 remove unused ioredis package 
it is loaded from redis-sharelatex, not here.
 2017-10-24 10:39:00 +01:00
Hayden Faulds
d1f0c4ceae add acceptance tests for project history API 2017-10-24 09:06:23 +01:00
Brian Gough
14f3d22071 upgrade to latest async ^2.5.0 2017-10-23 16:02:24 +01:00
Brian Gough
20655b6eeb remove unused requires of async 2017-10-23 15:59:32 +01:00
Hayden Faulds
04ecd1e7ee set pathname in acceptance tests 2017-10-23 15:32:25 +01:00
Hayden Faulds
be41a1614d decorate ops with pathname and doc_length 2017-10-23 15:32:25 +01:00
Hayden Faulds
962520fca8 flush track-changes and project-history in HistoryManager 2017-10-23 15:32:25 +01:00
Hayden Faulds
d003aef31c conditionally enqueue history updates for project 2017-10-23 15:32:25 +01:00
Hayden Faulds
c27df0bfef split apart multi and rclient in tests 2017-10-23 14:24:23 +01:00