Eric Mc Sween
819aa378d9
Decaf cleanup: remove Array.from()
2020-05-14 16:32:05 -04:00
Eric Mc Sween
16c0ed23db
Fix tests after decaf cleanup
...
* Camel casing in logs
* The Express request object always has query and body properties
2020-05-11 11:37:59 -04:00
Eric Mc Sween
0b1c7e90af
Decaf cleanup: remove unused variables
2020-05-11 11:26:28 -04:00
Eric Mc Sween
3acb970442
Decaf cleanup: simplify stubbed class
2020-05-11 11:24:07 -04:00
Eric Mc Sween
526ef25fcf
Decaf cleanup: unnecessary returns
2020-05-11 11:20:55 -04:00
Eric Mc Sween
a2a1914a53
Use max_doc_length setting to limit incoming doc size
2020-05-11 11:15:37 -04:00
Eric Mc Sween
cb959ddfc1
Decaf cleanup: use arrow functions for callbacks
2020-05-11 11:14:37 -04:00
Eric Mc Sween
e4ac63dd19
Decaf cleanup: move functions to top level
2020-05-11 11:12:15 -04:00
Eric Mc Sween
64a881461f
Decaf cleanup: camel case variables
2020-05-11 11:07:15 -04:00
Eric Mc Sween
fc73bbe1a5
Decaf cleanup: simplify null checks
2020-05-11 10:52:06 -04:00
Eric Mc Sween
80ea49c69c
Decaf cleanup: remove __guard__
2020-05-11 10:47:27 -04:00
Eric Mc Sween
814ac40e07
Decaf cleanup: unnecessary returns
2020-05-11 10:45:39 -04:00
Eric Mc Sween
3385ec5f26
Decaf cleanup: unnecessary Array.from()
2020-05-11 10:43:22 -04:00
Eric Mc Sween
41c0899b0c
Add a test for document size slightly over max doc length
2020-05-11 10:41:32 -04:00
Eric Mc Sween
f99125c65a
Decaf cleanup: camel case variables
2020-05-08 15:01:34 -04:00
Eric Mc Sween
75f9b0ff10
Decaf cleanup: handle errors
2020-05-08 15:01:34 -04:00
Eric Mc Sween
150c4a88f1
Decaf cleanup: simplify null checks
2020-05-08 15:01:34 -04:00
Eric Mc Sween
dc5d77998c
Decaf cleanup: remove unnecessary returns
2020-05-08 15:01:34 -04:00
Eric Mc Sween
ff2d31c066
Decaf cleanup: remove Array.from()
2020-05-08 15:01:34 -04:00
Eric Mc Sween
e8f935d046
Make max JSON request size configurable and default to 8 MB
...
This is to allow multi-document updates, for example when creating a new
project from a zip file.
2020-05-08 15:01:34 -04:00
Eric Mc Sween
e60d9237d0
Decaf cleanup: simplify null checks
2020-05-08 15:01:34 -04:00
Eric Mc Sween
18b92adcef
Decaf cleanup: remove __guard__()
2020-05-08 15:01:34 -04:00
Eric Mc Sween
7b2420413c
Decaf cleanup: unnecessary returns and arrow functions in callbacks
2020-05-08 15:01:33 -04:00
Eric Mc Sween
53d79d86a9
Decaf cleanup: remove Array.from()
2020-05-08 14:54:54 -04:00
Tim Alby
9f6ea07002
fix SyntaxError on export
var
2020-05-06 12:17:08 +02:00
Tim Alby
cf87daa754
fix Metrics module stub
2020-05-06 12:17:03 +02:00
Tim Alby
8176cb3e8d
remove requires for coffee-script package in tests
2020-05-06 12:16:59 +02:00
Tim Alby
ed7d277f98
lint & format app.js
2020-05-06 12:16:53 +02:00
Tim Alby
e089cfc93c
format config/settings.defaults.js & lib/diff_match_patch.js
2020-05-06 12:16:48 +02:00
Tim Alby
4184976099
fix lint errors in config/settings.defaults.js
2020-05-06 12:16:10 +02:00
Tim Alby
96e7a668b7
disable linting for lib/diff_match_patch.js
2020-05-06 12:16:04 +02:00
Tim Alby
6fdc094b5c
Uninstall coffee-script
2020-05-06 12:13:24 +02:00
Tim Alby
29d5e8f8b9
prettier: convert individual decaffeinated files to Prettier format
2020-05-06 12:13:11 +02:00
Tim Alby
877bae34b3
decaffeinate: convert individual files to js
2020-05-06 12:13:10 +02:00
Tim Alby
86c6c38a6e
decaffeinate: rename individual coffee files to js files
2020-05-06 12:13:07 +02:00
Tim Alby
0bae747b3e
prettier: convert test/stress decaffeinated files to Prettier format
2020-05-06 12:13:07 +02:00
Tim Alby
02b29bfaa1
decaffeinate: rename test/stress/coffee to test/stress/js
2020-05-06 12:13:03 +02:00
decaffeinate
86a3b0842a
decaffeinate: Run post-processing cleanups on run.coffee
2020-05-06 12:12:57 +02:00
decaffeinate
3b6c0d8ca6
decaffeinate: Convert run.coffee to JS
2020-05-06 12:12:55 +02:00
decaffeinate
b6cc463a1e
decaffeinate: Rename run.coffee from .coffee to .js
2020-05-06 12:12:53 +02:00
Tim Alby
d15738cb98
prettier: convert test/acceptance decaffeinated files to Prettier format
2020-05-06 12:12:47 +02:00
Tim Alby
9680e62bb1
decaffeinate: rename test/acceptance/coffee to test/acceptance/js
2020-05-06 12:12:41 +02:00
decaffeinate
24ac4d4935
decaffeinate: Run post-processing cleanups on ApplyingUpdatesToADocTests.coffee and 14 other files
2020-05-06 12:12:36 +02:00
decaffeinate
adffde3059
decaffeinate: Convert ApplyingUpdatesToADocTests.coffee and 14 other files to JS
2020-05-06 12:12:17 +02:00
decaffeinate
f46fe5be7f
decaffeinate: Rename ApplyingUpdatesToADocTests.coffee and 14 other files from .coffee to .js
2020-05-06 12:12:02 +02:00
Tim Alby
9a2f8ecbd4
prettier: convert test/unit decaffeinated files to Prettier format
2020-05-06 12:11:36 +02:00
Tim Alby
bb20394243
decaffeinate: rename test/unit/coffee to test/unit/js
2020-05-06 12:11:30 +02:00
decaffeinate
21850f1e6f
decaffeinate: Run post-processing cleanups on DiffCodecTests.coffee and 23 other files
2020-05-06 12:11:22 +02:00
decaffeinate
c781526af0
decaffeinate: Convert DiffCodecTests.coffee and 23 other files to JS
2020-05-06 12:10:51 +02:00
decaffeinate
6c4d7fb838
decaffeinate: Rename DiffCodecTests.coffee and 23 other files from .coffee to .js
2020-05-06 12:10:11 +02:00
Tim Alby
dbf9e88dc3
prettier: convert app/js decaffeinated files to Prettier format
2020-05-06 12:09:33 +02:00
Tim Alby
a519980c10
decaffeinate: rename app/coffee dir to app/js
2020-05-06 12:09:23 +02:00
decaffeinate
dad1d1212f
decaffeinate: Run post-processing cleanups on DeleteQueueManager.coffee and 58 other files
2020-05-06 12:09:15 +02:00
decaffeinate
1fa8882674
decaffeinate: Convert DeleteQueueManager.coffee and 58 other files to JS
2020-05-06 12:08:21 +02:00
decaffeinate
249b7247b5
decaffeinate: Rename DeleteQueueManager.coffee and 58 other files from .coffee to .js
2020-05-06 12:07:29 +02:00
Tim Alby
e2e8593f57
decaffeinate: add eslint and prettier packages
2020-05-06 12:06:28 +02:00
Tim Alby
16c98b3d15
decaffeinate: update .gitignore
2020-05-06 12:04:26 +02:00
Tim Alby
569a1f5824
decaffeinate: update build scripts to es
2020-05-06 12:04:19 +02:00
Tim Alby
bb9a5340f4
update build scripts to 2.2.0
2020-05-06 12:03:41 +02:00
Miguel Serrano
93a5ee063a
regenerated lockfile
2020-04-27 13:41:23 +02:00
Miguel Serrano
f935c392bc
Merge branch 'master' into sk-upgrade-dependencies
2020-04-27 13:38:21 +02:00
Brian Gough
79c934759c
add default redis settings for project history
2020-04-22 14:04:28 +01:00
Brian Gough
2e24d1670c
remove old unused ioredis keepalive option
2020-04-22 13:51:14 +01:00
Brian Gough
61da130cf4
keep maxRetriesPerRequest for project_history redis config
2020-04-22 13:50:39 +01:00
Brian Gough
a51f61a555
remove redis migration code
2020-04-21 14:48:47 +01:00
Brian Gough
248edc03fa
add comment about the two history clients
2020-04-21 14:44:19 +01:00
Brian Gough
af93193d6e
remove new_project_history and use project_history instead
2020-04-21 14:43:48 +01:00
Brian Gough
8e210fe441
update unit tests
2020-04-21 14:41:30 +01:00
Brian Gough
ac68f59487
Merge branch 'master' into bg-use-separate-redis-for-project-history
2020-04-16 15:48:21 +01:00
Brian Gough
55cbfd9962
Merge branch 'master' into bg-add-queue-metrics
2020-04-06 12:12:12 +01:00
Brian Gough
c095feaa06
upgrade logger-sharelatex
2020-04-06 10:43:53 +01:00
Brian Gough
beb3691795
add metrics for redis get/update
2020-04-06 10:00:48 +01:00
Brian Gough
2b72ec49a1
add comments for redis metrics
2020-04-02 11:33:52 +01:00
Brian Gough
21824d49da
Merge branch 'bg-add-queue-metrics' of github.com:overleaf/document-updater into bg-add-queue-metrics
2020-04-01 16:04:52 +01:00
Brian Gough
3a8c362fba
add doclines set/del metric
2020-04-01 15:59:25 +01:00
Brian Gough
00b11bda96
use separate loop for pendingUpdates metric
2020-04-01 14:50:55 +01:00
Henry Oswald
c2b050e286
bump redis to 1.0.12
2020-03-31 10:21:50 +01:00
Jakob Ackermann
17c2add0cf
[misc] track redis pub/sub payload sizes on publish
2020-03-30 11:31:43 +02:00
Shane Kilkelly
861ab5be80
update metrics and logger for latest bug fixes
2020-03-26 13:40:47 +00:00
Brian Gough
fcb72b9bf7
update tests
2020-03-25 14:27:41 +00:00
Brian Gough
1a0550364d
add metric for getdoc bytes
2020-03-25 14:27:41 +00:00
Brian Gough
891fcc696b
add metric for pending updates queue
2020-03-25 14:27:41 +00:00
Brian Gough
e293d86c14
add metric for project history queue
2020-03-25 12:15:16 +00:00
Jakob Ackermann
b2d1718a2e
[misc] bump logger-sharelatex to 1.9.1
2020-03-23 16:18:05 +01:00
Shane Kilkelly
ada4fba3dc
Fix express deprecations
2020-03-19 15:39:57 +00:00
Shane Kilkelly
21194e52ca
more upgrades
2020-03-19 15:23:45 +00:00
Shane Kilkelly
f069a2515d
upgrade express
2020-03-19 15:20:32 +00:00
Shane Kilkelly
aa324a1920
npm audit fix
2020-03-19 15:08:59 +00:00
Shane Kilkelly
b631a741ac
update build scripts
2020-03-19 15:06:36 +00:00
Brian Gough
e2564c9cdd
update metrics module to 2.5.1
...
reduce time window for summary metrics to 60s
2020-03-11 10:25:48 +00:00
Brian Gough
69125df44b
Merge pull request #121 from overleaf/bg-clean-up-package-json
...
clean up package.json
2020-03-11 10:20:30 +00:00
Eric Mc Sween
ff32104fe6
Merge pull request #123 from overleaf/em-doc-hard-delete
...
Add ignore_flush_errors option to the doc delete endpoint
2020-03-10 10:11:00 -04:00
Eric Mc Sween
d9caced0d6
Change skip_flush option to ignore_flush_errors in delete doc
...
Instead of skipping the flush, we'll still try to flush and proceed with
the doc deletion, even when the flush fails.
2020-03-10 09:40:49 -04:00
Eric Mc Sween
9b70eb75b3
Rename flush param to skip_flush in delete doc
...
Also move it to the query string instead of the body.
2020-03-09 16:27:32 -04:00
Eric Mc Sween
c09bc0e868
Add a "flush: false" option to the doc delete endpoint
...
This will delete the document from Redis without flushing to web,
docstore or history. To be used when something is broken.
2020-03-07 08:59:15 -05:00
nate stemen
ffd8d0745d
use empty object for ranges if it doesn't exist
2020-03-06 13:49:30 -05:00
Brian Gough
852ae5fbae
move bunyan out of devDependencies
...
(needed by @google-cloud/logging-bunyan)
2020-02-26 09:47:38 +00:00
Brian Gough
0419039d4d
Merge branch 'master' into bg-use-separate-redis-for-project-history
2020-02-21 14:13:33 +00:00
Brian Gough
2e178b0e2d
resolve merge conflicts
2020-02-21 14:06:58 +00:00
Brian Gough
922f237c39
Revert "add redislabs ca cert to repository"
...
This reverts commit eb7419b0f45992228393086caf7ed6f66296801f.
2020-02-19 09:26:59 +00:00
Brian Gough
412eabc306
Revert "add tls settings"
...
This reverts commit 72a4994cebab2731f99f0ada7a094c8a0acb3293.
2020-02-19 09:26:42 +00:00
Brian Gough
4e7c5c9046
update requestretry
2020-02-18 09:29:53 +00:00
Brian Gough
3aa997a0bc
update package-lock.json
2020-02-17 14:06:13 +00:00
Brian Gough
d9138c91f4
upgrade request to latest version
2020-02-17 14:05:59 +00:00
Brian Gough
583af0bfc4
update package-lock.json
2020-02-17 13:42:57 +00:00
Brian Gough
638688a854
remove unused lynx module
2020-02-17 13:41:53 +00:00
Brian Gough
908e916779
move sinon and sandboxed-module to devDependencies
2020-02-17 13:41:34 +00:00
Brian Gough
dec18e0773
Merge pull request #120 from overleaf/dependabot/npm_and_yarn/lodash-4.17.13
...
Bump lodash from 4.17.4 to 4.17.13
2020-02-17 11:09:49 +00:00
Brian Gough
0969a39291
Merge pull request #119 from overleaf/dependabot/npm_and_yarn/sshpk-1.16.1
...
Bump sshpk from 1.13.1 to 1.16.1
2020-02-17 11:02:45 +00:00
Brian Gough
30567ca2da
Merge pull request #118 from overleaf/dependabot/npm_and_yarn/stringstream-0.0.6
...
Bump stringstream from 0.0.5 to 0.0.6
2020-02-17 10:59:59 +00:00
dependabot[bot]
9a92cd7b31
Bump lodash from 4.17.4 to 4.17.13
...
Bumps [lodash](https://github.com/lodash/lodash ) from 4.17.4 to 4.17.13.
- [Release notes](https://github.com/lodash/lodash/releases )
- [Commits](https://github.com/lodash/lodash/compare/4.17.4...4.17.13 )
Signed-off-by: dependabot[bot] <support@github.com>
2020-02-14 09:56:26 +00:00
dependabot[bot]
3fb0ed414c
Bump sshpk from 1.13.1 to 1.16.1
...
Bumps [sshpk](https://github.com/joyent/node-sshpk ) from 1.13.1 to 1.16.1.
- [Release notes](https://github.com/joyent/node-sshpk/releases )
- [Commits](https://github.com/joyent/node-sshpk/compare/v1.13.1...v1.16.1 )
Signed-off-by: dependabot[bot] <support@github.com>
2020-02-14 09:56:25 +00:00
dependabot[bot]
2acd4c0f23
Bump stringstream from 0.0.5 to 0.0.6
...
Bumps [stringstream](https://github.com/mhart/StringStream ) from 0.0.5 to 0.0.6.
- [Release notes](https://github.com/mhart/StringStream/releases )
- [Commits](https://github.com/mhart/StringStream/compare/v0.0.5...v0.0.6 )
Signed-off-by: dependabot[bot] <support@github.com>
2020-02-14 09:56:24 +00:00
dependabot[bot]
c4bec0dcc7
Bump extend from 3.0.1 to 3.0.2
...
Bumps [extend](https://github.com/justmoon/node-extend ) from 3.0.1 to 3.0.2.
- [Release notes](https://github.com/justmoon/node-extend/releases )
- [Changelog](https://github.com/justmoon/node-extend/blob/master/CHANGELOG.md )
- [Commits](https://github.com/justmoon/node-extend/compare/v3.0.1...v3.0.2 )
Signed-off-by: dependabot[bot] <support@github.com>
2020-02-14 09:56:23 +00:00
Jakob Ackermann
5fb8c0a098
Merge pull request #116 from overleaf/jpa-dependencies-cleanup
...
[misc] rename npm-shrinkwrap.json to package-lock.json and run npm i
2020-02-14 10:55:54 +01:00
Jakob Ackermann
af4211d167
[misc] rename npm-shrinkwrap.json to package-lock.json and run npm i
2020-02-12 14:39:51 +01:00
Brian Gough
81e21c5e76
remove unused .travis.yml file
2020-02-12 12:37:00 +00:00
Jakob Ackermann
82ab2f7d14
Merge pull request #65 from das7pad/hotfix-redis-config
...
[redis] use the correct instance for the real time updates
2020-02-12 13:21:04 +01:00
Jakob Ackermann
e73978410c
Merge pull request #67 from das7pad/hotfix-static-queue-check
...
[misc] tests: fix a static test
2020-02-12 13:20:55 +01:00
Jakob Ackermann
776d85b5b0
Merge pull request #114 from overleaf/jpa-build-scripts-1-3-5
...
[misc] update the build scripts to 1.3.5
2020-02-12 13:20:44 +01:00
Brian Gough
26d8d07e7a
remove unused redis settings from rate limit unit test
2020-02-11 14:32:49 +00:00
Jakob Ackermann
9cfc59734a
[misc] update the build scripts to 1.3.5
2020-02-11 12:27:53 +01:00
Brian Gough
2d2e3bc2b1
Merge pull request #111 from overleaf/bg-add-comment-about-null-bytes
...
add comment about null byte check
2020-02-10 10:26:24 +00:00
Brian Gough
afe43fa252
update to node 10.19.0
2020-02-07 14:16:31 +00:00
Jakob Ackermann
4070b0ef6f
[misc] tests: fix a static test
...
`ProjectHistoryRedisManager.queueOps` is masked and can not push
changes into redis during the unittest.
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-02-07 12:36:46 +01:00
Jakob Ackermann
10d177f41b
[misc] test/unit: stub the correct redis settings
2020-02-07 12:24:57 +01:00
Brian Gough
338d3609f5
add comment about null byte check
2020-01-30 15:17:13 +00:00
Brian Gough
544ae05212
added note about rollback
2020-01-23 16:22:26 +00:00
Brian Gough
626e19ed1a
add logging of migration phase at startup
2020-01-23 15:46:54 +00:00
Brian Gough
d5a2b96df9
add note about deleting the migration key entries
2020-01-23 14:36:59 +00:00
Jakob Ackermann
fcfa3ecc9d
[misc] upgrade node to 10.18.1
2020-01-23 10:05:57 +01:00
Brian Gough
31324fb65a
add environment variable for migration_phase setting
2020-01-22 15:36:06 +00:00
Brian Gough
ad58fe76b2
add tls settings
2020-01-21 15:36:33 +00:00
Brian Gough
531d9b77b9
add redislabs ca cert to repository
2020-01-21 09:49:41 +00:00
Brian Gough
7036803acf
add missing argument to metrics.inc
...
also track retries rather than attempts (which is always 1 for a successful request)
2020-01-14 15:00:21 +00:00
Brian Gough
3caa0e7c05
add failure/retry metrics for web-api requests
2020-01-14 13:53:50 +00:00
Brian Gough
a638ef4251
add comment about locking in redis migration
2020-01-13 15:56:28 +00:00
Brian Gough
c2714f9ae9
add acceptance tests for RedisMigrationManager
2020-01-06 16:50:06 +00:00
Brian Gough
27044c2d02
allow migration phase to be modified at runtime for testing
2020-01-06 16:46:35 +00:00
Brian Gough
8ae95ebf60
fix rclient check in migration metrics
2020-01-06 16:45:36 +00:00
Brian Gough
97cbf46160
add metrics for migration
2019-12-16 11:46:35 +00:00
Brian Gough
a2e63d009e
fix migration phase check
2019-12-16 09:55:26 +00:00
Brian Gough
a85dffbcef
fix acceptance tests
2019-12-16 09:27:00 +00:00
Brian Gough
d0c5eb5698
support migration of project history keys to separate redis instance
2019-12-13 16:38:41 +00:00
Brian Gough
1bb12a5035
allow pending updates to clear in acceptance tests
2019-12-11 14:43:59 +00:00
Brian Gough
ad19fee667
add setting so that double flush is the default
...
can be disabled to stop flushing to track-changes
2019-11-25 13:36:25 +00:00
Brian Gough
4f6583bbf2
fix getDocVersion and add tests
2019-11-25 13:28:36 +00:00
Brian Gough
68e12f4d2d
add metrics for queue operations
2019-11-25 10:51:10 +00:00
Brian Gough
8b73bb9f13
Merge branch 'master' into bg-filter-track-changes-updates
2019-11-22 10:41:33 +00:00
Brian Gough
b7055eecee
add metrics for history flushes
2019-11-22 09:14:32 +00:00
Brian Gough
65cf4cf7c7
make flush to track-changes failsafe
2019-11-21 14:58:35 +00:00
Brian Gough
dcd7649bad
filter track-changes updates for projects using project-history
2019-11-19 10:02:56 +00:00
Brian Gough
08ee58abb8
remove console.log from tests
2019-11-15 09:34:13 +00:00
Brian Gough
82a72c9b73
fix missing bodyParser limit
2019-11-14 16:32:59 +00:00
dependabot[bot]
6871889bfb
Bump request from 2.25.0 to 2.47.0
...
Bumps [request](https://github.com/request/request ) from 2.25.0 to 2.47.0.
- [Release notes](https://github.com/request/request/releases )
- [Changelog](https://github.com/request/request/blob/master/CHANGELOG.md )
- [Commits](https://github.com/request/request/compare/v2.25.0...v2.47.0 )
Signed-off-by: dependabot[bot] <support@github.com>
2019-11-05 11:38:25 +00:00
dependabot[bot]
ccc072e9da
Bump express from 3.3.4 to 3.11.0
...
Bumps [express](https://github.com/expressjs/express ) from 3.3.4 to 3.11.0.
- [Release notes](https://github.com/expressjs/express/releases )
- [Changelog](https://github.com/expressjs/express/blob/master/History.md )
- [Commits](https://github.com/expressjs/express/compare/3.3.4...3.11.0 )
Signed-off-by: dependabot[bot] <support@github.com>
2019-10-30 13:38:50 +00:00
Nate Stemen
1d3a1d22be
bump build script to 1.1.24
2019-10-25 12:50:45 -04:00
Nate Stemen
1620956e2e
replace private link with public one
2019-10-25 12:50:33 -04:00
Brian Gough
a9428c9ece
Merge pull request #96 from overleaf/bg-add-ioredis-maxretriesperrequest
...
add ioredis maxRetriesPerRequest
2019-10-16 08:55:36 +01:00
Brian Gough
21a9126847
set maxRetriesPerRequest from environment variable
2019-10-16 08:44:08 +01:00
Brian Gough
781e01b8b4
Merge pull request #95 from overleaf/bg-update-ioredis
...
update to ioredis 4.14.1
2019-10-14 09:21:39 +01:00
Brian Gough
5a62632cfd
add ioredis option maxRetriesPerRequest: 0
...
see https://github.com/luin/ioredis/issues/965
2019-10-04 11:53:14 +01:00
Brian Gough
51a821c03c
remove old unused ioredis keepalive option
2019-10-04 11:51:37 +01:00
Brian Gough
18ccd11224
update to ioredis 4.14.1
2019-10-04 11:32:08 +01:00
Henry Oswald
e9efd6b93b
logs out when a redis event occurs
2019-10-03 11:10:31 +01:00
Brian Gough
d82b180b76
avoid project history queues building up with deferred flush
2019-10-03 04:05:24 +01:00
Brian Gough
c1454bc4ac
Merge pull request #92 from overleaf/bg-flush-queue-prod-fixes
...
add continuous background flush
2019-10-02 13:11:00 +01:00
Brian Gough
0c14b7d2f8
add comment about background flush limit
2019-10-01 15:06:01 +01:00
Brian Gough
ae3ebf2db6
start background flush after http server has started
2019-10-01 15:02:37 +01:00
Brian Gough
2845b23b70
add smoothing of delete spikes
2019-10-01 15:01:53 +01:00
Brian Gough
2c22a60052
add random jitter to cutoff time
2019-10-01 15:01:20 +01:00
Brian Gough
a32495d2b4
make background flush more adaptive
2019-10-01 14:09:41 +01:00
Brian Gough
73b4262186
add continuous background flush
2019-09-30 16:05:53 +01:00
Brian Gough
33fadf51c1
fix getDocTimestamps for multiple docs
2019-09-30 13:50:25 +01:00
Brian Gough
89a90399fe
Merge pull request #89 from overleaf/bg-queue-deletes
...
queue deletes for deferred processing
2019-09-30 11:14:08 +01:00
John Lees-Miller
c5a9105c33
Update config
2019-09-28 11:07:33 +01:00
Brian Gough
260923f291
keep flushQueuedProjects in the foreground
2019-09-27 10:46:24 +01:00
Brian Gough
7561e05660
check timestamps array length
2019-09-27 10:39:56 +01:00
Brian Gough
0f0682df43
allow flush to complete in acceptance test
2019-09-26 16:00:11 +01:00
Brian Gough
b7f3b848af
remove unused dryRun option
...
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
2019-09-26 15:50:55 +01:00
Brian Gough
3bc176259b
fix log line
2019-09-26 15:46:54 +01:00
Brian Gough
8cdc8c410a
fix error logging
2019-09-26 15:46:45 +01:00
Brian Gough
fc62abfcfa
run flush of queued projects in the background
2019-09-26 15:46:14 +01:00
Brian Gough
ba35c73cb6
add comment about ZPOPMIN
2019-09-26 15:18:10 +01:00
Brian Gough
a709a0adaa
for simplicity keep the cutoff time the same while flushing the queue
2019-09-26 15:05:38 +01:00
Brian Gough
eae4b352ca
remove unnecessary check
2019-09-26 14:59:03 +01:00
Brian Gough
9f358ead9f
add an acceptance test for flush with queue processing
2019-09-26 10:55:05 +01:00
Brian Gough
b49621b3e9
add comments
2019-09-26 10:14:49 +01:00
Brian Gough
83dd43b809
add metric for queue length
2019-09-25 17:04:36 +01:00
Brian Gough
f6b2ac7360
queue deletes for deferred processing
2019-09-25 16:42:49 +01:00
Brian Gough
c32101f0fb
upgrade redis-sharelatex to 1.0.10
2019-09-19 16:22:43 +01:00
Brian Gough
912a3a7753
remove redis server-side hashing for performance
...
we still compute the document hash in node, and check it on retrieval
but we don't check the hash at the point of writing it in redis which
was previously done with a redis Lua script.
2019-09-09 15:27:58 +01:00
Henry Oswald
aa15a76059
added log lines for all project flushing
2019-08-30 07:38:53 +01:00
Henry Oswald
0ae838dd2d
add logger into project flusher
2019-08-29 20:36:00 +01:00
Brian Gough
e75f2cc325
add unit tests for skipping history flush
2019-08-15 11:01:45 +01:00
Brian Gough
7493462154
add acceptance test for realtime shutdown
2019-08-15 10:38:07 +01:00
Brian Gough
a76e0dca88
skip history flush when project is cleared by realtime shutdown
...
history is flushed by a background cron job anyway
2019-08-15 09:51:16 +01:00
Simon Detheridge
f49f5cbf05
Merge pull request #79 from overleaf/ho-mongo-connection-string
...
use MONGO_CONNECTION_STRING in settings.defaults.coffee if set
2019-08-08 14:16:49 +01:00
Simon Detheridge
6721b904a7
Merge pull request #82 from overleaf/bg-mongo-health-check
...
add a combined health check for mongo and redis
2019-08-08 14:16:38 +01:00
Simon Detheridge
06444d2cc4
Improve/fix serializers for update logging ( #80 )
...
* Improve/fix serializers for update logging
2019-08-08 14:10:54 +01:00
Brian Gough
40f6494b19
add a combined health check for mongo and redis
2019-08-07 16:25:23 +01:00
Henry Oswald
aa1c74a979
use MONGO_CONNECTION_STRING in settings.defaults.coffee if set
2019-08-07 15:44:57 +01:00
Simon Detheridge
df9ca8b272
Add serializer to print only length of large fields in production
2019-07-31 16:42:28 +01:00
Brian Gough
618880f99d
remove unnecessary check for doc_id
2019-07-24 16:57:43 +01:00
Brian Gough
c9ccf62d71
support per-doc pubsub channels
2019-07-22 12:20:06 +01:00
Brian Gough
97487a077e
fix cluster/sentinel connection with real-time
2019-07-10 09:42:05 +01:00
Henry Oswald
06ad0f7acd
Merge pull request #75 from overleaf/ho-pubsub-connection
...
Remove real time redis connection and consolidate on pubsub
2019-07-08 13:58:41 +01:00
Henry Oswald
374585f47d
Merge pull request #72 from overleaf/sk-dep-upgrades-2019-06
...
Update logger, metrics, and redis
2019-07-08 13:58:28 +01:00
Henry Oswald
3b3b2da0f5
add pubsub redis connection and remove real time redis connection
2019-07-04 13:34:31 +01:00
Henry Oswald
eed8e215d7
bump redis-sharelatex (and io redis) to 1.0.9
2019-07-02 16:58:30 +01:00
Shane Kilkelly
e8a8c446ed
Increase timeout on test?
2019-06-19 11:34:38 +01:00
Shane Kilkelly
080b482e51
Update logger, metrics, and redis
...
Also fix acceptance tests, broken by a change in the redis driver
behaviour. It now returns promises from most operations, which confuses mocha.
2019-06-19 10:25:54 +01:00
Henry Oswald
fdef197271
Merge branch 'master' into ho-detailed-flush-status
2019-06-13 14:33:22 +01:00
Henry Oswald
d9a737f97c
return failed and succesfully flushed projects when flushing everything
2019-06-13 14:21:38 +01:00
Brian Gough
e8dd1aae9c
Merge pull request #70 from overleaf/bg-metric-for-invalid-hash
...
add metric for invalid hash and other sharejs errors
2019-06-12 13:50:34 +01:00
Brian Gough
d50b93df2f
add metric for invalid hash and other sharejs errors
2019-06-11 16:48:06 +01:00
Brian Gough
984b2d38e6
add unit tests
2019-06-11 16:33:14 +01:00
Brian Gough
966478cac4
fix hash check to use 'v' field instead of version
2019-06-11 14:11:46 +01:00
Brian Gough
e95059f98e
handle non-urgent flushes in background
2019-06-03 10:01:10 +01:00
Brian Gough
0bbfa7de27
Merge branch 'master' into bg-downgrade-delete-component-error
2019-05-08 09:07:02 +01:00
Henry Oswald
d169cca425
Merge branch 'master' into ho-flush-all-projects
2019-05-07 17:02:14 +01:00
Brian Gough
27a8248196
convert "Delete component" errors into warnings
2019-05-07 16:55:17 +01:00
Tim Alby
8b40da701e
update README
...
- remove build status badge
- change app name
- update copyright notice
- update links
2019-05-07 17:03:09 +02:00
Tim Alby
d316f172bf
update repo URL for Jenkins
2019-05-07 17:03:06 +02:00
Henry Oswald
e57741cb80
stub out project flusher for unit tests
2019-05-07 15:46:30 +01:00
Henry Oswald
4b8a27a220
change github url
2019-05-07 14:01:12 +01:00
Tim Alby
ca6bfd8929
Update README
2019-05-06 17:28:52 +02:00
Henry Oswald
daca83a057
add dryRun option to flush all projects
2019-05-02 16:54:22 +01:00
Henry Oswald
d5d1736a5e
adds /flush_all_projects project
2019-05-02 16:43:35 +01:00
Tim Alby
c1c23e4bee
record last author id on document flush
...
This is a multi-steps process:
* get a update's `user_id` from the metadata
* store the `user_id` (`lastUpdatedBy`) and current date (`lastUpdatedAt`) for
the document in Redis on every updates
* fetch `lastUpdatedAt` and `lastUpdatedBy` from Redis on document flush
* send the data to web to be persisted in Mongo
2019-05-02 11:10:02 +01:00
Brian Gough
68e7b9c4e9
Merge pull request #48 from sharelatex/bg-check-incoming-hash
...
check incoming hash when present
2019-04-29 10:15:44 +01:00
James Allen
52f3596e53
Review feedback
2019-04-16 11:05:17 +01:00
James Allen
33478f95fd
Fix package.json versions
2019-04-11 16:32:31 +01:00
James Allen
3d76f4b9bf
Record a snapshot to mongo when a doc's comments/changes get collapsed
2019-04-11 13:27:46 +01:00
Brian Gough
3c635c8d98
check version before it is modified by applyOp
2019-04-09 09:20:48 +01:00
Brian Gough
cc1f3fce5b
check incoming hash when present
2019-04-08 14:12:18 +01:00
Brian Gough
fd1425d83f
include a unique id in every message published to redis
2019-03-21 12:10:15 +00:00
Henry Oswald
4d7f9f3c1a
add sentry into settings.defaults
2019-03-05 17:34:27 +00:00
Brian Gough
11f07d1f09
Merge pull request #45 from sharelatex/bg-avoid-text-html-content-type-in-responses
...
use explicit json content-type to avoid security issues with text/html
2019-02-22 09:24:30 +00:00
Henry Oswald
937e8fe071
use redis cluster key in defaults for pendingupdates
2019-02-15 13:58:29 +00:00
Brian Gough
8c5d74faef
use explicit json content-type to avoid security issues with text/html
2019-02-12 16:45:11 +00:00
Henry Oswald
0a0fc91f28
Update app.coffee
2019-02-12 13:27:13 +00:00
Henry Oswald
3bc4cb492a
added log line
2019-02-07 16:30:53 +00:00
Henry Oswald
08723f8972
revert health check redis types
2019-02-07 15:53:26 +00:00
Henry Oswald
ecaef6485b
revert the removal of realtime keyspace
2019-02-07 15:27:51 +00:00
Henry Oswald
b5564095f3
add /health_check/redis route back in
2019-02-07 15:10:40 +00:00
Henry Oswald
73bd264401
remove realtime keys in settings, no longer used
2019-02-07 14:55:24 +00:00
Henry Oswald
2998750a33
fix redis version lock
2019-02-06 16:01:44 +00:00
Henry Oswald
4e1a2c787c
Revert "turn down logging, use logger.info for less important data"
...
This reverts commit c5f91428e3c7702fbbd3ffd1ef7a772d513f33f2.
2019-02-06 15:29:22 +00:00
Henry Oswald
8d3fb729c5
point to web_api
2019-02-01 20:10:47 +00:00
Henry Oswald
e44a02391a
standardise name of document-updater
2019-02-01 20:04:43 +00:00
Henry Oswald
98de529e42
make config look like chef config
2019-02-01 19:52:00 +00:00
Henry Oswald
3051d7db4b
bulk upgrade to 1.1.12, moved to npm rather than git
2019-01-31 16:00:24 +00:00
Henry Oswald
d279a9149c
add dispatcherCount to config file
2019-01-25 17:24:19 +00:00
Henry Oswald
76b054c7f4
bump node redis to 1.0.5
2019-01-25 17:01:59 +00:00
Christopher Hoskin
3eb2d27d4a
Add **/*.map to .gitignore
2019-01-08 15:01:48 +00:00
Christopher Hoskin
5db1913cec
Bump settings to v1.1.0
2019-01-08 15:00:26 +00:00
Christopher Hoskin
a783c6b3cc
Bump logger to 1.5.9
2019-01-04 09:33:08 +00:00
Christopher Hoskin
05a80c7cba
Add app.map to .gitignore
2019-01-04 09:31:00 +00:00
Christopher Hoskin
0aa7315c27
Bump buildscripts to 1.1.10
2019-01-04 09:24:09 +00:00
Christopher Hoskin
d9ec90f0e2
Move to metrics v2 and shrinkwrap
2019-01-04 09:22:09 +00:00
Christopher Hoskin
1217d8a80a
Merge branch 'master' into csh-ho-docker-issue-1338-bulk-upgrade
2019-01-04 09:18:40 +00:00
Henry Oswald
2505be283a
turn down logging, use logger.info for less important data
2018-12-13 12:46:10 +00:00
Henry Oswald
25e1ad7491
remove redis.realtime
...
real time pushes data into the document updater redis instance
2018-12-05 16:17:41 +00:00
Henry Oswald
9437cf8b02
explicity split redis config values
2018-12-05 15:16:55 +00:00
Henry Oswald
5f046ed329
install metrics route and bump metrics version
2018-12-05 12:35:46 +00:00
Brian Gough
aae1352519
ensure that project history is flushed when the project is deleted
2018-10-30 12:03:44 +00:00
Henry Oswald
a11d6dcc48
Merge branch 'master' into ho-docker
2018-10-15 15:17:40 +01:00
Alasdair Smith
c128e0ab1c
Use setting instead of hard-coding port
2018-10-09 11:46:27 +01:00
Henry Oswald
ff673c71d5
set redis port via env var
2018-09-28 17:04:34 +01:00
Henry Oswald
93cf574251
upgrade build scripts 1.1.9
2018-09-28 15:13:47 +01:00
Henry Oswald
3db7377d74
add process.env["REDIS_PASSWORD"]
2018-09-28 15:11:17 +01:00
Brian Gough
537bd9a0c0
Merge pull request #36 from sharelatex/bg-limit-resync-requests-to-web
...
limit resync requests to web (connects to overleaf/sharelatex#775 )
2018-08-16 12:25:16 +01:00
Brian Gough
910b27357d
add more logging to resync operations
2018-08-16 11:14:11 +01:00
Brian Gough
aa013f0bee
limit parallel resync doc requests to web
2018-08-16 11:13:11 +01:00
Brian Gough
fdcb806518
set a timestamp for the first entry in the projectHistory:Ops queue
2018-07-24 09:17:31 +01:00
Brian Gough
e471730efb
Revert "Merge pull request #33 from sharelatex/bg-add-timestamp-marker-to-project-history-queue"
...
This reverts commit 8ec398b133aa75c45384dd7ceb630cd743f4a15c, reversing
changes made to e3a6c934cf53fd245d7d0df737463cc90f954885.
2018-07-23 16:05:04 +01:00
Brian Gough
33e898f671
update to 1.1.9 build scripts
2018-07-23 15:35:54 +01:00
Brian Gough
6237577f87
set a timestamp for the first entry in the projectHistory:Ops queue
2018-07-20 10:43:31 +01:00
Hayden Faulds
62a10f25f2
move tk call to before sandboxed module call
2018-06-05 13:54:30 +01:00
Henry Oswald
581963a4af
Dockerised env. 1.1.3 build scripts
2018-05-23 13:52:20 +01:00
Brian Gough
7e526395fb
fix check for "delete component" message
2018-04-30 09:56:01 +01:00
Brian Gough
37f431f148
also block "delete component" messages from sentry
2018-04-27 16:03:46 +01:00
Brian Gough
64670f272a
stop OpRangeNotAvailable errors going to sentry
2018-04-27 15:45:28 +01:00
Hayden Faulds
af92ca70a1
coerce projectHistoryId to integer after reading from Redis
2018-04-23 15:19:06 +01:00
Hayden Faulds
1b63141e49
Revert "Revert "Add projectHistoryId to updates""
2018-04-23 12:08:04 +01:00
Hayden Faulds
3cc20d6311
Revert "Add projectHistoryId to updates"
2018-04-23 11:41:29 +01:00
Hayden Faulds
9629d0a11e
Merge pull request #26 from sharelatex/hof-deleted-project-history
...
Add projectHistoryId to updates
2018-04-23 11:01:53 +01:00
Alberto Fernández Capel
7d597fe59e
Do not specify node version in .travis.yml
...
So travis picks instead the version from .nvmrc
2018-04-19 14:03:35 +01:00
Hayden Faulds
25c0be4eaa
add projectHistryId to project structure updates
2018-04-13 15:22:05 +01:00
Hayden Faulds
a220794d32
add projectHistoryId to resync updates
2018-04-13 15:22:05 +01:00
Hayden Faulds
fb1852a593
cache projectHistoryId with doc in Redis
2018-04-13 15:22:05 +01:00
Brian Gough
dd0f8b880a
change update to projectUpdate in project related methods
2018-03-16 10:54:12 +00:00
Brian Gough
779f00f912
add acceptance test for ordering of project structure changes
2018-03-15 16:27:14 +00:00
Brian Gough
3385d2640a
fix structure ordering bug
2018-03-15 16:27:10 +00:00
Brian Gough
75a5428cbf
update acceptance tests
2018-03-15 16:24:14 +00:00
Brian Gough
b3887fd984
update unit tests for incoming project versions
2018-03-15 16:24:11 +00:00
Brian Gough
0642e3c8c9
support project version on incoming requests
2018-03-15 16:15:10 +00:00
Hayden Faulds
4a5731440a
Merge pull request #21 from sharelatex/hof-history-sync
...
resync project history endpoint
2018-03-14 10:40:59 +00:00
Hayden Faulds
1226f96fc3
extract ProjectHistoryRedisManager
2018-03-09 15:36:38 +00:00
Hayden Faulds
fc6ef6ea7a
improve logging
2018-03-09 12:21:48 +00:00
Hayden Faulds
088ab6e7ba
rename queueResyncDocContents -> queueResyncDocContent
2018-03-09 12:21:37 +00:00
Hayden Faulds
64d1fe13bc
additional logging for project history resync
2018-03-07 16:24:22 +00:00
Hayden Faulds
a72d228354
add unit tests
2018-03-07 13:29:53 +00:00
Hayden Faulds
152a25e96d
rename resyncProject -> resyncProjectHistory
2018-03-07 13:29:42 +00:00
Hayden Faulds
039f997e80
update project history resync url
2018-03-07 11:37:26 +00:00
Hayden Faulds
d46217be52
queue doc content sync updates
2018-03-02 15:02:27 +00:00
Hayden Faulds
52d6b710bd
add ability to resync project structure
2018-03-02 15:01:53 +00:00
James Allen
939eaa2d4b
Don't allow a document to be loaded without a pathname
2018-03-02 10:02:49 +00:00
Hayden Faulds
5a11332aa3
remove unecessary method wrapping
2018-02-22 10:17:59 +00:00
Hayden Faulds
0f87ae1f74
simplify comments
2018-02-22 10:16:29 +00:00
Hayden Faulds
ea0dd9700b
reduce the number of times we flush project history
2018-02-22 10:01:05 +00:00
James Allen
3d5740fd7d
Merge pull request #18 from sharelatex/msw-flush-project-ops
...
Flush project structure changes every 500
2018-02-16 10:30:00 +00:00
James Allen
5425fdef8a
update to 1.0.1 build scripts
2018-02-15 16:54:21 +00:00
James Allen
657da70d45
Fix unit test namespacing from mocha upgrade
2018-02-15 16:28:58 +00:00
James Allen
5d1659457e
Update to build 1.0.0 build scripts
2018-02-15 16:28:40 +00:00
James Allen
f6c79f3203
Fix and extend unit tests
2018-02-15 15:55:12 +00:00
Michael Walker
6c57317f8d
Add a test that sending too few updates does not flush history
2018-01-31 11:41:08 +00:00
Michael Walker
241d1b27d5
Remove _shouldFlushHistoryOps wrapper
2018-01-31 11:27:40 +00:00
Michael Walker
772ee4083a
Turn a unit test into an acceptance test
2018-01-31 11:17:56 +00:00
Michael Walker
bdaa2ffa77
Add an acceptance test for flushing project changes
2018-01-26 14:41:47 +00:00
Michael Walker
d2865f0650
Correctly count project_ops_length
2018-01-26 14:41:28 +00:00
Michael Walker
3028fb9c3d
Only flush project updates when crossing the threshold
2018-01-24 11:57:03 +00:00
Michael Walker
24c74db0dc
Flush project changes when queue hits limit
2018-01-23 11:18:20 +00:00
Michael Walker
3d313ebc53
Add method to get number of queued project updates
2018-01-23 11:10:54 +00:00
James Allen
ccf6cb1a18
Provide hosts as environment settings, add npm run start script and update coffeescript
2017-12-29 08:13:16 +00:00
Brian Gough
0b583dd4e6
fix log message
2017-12-20 13:27:10 +00:00
Brian Gough
a8b14552c0
allow flush to succeed when doc is already deleted
2017-12-19 12:27:57 +00:00
Brian Gough
41f15c4fa8
fix unit test
2017-12-14 14:46:27 +00:00
Brian Gough
947e18eccf
Merge branch 'bg-limit-ops-in-lock'
2017-12-14 14:26:44 +00:00
James Allen
6abf9c227d
Calculate and send doc length before each update
2017-12-11 17:26:32 +00:00
Hayden Faulds
bd45fcb5f8
Merge pull request #12 from sharelatex/hof-version-entity-creation
...
version entity creation; Connects to sharelatex/web-sharelatex-internal#104
2017-11-24 10:02:51 +00:00
Brian Gough
fb572670cd
Merge pull request #11 from sharelatex/bg-use-separate-flush-count-for-projects
...
use a separate parameter for doc and project flush
2017-11-21 13:54:03 +00:00
Brian Gough
0bb9f147aa
fix whitespace
2017-11-13 11:56:08 +00:00
Brian Gough
a4bbf5cf52
use a separate parameter for doc and project flush
2017-11-13 11:53:39 +00:00
Hayden Faulds
d0e56e5054
add acceptance tests for entity additions
2017-11-10 15:01:37 +00:00
Hayden Faulds
a3420b1236
version entity additions
2017-11-10 14:54:56 +00:00
Hayden Faulds
944e633bac
add acceptance test for entity renaming
2017-11-08 09:20:38 +00:00
Hayden Faulds
7e86afe55e
version file renames
2017-11-08 09:20:38 +00:00
Hayden Faulds
6d571e6d23
version document renames
2017-11-08 09:20:37 +00:00
Hayden Faulds
b8052e7612
allow settings.redis.project_history to be undefined
2017-10-25 13:56:38 +01:00
Hayden Faulds
5b7be93c79
Merge pull request #2 from sharelatex/hof-doc-pathname
...
Add metadata to updates for history
2017-10-25 13:13:00 +01:00
Brian Gough
e590543e42
Merge pull request #4 from sharelatex/bg-fix-listen-in-acceptance-tests
...
exit if mock servers fail to start
2017-10-25 09:13:48 +01:00
Brian Gough
f3098f7470
upgrade ioredis to 3.2.1 via redis-sharelatex
2017-10-24 12:20:14 +01:00
Brian Gough
367ed1e941
Merge pull request #8 from sharelatex/bg-upgrade-to-node-6x
...
upgrade to node 6.9.5
2017-10-24 11:11:23 +01:00
Brian Gough
56fd6e3d48
upgrade to node 6.9.5
2017-10-24 11:08:20 +01:00
Brian Gough
d9d58393d7
remove unused ioredis package
...
it is loaded from redis-sharelatex, not here.
2017-10-24 10:39:00 +01:00
Hayden Faulds
d1f0c4ceae
add acceptance tests for project history API
2017-10-24 09:06:23 +01:00
Brian Gough
14f3d22071
upgrade to latest async ^2.5.0
2017-10-23 16:02:24 +01:00
Brian Gough
20655b6eeb
remove unused requires of async
2017-10-23 15:59:32 +01:00