Eric Mc Sween
e41836028a
Decaf cleanup: error handling
2020-05-20 15:34:28 -04:00
Eric Mc Sween
cfc0d45ccd
Decaf cleanup: unnecessary returns
2020-05-20 15:32:57 -04:00
Eric Mc Sween
924cc0bf73
Decaf cleanup: simplify null checks
2020-05-20 15:31:41 -04:00
Eric Mc Sween
9799b94752
Accept ordered doc and file updates
...
Add an `updates` parameter to the project update endpoint. It can be
used instead of `docUpdates` and `fileUpdates` to provide a single list
of updates in the order they should be processed.
2020-05-20 07:57:32 -04:00
Eric Mc Sween
65fe74fd2d
Upgrade Sinon
2020-05-15 15:34:14 -04:00
Eric Mc Sween
759d385be8
Decaf cleanup: unused variables
2020-05-15 14:43:05 -04:00
Eric Mc Sween
86747ba64f
Decaf cleanup: stub setup in tests
2020-05-15 14:43:03 -04:00
Eric Mc Sween
9e4280916a
Decaf cleanup: unnecessary returns
2020-05-15 14:41:04 -04:00
Eric Mc Sween
7a5a782dc7
Decaf cleanup: camel case variables
2020-05-15 09:59:26 -04:00
Eric Mc Sween
16c0ed23db
Fix tests after decaf cleanup
...
* Camel casing in logs
* The Express request object always has query and body properties
2020-05-11 11:37:59 -04:00
Eric Mc Sween
0b1c7e90af
Decaf cleanup: remove unused variables
2020-05-11 11:26:28 -04:00
Eric Mc Sween
3acb970442
Decaf cleanup: simplify stubbed class
2020-05-11 11:24:07 -04:00
Eric Mc Sween
526ef25fcf
Decaf cleanup: unnecessary returns
2020-05-11 11:20:55 -04:00
Eric Mc Sween
41c0899b0c
Add a test for document size slightly over max doc length
2020-05-11 10:41:32 -04:00
Eric Mc Sween
f99125c65a
Decaf cleanup: camel case variables
2020-05-08 15:01:34 -04:00
Eric Mc Sween
75f9b0ff10
Decaf cleanup: handle errors
2020-05-08 15:01:34 -04:00
Eric Mc Sween
150c4a88f1
Decaf cleanup: simplify null checks
2020-05-08 15:01:34 -04:00
Eric Mc Sween
dc5d77998c
Decaf cleanup: remove unnecessary returns
2020-05-08 15:01:34 -04:00
Eric Mc Sween
ff2d31c066
Decaf cleanup: remove Array.from()
2020-05-08 15:01:34 -04:00
Eric Mc Sween
e8f935d046
Make max JSON request size configurable and default to 8 MB
...
This is to allow multi-document updates, for example when creating a new
project from a zip file.
2020-05-08 15:01:34 -04:00
Tim Alby
cf87daa754
fix Metrics module stub
2020-05-06 12:17:03 +02:00
Tim Alby
8176cb3e8d
remove requires for coffee-script package in tests
2020-05-06 12:16:59 +02:00
Tim Alby
0bae747b3e
prettier: convert test/stress decaffeinated files to Prettier format
2020-05-06 12:13:07 +02:00
Tim Alby
02b29bfaa1
decaffeinate: rename test/stress/coffee to test/stress/js
2020-05-06 12:13:03 +02:00
decaffeinate
86a3b0842a
decaffeinate: Run post-processing cleanups on run.coffee
2020-05-06 12:12:57 +02:00
decaffeinate
3b6c0d8ca6
decaffeinate: Convert run.coffee to JS
2020-05-06 12:12:55 +02:00
decaffeinate
b6cc463a1e
decaffeinate: Rename run.coffee from .coffee to .js
2020-05-06 12:12:53 +02:00
Tim Alby
d15738cb98
prettier: convert test/acceptance decaffeinated files to Prettier format
2020-05-06 12:12:47 +02:00
Tim Alby
9680e62bb1
decaffeinate: rename test/acceptance/coffee to test/acceptance/js
2020-05-06 12:12:41 +02:00
decaffeinate
24ac4d4935
decaffeinate: Run post-processing cleanups on ApplyingUpdatesToADocTests.coffee and 14 other files
2020-05-06 12:12:36 +02:00
decaffeinate
adffde3059
decaffeinate: Convert ApplyingUpdatesToADocTests.coffee and 14 other files to JS
2020-05-06 12:12:17 +02:00
decaffeinate
f46fe5be7f
decaffeinate: Rename ApplyingUpdatesToADocTests.coffee and 14 other files from .coffee to .js
2020-05-06 12:12:02 +02:00
Tim Alby
9a2f8ecbd4
prettier: convert test/unit decaffeinated files to Prettier format
2020-05-06 12:11:36 +02:00
Tim Alby
bb20394243
decaffeinate: rename test/unit/coffee to test/unit/js
2020-05-06 12:11:30 +02:00
decaffeinate
21850f1e6f
decaffeinate: Run post-processing cleanups on DiffCodecTests.coffee and 23 other files
2020-05-06 12:11:22 +02:00
decaffeinate
c781526af0
decaffeinate: Convert DiffCodecTests.coffee and 23 other files to JS
2020-05-06 12:10:51 +02:00
decaffeinate
6c4d7fb838
decaffeinate: Rename DiffCodecTests.coffee and 23 other files from .coffee to .js
2020-05-06 12:10:11 +02:00
Miguel Serrano
f935c392bc
Merge branch 'master' into sk-upgrade-dependencies
2020-04-27 13:38:21 +02:00
Brian Gough
a51f61a555
remove redis migration code
2020-04-21 14:48:47 +01:00
Brian Gough
af93193d6e
remove new_project_history and use project_history instead
2020-04-21 14:43:48 +01:00
Brian Gough
8e210fe441
update unit tests
2020-04-21 14:41:30 +01:00
Brian Gough
ac68f59487
Merge branch 'master' into bg-use-separate-redis-for-project-history
2020-04-16 15:48:21 +01:00
Brian Gough
21824d49da
Merge branch 'bg-add-queue-metrics' of github.com:overleaf/document-updater into bg-add-queue-metrics
2020-04-01 16:04:52 +01:00
Brian Gough
3a8c362fba
add doclines set/del metric
2020-04-01 15:59:25 +01:00
Jakob Ackermann
17c2add0cf
[misc] track redis pub/sub payload sizes on publish
2020-03-30 11:31:43 +02:00
Brian Gough
fcb72b9bf7
update tests
2020-03-25 14:27:41 +00:00
Shane Kilkelly
ada4fba3dc
Fix express deprecations
2020-03-19 15:39:57 +00:00
Shane Kilkelly
f069a2515d
upgrade express
2020-03-19 15:20:32 +00:00
Eric Mc Sween
d9caced0d6
Change skip_flush option to ignore_flush_errors in delete doc
...
Instead of skipping the flush, we'll still try to flush and proceed with
the doc deletion, even when the flush fails.
2020-03-10 09:40:49 -04:00
Eric Mc Sween
9b70eb75b3
Rename flush param to skip_flush in delete doc
...
Also move it to the query string instead of the body.
2020-03-09 16:27:32 -04:00
Eric Mc Sween
c09bc0e868
Add a "flush: false" option to the doc delete endpoint
...
This will delete the document from Redis without flushing to web,
docstore or history. To be used when something is broken.
2020-03-07 08:59:15 -05:00
Brian Gough
0419039d4d
Merge branch 'master' into bg-use-separate-redis-for-project-history
2020-02-21 14:13:33 +00:00
Jakob Ackermann
82ab2f7d14
Merge pull request #65 from das7pad/hotfix-redis-config
...
[redis] use the correct instance for the real time updates
2020-02-12 13:21:04 +01:00
Jakob Ackermann
e73978410c
Merge pull request #67 from das7pad/hotfix-static-queue-check
...
[misc] tests: fix a static test
2020-02-12 13:20:55 +01:00
Brian Gough
26d8d07e7a
remove unused redis settings from rate limit unit test
2020-02-11 14:32:49 +00:00
Jakob Ackermann
4070b0ef6f
[misc] tests: fix a static test
...
`ProjectHistoryRedisManager.queueOps` is masked and can not push
changes into redis during the unittest.
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-02-07 12:36:46 +01:00
Jakob Ackermann
10d177f41b
[misc] test/unit: stub the correct redis settings
2020-02-07 12:24:57 +01:00
Brian Gough
7036803acf
add missing argument to metrics.inc
...
also track retries rather than attempts (which is always 1 for a successful request)
2020-01-14 15:00:21 +00:00
Brian Gough
3caa0e7c05
add failure/retry metrics for web-api requests
2020-01-14 13:53:50 +00:00
Brian Gough
c2714f9ae9
add acceptance tests for RedisMigrationManager
2020-01-06 16:50:06 +00:00
Brian Gough
d0c5eb5698
support migration of project history keys to separate redis instance
2019-12-13 16:38:41 +00:00
Brian Gough
1bb12a5035
allow pending updates to clear in acceptance tests
2019-12-11 14:43:59 +00:00
Brian Gough
ad19fee667
add setting so that double flush is the default
...
can be disabled to stop flushing to track-changes
2019-11-25 13:36:25 +00:00
Brian Gough
4f6583bbf2
fix getDocVersion and add tests
2019-11-25 13:28:36 +00:00
Brian Gough
8b73bb9f13
Merge branch 'master' into bg-filter-track-changes-updates
2019-11-22 10:41:33 +00:00
Brian Gough
b7055eecee
add metrics for history flushes
2019-11-22 09:14:32 +00:00
Brian Gough
dcd7649bad
filter track-changes updates for projects using project-history
2019-11-19 10:02:56 +00:00
Brian Gough
08ee58abb8
remove console.log from tests
2019-11-15 09:34:13 +00:00
Brian Gough
82a72c9b73
fix missing bodyParser limit
2019-11-14 16:32:59 +00:00
Brian Gough
d82b180b76
avoid project history queues building up with deferred flush
2019-10-03 04:05:24 +01:00
Brian Gough
2845b23b70
add smoothing of delete spikes
2019-10-01 15:01:53 +01:00
Brian Gough
0f0682df43
allow flush to complete in acceptance test
2019-09-26 16:00:11 +01:00
Brian Gough
9f358ead9f
add an acceptance test for flush with queue processing
2019-09-26 10:55:05 +01:00
Brian Gough
f6b2ac7360
queue deletes for deferred processing
2019-09-25 16:42:49 +01:00
Brian Gough
912a3a7753
remove redis server-side hashing for performance
...
we still compute the document hash in node, and check it on retrieval
but we don't check the hash at the point of writing it in redis which
was previously done with a redis Lua script.
2019-09-09 15:27:58 +01:00
Brian Gough
e75f2cc325
add unit tests for skipping history flush
2019-08-15 11:01:45 +01:00
Brian Gough
7493462154
add acceptance test for realtime shutdown
2019-08-15 10:38:07 +01:00
Brian Gough
97487a077e
fix cluster/sentinel connection with real-time
2019-07-10 09:42:05 +01:00
Henry Oswald
3b3b2da0f5
add pubsub redis connection and remove real time redis connection
2019-07-04 13:34:31 +01:00
Shane Kilkelly
e8a8c446ed
Increase timeout on test?
2019-06-19 11:34:38 +01:00
Shane Kilkelly
080b482e51
Update logger, metrics, and redis
...
Also fix acceptance tests, broken by a change in the redis driver
behaviour. It now returns promises from most operations, which confuses mocha.
2019-06-19 10:25:54 +01:00
Brian Gough
984b2d38e6
add unit tests
2019-06-11 16:33:14 +01:00
Brian Gough
e95059f98e
handle non-urgent flushes in background
2019-06-03 10:01:10 +01:00
Brian Gough
0bbfa7de27
Merge branch 'master' into bg-downgrade-delete-component-error
2019-05-08 09:07:02 +01:00
Brian Gough
27a8248196
convert "Delete component" errors into warnings
2019-05-07 16:55:17 +01:00
Henry Oswald
e57741cb80
stub out project flusher for unit tests
2019-05-07 15:46:30 +01:00
Tim Alby
c1c23e4bee
record last author id on document flush
...
This is a multi-steps process:
* get a update's `user_id` from the metadata
* store the `user_id` (`lastUpdatedBy`) and current date (`lastUpdatedAt`) for
the document in Redis on every updates
* fetch `lastUpdatedAt` and `lastUpdatedBy` from Redis on document flush
* send the data to web to be persisted in Mongo
2019-05-02 11:10:02 +01:00
James Allen
52f3596e53
Review feedback
2019-04-16 11:05:17 +01:00
James Allen
3d76f4b9bf
Record a snapshot to mongo when a doc's comments/changes get collapsed
2019-04-11 13:27:46 +01:00
Brian Gough
fd1425d83f
include a unique id in every message published to redis
2019-03-21 12:10:15 +00:00
Brian Gough
8c5d74faef
use explicit json content-type to avoid security issues with text/html
2019-02-12 16:45:11 +00:00
Henry Oswald
ecaef6485b
revert the removal of realtime keyspace
2019-02-07 15:27:51 +00:00
Henry Oswald
4e1a2c787c
Revert "turn down logging, use logger.info for less important data"
...
This reverts commit c5f91428e3c7702fbbd3ffd1ef7a772d513f33f2.
2019-02-06 15:29:22 +00:00
Christopher Hoskin
1217d8a80a
Merge branch 'master' into csh-ho-docker-issue-1338-bulk-upgrade
2019-01-04 09:18:40 +00:00
Henry Oswald
2505be283a
turn down logging, use logger.info for less important data
2018-12-13 12:46:10 +00:00
Henry Oswald
25e1ad7491
remove redis.realtime
...
real time pushes data into the document updater redis instance
2018-12-05 16:17:41 +00:00
Brian Gough
aae1352519
ensure that project history is flushed when the project is deleted
2018-10-30 12:03:44 +00:00
Brian Gough
fdcb806518
set a timestamp for the first entry in the projectHistory:Ops queue
2018-07-24 09:17:31 +01:00
Brian Gough
e471730efb
Revert "Merge pull request #33 from sharelatex/bg-add-timestamp-marker-to-project-history-queue"
...
This reverts commit 8ec398b133aa75c45384dd7ceb630cd743f4a15c, reversing
changes made to e3a6c934cf53fd245d7d0df737463cc90f954885.
2018-07-23 16:05:04 +01:00
Brian Gough
6237577f87
set a timestamp for the first entry in the projectHistory:Ops queue
2018-07-20 10:43:31 +01:00