github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-21 20:47:08 -05:00
Code Issues Projects Releases Packages Wiki Activity
23871 commits 3 branches 7 tags 206 MiB
Commit graph

459 commits

Author SHA1 Message Date
Jakob Ackermann
178440395f [perf] switch write sequence for doc contents and doc tracking 
Doc contents are added only after the tracking has been setup.
All read paths on the tracking have been checked to gracefully handle
 the case of existing doc_id but missing doc contents.

- getDoc: -1 operation

REF: 0a2b47c660c60b95e360d8f3b3e30b862ceb6d79
 2021-04-13 11:46:44 +01:00
Jakob Ackermann
50b24043b7 [perf] use MGET for fetching multiple keys in one operation 
- getDoc: from 13 down to 2 operations
 2021-04-09 08:42:35 +01:00
Eric Mc Sween
4d70bd664f Reintroduce Node 12 and metrics upgrades 
These changes were previously merged, not deployed, and reverted. This
reverts the revert.

This reverts commit a6b8c6c658b33b6eee78b8b99e43308f32211ae2, reversing
changes made to 93c98921372eed4244d22fce800716cb27eca299.
 2021-04-01 15:51:00 -04:00
Eric Mc Sween
4dd1b26b2e Revert "Merge pull request #161 from overleaf/em-upgrade-node-12" 
This reverts commit d44102751b9436ad89c5b3b05e7abdff51fcc78a, reversing
changes made to 6c2f5b8d053b75c677da2b7ddd04f998d2be6fff.
 2021-03-31 12:07:11 -04:00
Eric Mc Sween
dc5bfea181 Global test setup 
Configure chai and SandboxedModule globally with options used in many
tests.

That required upgrading mocha and SandboxedModule.
 2021-03-16 16:48:45 -04:00
Thomas
23738540ed Fix API request errors which could contain API hostname or address (#160) 
Wrap errors produced by failing requests to web API, and remove the url/hostname from thrown error messages. (But keep the URL path for info.)
 2021-02-24 15:09:19 +01:00
Henry Oswald
0cdeffae6c fix off by 1 error in Doc updater client helper file 2021-02-09 10:50:37 +00:00
Henry Oswald
11c8cfc939 shard the pending-updates-list queue 2021-02-02 16:38:25 +00:00
Jakob Ackermann
08ed5f6c9b [misc] bump @overleaf/redis-wrapper to version 2.0.0 2020-11-11 16:20:32 +00:00
Jakob Ackermann
1d57706d44 [misc] migrate acceptance tests to the native mongo driver, drop mongojs 2020-08-28 17:06:25 +01:00
Eric Mc Sween
1d1f204021 Remove backwards-compat project update API 
The project update endpoint accepted updates both in two array params:
docUpdates and fileUpdates, and in a single array: updates. This commit
removes the docUpdates/fileUpdates params now that web uses the updates
param.
 2020-05-20 16:26:22 -04:00
Eric Mc Sween
c018fee72c Decaf cleanup: camel case variables 2020-05-20 16:12:27 -04:00
Eric Mc Sween
abb7e8fa20 Decaf cleanup: error handling 2020-05-20 16:08:03 -04:00
Eric Mc Sween
05a2cf829c Decaf cleanup: simplify loops 2020-05-20 16:06:42 -04:00
Eric Mc Sween
e9df9714e5 Decaf cleanup: unnecessary returns 2020-05-20 15:54:36 -04:00
Eric Mc Sween
8bbfd25d47 Decaf cleanup: simplify null checks 2020-05-20 15:53:06 -04:00
Eric Mc Sween
3830b8029a Decaf cleanup: camel case variables 2020-05-20 15:35:12 -04:00
Eric Mc Sween
e41836028a Decaf cleanup: error handling 2020-05-20 15:34:28 -04:00
Eric Mc Sween
cfc0d45ccd Decaf cleanup: unnecessary returns 2020-05-20 15:32:57 -04:00
Eric Mc Sween
924cc0bf73 Decaf cleanup: simplify null checks 2020-05-20 15:31:41 -04:00
Eric Mc Sween
9799b94752 Accept ordered doc and file updates 
Add an `updates` parameter to the project update endpoint. It can be
used instead of `docUpdates` and `fileUpdates` to provide a single list
of updates in the order they should be processed.
 2020-05-20 07:57:32 -04:00
Eric Mc Sween
65fe74fd2d Upgrade Sinon 2020-05-15 15:34:14 -04:00
Eric Mc Sween
759d385be8 Decaf cleanup: unused variables 2020-05-15 14:43:05 -04:00
Eric Mc Sween
86747ba64f Decaf cleanup: stub setup in tests 2020-05-15 14:43:03 -04:00
Eric Mc Sween
9e4280916a Decaf cleanup: unnecessary returns 2020-05-15 14:41:04 -04:00
Eric Mc Sween
7a5a782dc7 Decaf cleanup: camel case variables 2020-05-15 09:59:26 -04:00
Eric Mc Sween
16c0ed23db Fix tests after decaf cleanup 
* Camel casing in logs
* The Express request object always has query and body properties
 2020-05-11 11:37:59 -04:00
Eric Mc Sween
0b1c7e90af Decaf cleanup: remove unused variables 2020-05-11 11:26:28 -04:00
Eric Mc Sween
3acb970442 Decaf cleanup: simplify stubbed class 2020-05-11 11:24:07 -04:00
Eric Mc Sween
526ef25fcf Decaf cleanup: unnecessary returns 2020-05-11 11:20:55 -04:00
Eric Mc Sween
41c0899b0c Add a test for document size slightly over max doc length 2020-05-11 10:41:32 -04:00
Eric Mc Sween
f99125c65a Decaf cleanup: camel case variables 2020-05-08 15:01:34 -04:00
Eric Mc Sween
75f9b0ff10 Decaf cleanup: handle errors 2020-05-08 15:01:34 -04:00
Eric Mc Sween
150c4a88f1 Decaf cleanup: simplify null checks 2020-05-08 15:01:34 -04:00
Eric Mc Sween
dc5d77998c Decaf cleanup: remove unnecessary returns 2020-05-08 15:01:34 -04:00
Eric Mc Sween
ff2d31c066 Decaf cleanup: remove Array.from() 2020-05-08 15:01:34 -04:00
Eric Mc Sween
e8f935d046 Make max JSON request size configurable and default to 8 MB 
This is to allow multi-document updates, for example when creating a new
project from a zip file.
 2020-05-08 15:01:34 -04:00
Tim Alby
cf87daa754 fix Metrics module stub 2020-05-06 12:17:03 +02:00
Tim Alby
8176cb3e8d remove requires for coffee-script package in tests 2020-05-06 12:16:59 +02:00
Tim Alby
0bae747b3e prettier: convert test/stress decaffeinated files to Prettier format 2020-05-06 12:13:07 +02:00
Tim Alby
02b29bfaa1 decaffeinate: rename test/stress/coffee to test/stress/js 2020-05-06 12:13:03 +02:00
decaffeinate
86a3b0842a decaffeinate: Run post-processing cleanups on run.coffee 2020-05-06 12:12:57 +02:00
decaffeinate
3b6c0d8ca6 decaffeinate: Convert run.coffee to JS 2020-05-06 12:12:55 +02:00
decaffeinate
b6cc463a1e decaffeinate: Rename run.coffee from .coffee to .js 2020-05-06 12:12:53 +02:00
Tim Alby
d15738cb98 prettier: convert test/acceptance decaffeinated files to Prettier format 2020-05-06 12:12:47 +02:00
Tim Alby
9680e62bb1 decaffeinate: rename test/acceptance/coffee to test/acceptance/js 2020-05-06 12:12:41 +02:00
decaffeinate
24ac4d4935 decaffeinate: Run post-processing cleanups on ApplyingUpdatesToADocTests.coffee and 14 other files 2020-05-06 12:12:36 +02:00
decaffeinate
adffde3059 decaffeinate: Convert ApplyingUpdatesToADocTests.coffee and 14 other files to JS 2020-05-06 12:12:17 +02:00
decaffeinate
f46fe5be7f decaffeinate: Rename ApplyingUpdatesToADocTests.coffee and 14 other files from .coffee to .js 2020-05-06 12:12:02 +02:00
Tim Alby
9a2f8ecbd4 prettier: convert test/unit decaffeinated files to Prettier format 2020-05-06 12:11:36 +02:00
Tim Alby
bb20394243 decaffeinate: rename test/unit/coffee to test/unit/js 2020-05-06 12:11:30 +02:00
decaffeinate
21850f1e6f decaffeinate: Run post-processing cleanups on DiffCodecTests.coffee and 23 other files 2020-05-06 12:11:22 +02:00
decaffeinate
c781526af0 decaffeinate: Convert DiffCodecTests.coffee and 23 other files to JS 2020-05-06 12:10:51 +02:00
decaffeinate
6c4d7fb838 decaffeinate: Rename DiffCodecTests.coffee and 23 other files from .coffee to .js 2020-05-06 12:10:11 +02:00
Miguel Serrano
f935c392bc Merge branch 'master' into sk-upgrade-dependencies 2020-04-27 13:38:21 +02:00
Brian Gough
a51f61a555 remove redis migration code 2020-04-21 14:48:47 +01:00
Brian Gough
af93193d6e remove new_project_history and use project_history instead 2020-04-21 14:43:48 +01:00
Brian Gough
8e210fe441 update unit tests 2020-04-21 14:41:30 +01:00
Brian Gough
ac68f59487 Merge branch 'master' into bg-use-separate-redis-for-project-history 2020-04-16 15:48:21 +01:00
Brian Gough
21824d49da Merge branch 'bg-add-queue-metrics' of github.com:overleaf/document-updater into bg-add-queue-metrics 2020-04-01 16:04:52 +01:00
Brian Gough
3a8c362fba add doclines set/del metric 2020-04-01 15:59:25 +01:00
Jakob Ackermann
17c2add0cf [misc] track redis pub/sub payload sizes on publish 2020-03-30 11:31:43 +02:00
Brian Gough
fcb72b9bf7 update tests 2020-03-25 14:27:41 +00:00
Shane Kilkelly
ada4fba3dc Fix express deprecations 2020-03-19 15:39:57 +00:00
Shane Kilkelly
f069a2515d upgrade express 2020-03-19 15:20:32 +00:00
Eric Mc Sween
d9caced0d6 Change skip_flush option to ignore_flush_errors in delete doc 
Instead of skipping the flush, we'll still try to flush and proceed with
the doc deletion, even when the flush fails.
 2020-03-10 09:40:49 -04:00
Eric Mc Sween
9b70eb75b3 Rename flush param to skip_flush in delete doc 
Also move it to the query string instead of the body.
 2020-03-09 16:27:32 -04:00
Eric Mc Sween
c09bc0e868 Add a "flush: false" option to the doc delete endpoint 
This will delete the document from Redis without flushing to web,
docstore or history. To be used when something is broken.
 2020-03-07 08:59:15 -05:00
Brian Gough
0419039d4d Merge branch 'master' into bg-use-separate-redis-for-project-history 2020-02-21 14:13:33 +00:00
Jakob Ackermann
82ab2f7d14 Merge pull request #65 from das7pad/hotfix-redis-config 
[redis] use the correct instance for the real time updates
 2020-02-12 13:21:04 +01:00
Jakob Ackermann
e73978410c Merge pull request #67 from das7pad/hotfix-static-queue-check 
[misc] tests: fix a static test
 2020-02-12 13:20:55 +01:00
Brian Gough
26d8d07e7a remove unused redis settings from rate limit unit test 2020-02-11 14:32:49 +00:00
Jakob Ackermann
4070b0ef6f [misc] tests: fix a static test 
`ProjectHistoryRedisManager.queueOps` is masked and can not push
 changes into redis during the unittest.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-02-07 12:36:46 +01:00
Jakob Ackermann
10d177f41b [misc] test/unit: stub the correct redis settings 2020-02-07 12:24:57 +01:00
Brian Gough
7036803acf add missing argument to metrics.inc 
also track retries rather than attempts (which is always 1 for a successful request)
 2020-01-14 15:00:21 +00:00
Brian Gough
3caa0e7c05 add failure/retry metrics for web-api requests 2020-01-14 13:53:50 +00:00
Brian Gough
c2714f9ae9 add acceptance tests for RedisMigrationManager 2020-01-06 16:50:06 +00:00
Brian Gough
d0c5eb5698 support migration of project history keys to separate redis instance 2019-12-13 16:38:41 +00:00
Brian Gough
1bb12a5035 allow pending updates to clear in acceptance tests 2019-12-11 14:43:59 +00:00
Brian Gough
ad19fee667 add setting so that double flush is the default 
can be disabled to stop flushing to track-changes
 2019-11-25 13:36:25 +00:00
Brian Gough
4f6583bbf2 fix getDocVersion and add tests 2019-11-25 13:28:36 +00:00
Brian Gough
8b73bb9f13 Merge branch 'master' into bg-filter-track-changes-updates 2019-11-22 10:41:33 +00:00
Brian Gough
b7055eecee add metrics for history flushes 2019-11-22 09:14:32 +00:00
Brian Gough
dcd7649bad filter track-changes updates for projects using project-history 2019-11-19 10:02:56 +00:00
Brian Gough
08ee58abb8 remove console.log from tests 2019-11-15 09:34:13 +00:00
Brian Gough
82a72c9b73 fix missing bodyParser limit 2019-11-14 16:32:59 +00:00
Brian Gough
d82b180b76 avoid project history queues building up with deferred flush 2019-10-03 04:05:24 +01:00
Brian Gough
2845b23b70 add smoothing of delete spikes 2019-10-01 15:01:53 +01:00
Brian Gough
0f0682df43 allow flush to complete in acceptance test 2019-09-26 16:00:11 +01:00
Brian Gough
9f358ead9f add an acceptance test for flush with queue processing 2019-09-26 10:55:05 +01:00
Brian Gough
f6b2ac7360 queue deletes for deferred processing 2019-09-25 16:42:49 +01:00
Brian Gough
912a3a7753 remove redis server-side hashing for performance 
we still  compute the document hash in node, and check it on retrieval
but we don't check the hash at the point of writing it in redis which
was previously done with a redis Lua script.
 2019-09-09 15:27:58 +01:00
Brian Gough
e75f2cc325 add unit tests for skipping history flush 2019-08-15 11:01:45 +01:00
Brian Gough
7493462154 add acceptance test for realtime shutdown 2019-08-15 10:38:07 +01:00
Brian Gough
97487a077e fix cluster/sentinel connection with real-time 2019-07-10 09:42:05 +01:00
Henry Oswald
3b3b2da0f5 add pubsub redis connection and remove real time redis connection 2019-07-04 13:34:31 +01:00
Shane Kilkelly
e8a8c446ed Increase timeout on test? 2019-06-19 11:34:38 +01:00
Shane Kilkelly
080b482e51 Update logger, metrics, and redis 
Also fix acceptance tests, broken by a change in the redis driver
behaviour. It now returns promises from most operations, which confuses mocha.
 2019-06-19 10:25:54 +01:00
Brian Gough
984b2d38e6 add unit tests 2019-06-11 16:33:14 +01:00
Brian Gough
e95059f98e handle non-urgent flushes in background 2019-06-03 10:01:10 +01:00
Brian Gough
0bbfa7de27 Merge branch 'master' into bg-downgrade-delete-component-error 2019-05-08 09:07:02 +01:00
Brian Gough
27a8248196 convert "Delete component" errors into warnings 2019-05-07 16:55:17 +01:00
Henry Oswald
e57741cb80 stub out project flusher for unit tests 2019-05-07 15:46:30 +01:00
Tim Alby
c1c23e4bee record last author id on document flush 
This is a multi-steps process:
* get a update's `user_id` from the metadata
* store the `user_id` (`lastUpdatedBy`) and current date (`lastUpdatedAt`) for
  the document in Redis on every updates
* fetch `lastUpdatedAt` and `lastUpdatedBy` from Redis on document flush
* send the data to web to be persisted in Mongo
 2019-05-02 11:10:02 +01:00
James Allen
52f3596e53 Review feedback 2019-04-16 11:05:17 +01:00
James Allen
3d76f4b9bf Record a snapshot to mongo when a doc's comments/changes get collapsed 2019-04-11 13:27:46 +01:00
Brian Gough
fd1425d83f include a unique id in every message published to redis 2019-03-21 12:10:15 +00:00
Brian Gough
8c5d74faef use explicit json content-type to avoid security issues with text/html 2019-02-12 16:45:11 +00:00
Henry Oswald
ecaef6485b revert the removal of realtime keyspace 2019-02-07 15:27:51 +00:00
Henry Oswald
4e1a2c787c Revert "turn down logging, use logger.info for less important data" 
This reverts commit c5f91428e3c7702fbbd3ffd1ef7a772d513f33f2.
 2019-02-06 15:29:22 +00:00
Christopher Hoskin
1217d8a80a Merge branch 'master' into csh-ho-docker-issue-1338-bulk-upgrade 2019-01-04 09:18:40 +00:00
Henry Oswald
2505be283a turn down logging, use logger.info for less important data 2018-12-13 12:46:10 +00:00
Henry Oswald
25e1ad7491 remove redis.realtime 
real time pushes data into the document updater redis instance
 2018-12-05 16:17:41 +00:00
Brian Gough
aae1352519 ensure that project history is flushed when the project is deleted 2018-10-30 12:03:44 +00:00
Brian Gough
fdcb806518 set a timestamp for the first entry in the projectHistory:Ops queue 2018-07-24 09:17:31 +01:00
Brian Gough
e471730efb Revert "Merge pull request #33 from sharelatex/bg-add-timestamp-marker-to-project-history-queue" 
This reverts commit 8ec398b133aa75c45384dd7ceb630cd743f4a15c, reversing
changes made to e3a6c934cf53fd245d7d0df737463cc90f954885.
 2018-07-23 16:05:04 +01:00
Brian Gough
6237577f87 set a timestamp for the first entry in the projectHistory:Ops queue 2018-07-20 10:43:31 +01:00
Hayden Faulds
62a10f25f2 move tk call to before sandboxed module call 2018-06-05 13:54:30 +01:00
Hayden Faulds
af92ca70a1 coerce projectHistoryId to integer after reading from Redis 2018-04-23 15:19:06 +01:00
Hayden Faulds
1b63141e49 Revert "Revert "Add projectHistoryId to updates"" 2018-04-23 12:08:04 +01:00
Hayden Faulds
3cc20d6311 Revert "Add projectHistoryId to updates" 2018-04-23 11:41:29 +01:00
Hayden Faulds
25c0be4eaa add projectHistryId to project structure updates 2018-04-13 15:22:05 +01:00
Hayden Faulds
a220794d32 add projectHistoryId to resync updates 2018-04-13 15:22:05 +01:00
Hayden Faulds
fb1852a593 cache projectHistoryId with doc in Redis 2018-04-13 15:22:05 +01:00
Brian Gough
779f00f912 add acceptance test for ordering of project structure changes 2018-03-15 16:27:14 +00:00
Brian Gough
3385d2640a fix structure ordering bug 2018-03-15 16:27:10 +00:00
Brian Gough
75a5428cbf update acceptance tests 2018-03-15 16:24:14 +00:00
Brian Gough
b3887fd984 update unit tests for incoming project versions 2018-03-15 16:24:11 +00:00
Brian Gough
0642e3c8c9 support project version on incoming requests 2018-03-15 16:15:10 +00:00
Hayden Faulds
4a5731440a Merge pull request #21 from sharelatex/hof-history-sync 
resync project history endpoint
 2018-03-14 10:40:59 +00:00
Hayden Faulds
1226f96fc3 extract ProjectHistoryRedisManager 2018-03-09 15:36:38 +00:00
Hayden Faulds
a72d228354 add unit tests 2018-03-07 13:29:53 +00:00
James Allen
939eaa2d4b Don't allow a document to be loaded without a pathname 2018-03-02 10:02:49 +00:00
Hayden Faulds
5a11332aa3 remove unecessary method wrapping 2018-02-22 10:17:59 +00:00
Hayden Faulds
ea0dd9700b reduce the number of times we flush project history 2018-02-22 10:01:05 +00:00
James Allen
3d5740fd7d Merge pull request #18 from sharelatex/msw-flush-project-ops 
Flush project structure changes every 500
 2018-02-16 10:30:00 +00:00
James Allen
657da70d45 Fix unit test namespacing from mocha upgrade 2018-02-15 16:28:58 +00:00
James Allen
5d1659457e Update to build 1.0.0 build scripts 2018-02-15 16:28:40 +00:00
James Allen
f6c79f3203 Fix and extend unit tests 2018-02-15 15:55:12 +00:00
Michael Walker
6c57317f8d Add a test that sending too few updates does not flush history 2018-01-31 11:41:08 +00:00
Michael Walker
241d1b27d5 Remove _shouldFlushHistoryOps wrapper 2018-01-31 11:27:40 +00:00
Michael Walker
772ee4083a Turn a unit test into an acceptance test 2018-01-31 11:17:56 +00:00
Michael Walker
bdaa2ffa77 Add an acceptance test for flushing project changes 2018-01-26 14:41:47 +00:00
Michael Walker
3028fb9c3d Only flush project updates when crossing the threshold 2018-01-24 11:57:03 +00:00
Brian Gough
41f15c4fa8 fix unit test 2017-12-14 14:46:27 +00:00
Brian Gough
947e18eccf Merge branch 'bg-limit-ops-in-lock' 2017-12-14 14:26:44 +00:00
James Allen
6abf9c227d Calculate and send doc length before each update 2017-12-11 17:26:32 +00:00
Hayden Faulds
d0e56e5054 add acceptance tests for entity additions 2017-11-10 15:01:37 +00:00
Hayden Faulds
a3420b1236 version entity additions 2017-11-10 14:54:56 +00:00
Hayden Faulds
944e633bac add acceptance test for entity renaming 2017-11-08 09:20:38 +00:00