github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-10-24 21:12:38 -04:00
Code Issues Projects Releases Packages Wiki Activity
18947 commits 3 branches 7 tags 197 MiB
Commit graph

469 commits

Author SHA1 Message Date
Jakob Ackermann
16ef0d9610 [misc] mongodb: use the new db connector by default 
mongojs was enabling it by default as well.
 2020-09-10 10:40:05 +01:00
Jakob Ackermann
c337cf1c4f [misc] mongodb: refactor the process of setting up the db construct 
Co-Authored-By: John Lees-Miller <jdleesmiller@gmail.com>
 2020-09-07 09:49:06 +01:00
Jakob Ackermann
1d57706d44 [misc] migrate acceptance tests to the native mongo driver, drop mongojs 2020-08-28 17:06:25 +01:00
Jakob Ackermann
f80a92ce46 [misc] migrate the app to the native mongo driver 
acceptance tests to follow in a separate commit
 2020-08-28 15:12:30 +01:00
Eric Mc Sween
1d1f204021 Remove backwards-compat project update API 
The project update endpoint accepted updates both in two array params:
docUpdates and fileUpdates, and in a single array: updates. This commit
removes the docUpdates/fileUpdates params now that web uses the updates
param.
 2020-05-20 16:26:22 -04:00
Eric Mc Sween
9799b94752 Accept ordered doc and file updates 
Add an `updates` parameter to the project update endpoint. It can be
used instead of `docUpdates` and `fileUpdates` to provide a single list
of updates in the order they should be processed.
 2020-05-20 07:57:32 -04:00
Eric Mc Sween
6269ace987 Decaf cleanup: move functions to top level 2020-05-15 09:59:32 -04:00
Eric Mc Sween
3d000bcbe6 Decaf cleanup: use arrow functions for callbacks 2020-05-15 09:59:32 -04:00
Eric Mc Sween
751d9ea718 Decaf cleanup: simplify loops 2020-05-15 09:59:32 -04:00
Eric Mc Sween
7a5a782dc7 Decaf cleanup: camel case variables 2020-05-15 09:59:26 -04:00
Eric Mc Sween
ceb77f4c93 Decaf cleanup: remove unused variables 2020-05-14 16:54:08 -04:00
Eric Mc Sween
6b5760ca28 Decaf cleanup: simplify null checks 2020-05-14 16:53:22 -04:00
Eric Mc Sween
f2c67b66fa Decaf cleanup: remove default callbacks 2020-05-14 16:50:04 -04:00
Eric Mc Sween
2bff83137c Decaf cleanup: unnecessary returns 2020-05-14 16:35:10 -04:00
Eric Mc Sween
819aa378d9 Decaf cleanup: remove Array.from() 2020-05-14 16:32:05 -04:00
Eric Mc Sween
a2a1914a53 Use max_doc_length setting to limit incoming doc size 2020-05-11 11:15:37 -04:00
Eric Mc Sween
cb959ddfc1 Decaf cleanup: use arrow functions for callbacks 2020-05-11 11:14:37 -04:00
Eric Mc Sween
e4ac63dd19 Decaf cleanup: move functions to top level 2020-05-11 11:12:15 -04:00
Eric Mc Sween
64a881461f Decaf cleanup: camel case variables 2020-05-11 11:07:15 -04:00
Eric Mc Sween
fc73bbe1a5 Decaf cleanup: simplify null checks 2020-05-11 10:52:06 -04:00
Eric Mc Sween
80ea49c69c Decaf cleanup: remove __guard__ 2020-05-11 10:47:27 -04:00
Eric Mc Sween
814ac40e07 Decaf cleanup: unnecessary returns 2020-05-11 10:45:39 -04:00
Eric Mc Sween
3385ec5f26 Decaf cleanup: unnecessary Array.from() 2020-05-11 10:43:22 -04:00
Tim Alby
9f6ea07002 fix SyntaxError on export var 2020-05-06 12:17:08 +02:00
Tim Alby
e089cfc93c format config/settings.defaults.js & lib/diff_match_patch.js 2020-05-06 12:16:48 +02:00
Tim Alby
96e7a668b7 disable linting for lib/diff_match_patch.js 2020-05-06 12:16:04 +02:00
Tim Alby
dbf9e88dc3 prettier: convert app/js decaffeinated files to Prettier format 2020-05-06 12:09:33 +02:00
Tim Alby
a519980c10 decaffeinate: rename app/coffee dir to app/js 2020-05-06 12:09:23 +02:00
decaffeinate
dad1d1212f decaffeinate: Run post-processing cleanups on DeleteQueueManager.coffee and 58 other files 2020-05-06 12:09:15 +02:00
decaffeinate
1fa8882674 decaffeinate: Convert DeleteQueueManager.coffee and 58 other files to JS 2020-05-06 12:08:21 +02:00
decaffeinate
249b7247b5 decaffeinate: Rename DeleteQueueManager.coffee and 58 other files from .coffee to .js 2020-05-06 12:07:29 +02:00
Miguel Serrano
f935c392bc Merge branch 'master' into sk-upgrade-dependencies 2020-04-27 13:38:21 +02:00
Brian Gough
a51f61a555 remove redis migration code 2020-04-21 14:48:47 +01:00
Brian Gough
248edc03fa add comment about the two history clients 2020-04-21 14:44:19 +01:00
Brian Gough
af93193d6e remove new_project_history and use project_history instead 2020-04-21 14:43:48 +01:00
Brian Gough
ac68f59487 Merge branch 'master' into bg-use-separate-redis-for-project-history 2020-04-16 15:48:21 +01:00
Brian Gough
beb3691795 add metrics for redis get/update 2020-04-06 10:00:48 +01:00
Brian Gough
2b72ec49a1 add comments for redis metrics 2020-04-02 11:33:52 +01:00
Brian Gough
21824d49da Merge branch 'bg-add-queue-metrics' of github.com:overleaf/document-updater into bg-add-queue-metrics 2020-04-01 16:04:52 +01:00
Brian Gough
3a8c362fba add doclines set/del metric 2020-04-01 15:59:25 +01:00
Brian Gough
00b11bda96 use separate loop for pendingUpdates metric 2020-04-01 14:50:55 +01:00
Jakob Ackermann
17c2add0cf [misc] track redis pub/sub payload sizes on publish 2020-03-30 11:31:43 +02:00
Brian Gough
1a0550364d add metric for getdoc bytes 2020-03-25 14:27:41 +00:00
Brian Gough
891fcc696b add metric for pending updates queue 2020-03-25 14:27:41 +00:00
Brian Gough
e293d86c14 add metric for project history queue 2020-03-25 12:15:16 +00:00
Shane Kilkelly
ada4fba3dc Fix express deprecations 2020-03-19 15:39:57 +00:00
Eric Mc Sween
ff32104fe6 Merge pull request #123 from overleaf/em-doc-hard-delete 
Add ignore_flush_errors option to the doc delete endpoint
 2020-03-10 10:11:00 -04:00
Eric Mc Sween
d9caced0d6 Change skip_flush option to ignore_flush_errors in delete doc 
Instead of skipping the flush, we'll still try to flush and proceed with
the doc deletion, even when the flush fails.
 2020-03-10 09:40:49 -04:00
Eric Mc Sween
9b70eb75b3 Rename flush param to skip_flush in delete doc 
Also move it to the query string instead of the body.
 2020-03-09 16:27:32 -04:00
Eric Mc Sween
c09bc0e868 Add a "flush: false" option to the doc delete endpoint 
This will delete the document from Redis without flushing to web,
docstore or history. To be used when something is broken.
 2020-03-07 08:59:15 -05:00
nate stemen
ffd8d0745d use empty object for ranges if it doesn't exist 2020-03-06 13:49:30 -05:00
Brian Gough
0419039d4d Merge branch 'master' into bg-use-separate-redis-for-project-history 2020-02-21 14:13:33 +00:00
Brian Gough
338d3609f5 add comment about null byte check 2020-01-30 15:17:13 +00:00
Brian Gough
544ae05212 added note about rollback 2020-01-23 16:22:26 +00:00
Brian Gough
626e19ed1a add logging of migration phase at startup 2020-01-23 15:46:54 +00:00
Brian Gough
d5a2b96df9 add note about deleting the migration key entries 2020-01-23 14:36:59 +00:00
Brian Gough
7036803acf add missing argument to metrics.inc 
also track retries rather than attempts (which is always 1 for a successful request)
 2020-01-14 15:00:21 +00:00
Brian Gough
3caa0e7c05 add failure/retry metrics for web-api requests 2020-01-14 13:53:50 +00:00
Brian Gough
a638ef4251 add comment about locking in redis migration 2020-01-13 15:56:28 +00:00
Brian Gough
27044c2d02 allow migration phase to be modified at runtime for testing 2020-01-06 16:46:35 +00:00
Brian Gough
8ae95ebf60 fix rclient check in migration metrics 2020-01-06 16:45:36 +00:00
Brian Gough
97cbf46160 add metrics for migration 2019-12-16 11:46:35 +00:00
Brian Gough
a2e63d009e fix migration phase check 2019-12-16 09:55:26 +00:00
Brian Gough
d0c5eb5698 support migration of project history keys to separate redis instance 2019-12-13 16:38:41 +00:00
Brian Gough
ad19fee667 add setting so that double flush is the default 
can be disabled to stop flushing to track-changes
 2019-11-25 13:36:25 +00:00
Brian Gough
4f6583bbf2 fix getDocVersion and add tests 2019-11-25 13:28:36 +00:00
Brian Gough
68e12f4d2d add metrics for queue operations 2019-11-25 10:51:10 +00:00
Brian Gough
8b73bb9f13 Merge branch 'master' into bg-filter-track-changes-updates 2019-11-22 10:41:33 +00:00
Brian Gough
b7055eecee add metrics for history flushes 2019-11-22 09:14:32 +00:00
Brian Gough
65cf4cf7c7 make flush to track-changes failsafe 2019-11-21 14:58:35 +00:00
Brian Gough
dcd7649bad filter track-changes updates for projects using project-history 2019-11-19 10:02:56 +00:00
Brian Gough
d82b180b76 avoid project history queues building up with deferred flush 2019-10-03 04:05:24 +01:00
Brian Gough
c1454bc4ac Merge pull request #92 from overleaf/bg-flush-queue-prod-fixes 
add continuous background flush
 2019-10-02 13:11:00 +01:00
Brian Gough
0c14b7d2f8 add comment about background flush limit 2019-10-01 15:06:01 +01:00
Brian Gough
2845b23b70 add smoothing of delete spikes 2019-10-01 15:01:53 +01:00
Brian Gough
2c22a60052 add random jitter to cutoff time 2019-10-01 15:01:20 +01:00
Brian Gough
a32495d2b4 make background flush more adaptive 2019-10-01 14:09:41 +01:00
Brian Gough
73b4262186 add continuous background flush 2019-09-30 16:05:53 +01:00
Brian Gough
33fadf51c1 fix getDocTimestamps for multiple docs 2019-09-30 13:50:25 +01:00
Brian Gough
260923f291 keep flushQueuedProjects in the foreground 2019-09-27 10:46:24 +01:00
Brian Gough
7561e05660 check timestamps array length 2019-09-27 10:39:56 +01:00
Brian Gough
b7f3b848af remove unused dryRun option 
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
 2019-09-26 15:50:55 +01:00
Brian Gough
3bc176259b fix log line 2019-09-26 15:46:54 +01:00
Brian Gough
8cdc8c410a fix error logging 2019-09-26 15:46:45 +01:00
Brian Gough
fc62abfcfa run flush of queued projects in the background 2019-09-26 15:46:14 +01:00
Brian Gough
ba35c73cb6 add comment about ZPOPMIN 2019-09-26 15:18:10 +01:00
Brian Gough
a709a0adaa for simplicity keep the cutoff time the same while flushing the queue 2019-09-26 15:05:38 +01:00
Brian Gough
eae4b352ca remove unnecessary check 2019-09-26 14:59:03 +01:00
Brian Gough
b49621b3e9 add comments 2019-09-26 10:14:49 +01:00
Brian Gough
83dd43b809 add metric for queue length 2019-09-25 17:04:36 +01:00
Brian Gough
f6b2ac7360 queue deletes for deferred processing 2019-09-25 16:42:49 +01:00
Brian Gough
912a3a7753 remove redis server-side hashing for performance 
we still  compute the document hash in node, and check it on retrieval
but we don't check the hash at the point of writing it in redis which
was previously done with a redis Lua script.
 2019-09-09 15:27:58 +01:00
Henry Oswald
aa15a76059 added log lines for all project flushing 2019-08-30 07:38:53 +01:00
Henry Oswald
0ae838dd2d add logger into project flusher 2019-08-29 20:36:00 +01:00
Brian Gough
a76e0dca88 skip history flush when project is cleared by realtime shutdown 
history is flushed by a background cron job anyway
 2019-08-15 09:51:16 +01:00
Simon Detheridge
6721b904a7 Merge pull request #82 from overleaf/bg-mongo-health-check 
add a combined health check for mongo and redis
 2019-08-08 14:16:38 +01:00
Simon Detheridge
06444d2cc4 Improve/fix serializers for update logging (#80) 
* Improve/fix serializers for update logging
 2019-08-08 14:10:54 +01:00
Brian Gough
40f6494b19 add a combined health check for mongo and redis 2019-08-07 16:25:23 +01:00
Simon Detheridge
df9ca8b272 Add serializer to print only length of large fields in production 2019-07-31 16:42:28 +01:00
Brian Gough
618880f99d remove unnecessary check for doc_id 2019-07-24 16:57:43 +01:00