github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-21 20:47:08 -05:00
Code Issues Projects Releases Packages Wiki Activity
21799 commits 3 branches 7 tags 206 MiB
Commit graph

490 commits

Author SHA1 Message Date
Jakob Ackermann
6e551f9b34 [perf] use MGET/MSET/DEL for manipulating multiple keys in one operation 
In some cases we can get rid of MULTI/EXEC operations too.
- putDocInMemory: from 10 down to 2 operations
- removeDocFromMemory: from 14+4 down to 4+4 operations
- updateDoc: from 13 down to 8 operations
 2021-04-13 11:47:10 +01:00
Jakob Ackermann
178440395f [perf] switch write sequence for doc contents and doc tracking 
Doc contents are added only after the tracking has been setup.
All read paths on the tracking have been checked to gracefully handle
 the case of existing doc_id but missing doc contents.

- getDoc: -1 operation

REF: 0a2b47c660c60b95e360d8f3b3e30b862ceb6d79
 2021-04-13 11:46:44 +01:00
Jakob Ackermann
50b24043b7 [perf] use MGET for fetching multiple keys in one operation 
- getDoc: from 13 down to 2 operations
 2021-04-09 08:42:35 +01:00
Eric Mc Sween
4d70bd664f Reintroduce Node 12 and metrics upgrades 
These changes were previously merged, not deployed, and reverted. This
reverts the revert.

This reverts commit a6b8c6c658b33b6eee78b8b99e43308f32211ae2, reversing
changes made to 93c98921372eed4244d22fce800716cb27eca299.
 2021-04-01 15:51:00 -04:00
Eric Mc Sween
4dd1b26b2e Revert "Merge pull request #161 from overleaf/em-upgrade-node-12" 
This reverts commit d44102751b9436ad89c5b3b05e7abdff51fcc78a, reversing
changes made to 6c2f5b8d053b75c677da2b7ddd04f998d2be6fff.
 2021-03-31 12:07:11 -04:00
Eric Mc Sween
1a2235a219 Upgrade to Node 12 2021-03-16 16:48:45 -04:00
Thomas
23738540ed Fix API request errors which could contain API hostname or address (#160) 
Wrap errors produced by failing requests to web API, and remove the url/hostname from thrown error messages. (But keep the URL path for info.)
 2021-02-24 15:09:19 +01:00
Henry Oswald
20a373d95c stop listening on the pending updates channels 10 times 2021-02-23 08:27:29 +00:00
Henry Oswald
c7e57cd28f add Dispatchers running on old queue while we migrate 
revert once migrated
 2021-02-15 14:16:45 +00:00
Henry Oswald
854e24bb57 remove unneeded anonymous func 2021-02-15 14:12:28 +00:00
Henry Oswald
11c8cfc939 shard the pending-updates-list queue 2021-02-02 16:38:25 +00:00
Eric Mc Sween
98f8d7f51c Set the diff-match-patch timeout to 100ms 
This might result in worse diffs, but we don't want to spend a second
blocking the event loop while we figure out nicer diffs when comparing
documents.
 2021-01-14 15:11:15 -05:00
Eric Mc Sween
de247302b1 Use a centralized diff-match-patch package 
We use our own fork of the diff-match-patch npm package, which adds an
optimization for the semantic alignment loop.
 2020-12-07 16:15:19 -05:00
Eric Mc Sween
dce5b8759a Decaf cleanup: capitalize class names 2020-12-07 15:30:02 -05:00
Eric Mc Sween
db4b0a6f38 Decaf cleanup: do not throw strings 2020-12-07 15:28:25 -05:00
Eric Mc Sween
8c70e72bfa Decaf cleanup: unused variable 2020-12-07 15:27:41 -05:00
Eric Mc Sween
9f17f3ea0a Decaf cleanup: remove default callback 2020-12-07 15:27:01 -05:00
Eric Mc Sween
b74e7f6feb Decaf cleanup: unnecessary returns 2020-12-07 15:25:52 -05:00
Eric Mc Sween
a91770e979 Decaf cleanup: remove Array.from() 2020-12-07 15:25:20 -05:00
Jakob Ackermann
5e00684dbb [misc] bump metrics module to 3.4.1 
- renamed package from `metrics-sharelatex` to `@overleaf/metrics`
- drop support for statsd backend
- decaffeinate
- compress `/metrics` response using gzip
- bump debugging agents to latest versions
- expose prometheus interfaces for custom metrics (custom tags)
- cleanup of open sockets metrics
- fix deprecation warnings for header access
 2020-11-25 11:57:19 +00:00
Jakob Ackermann
08ed5f6c9b [misc] bump @overleaf/redis-wrapper to version 2.0.0 2020-11-11 16:20:32 +00:00
Jakob Ackermann
16ef0d9610 [misc] mongodb: use the new db connector by default 
mongojs was enabling it by default as well.
 2020-09-10 10:40:05 +01:00
Jakob Ackermann
c337cf1c4f [misc] mongodb: refactor the process of setting up the db construct 
Co-Authored-By: John Lees-Miller <jdleesmiller@gmail.com>
 2020-09-07 09:49:06 +01:00
Jakob Ackermann
1d57706d44 [misc] migrate acceptance tests to the native mongo driver, drop mongojs 2020-08-28 17:06:25 +01:00
Jakob Ackermann
f80a92ce46 [misc] migrate the app to the native mongo driver 
acceptance tests to follow in a separate commit
 2020-08-28 15:12:30 +01:00
Eric Mc Sween
1d1f204021 Remove backwards-compat project update API 
The project update endpoint accepted updates both in two array params:
docUpdates and fileUpdates, and in a single array: updates. This commit
removes the docUpdates/fileUpdates params now that web uses the updates
param.
 2020-05-20 16:26:22 -04:00
Eric Mc Sween
9799b94752 Accept ordered doc and file updates 
Add an `updates` parameter to the project update endpoint. It can be
used instead of `docUpdates` and `fileUpdates` to provide a single list
of updates in the order they should be processed.
 2020-05-20 07:57:32 -04:00
Eric Mc Sween
6269ace987 Decaf cleanup: move functions to top level 2020-05-15 09:59:32 -04:00
Eric Mc Sween
3d000bcbe6 Decaf cleanup: use arrow functions for callbacks 2020-05-15 09:59:32 -04:00
Eric Mc Sween
751d9ea718 Decaf cleanup: simplify loops 2020-05-15 09:59:32 -04:00
Eric Mc Sween
7a5a782dc7 Decaf cleanup: camel case variables 2020-05-15 09:59:26 -04:00
Eric Mc Sween
ceb77f4c93 Decaf cleanup: remove unused variables 2020-05-14 16:54:08 -04:00
Eric Mc Sween
6b5760ca28 Decaf cleanup: simplify null checks 2020-05-14 16:53:22 -04:00
Eric Mc Sween
f2c67b66fa Decaf cleanup: remove default callbacks 2020-05-14 16:50:04 -04:00
Eric Mc Sween
2bff83137c Decaf cleanup: unnecessary returns 2020-05-14 16:35:10 -04:00
Eric Mc Sween
819aa378d9 Decaf cleanup: remove Array.from() 2020-05-14 16:32:05 -04:00
Eric Mc Sween
a2a1914a53 Use max_doc_length setting to limit incoming doc size 2020-05-11 11:15:37 -04:00
Eric Mc Sween
cb959ddfc1 Decaf cleanup: use arrow functions for callbacks 2020-05-11 11:14:37 -04:00
Eric Mc Sween
e4ac63dd19 Decaf cleanup: move functions to top level 2020-05-11 11:12:15 -04:00
Eric Mc Sween
64a881461f Decaf cleanup: camel case variables 2020-05-11 11:07:15 -04:00
Eric Mc Sween
fc73bbe1a5 Decaf cleanup: simplify null checks 2020-05-11 10:52:06 -04:00
Eric Mc Sween
80ea49c69c Decaf cleanup: remove __guard__ 2020-05-11 10:47:27 -04:00
Eric Mc Sween
814ac40e07 Decaf cleanup: unnecessary returns 2020-05-11 10:45:39 -04:00
Eric Mc Sween
3385ec5f26 Decaf cleanup: unnecessary Array.from() 2020-05-11 10:43:22 -04:00
Tim Alby
9f6ea07002 fix SyntaxError on export var 2020-05-06 12:17:08 +02:00
Tim Alby
e089cfc93c format config/settings.defaults.js & lib/diff_match_patch.js 2020-05-06 12:16:48 +02:00
Tim Alby
96e7a668b7 disable linting for lib/diff_match_patch.js 2020-05-06 12:16:04 +02:00
Tim Alby
dbf9e88dc3 prettier: convert app/js decaffeinated files to Prettier format 2020-05-06 12:09:33 +02:00
Tim Alby
a519980c10 decaffeinate: rename app/coffee dir to app/js 2020-05-06 12:09:23 +02:00
decaffeinate
dad1d1212f decaffeinate: Run post-processing cleanups on DeleteQueueManager.coffee and 58 other files 2020-05-06 12:09:15 +02:00
decaffeinate
1fa8882674 decaffeinate: Convert DeleteQueueManager.coffee and 58 other files to JS 2020-05-06 12:08:21 +02:00
decaffeinate
249b7247b5 decaffeinate: Rename DeleteQueueManager.coffee and 58 other files from .coffee to .js 2020-05-06 12:07:29 +02:00
Miguel Serrano
f935c392bc Merge branch 'master' into sk-upgrade-dependencies 2020-04-27 13:38:21 +02:00
Brian Gough
a51f61a555 remove redis migration code 2020-04-21 14:48:47 +01:00
Brian Gough
248edc03fa add comment about the two history clients 2020-04-21 14:44:19 +01:00
Brian Gough
af93193d6e remove new_project_history and use project_history instead 2020-04-21 14:43:48 +01:00
Brian Gough
ac68f59487 Merge branch 'master' into bg-use-separate-redis-for-project-history 2020-04-16 15:48:21 +01:00
Brian Gough
beb3691795 add metrics for redis get/update 2020-04-06 10:00:48 +01:00
Brian Gough
2b72ec49a1 add comments for redis metrics 2020-04-02 11:33:52 +01:00
Brian Gough
21824d49da Merge branch 'bg-add-queue-metrics' of github.com:overleaf/document-updater into bg-add-queue-metrics 2020-04-01 16:04:52 +01:00
Brian Gough
3a8c362fba add doclines set/del metric 2020-04-01 15:59:25 +01:00
Brian Gough
00b11bda96 use separate loop for pendingUpdates metric 2020-04-01 14:50:55 +01:00
Jakob Ackermann
17c2add0cf [misc] track redis pub/sub payload sizes on publish 2020-03-30 11:31:43 +02:00
Brian Gough
1a0550364d add metric for getdoc bytes 2020-03-25 14:27:41 +00:00
Brian Gough
891fcc696b add metric for pending updates queue 2020-03-25 14:27:41 +00:00
Brian Gough
e293d86c14 add metric for project history queue 2020-03-25 12:15:16 +00:00
Shane Kilkelly
ada4fba3dc Fix express deprecations 2020-03-19 15:39:57 +00:00
Eric Mc Sween
ff32104fe6 Merge pull request #123 from overleaf/em-doc-hard-delete 
Add ignore_flush_errors option to the doc delete endpoint
 2020-03-10 10:11:00 -04:00
Eric Mc Sween
d9caced0d6 Change skip_flush option to ignore_flush_errors in delete doc 
Instead of skipping the flush, we'll still try to flush and proceed with
the doc deletion, even when the flush fails.
 2020-03-10 09:40:49 -04:00
Eric Mc Sween
9b70eb75b3 Rename flush param to skip_flush in delete doc 
Also move it to the query string instead of the body.
 2020-03-09 16:27:32 -04:00
Eric Mc Sween
c09bc0e868 Add a "flush: false" option to the doc delete endpoint 
This will delete the document from Redis without flushing to web,
docstore or history. To be used when something is broken.
 2020-03-07 08:59:15 -05:00
nate stemen
ffd8d0745d use empty object for ranges if it doesn't exist 2020-03-06 13:49:30 -05:00
Brian Gough
0419039d4d Merge branch 'master' into bg-use-separate-redis-for-project-history 2020-02-21 14:13:33 +00:00
Brian Gough
338d3609f5 add comment about null byte check 2020-01-30 15:17:13 +00:00
Brian Gough
544ae05212 added note about rollback 2020-01-23 16:22:26 +00:00
Brian Gough
626e19ed1a add logging of migration phase at startup 2020-01-23 15:46:54 +00:00
Brian Gough
d5a2b96df9 add note about deleting the migration key entries 2020-01-23 14:36:59 +00:00
Brian Gough
7036803acf add missing argument to metrics.inc 
also track retries rather than attempts (which is always 1 for a successful request)
 2020-01-14 15:00:21 +00:00
Brian Gough
3caa0e7c05 add failure/retry metrics for web-api requests 2020-01-14 13:53:50 +00:00
Brian Gough
a638ef4251 add comment about locking in redis migration 2020-01-13 15:56:28 +00:00
Brian Gough
27044c2d02 allow migration phase to be modified at runtime for testing 2020-01-06 16:46:35 +00:00
Brian Gough
8ae95ebf60 fix rclient check in migration metrics 2020-01-06 16:45:36 +00:00
Brian Gough
97cbf46160 add metrics for migration 2019-12-16 11:46:35 +00:00
Brian Gough
a2e63d009e fix migration phase check 2019-12-16 09:55:26 +00:00
Brian Gough
d0c5eb5698 support migration of project history keys to separate redis instance 2019-12-13 16:38:41 +00:00
Brian Gough
ad19fee667 add setting so that double flush is the default 
can be disabled to stop flushing to track-changes
 2019-11-25 13:36:25 +00:00
Brian Gough
4f6583bbf2 fix getDocVersion and add tests 2019-11-25 13:28:36 +00:00
Brian Gough
68e12f4d2d add metrics for queue operations 2019-11-25 10:51:10 +00:00
Brian Gough
8b73bb9f13 Merge branch 'master' into bg-filter-track-changes-updates 2019-11-22 10:41:33 +00:00
Brian Gough
b7055eecee add metrics for history flushes 2019-11-22 09:14:32 +00:00
Brian Gough
65cf4cf7c7 make flush to track-changes failsafe 2019-11-21 14:58:35 +00:00
Brian Gough
dcd7649bad filter track-changes updates for projects using project-history 2019-11-19 10:02:56 +00:00
Brian Gough
d82b180b76 avoid project history queues building up with deferred flush 2019-10-03 04:05:24 +01:00
Brian Gough
c1454bc4ac Merge pull request #92 from overleaf/bg-flush-queue-prod-fixes 
add continuous background flush
 2019-10-02 13:11:00 +01:00
Brian Gough
0c14b7d2f8 add comment about background flush limit 2019-10-01 15:06:01 +01:00
Brian Gough
2845b23b70 add smoothing of delete spikes 2019-10-01 15:01:53 +01:00
Brian Gough
2c22a60052 add random jitter to cutoff time 2019-10-01 15:01:20 +01:00
Brian Gough
a32495d2b4 make background flush more adaptive 2019-10-01 14:09:41 +01:00
Brian Gough
73b4262186 add continuous background flush 2019-09-30 16:05:53 +01:00
Brian Gough
33fadf51c1 fix getDocTimestamps for multiple docs 2019-09-30 13:50:25 +01:00