github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-21 20:47:08 -05:00
Code Issues Projects Releases Packages Wiki Activity
20293 commits 3 branches 7 tags 206 MiB
Commit graph

482 commits

Author SHA1 Message Date
Henry Oswald
c7e57cd28f add Dispatchers running on old queue while we migrate 
revert once migrated
 2021-02-15 14:16:45 +00:00
Henry Oswald
854e24bb57 remove unneeded anonymous func 2021-02-15 14:12:28 +00:00
Henry Oswald
11c8cfc939 shard the pending-updates-list queue 2021-02-02 16:38:25 +00:00
Eric Mc Sween
98f8d7f51c Set the diff-match-patch timeout to 100ms 
This might result in worse diffs, but we don't want to spend a second
blocking the event loop while we figure out nicer diffs when comparing
documents.
 2021-01-14 15:11:15 -05:00
Eric Mc Sween
de247302b1 Use a centralized diff-match-patch package 
We use our own fork of the diff-match-patch npm package, which adds an
optimization for the semantic alignment loop.
 2020-12-07 16:15:19 -05:00
Eric Mc Sween
dce5b8759a Decaf cleanup: capitalize class names 2020-12-07 15:30:02 -05:00
Eric Mc Sween
db4b0a6f38 Decaf cleanup: do not throw strings 2020-12-07 15:28:25 -05:00
Eric Mc Sween
8c70e72bfa Decaf cleanup: unused variable 2020-12-07 15:27:41 -05:00
Eric Mc Sween
9f17f3ea0a Decaf cleanup: remove default callback 2020-12-07 15:27:01 -05:00
Eric Mc Sween
b74e7f6feb Decaf cleanup: unnecessary returns 2020-12-07 15:25:52 -05:00
Eric Mc Sween
a91770e979 Decaf cleanup: remove Array.from() 2020-12-07 15:25:20 -05:00
Jakob Ackermann
5e00684dbb [misc] bump metrics module to 3.4.1 
- renamed package from `metrics-sharelatex` to `@overleaf/metrics`
- drop support for statsd backend
- decaffeinate
- compress `/metrics` response using gzip
- bump debugging agents to latest versions
- expose prometheus interfaces for custom metrics (custom tags)
- cleanup of open sockets metrics
- fix deprecation warnings for header access
 2020-11-25 11:57:19 +00:00
Jakob Ackermann
08ed5f6c9b [misc] bump @overleaf/redis-wrapper to version 2.0.0 2020-11-11 16:20:32 +00:00
Jakob Ackermann
16ef0d9610 [misc] mongodb: use the new db connector by default 
mongojs was enabling it by default as well.
 2020-09-10 10:40:05 +01:00
Jakob Ackermann
c337cf1c4f [misc] mongodb: refactor the process of setting up the db construct 
Co-Authored-By: John Lees-Miller <jdleesmiller@gmail.com>
 2020-09-07 09:49:06 +01:00
Jakob Ackermann
1d57706d44 [misc] migrate acceptance tests to the native mongo driver, drop mongojs 2020-08-28 17:06:25 +01:00
Jakob Ackermann
f80a92ce46 [misc] migrate the app to the native mongo driver 
acceptance tests to follow in a separate commit
 2020-08-28 15:12:30 +01:00
Eric Mc Sween
1d1f204021 Remove backwards-compat project update API 
The project update endpoint accepted updates both in two array params:
docUpdates and fileUpdates, and in a single array: updates. This commit
removes the docUpdates/fileUpdates params now that web uses the updates
param.
 2020-05-20 16:26:22 -04:00
Eric Mc Sween
9799b94752 Accept ordered doc and file updates 
Add an `updates` parameter to the project update endpoint. It can be
used instead of `docUpdates` and `fileUpdates` to provide a single list
of updates in the order they should be processed.
 2020-05-20 07:57:32 -04:00
Eric Mc Sween
6269ace987 Decaf cleanup: move functions to top level 2020-05-15 09:59:32 -04:00
Eric Mc Sween
3d000bcbe6 Decaf cleanup: use arrow functions for callbacks 2020-05-15 09:59:32 -04:00
Eric Mc Sween
751d9ea718 Decaf cleanup: simplify loops 2020-05-15 09:59:32 -04:00
Eric Mc Sween
7a5a782dc7 Decaf cleanup: camel case variables 2020-05-15 09:59:26 -04:00
Eric Mc Sween
ceb77f4c93 Decaf cleanup: remove unused variables 2020-05-14 16:54:08 -04:00
Eric Mc Sween
6b5760ca28 Decaf cleanup: simplify null checks 2020-05-14 16:53:22 -04:00
Eric Mc Sween
f2c67b66fa Decaf cleanup: remove default callbacks 2020-05-14 16:50:04 -04:00
Eric Mc Sween
2bff83137c Decaf cleanup: unnecessary returns 2020-05-14 16:35:10 -04:00
Eric Mc Sween
819aa378d9 Decaf cleanup: remove Array.from() 2020-05-14 16:32:05 -04:00
Eric Mc Sween
a2a1914a53 Use max_doc_length setting to limit incoming doc size 2020-05-11 11:15:37 -04:00
Eric Mc Sween
cb959ddfc1 Decaf cleanup: use arrow functions for callbacks 2020-05-11 11:14:37 -04:00
Eric Mc Sween
e4ac63dd19 Decaf cleanup: move functions to top level 2020-05-11 11:12:15 -04:00
Eric Mc Sween
64a881461f Decaf cleanup: camel case variables 2020-05-11 11:07:15 -04:00
Eric Mc Sween
fc73bbe1a5 Decaf cleanup: simplify null checks 2020-05-11 10:52:06 -04:00
Eric Mc Sween
80ea49c69c Decaf cleanup: remove __guard__ 2020-05-11 10:47:27 -04:00
Eric Mc Sween
814ac40e07 Decaf cleanup: unnecessary returns 2020-05-11 10:45:39 -04:00
Eric Mc Sween
3385ec5f26 Decaf cleanup: unnecessary Array.from() 2020-05-11 10:43:22 -04:00
Tim Alby
9f6ea07002 fix SyntaxError on export var 2020-05-06 12:17:08 +02:00
Tim Alby
e089cfc93c format config/settings.defaults.js & lib/diff_match_patch.js 2020-05-06 12:16:48 +02:00
Tim Alby
96e7a668b7 disable linting for lib/diff_match_patch.js 2020-05-06 12:16:04 +02:00
Tim Alby
dbf9e88dc3 prettier: convert app/js decaffeinated files to Prettier format 2020-05-06 12:09:33 +02:00
Tim Alby
a519980c10 decaffeinate: rename app/coffee dir to app/js 2020-05-06 12:09:23 +02:00
decaffeinate
dad1d1212f decaffeinate: Run post-processing cleanups on DeleteQueueManager.coffee and 58 other files 2020-05-06 12:09:15 +02:00
decaffeinate
1fa8882674 decaffeinate: Convert DeleteQueueManager.coffee and 58 other files to JS 2020-05-06 12:08:21 +02:00
decaffeinate
249b7247b5 decaffeinate: Rename DeleteQueueManager.coffee and 58 other files from .coffee to .js 2020-05-06 12:07:29 +02:00
Miguel Serrano
f935c392bc Merge branch 'master' into sk-upgrade-dependencies 2020-04-27 13:38:21 +02:00
Brian Gough
a51f61a555 remove redis migration code 2020-04-21 14:48:47 +01:00
Brian Gough
248edc03fa add comment about the two history clients 2020-04-21 14:44:19 +01:00
Brian Gough
af93193d6e remove new_project_history and use project_history instead 2020-04-21 14:43:48 +01:00
Brian Gough
ac68f59487 Merge branch 'master' into bg-use-separate-redis-for-project-history 2020-04-16 15:48:21 +01:00
Brian Gough
beb3691795 add metrics for redis get/update 2020-04-06 10:00:48 +01:00
Brian Gough
2b72ec49a1 add comments for redis metrics 2020-04-02 11:33:52 +01:00
Brian Gough
21824d49da Merge branch 'bg-add-queue-metrics' of github.com:overleaf/document-updater into bg-add-queue-metrics 2020-04-01 16:04:52 +01:00
Brian Gough
3a8c362fba add doclines set/del metric 2020-04-01 15:59:25 +01:00
Brian Gough
00b11bda96 use separate loop for pendingUpdates metric 2020-04-01 14:50:55 +01:00
Jakob Ackermann
17c2add0cf [misc] track redis pub/sub payload sizes on publish 2020-03-30 11:31:43 +02:00
Brian Gough
1a0550364d add metric for getdoc bytes 2020-03-25 14:27:41 +00:00
Brian Gough
891fcc696b add metric for pending updates queue 2020-03-25 14:27:41 +00:00
Brian Gough
e293d86c14 add metric for project history queue 2020-03-25 12:15:16 +00:00
Shane Kilkelly
ada4fba3dc Fix express deprecations 2020-03-19 15:39:57 +00:00
Eric Mc Sween
ff32104fe6 Merge pull request #123 from overleaf/em-doc-hard-delete 
Add ignore_flush_errors option to the doc delete endpoint
 2020-03-10 10:11:00 -04:00
Eric Mc Sween
d9caced0d6 Change skip_flush option to ignore_flush_errors in delete doc 
Instead of skipping the flush, we'll still try to flush and proceed with
the doc deletion, even when the flush fails.
 2020-03-10 09:40:49 -04:00
Eric Mc Sween
9b70eb75b3 Rename flush param to skip_flush in delete doc 
Also move it to the query string instead of the body.
 2020-03-09 16:27:32 -04:00
Eric Mc Sween
c09bc0e868 Add a "flush: false" option to the doc delete endpoint 
This will delete the document from Redis without flushing to web,
docstore or history. To be used when something is broken.
 2020-03-07 08:59:15 -05:00
nate stemen
ffd8d0745d use empty object for ranges if it doesn't exist 2020-03-06 13:49:30 -05:00
Brian Gough
0419039d4d Merge branch 'master' into bg-use-separate-redis-for-project-history 2020-02-21 14:13:33 +00:00
Brian Gough
338d3609f5 add comment about null byte check 2020-01-30 15:17:13 +00:00
Brian Gough
544ae05212 added note about rollback 2020-01-23 16:22:26 +00:00
Brian Gough
626e19ed1a add logging of migration phase at startup 2020-01-23 15:46:54 +00:00
Brian Gough
d5a2b96df9 add note about deleting the migration key entries 2020-01-23 14:36:59 +00:00
Brian Gough
7036803acf add missing argument to metrics.inc 
also track retries rather than attempts (which is always 1 for a successful request)
 2020-01-14 15:00:21 +00:00
Brian Gough
3caa0e7c05 add failure/retry metrics for web-api requests 2020-01-14 13:53:50 +00:00
Brian Gough
a638ef4251 add comment about locking in redis migration 2020-01-13 15:56:28 +00:00
Brian Gough
27044c2d02 allow migration phase to be modified at runtime for testing 2020-01-06 16:46:35 +00:00
Brian Gough
8ae95ebf60 fix rclient check in migration metrics 2020-01-06 16:45:36 +00:00
Brian Gough
97cbf46160 add metrics for migration 2019-12-16 11:46:35 +00:00
Brian Gough
a2e63d009e fix migration phase check 2019-12-16 09:55:26 +00:00
Brian Gough
d0c5eb5698 support migration of project history keys to separate redis instance 2019-12-13 16:38:41 +00:00
Brian Gough
ad19fee667 add setting so that double flush is the default 
can be disabled to stop flushing to track-changes
 2019-11-25 13:36:25 +00:00
Brian Gough
4f6583bbf2 fix getDocVersion and add tests 2019-11-25 13:28:36 +00:00
Brian Gough
68e12f4d2d add metrics for queue operations 2019-11-25 10:51:10 +00:00
Brian Gough
8b73bb9f13 Merge branch 'master' into bg-filter-track-changes-updates 2019-11-22 10:41:33 +00:00
Brian Gough
b7055eecee add metrics for history flushes 2019-11-22 09:14:32 +00:00
Brian Gough
65cf4cf7c7 make flush to track-changes failsafe 2019-11-21 14:58:35 +00:00
Brian Gough
dcd7649bad filter track-changes updates for projects using project-history 2019-11-19 10:02:56 +00:00
Brian Gough
d82b180b76 avoid project history queues building up with deferred flush 2019-10-03 04:05:24 +01:00
Brian Gough
c1454bc4ac Merge pull request #92 from overleaf/bg-flush-queue-prod-fixes 
add continuous background flush
 2019-10-02 13:11:00 +01:00
Brian Gough
0c14b7d2f8 add comment about background flush limit 2019-10-01 15:06:01 +01:00
Brian Gough
2845b23b70 add smoothing of delete spikes 2019-10-01 15:01:53 +01:00
Brian Gough
2c22a60052 add random jitter to cutoff time 2019-10-01 15:01:20 +01:00
Brian Gough
a32495d2b4 make background flush more adaptive 2019-10-01 14:09:41 +01:00
Brian Gough
73b4262186 add continuous background flush 2019-09-30 16:05:53 +01:00
Brian Gough
33fadf51c1 fix getDocTimestamps for multiple docs 2019-09-30 13:50:25 +01:00
Brian Gough
260923f291 keep flushQueuedProjects in the foreground 2019-09-27 10:46:24 +01:00
Brian Gough
7561e05660 check timestamps array length 2019-09-27 10:39:56 +01:00
Brian Gough
b7f3b848af remove unused dryRun option 
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
 2019-09-26 15:50:55 +01:00
Brian Gough
3bc176259b fix log line 2019-09-26 15:46:54 +01:00
Brian Gough
8cdc8c410a fix error logging 2019-09-26 15:46:45 +01:00
Brian Gough
fc62abfcfa run flush of queued projects in the background 2019-09-26 15:46:14 +01:00
Brian Gough
ba35c73cb6 add comment about ZPOPMIN 2019-09-26 15:18:10 +01:00
Brian Gough
a709a0adaa for simplicity keep the cutoff time the same while flushing the queue 2019-09-26 15:05:38 +01:00