Commit graph

647 commits

Author SHA1 Message Date
Brian Gough
2845b23b70 add smoothing of delete spikes 2019-10-01 15:01:53 +01:00
Brian Gough
2c22a60052 add random jitter to cutoff time 2019-10-01 15:01:20 +01:00
Brian Gough
a32495d2b4 make background flush more adaptive 2019-10-01 14:09:41 +01:00
Brian Gough
73b4262186 add continuous background flush 2019-09-30 16:05:53 +01:00
Brian Gough
89a90399fe Merge pull request #89 from overleaf/bg-queue-deletes
queue deletes for deferred processing
2019-09-30 11:14:08 +01:00
John Lees-Miller
8b99551d9f Merge pull request #90 from overleaf/config-update
Update config
2019-09-28 16:10:28 +01:00
John Lees-Miller
c5a9105c33 Update config 2019-09-28 11:07:33 +01:00
Brian Gough
260923f291 keep flushQueuedProjects in the foreground 2019-09-27 10:46:24 +01:00
Brian Gough
7561e05660 check timestamps array length 2019-09-27 10:39:56 +01:00
Brian Gough
0f0682df43 allow flush to complete in acceptance test 2019-09-26 16:00:11 +01:00
Brian Gough
b7f3b848af remove unused dryRun option
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
2019-09-26 15:50:55 +01:00
Brian Gough
3bc176259b fix log line 2019-09-26 15:46:54 +01:00
Brian Gough
8cdc8c410a fix error logging 2019-09-26 15:46:45 +01:00
Brian Gough
fc62abfcfa run flush of queued projects in the background 2019-09-26 15:46:14 +01:00
Brian Gough
ba35c73cb6 add comment about ZPOPMIN 2019-09-26 15:18:10 +01:00
Brian Gough
a709a0adaa for simplicity keep the cutoff time the same while flushing the queue 2019-09-26 15:05:38 +01:00
Brian Gough
eae4b352ca remove unnecessary check 2019-09-26 14:59:03 +01:00
Brian Gough
9f358ead9f add an acceptance test for flush with queue processing 2019-09-26 10:55:05 +01:00
Brian Gough
b49621b3e9 add comments 2019-09-26 10:14:49 +01:00
Brian Gough
83dd43b809 add metric for queue length 2019-09-25 17:04:36 +01:00
Brian Gough
f6b2ac7360 queue deletes for deferred processing 2019-09-25 16:42:49 +01:00
Brian Gough
f05e048203 Merge pull request #88 from overleaf/bg-upgrade-redis-wrapper
upgrade redis-sharelatex to 1.0.10
2019-09-19 16:26:57 +01:00
Brian Gough
c32101f0fb upgrade redis-sharelatex to 1.0.10 2019-09-19 16:22:43 +01:00
Brian Gough
adafbb0cf9 Merge pull request #87 from overleaf/bg-remove-redis-server-hashing
remove redis server-side hashing for performance
2019-09-10 14:38:49 +01:00
Brian Gough
912a3a7753 remove redis server-side hashing for performance
we still  compute the document hash in node, and check it on retrieval
but we don't check the hash at the point of writing it in redis which
was previously done with a redis Lua script.
2019-09-09 15:27:58 +01:00
Henry Oswald
c64cae2248 Merge pull request #86 from overleaf/ho-add-logging
Add logging for all project flushing
2019-08-30 09:46:46 +01:00
Henry Oswald
aa15a76059 added log lines for all project flushing 2019-08-30 07:38:53 +01:00
Henry Oswald
0ae838dd2d add logger into project flusher 2019-08-29 20:36:00 +01:00
Brian Gough
a9ba7937c5 Merge pull request #84 from overleaf/bg-skip-history-flush-on-realtime-shutdown
skip history flush when project is cleared by realtime shutdown
2019-08-15 11:06:15 +01:00
Brian Gough
e75f2cc325 add unit tests for skipping history flush 2019-08-15 11:01:45 +01:00
Brian Gough
7493462154 add acceptance test for realtime shutdown 2019-08-15 10:38:07 +01:00
Brian Gough
a76e0dca88 skip history flush when project is cleared by realtime shutdown
history is flushed by a background cron job anyway
2019-08-15 09:51:16 +01:00
Simon Detheridge
f49f5cbf05 Merge pull request #79 from overleaf/ho-mongo-connection-string
use MONGO_CONNECTION_STRING in settings.defaults.coffee if set
2019-08-08 14:16:49 +01:00
Simon Detheridge
6721b904a7 Merge pull request #82 from overleaf/bg-mongo-health-check
add a combined health check for mongo and redis
2019-08-08 14:16:38 +01:00
Simon Detheridge
06444d2cc4 Improve/fix serializers for update logging (#80)
* Improve/fix serializers for update logging
2019-08-08 14:10:54 +01:00
Brian Gough
40f6494b19 add a combined health check for mongo and redis 2019-08-07 16:25:23 +01:00
Henry Oswald
aa1c74a979 use MONGO_CONNECTION_STRING in settings.defaults.coffee if set 2019-08-07 15:44:57 +01:00
Simon Detheridge
ad485709cf Merge pull request #78 from overleaf/spd-such-logs
Don't log document lines in log messages
2019-08-02 12:27:49 +01:00
Simon Detheridge
df9ca8b272 Add serializer to print only length of large fields in production 2019-07-31 16:42:28 +01:00
Brian Gough
2c0c3dc573 Merge pull request #77 from overleaf/bg-use-individual-channels
support per-doc pubsub channels
2019-07-30 09:35:38 +01:00
Brian Gough
618880f99d remove unnecessary check for doc_id 2019-07-24 16:57:43 +01:00
Brian Gough
c9ccf62d71 support per-doc pubsub channels 2019-07-22 12:20:06 +01:00
Brian Gough
0c8922c769 Merge pull request #76 from overleaf/bg-get-updates-from-cluster
fix cluster/sentinel connection with real-time
2019-07-10 09:53:18 +01:00
Brian Gough
97487a077e fix cluster/sentinel connection with real-time 2019-07-10 09:42:05 +01:00
Henry Oswald
06ad0f7acd Merge pull request #75 from overleaf/ho-pubsub-connection
Remove real time redis connection and consolidate on pubsub
2019-07-08 13:58:41 +01:00
Henry Oswald
374585f47d Merge pull request #72 from overleaf/sk-dep-upgrades-2019-06
Update logger, metrics, and redis
2019-07-08 13:58:28 +01:00
Henry Oswald
94b787f467 Merge pull request #71 from overleaf/ho-detailed-flush-status
return failed and succesfully flushed projects when flushing everything
2019-07-08 13:58:17 +01:00
Henry Oswald
3b3b2da0f5 add pubsub redis connection and remove real time redis connection 2019-07-04 13:34:31 +01:00
Henry Oswald
a42aa45a36 Merge branch 'sk-dep-upgrades-2019-06' into ho-pubsub-connection 2019-07-03 10:18:13 +01:00
Henry Oswald
eed8e215d7 bump redis-sharelatex (and io redis) to 1.0.9 2019-07-02 16:58:30 +01:00