570 commits

Author	SHA1	Message	Date
Brian Gough	781e01b8b4	Merge pull request #95 from overleaf/bg-update-ioredis ... update to ioredis 4.14.1	2019-10-14 09:21:39 +01:00
Brian Gough	18ccd11224	update to ioredis 4.14.1	2019-10-04 11:32:08 +01:00
Henry Oswald	e9efd6b93b	logs out when a redis event occurs	2019-10-03 11:10:31 +01:00
Brian Gough	d82b180b76	avoid project history queues building up with deferred flush	2019-10-03 04:05:24 +01:00
Brian Gough	c1454bc4ac	Merge pull request #92 from overleaf/bg-flush-queue-prod-fixes ... add continuous background flush	2019-10-02 13:11:00 +01:00
Brian Gough	0c14b7d2f8	add comment about background flush limit	2019-10-01 15:06:01 +01:00
Brian Gough	ae3ebf2db6	start background flush after http server has started	2019-10-01 15:02:37 +01:00
Brian Gough	2845b23b70	add smoothing of delete spikes	2019-10-01 15:01:53 +01:00
Brian Gough	2c22a60052	add random jitter to cutoff time	2019-10-01 15:01:20 +01:00
Brian Gough	a32495d2b4	make background flush more adaptive	2019-10-01 14:09:41 +01:00
Brian Gough	73b4262186	add continuous background flush	2019-09-30 16:05:53 +01:00
Brian Gough	33fadf51c1	fix getDocTimestamps for multiple docs	2019-09-30 13:50:25 +01:00
Brian Gough	89a90399fe	Merge pull request #89 from overleaf/bg-queue-deletes ... queue deletes for deferred processing	2019-09-30 11:14:08 +01:00
John Lees-Miller	c5a9105c33	Update config	2019-09-28 11:07:33 +01:00
Brian Gough	260923f291	keep flushQueuedProjects in the foreground	2019-09-27 10:46:24 +01:00
Brian Gough	7561e05660	check timestamps array length	2019-09-27 10:39:56 +01:00
Brian Gough	0f0682df43	allow flush to complete in acceptance test	2019-09-26 16:00:11 +01:00
Brian Gough	b7f3b848af	remove unused dryRun option ... Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>	2019-09-26 15:50:55 +01:00
Brian Gough	3bc176259b	fix log line	2019-09-26 15:46:54 +01:00
Brian Gough	8cdc8c410a	fix error logging	2019-09-26 15:46:45 +01:00
Brian Gough	fc62abfcfa	run flush of queued projects in the background	2019-09-26 15:46:14 +01:00
Brian Gough	ba35c73cb6	add comment about ZPOPMIN	2019-09-26 15:18:10 +01:00
Brian Gough	a709a0adaa	for simplicity keep the cutoff time the same while flushing the queue	2019-09-26 15:05:38 +01:00
Brian Gough	eae4b352ca	remove unnecessary check	2019-09-26 14:59:03 +01:00
Brian Gough	9f358ead9f	add an acceptance test for flush with queue processing	2019-09-26 10:55:05 +01:00
Brian Gough	b49621b3e9	add comments	2019-09-26 10:14:49 +01:00
Brian Gough	83dd43b809	add metric for queue length	2019-09-25 17:04:36 +01:00
Brian Gough	f6b2ac7360	queue deletes for deferred processing	2019-09-25 16:42:49 +01:00
Brian Gough	c32101f0fb	upgrade redis-sharelatex to 1.0.10	2019-09-19 16:22:43 +01:00
Brian Gough	912a3a7753	remove redis server-side hashing for performance ... we still compute the document hash in node, and check it on retrieval but we don't check the hash at the point of writing it in redis which was previously done with a redis Lua script.	2019-09-09 15:27:58 +01:00
Henry Oswald	aa15a76059	added log lines for all project flushing	2019-08-30 07:38:53 +01:00
Henry Oswald	0ae838dd2d	add logger into project flusher	2019-08-29 20:36:00 +01:00
Brian Gough	e75f2cc325	add unit tests for skipping history flush	2019-08-15 11:01:45 +01:00
Brian Gough	7493462154	add acceptance test for realtime shutdown	2019-08-15 10:38:07 +01:00
Brian Gough	a76e0dca88	skip history flush when project is cleared by realtime shutdown ... history is flushed by a background cron job anyway	2019-08-15 09:51:16 +01:00
Simon Detheridge	f49f5cbf05	Merge pull request #79 from overleaf/ho-mongo-connection-string ... use MONGO_CONNECTION_STRING in settings.defaults.coffee if set	2019-08-08 14:16:49 +01:00
Simon Detheridge	6721b904a7	Merge pull request #82 from overleaf/bg-mongo-health-check ... add a combined health check for mongo and redis	2019-08-08 14:16:38 +01:00
Simon Detheridge	06444d2cc4	Improve/fix serializers for update logging (#80) ... * Improve/fix serializers for update logging	2019-08-08 14:10:54 +01:00
Brian Gough	40f6494b19	add a combined health check for mongo and redis	2019-08-07 16:25:23 +01:00
Henry Oswald	aa1c74a979	use MONGO_CONNECTION_STRING in settings.defaults.coffee if set	2019-08-07 15:44:57 +01:00
Simon Detheridge	df9ca8b272	Add serializer to print only length of large fields in production	2019-07-31 16:42:28 +01:00
Brian Gough	618880f99d	remove unnecessary check for doc_id	2019-07-24 16:57:43 +01:00
Brian Gough	c9ccf62d71	support per-doc pubsub channels	2019-07-22 12:20:06 +01:00
Brian Gough	97487a077e	fix cluster/sentinel connection with real-time	2019-07-10 09:42:05 +01:00
Henry Oswald	06ad0f7acd	Merge pull request #75 from overleaf/ho-pubsub-connection ... Remove real time redis connection and consolidate on pubsub	2019-07-08 13:58:41 +01:00
Henry Oswald	374585f47d	Merge pull request #72 from overleaf/sk-dep-upgrades-2019-06 ... Update logger, metrics, and redis	2019-07-08 13:58:28 +01:00
Henry Oswald	3b3b2da0f5	add pubsub redis connection and remove real time redis connection	2019-07-04 13:34:31 +01:00
Henry Oswald	eed8e215d7	bump redis-sharelatex (and io redis) to 1.0.9	2019-07-02 16:58:30 +01:00
Shane Kilkelly	e8a8c446ed	Increase timeout on test?	2019-06-19 11:34:38 +01:00
Shane Kilkelly	080b482e51	Update logger, metrics, and redis ... Also fix acceptance tests, broken by a change in the redis driver behaviour. It now returns promises from most operations, which confuses mocha.	2019-06-19 10:25:54 +01:00