360 commits

Author	SHA1	Message	Date
Eric Mc Sween	d9caced0d6	Change skip_flush option to ignore_flush_errors in delete doc ... Instead of skipping the flush, we'll still try to flush and proceed with the doc deletion, even when the flush fails.	2020-03-10 09:40:49 -04:00
Eric Mc Sween	9b70eb75b3	Rename flush param to skip_flush in delete doc ... Also move it to the query string instead of the body.	2020-03-09 16:27:32 -04:00
Eric Mc Sween	c09bc0e868	Add a "flush: false" option to the doc delete endpoint ... This will delete the document from Redis without flushing to web, docstore or history. To be used when something is broken.	2020-03-07 08:59:15 -05:00
Brian Gough	338d3609f5	add comment about null byte check	2020-01-30 15:17:13 +00:00
Brian Gough	7036803acf	add missing argument to metrics.inc ... also track retries rather than attempts (which is always 1 for a successful request)	2020-01-14 15:00:21 +00:00
Brian Gough	3caa0e7c05	add failure/retry metrics for web-api requests	2020-01-14 13:53:50 +00:00
Brian Gough	ad19fee667	add setting so that double flush is the default ... can be disabled to stop flushing to track-changes	2019-11-25 13:36:25 +00:00
Brian Gough	4f6583bbf2	fix getDocVersion and add tests	2019-11-25 13:28:36 +00:00
Brian Gough	68e12f4d2d	add metrics for queue operations	2019-11-25 10:51:10 +00:00
Brian Gough	8b73bb9f13	Merge branch 'master' into bg-filter-track-changes-updates	2019-11-22 10:41:33 +00:00
Brian Gough	b7055eecee	add metrics for history flushes	2019-11-22 09:14:32 +00:00
Brian Gough	65cf4cf7c7	make flush to track-changes failsafe	2019-11-21 14:58:35 +00:00
Brian Gough	dcd7649bad	filter track-changes updates for projects using project-history	2019-11-19 10:02:56 +00:00
Brian Gough	d82b180b76	avoid project history queues building up with deferred flush	2019-10-03 04:05:24 +01:00
Brian Gough	c1454bc4ac	Merge pull request #92 from overleaf/bg-flush-queue-prod-fixes ... add continuous background flush	2019-10-02 13:11:00 +01:00
Brian Gough	0c14b7d2f8	add comment about background flush limit	2019-10-01 15:06:01 +01:00
Brian Gough	2845b23b70	add smoothing of delete spikes	2019-10-01 15:01:53 +01:00
Brian Gough	2c22a60052	add random jitter to cutoff time	2019-10-01 15:01:20 +01:00
Brian Gough	a32495d2b4	make background flush more adaptive	2019-10-01 14:09:41 +01:00
Brian Gough	73b4262186	add continuous background flush	2019-09-30 16:05:53 +01:00
Brian Gough	33fadf51c1	fix getDocTimestamps for multiple docs	2019-09-30 13:50:25 +01:00
Brian Gough	260923f291	keep flushQueuedProjects in the foreground	2019-09-27 10:46:24 +01:00
Brian Gough	7561e05660	check timestamps array length	2019-09-27 10:39:56 +01:00
Brian Gough	b7f3b848af	remove unused dryRun option ... Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>	2019-09-26 15:50:55 +01:00
Brian Gough	3bc176259b	fix log line	2019-09-26 15:46:54 +01:00
Brian Gough	8cdc8c410a	fix error logging	2019-09-26 15:46:45 +01:00
Brian Gough	fc62abfcfa	run flush of queued projects in the background	2019-09-26 15:46:14 +01:00
Brian Gough	ba35c73cb6	add comment about ZPOPMIN	2019-09-26 15:18:10 +01:00
Brian Gough	a709a0adaa	for simplicity keep the cutoff time the same while flushing the queue	2019-09-26 15:05:38 +01:00
Brian Gough	eae4b352ca	remove unnecessary check	2019-09-26 14:59:03 +01:00
Brian Gough	b49621b3e9	add comments	2019-09-26 10:14:49 +01:00
Brian Gough	83dd43b809	add metric for queue length	2019-09-25 17:04:36 +01:00
Brian Gough	f6b2ac7360	queue deletes for deferred processing	2019-09-25 16:42:49 +01:00
Brian Gough	912a3a7753	remove redis server-side hashing for performance ... we still compute the document hash in node, and check it on retrieval but we don't check the hash at the point of writing it in redis which was previously done with a redis Lua script.	2019-09-09 15:27:58 +01:00
Henry Oswald	aa15a76059	added log lines for all project flushing	2019-08-30 07:38:53 +01:00
Henry Oswald	0ae838dd2d	add logger into project flusher	2019-08-29 20:36:00 +01:00
Brian Gough	a76e0dca88	skip history flush when project is cleared by realtime shutdown ... history is flushed by a background cron job anyway	2019-08-15 09:51:16 +01:00
Simon Detheridge	6721b904a7	Merge pull request #82 from overleaf/bg-mongo-health-check ... add a combined health check for mongo and redis	2019-08-08 14:16:38 +01:00
Simon Detheridge	06444d2cc4	Improve/fix serializers for update logging (#80) ... * Improve/fix serializers for update logging	2019-08-08 14:10:54 +01:00
Brian Gough	40f6494b19	add a combined health check for mongo and redis	2019-08-07 16:25:23 +01:00
Simon Detheridge	df9ca8b272	Add serializer to print only length of large fields in production	2019-07-31 16:42:28 +01:00
Brian Gough	618880f99d	remove unnecessary check for doc_id	2019-07-24 16:57:43 +01:00
Brian Gough	c9ccf62d71	support per-doc pubsub channels	2019-07-22 12:20:06 +01:00
Brian Gough	97487a077e	fix cluster/sentinel connection with real-time	2019-07-10 09:42:05 +01:00
Henry Oswald	06ad0f7acd	Merge pull request #75 from overleaf/ho-pubsub-connection ... Remove real time redis connection and consolidate on pubsub	2019-07-08 13:58:41 +01:00
Henry Oswald	3b3b2da0f5	add pubsub redis connection and remove real time redis connection	2019-07-04 13:34:31 +01:00
Brian Gough	16fb297043	Revert "skip hash check when non-BMP characters replaced"	2019-06-27 11:39:45 +01:00
Brian Gough	f37860599d	skip hash check when non-BMP characters replaced	2019-06-25 16:36:10 +01:00
Henry Oswald	fdef197271	Merge branch 'master' into ho-detailed-flush-status	2019-06-13 14:33:22 +01:00
Henry Oswald	d9a737f97c	return failed and succesfully flushed projects when flushing everything	2019-06-13 14:21:38 +01:00