344 commits

Author	SHA1	Message	Date
Brian Gough	0c14b7d2f8	add comment about background flush limit	2019-10-01 15:06:01 +01:00
Brian Gough	2845b23b70	add smoothing of delete spikes	2019-10-01 15:01:53 +01:00
Brian Gough	2c22a60052	add random jitter to cutoff time	2019-10-01 15:01:20 +01:00
Brian Gough	a32495d2b4	make background flush more adaptive	2019-10-01 14:09:41 +01:00
Brian Gough	73b4262186	add continuous background flush	2019-09-30 16:05:53 +01:00
Brian Gough	260923f291	keep flushQueuedProjects in the foreground	2019-09-27 10:46:24 +01:00
Brian Gough	7561e05660	check timestamps array length	2019-09-27 10:39:56 +01:00
Brian Gough	b7f3b848af	remove unused dryRun option ... Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>	2019-09-26 15:50:55 +01:00
Brian Gough	3bc176259b	fix log line	2019-09-26 15:46:54 +01:00
Brian Gough	8cdc8c410a	fix error logging	2019-09-26 15:46:45 +01:00
Brian Gough	fc62abfcfa	run flush of queued projects in the background	2019-09-26 15:46:14 +01:00
Brian Gough	ba35c73cb6	add comment about ZPOPMIN	2019-09-26 15:18:10 +01:00
Brian Gough	a709a0adaa	for simplicity keep the cutoff time the same while flushing the queue	2019-09-26 15:05:38 +01:00
Brian Gough	eae4b352ca	remove unnecessary check	2019-09-26 14:59:03 +01:00
Brian Gough	b49621b3e9	add comments	2019-09-26 10:14:49 +01:00
Brian Gough	83dd43b809	add metric for queue length	2019-09-25 17:04:36 +01:00
Brian Gough	f6b2ac7360	queue deletes for deferred processing	2019-09-25 16:42:49 +01:00
Brian Gough	912a3a7753	remove redis server-side hashing for performance ... we still compute the document hash in node, and check it on retrieval but we don't check the hash at the point of writing it in redis which was previously done with a redis Lua script.	2019-09-09 15:27:58 +01:00
Henry Oswald	aa15a76059	added log lines for all project flushing	2019-08-30 07:38:53 +01:00
Henry Oswald	0ae838dd2d	add logger into project flusher	2019-08-29 20:36:00 +01:00
Brian Gough	a76e0dca88	skip history flush when project is cleared by realtime shutdown ... history is flushed by a background cron job anyway	2019-08-15 09:51:16 +01:00
Simon Detheridge	6721b904a7	Merge pull request #82 from overleaf/bg-mongo-health-check ... add a combined health check for mongo and redis	2019-08-08 14:16:38 +01:00
Simon Detheridge	06444d2cc4	Improve/fix serializers for update logging (#80) ... * Improve/fix serializers for update logging	2019-08-08 14:10:54 +01:00
Brian Gough	40f6494b19	add a combined health check for mongo and redis	2019-08-07 16:25:23 +01:00
Simon Detheridge	df9ca8b272	Add serializer to print only length of large fields in production	2019-07-31 16:42:28 +01:00
Brian Gough	618880f99d	remove unnecessary check for doc_id	2019-07-24 16:57:43 +01:00
Brian Gough	c9ccf62d71	support per-doc pubsub channels	2019-07-22 12:20:06 +01:00
Brian Gough	97487a077e	fix cluster/sentinel connection with real-time	2019-07-10 09:42:05 +01:00
Henry Oswald	06ad0f7acd	Merge pull request #75 from overleaf/ho-pubsub-connection ... Remove real time redis connection and consolidate on pubsub	2019-07-08 13:58:41 +01:00
Henry Oswald	3b3b2da0f5	add pubsub redis connection and remove real time redis connection	2019-07-04 13:34:31 +01:00
Brian Gough	16fb297043	Revert "skip hash check when non-BMP characters replaced"	2019-06-27 11:39:45 +01:00
Brian Gough	f37860599d	skip hash check when non-BMP characters replaced	2019-06-25 16:36:10 +01:00
Henry Oswald	fdef197271	Merge branch 'master' into ho-detailed-flush-status	2019-06-13 14:33:22 +01:00
Henry Oswald	d9a737f97c	return failed and succesfully flushed projects when flushing everything	2019-06-13 14:21:38 +01:00
Brian Gough	e8dd1aae9c	Merge pull request #70 from overleaf/bg-metric-for-invalid-hash ... add metric for invalid hash and other sharejs errors	2019-06-12 13:50:34 +01:00
Brian Gough	d50b93df2f	add metric for invalid hash and other sharejs errors	2019-06-11 16:48:06 +01:00
Brian Gough	966478cac4	fix hash check to use 'v' field instead of version	2019-06-11 14:11:46 +01:00
Brian Gough	e95059f98e	handle non-urgent flushes in background	2019-06-03 10:01:10 +01:00
Brian Gough	0bbfa7de27	Merge branch 'master' into bg-downgrade-delete-component-error	2019-05-08 09:07:02 +01:00
Brian Gough	27a8248196	convert "Delete component" errors into warnings	2019-05-07 16:55:17 +01:00
Henry Oswald	daca83a057	add dryRun option to flush all projects	2019-05-02 16:54:22 +01:00
Henry Oswald	d5d1736a5e	adds /flush_all_projects project	2019-05-02 16:43:35 +01:00
Tim Alby	c1c23e4bee	record last author id on document flush ... This is a multi-steps process: * get a update's `user_id` from the metadata * store the `user_id` (`lastUpdatedBy`) and current date (`lastUpdatedAt`) for the document in Redis on every updates * fetch `lastUpdatedAt` and `lastUpdatedBy` from Redis on document flush * send the data to web to be persisted in Mongo	2019-05-02 11:10:02 +01:00
Brian Gough	68e7b9c4e9	Merge pull request #48 from sharelatex/bg-check-incoming-hash ... check incoming hash when present	2019-04-29 10:15:44 +01:00
James Allen	52f3596e53	Review feedback	2019-04-16 11:05:17 +01:00
James Allen	3d76f4b9bf	Record a snapshot to mongo when a doc's comments/changes get collapsed	2019-04-11 13:27:46 +01:00
Brian Gough	3c635c8d98	check version before it is modified by applyOp	2019-04-09 09:20:48 +01:00
Brian Gough	cc1f3fce5b	check incoming hash when present	2019-04-08 14:12:18 +01:00
Brian Gough	fd1425d83f	include a unique id in every message published to redis	2019-03-21 12:10:15 +00:00
Brian Gough	8c5d74faef	use explicit json content-type to avoid security issues with text/html	2019-02-12 16:45:11 +00:00