447 commits

Author	SHA1	Message	Date
Brian Gough	fd1425d83f	include a unique id in every message published to redis	2019-03-21 12:10:15 +00:00
Brian Gough	8c5d74faef	use explicit json content-type to avoid security issues with text/html	2019-02-12 16:45:11 +00:00
Henry Oswald	3bc4cb492a	added log line	2019-02-07 16:30:53 +00:00
Henry Oswald	ecaef6485b	revert the removal of realtime keyspace	2019-02-07 15:27:51 +00:00
Henry Oswald	4e1a2c787c	Revert "turn down logging, use logger.info for less important data" ... This reverts commit c5f91428e3c7702fbbd3ffd1ef7a772d513f33f2.	2019-02-06 15:29:22 +00:00
Christopher Hoskin	1217d8a80a	Merge branch 'master' into csh-ho-docker-issue-1338-bulk-upgrade	2019-01-04 09:18:40 +00:00
Henry Oswald	2505be283a	turn down logging, use logger.info for less important data	2018-12-13 12:46:10 +00:00
Henry Oswald	25e1ad7491	remove redis.realtime ... real time pushes data into the document updater redis instance	2018-12-05 16:17:41 +00:00
Brian Gough	aae1352519	ensure that project history is flushed when the project is deleted	2018-10-30 12:03:44 +00:00
Brian Gough	537bd9a0c0	Merge pull request #36 from sharelatex/bg-limit-resync-requests-to-web ... limit resync requests to web (connects to overleaf/sharelatex#775)	2018-08-16 12:25:16 +01:00
Brian Gough	910b27357d	add more logging to resync operations	2018-08-16 11:14:11 +01:00
Brian Gough	aa013f0bee	limit parallel resync doc requests to web	2018-08-16 11:13:11 +01:00
Brian Gough	fdcb806518	set a timestamp for the first entry in the projectHistory:Ops queue	2018-07-24 09:17:31 +01:00
Brian Gough	e471730efb	Revert "Merge pull request #33 from sharelatex/bg-add-timestamp-marker-to-project-history-queue" ... This reverts commit 8ec398b133aa75c45384dd7ceb630cd743f4a15c, reversing changes made to e3a6c934cf53fd245d7d0df737463cc90f954885.	2018-07-23 16:05:04 +01:00
Brian Gough	6237577f87	set a timestamp for the first entry in the projectHistory:Ops queue	2018-07-20 10:43:31 +01:00
Brian Gough	7e526395fb	fix check for "delete component" message	2018-04-30 09:56:01 +01:00
Brian Gough	37f431f148	also block "delete component" messages from sentry	2018-04-27 16:03:46 +01:00
Brian Gough	64670f272a	stop OpRangeNotAvailable errors going to sentry	2018-04-27 15:45:28 +01:00
Hayden Faulds	af92ca70a1	coerce projectHistoryId to integer after reading from Redis	2018-04-23 15:19:06 +01:00
Hayden Faulds	1b63141e49	Revert "Revert "Add projectHistoryId to updates""	2018-04-23 12:08:04 +01:00
Hayden Faulds	3cc20d6311	Revert "Add projectHistoryId to updates"	2018-04-23 11:41:29 +01:00
Hayden Faulds	25c0be4eaa	add projectHistryId to project structure updates	2018-04-13 15:22:05 +01:00
Hayden Faulds	a220794d32	add projectHistoryId to resync updates	2018-04-13 15:22:05 +01:00
Hayden Faulds	fb1852a593	cache projectHistoryId with doc in Redis	2018-04-13 15:22:05 +01:00
Brian Gough	dd0f8b880a	change update to projectUpdate in project related methods	2018-03-16 10:54:12 +00:00
Brian Gough	3385d2640a	fix structure ordering bug	2018-03-15 16:27:10 +00:00
Brian Gough	0642e3c8c9	support project version on incoming requests	2018-03-15 16:15:10 +00:00
Hayden Faulds	4a5731440a	Merge pull request #21 from sharelatex/hof-history-sync ... resync project history endpoint	2018-03-14 10:40:59 +00:00
Hayden Faulds	1226f96fc3	extract ProjectHistoryRedisManager	2018-03-09 15:36:38 +00:00
Hayden Faulds	fc6ef6ea7a	improve logging	2018-03-09 12:21:48 +00:00
Hayden Faulds	088ab6e7ba	rename queueResyncDocContents -> queueResyncDocContent	2018-03-09 12:21:37 +00:00
Hayden Faulds	64d1fe13bc	additional logging for project history resync	2018-03-07 16:24:22 +00:00
Hayden Faulds	152a25e96d	rename resyncProject -> resyncProjectHistory	2018-03-07 13:29:42 +00:00
Hayden Faulds	d46217be52	queue doc content sync updates	2018-03-02 15:02:27 +00:00
Hayden Faulds	52d6b710bd	add ability to resync project structure	2018-03-02 15:01:53 +00:00
James Allen	939eaa2d4b	Don't allow a document to be loaded without a pathname	2018-03-02 10:02:49 +00:00
Hayden Faulds	5a11332aa3	remove unecessary method wrapping	2018-02-22 10:17:59 +00:00
Hayden Faulds	0f87ae1f74	simplify comments	2018-02-22 10:16:29 +00:00
Hayden Faulds	ea0dd9700b	reduce the number of times we flush project history	2018-02-22 10:01:05 +00:00
Michael Walker	241d1b27d5	Remove _shouldFlushHistoryOps wrapper	2018-01-31 11:27:40 +00:00
Michael Walker	d2865f0650	Correctly count project_ops_length	2018-01-26 14:41:28 +00:00
Michael Walker	3028fb9c3d	Only flush project updates when crossing the threshold	2018-01-24 11:57:03 +00:00
Michael Walker	24c74db0dc	Flush project changes when queue hits limit	2018-01-23 11:18:20 +00:00
Michael Walker	3d313ebc53	Add method to get number of queued project updates	2018-01-23 11:10:54 +00:00
James Allen	ccf6cb1a18	Provide hosts as environment settings, add npm run start script and update coffeescript	2017-12-29 08:13:16 +00:00
Brian Gough	0b583dd4e6	fix log message	2017-12-20 13:27:10 +00:00
Brian Gough	a8b14552c0	allow flush to succeed when doc is already deleted	2017-12-19 12:27:57 +00:00
Brian Gough	947e18eccf	Merge branch 'bg-limit-ops-in-lock'	2017-12-14 14:26:44 +00:00
James Allen	6abf9c227d	Calculate and send doc length before each update	2017-12-11 17:26:32 +00:00
Hayden Faulds	bd45fcb5f8	Merge pull request #12 from sharelatex/hof-version-entity-creation ... version entity creation; Connects to sharelatex/web-sharelatex-internal#104	2017-11-24 10:02:51 +00:00
Brian Gough	fb572670cd	Merge pull request #11 from sharelatex/bg-use-separate-flush-count-for-projects ... use a separate parameter for doc and project flush	2017-11-21 13:54:03 +00:00
Brian Gough	0bb9f147aa	fix whitespace	2017-11-13 11:56:08 +00:00
Brian Gough	a4bbf5cf52	use a separate parameter for doc and project flush	2017-11-13 11:53:39 +00:00
Hayden Faulds	a3420b1236	version entity additions	2017-11-10 14:54:56 +00:00
Hayden Faulds	944e633bac	add acceptance test for entity renaming	2017-11-08 09:20:38 +00:00
Hayden Faulds	7e86afe55e	version file renames	2017-11-08 09:20:38 +00:00
Hayden Faulds	6d571e6d23	version document renames	2017-11-08 09:20:37 +00:00
Hayden Faulds	b8052e7612	allow settings.redis.project_history to be undefined	2017-10-25 13:56:38 +01:00
Hayden Faulds	5b7be93c79	Merge pull request #2 from sharelatex/hof-doc-pathname ... Add metadata to updates for history	2017-10-25 13:13:00 +01:00
Brian Gough	20655b6eeb	remove unused requires of async	2017-10-23 15:59:32 +01:00
Hayden Faulds	be41a1614d	decorate ops with pathname and doc_length	2017-10-23 15:32:25 +01:00
Hayden Faulds	962520fca8	flush track-changes and project-history in HistoryManager	2017-10-23 15:32:25 +01:00
Hayden Faulds	d003aef31c	conditionally enqueue history updates for project	2017-10-23 15:32:25 +01:00
Hayden Faulds	81f998afe9	return pathname from DocumentManager.getDoc	2017-10-23 14:23:21 +01:00
Hayden Faulds	748315aadc	handle pathname in DocumentManager.getDoc	2017-10-23 14:23:21 +01:00
Hayden Faulds	cbdace7386	store pathname in Redis cache	2017-10-23 14:23:21 +01:00
Hayden Faulds	05b93a629a	return pathname from PersistenceManager	2017-10-23 14:17:14 +01:00
Brian Gough	7cbb3e7af8	change getProjectDocs endpoint from GET to POST ... also note that it flushes docs if they are considered old (i.e. not recently flushed)	2017-10-11 15:30:29 +01:00
Brian Gough	8d3f82360c	update DocumentManager.getDoc signature ... keep alreadyLoaded flag at the end for consistency	2017-10-11 14:25:33 +01:00
Brian Gough	2bbbf3c005	add unflushed time to doc in redis	2017-10-06 12:34:28 +01:00
Brian Gough	ad745f133d	Merge pull request #59 from sharelatex/bg-reset-project-state ... add endpoint to clear project state	2017-09-13 10:39:47 +01:00
Brian Gough	f34c12fdc7	add endpoint to clear project state	2017-09-08 15:50:26 +01:00
Brian Gough	215a939d4e	add comment about log.error for excludeVersions	2017-09-08 13:43:22 +01:00
Brian Gough	1e54b4c72c	log all errors in getProjectDocs	2017-09-07 14:12:48 +01:00
Brian Gough	8372911a1b	return doclines as array from getProjectDocs ... for compatibility with getDoc	2017-08-11 16:56:30 +01:00
Brian Gough	f696ccb0d9	add unit tests for getProjectDocs	2017-08-10 16:41:28 +01:00
Brian Gough	1eb80936f2	fix log line	2017-08-10 14:57:27 +01:00
Brian Gough	47bebf9638	no need to support optional keys.projectState	2017-08-09 16:47:09 +01:00
Brian Gough	bd6b5b2c59	use v instead of rev in getProjectDocs	2017-08-09 15:30:53 +01:00
Brian Gough	9f3ec72f81	switch to single get/set method for getProjectDocs ... if project state hasn't changed, return the docs. Otherwise set the hash and return a 409 Conflict response.	2017-08-07 14:43:28 +01:00
Brian Gough	f5f516a910	delete clsi state when deleting doc (if used)	2017-08-03 15:04:12 +01:00
Brian Gough	860537146b	avoid logging doclines in getProjectDocs	2017-08-03 15:04:12 +01:00
Brian Gough	80be5adc0d	change filter to exclude in getProjectDocs	2017-08-03 15:04:06 +01:00
Brian Gough	0109585092	provide endpoint for current project docs in redis	2017-08-02 13:42:10 +01:00
Shane Kilkelly	c3465cf3d4	Reduce retry delay to 10ms, and refactor	2017-07-18 15:28:18 +01:00
Shane Kilkelly	0d46c3e2b3	WIP: auto-retry web requests	2017-07-17 10:08:21 +01:00
Brian Gough	c21cc013f3	use null instead of err in callbacks on success	2017-07-12 10:48:32 +01:00
Brian Gough	bb0dc4b4d5	release the lock if it took too long to acquire it	2017-07-12 10:48:32 +01:00
Brian Gough	54c0fc1180	put a 5 second timeout in getting redis lock	2017-07-11 14:28:58 +01:00
Brian Gough	ffea31290c	return total time from the profile .end() method	2017-07-11 14:28:58 +01:00
Brian Gough	59ace9d5c5	avoid long lines in timeout check	2017-06-26 16:36:05 +01:00
Brian Gough	963e513057	add a timeout for getPreviousDocOps ... it uses several redis operations and this makes it prone to subsequent timeouts if getDoc succeeds but is slow	2017-06-23 15:50:21 +01:00
James Allen	54f128289f	Merge pull request #50 from sharelatex/bg-metric-for-unlock-errors ... add metric for unlock errors	2017-06-13 14:46:40 +01:00
James Allen	7d22e4422c	Merge pull request #49 from sharelatex/bg-add-redis-timeout ... add redis timeout	2017-06-13 14:46:32 +01:00
James Allen	a3d7260619	Ensure expires is called after the key is created	2017-06-08 16:43:06 +01:00
Brian Gough	2e5d57cd99	don't increase rate limit when tasks are failing	2017-06-07 11:47:28 +01:00
Brian Gough	1ef258c878	add a timeout on the redis getDoc request	2017-06-07 11:34:42 +01:00
Brian Gough	673f4228ce	add metric for unlock errors ... sentry does not record them reliably, due to rate limiting	2017-06-05 16:29:58 +01:00
Brian Gough	12e8eaa9b6	fix bug in doclines check	2017-05-31 16:08:33 +01:00
Brian Gough	bed64d26c8	check if doc is missing from DocsIn set	2017-05-31 15:33:59 +01:00
Brian Gough	fa089dc28d	avoid graphite drift in active worker count	2017-05-30 14:37:01 +01:00
Brian Gough	891ffda3bf	put a rate limit on worker calls	2017-05-26 15:08:35 +01:00
Brian Gough	c43f97e8e3	add metric for active workers	2017-05-24 12:12:15 +01:00
Brian Gough	a3947c587f	fix bug, jsonOps is array not string	2017-05-22 15:20:28 +01:00
Brian Gough	54ad45c870	add null byte check for ranges	2017-05-22 15:00:05 +01:00
Brian Gough	a13f055d62	add null byte check for ops	2017-05-22 15:00:00 +01:00
Brian Gough	78828ebbc5	fine grained logging for cpu usage in applyUpdate	2017-05-19 16:00:16 +01:00
Brian Gough	a550078071	replace verbose logging by slow query log	2017-05-18 15:04:12 +01:00
Brian Gough	95596061d4	clarify calculation of process.hrtime	2017-05-18 15:02:54 +01:00
Brian Gough	e00f4dde8e	log profilers start and end times for reference	2017-05-18 15:02:44 +01:00
Brian Gough	e11f64f83a	increase profiling cutoff to 1s	2017-05-18 15:02:08 +01:00
Brian Gough	b6efb051e1	add profiling	2017-05-18 11:17:08 +01:00
Brian Gough	387ecac6cb	log for each get/try/release lock	2017-05-18 11:17:08 +01:00
Brian Gough	dff6e2c3da	bug fix in backoff	2017-05-16 16:31:28 +01:00
Brian Gough	c7d8fbbb8a	exponential backoff for lock up to maximum time	2017-05-16 15:22:02 +01:00
Brian Gough	b15d2ef796	Revert "fail safely if doc cannot be loaded"	2017-05-15 16:18:40 +01:00
Paulo Jorge Reis	2c3f0aa6b4	Merge pull request #34 from sharelatex/pr-bulk-actions ... Add methods to bulk accept changes.	2017-05-15 15:14:33 +01:00
Paulo Reis	ef7fa59257	Update ranges tracker.	2017-05-15 11:11:14 +01:00
Paulo Reis	05d7d1b8c6	Merge branch 'master' into pr-bulk-actions	2017-05-15 10:35:16 +01:00
Brian Gough	2ee40d0748	ensure document is in redis before consuming ops	2017-05-12 14:22:18 +01:00
Brian Gough	be96548199	log number of updates for future debugging	2017-05-12 14:07:59 +01:00
Brian Gough	d4a8d88750	put a limit on the number of ops per iteration	2017-05-12 13:15:59 +01:00
Paulo Reis	5677c7ad9a	Update range tracker.	2017-05-11 11:31:19 +01:00
Brian Gough	36407ac726	rename HistoryManager pushUncompressedHistoryOps	2017-05-09 12:02:27 +01:00
Brian Gough	7ce6285e3d	increase flush threshold to 100 ops	2017-05-09 10:44:26 +01:00
Brian Gough	fdf5e8e0b8	get history ops length directly from redis update	2017-05-09 10:34:31 +01:00
Brian Gough	2d158b03d7	rename pushUncompressedHistoryOps	2017-05-09 09:32:56 +01:00
Paulo Reis	144804e516	Update tests with new methods.	2017-05-08 16:34:17 +01:00
Paulo Reis	d7ac57d054	Use new bulk method.	2017-05-08 16:08:42 +01:00
Paulo Reis	70341e4e80	Add bulk method from range tracker.	2017-05-08 16:08:34 +01:00
Brian Gough	79d8fced49	make history update more atomic	2017-05-08 15:58:29 +01:00
Paulo Reis	e96e6d6a29	Remove unused method.	2017-05-05 15:17:19 +01:00
Paulo Reis	9b8a2c3030	Consolidate methods to accept changes.	2017-05-05 15:12:06 +01:00
Paulo Reis	d530ee9501	Add methods to bulk accept changes.	2017-05-04 15:32:54 +01:00
James Allen	4104ca4889	Add in separate redis config for the lock and fix a few web -> realtime	2017-05-03 16:27:32 +01:00
James Allen	ed493d8ad3	Rename web -> realtime for consistency with realtime	2017-05-02 15:38:33 +01:00
James Allen	c5449ae282	Split out redis config for real-time and track-changes into separate cluster-compatible configs	2017-04-13 17:00:42 +01:00
James Allen	f21208e841	Use new redis-sharelatex instead of RedisBackend for cluster abstraction	2017-04-12 14:53:03 +01:00
Brian Gough	5583764d05	fix callback prototype in getLock	2017-03-30 15:31:47 +01:00
Brian Gough	c11618b475	improve unlock error handling	2017-03-30 15:31:34 +01:00
Brian Gough	527c02a00b	log op versions pushed to track changes	2017-03-30 11:20:50 +01:00
James Allen	0245bfd031	Support an incoming undo flag for history restores	2017-03-21 11:20:38 +00:00
James Allen	199079a5b7	Merge branch 'master' into ja-cut-and-paste-comments	2017-03-20 10:29:01 +00:00
James Allen	9c5299ec7c	Update RangeTracker to support upserting and moving comments for cut and paste	2017-03-16 15:49:41 +00:00
James Allen	5499a67d78	Add in a consistency check after applying updates that ranges still match	2017-03-15 14:12:06 +00:00
James Allen	b9e4492001	Wrap RangeTracker in try catch	2017-03-06 19:05:11 +00:00
James Allen	d8135749d2	Merge branch 'master' into ja-increase-limits-on-changes ... Conflicts: app/coffee/RedisManager.coffee	2017-03-06 11:57:40 +00:00
James Allen	269433f250	Merge pull request #25 from sharelatex/bg-log-doclines-on-hash-mismatch ... log doclines on hash mismatch	2017-03-06 11:53:24 +00:00
Brian Gough	d086e0b61b	log doclines on hash mismatch	2017-03-03 15:57:44 +00:00
Brian Gough	b186a01c04	don't log errors from redis backend ... this also picks up errors from RedisManager like "doc ops range is not loaded in redis"	2017-03-03 15:27:42 +00:00
James Allen	d56bb55953	Revert PR #19	2017-03-01 16:49:46 +00:00
James Allen	9cfe651930	Merge pull request #19 from sharelatex/ja-track-changes ... Don't consume partial parts of delete when inserting a change	2017-03-01 10:20:06 +01:00
James Allen	f544814dda	Add in an absolute size limit on the ranges JSON object	2017-02-27 14:34:20 +01:00
James Allen	f211c282b2	Merge branch 'master' into ja-increase-limits-on-changes	2017-02-27 13:50:42 +01:00
James Allen	dd45a8dd7d	Improve logging around hashes and null byte errors	2017-02-27 10:06:24 +01:00
James Allen	60f3393c44	Do a naive check for null bytes after JSON.stringify on doc lines	2017-02-27 09:55:04 +01:00
James Allen	fd3dee0af5	Increase changes count limit to 2000	2017-02-24 14:58:47 +01:00
James Allen	80284e1b01	Only cancel deletes with inserts on undo and reject	2017-02-24 14:21:06 +01:00
James Allen	0420929e2a	Reduce the data logged when updating ranges	2017-02-23 13:57:47 +01:00
James Allen	e1481df76a	Don't consume partial parts of delete when inserting a change	2017-02-23 11:39:19 +01:00
James Allen	d6b5cc6389	Increase changes limit to 1200	2017-02-22 18:11:55 +01:00
Brian Gough	168bab6960	make hash error logging fine grained	2017-02-22 14:32:51 +00:00
Brian Gough	19e7522d3f	make logging of hash errors optional	2017-02-22 13:46:34 +00:00
Brian Gough	8d1b11f37a	Merge branch 'bg-verify-writes-II'	2017-02-21 16:03:48 +00:00
Brian Gough	5eb339e525	fix utf8 encoding in sha1 hash	2017-02-21 16:03:06 +00:00
Brian Gough	6b808522dc	Merge pull request #18 from sharelatex/bg-verify-writes ... merge redis hash check	2017-02-20 15:55:19 +00:00
Brian Gough	590f8e7ced	Merge pull request #15 from sharelatex/bg-verify-writes ... store sha1 hash of docLines in redis	2017-02-20 15:53:56 +00:00
Brian Gough	c57d3ce31c	compute hash on write in redis server	2017-02-20 14:09:09 +00:00
James Allen	660752dfa6	Increase change limit to 1000	2017-02-17 15:32:44 +01:00
James Allen	89537927b5	Update Ranges Tracker	2017-02-17 09:24:19 +01:00
James Allen	de7f760ea4	Update Ranges Tracker	2017-02-16 17:09:08 +01:00
Brian Gough	b5a4458b68	check sha1 hash value only if present	2017-02-15 14:12:36 +00:00
Brian Gough	bd70aaa76c	add sha1 hash support on writes	2017-02-14 16:11:43 +00:00
James Allen	8725aeab03	Allow deleting of comment ranges	2017-01-24 15:57:11 +01:00
James Allen	dc976fd9e9	Use thread id for comment id	2017-01-23 09:45:46 +01:00
James Allen	0705242a55	Increase max op age to 80	2017-01-17 11:45:10 +01:00
James Allen	0706feb26b	Add max limit on number of comments and changes per doc	2017-01-10 16:58:11 +01:00
James Allen	a3d5971d54	Update RangesTRacker	2017-01-10 11:59:09 +01:00
James Allen	540d0f7ec7	Make sure comment ids are unique	2017-01-10 11:55:38 +01:00
James Allen	be19532a1d	Add HTTP end point for accepting changes	2017-01-09 14:41:18 +01:00
James Allen	593e7260d4	Update RangesTracker	2017-01-09 10:52:06 +01:00
James Allen	7cac2f7d76	Generate deterministic range ids based on seed	2017-01-09 10:46:58 +01:00
James Allen	2c7029cc50	Revert "Support a {dr:...} op for deleting ranges" ... This reverts commit 24c58e5ad430e0240533cc1e5c21122859fe8dc9.	2017-01-09 09:24:19 +01:00
James Allen	0f13cb3aa7	Support a {dr:...} op for deleting ranges	2017-01-06 16:58:51 +01:00
James Allen	59a06cd798	Accept comments with thread id as an op type	2016-12-13 15:51:47 +00:00
James Allen	47b19818ff	Add in new comment op type	2016-12-12 17:53:43 +00:00
James Allen	e3fee1a1d1	Rename 'track changes entries' -> 'ranges'	2016-12-08 12:31:43 +00:00
James Allen	3ea2e07993	Add tests for fetching and flushing track changes entries to persistence layer	2016-12-02 12:01:23 +00:00
James Allen	418405e8b9	Return track changes entries in HTTP request	2016-12-02 11:37:27 +00:00
James Allen	4fadd75ef3	Track changes based on flag on op, not global setting	2016-12-02 11:04:21 +00:00
James Allen	889f5fdf9f	Fix ShareJsDB tests	2016-12-01 17:14:40 +00:00
James Allen	9ee913be39	Update PersistenceManagerTests	2016-12-01 16:40:15 +00:00
James Allen	d36f1a86f1	Merge branch 'master' into ja-track-changes ... Conflicts: app/coffee/PersistenceManager.coffee	2016-12-01 16:06:13 +00:00
James Allen	efab68e6b2	Add in guards on bad data from web API	2016-11-29 17:13:16 +00:00
James Allen	2852043a6d	Don't store doc version in Mongo directly, instead use docstore	2016-11-29 17:06:23 +00:00
James Allen	e739e86c48	Get basic ChangeTracker hooked up. WIP	2016-11-28 10:14:42 +00:00
James Allen	5ce15c4d60	Move check of zero length op array up a level	2016-09-12 10:37:41 +01:00
James Allen	993aab7a78	Don't try to redis rpush with no arguments	2016-09-12 10:37:32 +01:00
James Allen	6ca48523d9	mapSeries -> eachSeries	2016-09-09 15:28:27 +01:00
James Allen	2d82d56f93	Process updates one at a time and write into redis after each one	2016-09-09 11:01:14 +01:00