99 commits

Author	SHA1	Message	Date
James Allen	6ca48523d9	mapSeries -> eachSeries	2016-09-09 15:28:27 +01:00
James Allen	2d82d56f93	Process updates one at a time and write into redis after each one	2016-09-09 11:01:14 +01:00
James Allen	c85fd7ef41	Merge branch 'master' into ja-consistency-refactor	2016-09-07 11:08:04 +01:00
Brian Gough	9bc7594226	clean up redis query	2016-09-05 10:31:54 +01:00
James Allen	8779f3f686	Only write DocOps atomically with version and lines, after all docs are applied	2016-08-23 16:00:46 +01:00
James Allen	87f3e5e809	Remove old migration code	2016-08-23 11:03:37 +01:00
James Allen	a1bac9719a	Update secondary timeout for multi commands too	2016-07-12 11:45:10 +01:00
James Allen	d0c54f1be6	Increase secondary timeout to 600ms	2016-07-12 11:11:55 +01:00
James Allen	4f34cb7363	Only set keys in migration if they exist	2016-07-08 11:45:36 +01:00
James Allen	8b090c0a60	Don't call sadd with blank args	2016-07-07 16:41:38 +01:00
James Allen	6bc78ccf77	Add in some null checks on comparing backend results	2016-07-07 11:24:33 +01:00
James Allen	1c62a1c5a1	Don't propagate error on secondary timeout	2016-07-07 09:46:15 +01:00
James Allen	edf1e1ab49	Initialize primaryResult to an array, not null	2016-07-06 15:02:05 +01:00
James Allen	59883023ca	Ignore different order of results from smembers	2016-07-06 14:24:27 +01:00
James Allen	89f90c1b04	Timeout secondary requests if they take longer than 200ms	2016-07-06 12:25:36 +01:00
James Allen	508a95c19b	Use the main redis instance for locks	2016-07-06 11:50:02 +01:00
James Allen	e912ccc562	Wait for both backends to return so that they are always in sync	2016-07-05 16:07:47 +01:00
James Allen	dfd45bd23c	Add timers to time how long each redis request takes	2016-07-04 11:14:23 +01:00
James Allen	149351aa49	Remove debugging log lines	2016-06-29 21:18:13 +01:00
James Allen	fde334acc4	Create script that will ensure data is migrate between redis backends	2016-06-29 12:57:56 +01:00
James Allen	dbe03e2757	Track metrics of when backends match or disagree	2016-06-28 16:49:15 +01:00
James Allen	da89ff7172	Add in external health check rather than internal	2016-06-23 18:00:03 +01:00
James Allen	414ab5d6a9	Refactor TCP redis monitoring to be more explicit	2016-06-23 15:59:25 +01:00
James Allen	8ef03c3d2f	Add in application layer monitoring of the health of each cluster node	2016-06-23 15:38:51 +01:00
James Allen	39d920682a	Only pass through keys to redis that are explicitly set in the config	2016-06-17 14:11:22 +01:00
James Allen	40b7da8edb	Support endpoints and masterName in redis config	2016-06-17 14:05:54 +01:00
James Allen	bc00aab7b1	Explicitly separate calls to web and docupdater redis instances	2016-06-17 12:17:22 +01:00
James Allen	9ff3026807	Merge branch 'master' into ja_redis_cluster	2016-06-09 10:00:58 +01:00
James Allen	48a92b28e5	Don't run redis commands in parallel for easier consistency reasoning	2016-06-09 09:54:13 +01:00
James Allen	c823e06912	Don't run redis commands in parallel for easier consistency reasoning	2016-06-08 16:42:09 +01:00
James Allen	c6605ed5f0	Fix misapplied arguments to multi commands	2016-06-08 16:41:58 +01:00
James Allen	b4936f62af	Check that return values from different redis backends match	2016-06-08 12:18:37 +01:00
James Allen	ef43e2b325	Configure backend database as redis cluster	2016-06-07 18:38:32 +01:00
James Allen	1db6f8f159	Pull out rclient into RedisBackend that supports sending requests to multiple rclients	2016-06-07 17:58:18 +01:00
James Allen	ea8f4a6d7a	Remove unused getDocOpsLength method	2016-06-01 11:58:15 +01:00
James Allen	05b09a447e	Ensure that all multi call keys will hash to the same node in cluster	2016-06-01 11:40:12 +01:00
James Allen	f707783aba	Remove old methods of triggering doc updates ... Remove the old pub/sub listener which is no longer used. Also remove the DocsWithPendingUpdates set, which used to track docs waiting to be updated. This was necessary incase messages were missed on the pub/sub channel, so we knew which docs still had pending updates. However, now we use the BLPOP queue, these updates just sit in the queue until a consumer comes back to continue consuming them.	2016-06-01 11:28:23 +01:00
James Allen	b2e58266ae	Remove unused total doc counting redis commands	2016-06-01 11:03:39 +01:00
James Allen	32ce44f9c6	Remove unused gzipping logic	2016-06-01 10:53:50 +01:00
James Allen	11667e8250	Return semantic error if the requested ops are not available when getting a document	2016-05-31 13:24:19 +01:00
Shane Kilkelly	048fd19418	Add one to size of line. ... To account for newline characters in the original document	2016-05-12 09:26:50 +01:00
Shane Kilkelly	82d5a7fafd	check total size of lines, rather than content-length	2016-05-11 15:55:21 +01:00
Shane Kilkelly	210a61112b	reject payloads larger than 2MB for setDoc	2016-04-29 15:08:21 +01:00
James Allen	4ab4e27724	Include COUNT in lock and make RND per-process incase randomBytes blocks	2016-04-13 16:27:53 +01:00
James Allen	d15a22be26	Remove misleading log line about lock value	2016-04-13 12:20:53 +01:00
James Allen	945c728db2	Use signed locks so only the locking party can remove their lock	2016-04-13 11:59:56 +01:00
James Allen	6c79ab4321	Don't let HTTP calls take longer than 5 seconds since we're inside a 30 second lock	2016-04-12 17:10:39 +01:00
James Allen	af5d01e440	Flush track changes when unloading data from redis	2016-01-20 15:05:31 +00:00
James Allen	184c9031cd	Delete doc from redis after a full set if it wasn't in redis before	2016-01-20 14:31:25 +00:00
Brian Gough	e8f09f3357	added comment about flushing to track changes	2015-12-01 16:03:05 +00:00