299 commits

Author	SHA1	Message	Date
Brian Gough	10932eb4a9	remove debug logs	2016-03-10 15:20:32 +00:00
Brian Gough	5ea7a31ad9	update archiving acceptance tests	2016-03-10 15:15:57 +00:00
Brian Gough	98738d1344	fix for acceptance test	2016-03-10 15:15:29 +00:00
Brian Gough	f01bf99682	acceptance tests - work in progress	2016-03-09 16:56:49 +00:00
Brian Gough	e2e8292590	update tests	2016-03-09 15:24:28 +00:00
Brian Gough	6d43273f04	working on packmanager tests	2016-03-09 14:44:59 +00:00
Brian Gough	f6367e21b8	give separate error for archive in progress vs completed	2016-03-09 14:44:59 +00:00
Brian Gough	7350ab531d	exclude already cached packs from archival	2016-03-09 14:44:59 +00:00
Brian Gough	28b184e0ca	fix incorrect use of _.union (argument must be array)	2016-03-09 14:44:59 +00:00
Brian Gough	8922b97bd7	avoid duplicate filling of UserInfo in getDocUpdates	2016-03-09 14:44:59 +00:00
Brian Gough	bfc9494cc1	update UpdatesManager tests	2016-03-09 14:44:59 +00:00
Brian Gough	4e63c50a75	update MongoManager tests	2016-03-09 14:44:59 +00:00
Brian Gough	b0d2e22c58	update config file for changes s3 settings	2016-03-09 13:28:22 +00:00
Brian Gough	7e6ea2793b	remove startup dependency on s3 settings	2016-03-09 13:28:02 +00:00
Brian Gough	b780df6801	removed unused tests	2016-03-04 15:47:59 +00:00
Brian Gough	4d58c14573	updated MongoAWS tests	2016-03-04 15:47:47 +00:00
Brian Gough	1419d20b1f	fix indentation	2016-03-04 15:43:32 +00:00
Brian Gough	3175f6d3a6	handle case where index does not exist	2016-03-03 14:36:16 +00:00
Henry Oswald	e8b3fb5be6	added more logging to failed health checks	2016-03-03 10:50:55 +00:00
Brian Gough	795f717bab	added index definitions	2016-03-01 11:38:23 +00:00
Brian Gough	ebd5628e53	temporary workaround to get tests passing ... temporary workaround to get tests passing	2016-03-01 10:10:12 +00:00
Brian Gough	3d9dfeccc3	remove pack worker ... remove the op-specific code remove tests for ops, now only packing remove unused packing code work in progress store index for completed packs only support archiving and unarchiving of individual packs remove support for archiving whole document history split out ArchiveManager, IndexManager remove old DocArchive code remove docHistoryStats collection comment about archiving added method to look at index when last pack has been archived added start of iterator for project results use a proper iterator added heap module getting it working increase pack size since bulk operations no longer needed remove unused MongoAWSexternal cleanup added doc iterator remove old query code added missing files cleanup clean upclean up started adding pack worker for archiving work in progress work in progress getting pack worker working updating worker getting packworker working added lock use correct key name for track changes aws access use correct key name for track changes aws access always send back users array fix up comparison of retrieved objects handle op ids inside packs log when s3 download completes comments cleanup, remove finalisation ideacleanup, remove finalisation idea remove logging	2016-03-01 10:10:02 +00:00
Brian Gough	a23ddf31c0	allow packing of temporary ops	2016-01-29 12:36:03 +00:00
Brian Gough	612079ef8a	started adding more pack tests and packing of temp ops	2016-01-28 16:56:32 +00:00
Brian Gough	77cafa36af	support continuing from last packed doc	2016-01-28 16:40:20 +00:00
Brian - Work	666a07e5ba	move lock check into HealthChecker ... to avoid dependency of HttpController on LockManager in unit tests	2016-01-27 16:04:55 +00:00
Brian - Work	64545d6540	improve packing script	2016-01-27 15:48:36 +00:00
Brian Gough	199d2aaa92	script to pack existing docs	2016-01-27 15:14:23 +00:00
Brian Gough	ba0de92081	improve logging by avoiding string escapes	2016-01-26 14:54:06 +00:00
Brian Gough	b44a7b9aa6	reject very large ops	2016-01-26 14:52:40 +00:00
Brian Gough	b7a4c72f9c	avoid compressing updates if the result would be too big	2016-01-26 12:23:21 +00:00
Brian Gough	ed0aaa189d	add test for non-overlapping insert-delete case	2016-01-26 12:13:43 +00:00
Brian Gough	b3ddd839e6	add logging of raw updates	2016-01-26 11:28:02 +00:00
Brian Gough	29c7c5e249	enable packs by default for new docs	2016-01-25 09:55:55 +00:00
Brian Gough	d10123d3c4	include n parameter when packing	2016-01-25 09:45:25 +00:00
Brian Gough	9b2cd11cd4	don't try to append to packs when using the old op code	2016-01-22 10:45:24 +00:00
Brian Gough	84ace7f4c7	use packs only for temporary ops	2016-01-20 14:22:13 +00:00
Brian Gough	ae61d1261e	added tests for pack updates	2016-01-19 15:58:09 +00:00
Brian Gough	61d22f027a	added test for inserting as pack	2016-01-19 15:58:09 +00:00
Brian Gough	399a8f0a29	update tests to assume packs are created	2016-01-19 15:58:09 +00:00
Brian Gough	78b3412ca8	decrease delay when packing	2016-01-19 15:58:09 +00:00
Brian Gough	679a81564e	respect mongo 3 limit of 1000 bulk operations	2016-01-19 15:58:09 +00:00
Brian Gough	f592611cac	always create a new pack, never keep as op	2016-01-19 15:58:09 +00:00
Brian Gough	c6be12f3d5	set v_end on pack creation	2016-01-19 15:58:09 +00:00
Brian - Work	f64969c784	added comment about query memory usage for toArray	2016-01-19 15:58:09 +00:00
Brian Gough	0532a4daaa	use compound index to replace separate index for packs	2016-01-19 15:56:09 +00:00
Brian Gough	0ba00a9eb7	expire temporary packs and roll over to a new pack each day	2016-01-19 15:56:09 +00:00
Brian Gough	5e830cbbdb	put all new ops into packs	2016-01-19 15:56:09 +00:00
Brian Gough	dc564fd5d0	archiving document history now sends all changes to s3	2016-01-15 15:54:46 +00:00
Brian Gough	5153ed8217	make peekLastUpdate alway return lastVersion when available	2016-01-15 15:54:44 +00:00