130 commits

Author	SHA1	Message	Date
Brian Gough	82d0f4fce8	make unarchive more responsive by downloading documents in parallel ... unarchive is triggered interactively so we should try to make it reasonably fast	2015-09-16 15:33:59 +01:00
Brian Gough	dfa0036507	pause stream while writing to mongo	2015-09-16 15:32:36 +01:00
Brian Gough	70200a9cf1	only log document ids, not document content ... avoid filling the log with large documents	2015-09-16 15:31:43 +01:00
Brian Gough	d3dff28bea	Merge remote-tracking branch 'origin/master' into heukirne-s3-archive	2015-09-15 15:19:43 +01:00
Brian Gough	092f98d3ad	suppress error in normal shutdown case	2015-09-12 11:07:54 +01:00
Shane Kilkelly	eab8b4b6c8	Null safe access of id property, needed as user can be null.	2015-09-11 14:07:06 +01:00
Shane Kilkelly	0ad374556d	Add a comment for clarity.	2015-09-10 16:43:40 +01:00
Shane Kilkelly	8387383cb4	In _summarizeUpdates, allow null users through. ... A null value represents a deleted or otherwise missing user record.	2015-09-10 14:32:47 +01:00
Shane Kilkelly	810bddb2cb	Log a message when the web api produces a 404 response.	2015-09-10 14:32:35 +01:00
Shane Kilkelly	522786d45e	Produce a null value, rather than crashing when the user info service returns 404.	2015-09-09 15:48:22 +01:00
Henry Oswald	18d817ee0a	added some missing error handling	2015-09-08 16:33:45 +01:00
Henry Oswald	17b0d99a65	rework the archiveDocChangesWithLock function ... make it a bit more readable for me, struggle to trust indentation based calls in coffeescript	2015-09-08 16:26:01 +01:00
Henry Oswald	0b3ebcff06	remove if statments checking if s3 is a backend ... if its not enable then it can crash. In prod it should always be there or not used at all	2015-09-08 16:23:15 +01:00
Henrique Dias	c5a8a249c6	add unarchive acceptance tests	2015-09-03 08:36:32 -03:00
Henrique Dias	da9e7dc7e1	init archive acceptance tests	2015-09-02 18:47:34 -03:00
Henrique Dias	d2b1243701	split MongoAWS files	2015-09-02 15:45:29 -03:00
Henrique Dias	1abcea1a66	add some unit test	2015-08-31 18:13:18 -03:00
Henrique Dias	efff026a79	handle easier propagation	2015-08-25 16:52:28 -03:00
Henrique Dias	f910e63e90	fix null case	2015-08-24 12:22:17 -03:00
Henrique Dias	fcbe4aa925	fix inS3 propagation	2015-08-24 12:19:19 -03:00
Henrique Dias	1ccba422c8	remove unused function	2015-08-24 10:55:27 -03:00
Henrique Dias	98ce03b2f2	replace docs collection to DocstoreHandler	2015-08-24 10:38:31 -03:00
Henrique Dias	04ec45529f	restore updates from S3 when exists ... fix: avoid rearchiving	2015-08-18 17:11:19 -03:00
Henrique Dias	20c3e15f93	fix bulk insert limit	2015-08-14 19:58:38 -03:00
Henrique Dias	26c8048729	change mongo stream method (still have a bug in bulk insert limit)	2015-08-14 19:19:54 -03:00
Henrique Dias	fd4afb3574	Archive changes, care about: versioin, expiresAt and Lock	2015-08-14 15:07:16 -03:00
Henrique Dias	6bc9c9010a	handle auto unarchive track changes	2015-08-09 19:52:32 -03:00
Henrique Dias	3bc5380468	handle inS3 flag	2015-08-09 17:50:15 -03:00
Henrique Dias	daa42bcea0	change s3Stream lib	2015-08-09 15:47:47 -03:00
Henrique Dias	bca48ac117	add unarchive doc track from s3	2015-08-06 17:09:36 -03:00
Henrique Dias	438c4f4d0c	using mongoexport for s3 archive	2015-08-06 15:46:44 -03:00
Henrique Dias	028fe2fa03	archive docChanges list to s3	2015-08-06 11:11:43 -03:00
Henrique Dias	ae047ecf76	init s3 feature	2015-08-06 10:00:09 -03:00
Brian Gough	775f5ebbe1	add configurable limit, delay and timeout to /pack via query string	2015-06-05 13:38:47 +01:00
Brian Gough	23d2518ebb	added a /pack endpoint to fork a pack worker	2015-06-04 16:36:56 +01:00
Brian Gough	289616d3db	added a /doc/:doc_id/pack endpoint	2015-06-04 16:23:49 +01:00
Brian Gough	3f2e4b0c11	move pack script functionality into PackManager	2015-06-04 16:21:40 +01:00
Brian Gough	27a3511b37	update docHistoryStats after packing ops	2015-06-03 10:36:07 +01:00
Brian Gough	19d812734e	make PackManager parameters configurable	2015-06-03 10:35:36 +01:00
Brian Gough	66bca8d05c	include the current date in the updates to docHistoryStats	2015-05-26 11:00:55 +01:00
Brian Gough	1811ac2145	added support for cleaning old expired ops in packs	2015-05-22 15:40:02 +01:00
Brian Gough	5c4afd5303	add docHistoryStats collection to keep track of updates to docs	2015-05-22 15:40:01 +01:00
Brian Gough	78f0bdbae3	fix name of temporary parameter to match other methods	2015-05-22 15:40:01 +01:00
Brian Gough	adc2866a7d	add check to exclude temporary ops from packs	2015-05-22 15:40:01 +01:00
Brian Gough	ffeb1cccb6	move pack migration code into PackManager	2015-05-22 15:40:01 +01:00
Brian Gough	4c0eea9916	return an error if trying to pop the last update from a pack	2015-02-17 14:53:50 +00:00
Brian Gough	24701219ff	clean up comments and remove console.logs	2015-02-17 13:47:55 +00:00
Brian Gough	76cbf32e6a	cleanup ... remove unnecessary comments, whitespace and unused modules	2015-02-17 13:41:31 +00:00
Brian Gough	448a7b7f36	rename MongoPackManager to PackManager	2015-02-17 11:14:13 +00:00
Brian Gough	92e67511d5	sort by doc_id to ensure consistent order for all changes with the same timestamp	2015-02-16 16:44:06 +00:00