149 commits

Author	SHA1	Message	Date
Brian Gough	e683b0275a	bug fix for clear archive in progress flag	2015-09-24 09:09:49 +01:00
Brian Gough	692e8c657c	Revert to the default lock timeout now we have write barriers ... Revert "increase lock timeouts for archiving" This reverts commit 9eee1b383772adf058130d6e5eab409f57ce03cd.	2015-09-24 08:53:09 +01:00
Brian Gough	2ab1778dd9	move default value of lastVersion into function body	2015-09-23 16:31:33 +01:00
Brian Gough	dc0044020f	only archive entries older than the current update ... to avoid a stale version of the current update ever being pulled back from S3	2015-09-23 14:33:40 +01:00
Brian Gough	696a866b67	pause the stream of ops, not the download ... the download is buffered in the lineStream so a lot comes out even after pausing the S3 download.	2015-09-23 13:38:57 +01:00
Brian Gough	847a553344	prevent double archiving by checking if any inS3 field is already present	2015-09-23 13:29:32 +01:00
Brian Gough	e49f260507	allow rollback/locking by setting inS3:false when starting the archive process	2015-09-23 13:28:07 +01:00
Brian Gough	551e8334cf	compressedUpdates are now never inserted with inS3 ... it is now always added later, and a new update is forced for any addition to an archived update	2015-09-23 13:25:10 +01:00
Brian Gough	d6b827426c	support forcing new compressed update in popLastCompressedUpdate ... callback with a null update, passing the version as an additional argument	2015-09-23 13:22:38 +01:00
Brian Gough	a10dc4f898	Merge pull request #6 from heukirne/s3-archive ... Add S3 archive track changes feature	2015-09-21 11:25:06 +01:00
Brian Gough	0e627c92d8	avoid clobbering global _ in loop	2015-09-18 16:26:05 +01:00
Henrique Dias	aa66c5ee8c	improve size function	2015-09-17 10:41:53 -03:00
Henrique Dias	3f712c452a	add size bulk limit	2015-09-17 09:23:13 -03:00
Brian Gough	7af5050370	add lock to unarchive doc	2015-09-16 16:18:36 +01:00
Brian Gough	18f06a3daf	increase lock timeouts for archiving	2015-09-16 16:09:38 +01:00
Brian Gough	b4ffa7d57e	share the document lock between archiving and packing	2015-09-16 16:03:55 +01:00
Brian Gough	9d39012b49	add error handler to each stage of download pipeline	2015-09-16 16:00:37 +01:00
Brian Gough	d9085a5e5e	add error handler to each stage of upload pipeline	2015-09-16 16:00:25 +01:00
Brian Gough	1c1b1d9595	log the case where there are no entries in the document history	2015-09-16 15:34:30 +01:00
Brian Gough	82d0f4fce8	make unarchive more responsive by downloading documents in parallel ... unarchive is triggered interactively so we should try to make it reasonably fast	2015-09-16 15:33:59 +01:00
Brian Gough	dfa0036507	pause stream while writing to mongo	2015-09-16 15:32:36 +01:00
Brian Gough	70200a9cf1	only log document ids, not document content ... avoid filling the log with large documents	2015-09-16 15:31:43 +01:00
Brian Gough	d3dff28bea	Merge remote-tracking branch 'origin/master' into heukirne-s3-archive	2015-09-15 15:19:43 +01:00
Brian Gough	092f98d3ad	suppress error in normal shutdown case	2015-09-12 11:07:54 +01:00
Shane Kilkelly	eab8b4b6c8	Null safe access of id property, needed as user can be null.	2015-09-11 14:07:06 +01:00
Shane Kilkelly	0ad374556d	Add a comment for clarity.	2015-09-10 16:43:40 +01:00
Shane Kilkelly	8387383cb4	In _summarizeUpdates, allow null users through. ... A null value represents a deleted or otherwise missing user record.	2015-09-10 14:32:47 +01:00
Shane Kilkelly	810bddb2cb	Log a message when the web api produces a 404 response.	2015-09-10 14:32:35 +01:00
Shane Kilkelly	522786d45e	Produce a null value, rather than crashing when the user info service returns 404.	2015-09-09 15:48:22 +01:00
Henry Oswald	18d817ee0a	added some missing error handling	2015-09-08 16:33:45 +01:00
Henry Oswald	17b0d99a65	rework the archiveDocChangesWithLock function ... make it a bit more readable for me, struggle to trust indentation based calls in coffeescript	2015-09-08 16:26:01 +01:00
Henry Oswald	0b3ebcff06	remove if statments checking if s3 is a backend ... if its not enable then it can crash. In prod it should always be there or not used at all	2015-09-08 16:23:15 +01:00
Henrique Dias	c5a8a249c6	add unarchive acceptance tests	2015-09-03 08:36:32 -03:00
Henrique Dias	da9e7dc7e1	init archive acceptance tests	2015-09-02 18:47:34 -03:00
Henrique Dias	d2b1243701	split MongoAWS files	2015-09-02 15:45:29 -03:00
Henrique Dias	1abcea1a66	add some unit test	2015-08-31 18:13:18 -03:00
Henrique Dias	efff026a79	handle easier propagation	2015-08-25 16:52:28 -03:00
Henrique Dias	f910e63e90	fix null case	2015-08-24 12:22:17 -03:00
Henrique Dias	fcbe4aa925	fix inS3 propagation	2015-08-24 12:19:19 -03:00
Henrique Dias	1ccba422c8	remove unused function	2015-08-24 10:55:27 -03:00
Henrique Dias	98ce03b2f2	replace docs collection to DocstoreHandler	2015-08-24 10:38:31 -03:00
Henrique Dias	04ec45529f	restore updates from S3 when exists ... fix: avoid rearchiving	2015-08-18 17:11:19 -03:00
Henrique Dias	20c3e15f93	fix bulk insert limit	2015-08-14 19:58:38 -03:00
Henrique Dias	26c8048729	change mongo stream method (still have a bug in bulk insert limit)	2015-08-14 19:19:54 -03:00
Henrique Dias	fd4afb3574	Archive changes, care about: versioin, expiresAt and Lock	2015-08-14 15:07:16 -03:00
Henrique Dias	6bc9c9010a	handle auto unarchive track changes	2015-08-09 19:52:32 -03:00
Henrique Dias	3bc5380468	handle inS3 flag	2015-08-09 17:50:15 -03:00
Henrique Dias	daa42bcea0	change s3Stream lib	2015-08-09 15:47:47 -03:00
Henrique Dias	bca48ac117	add unarchive doc track from s3	2015-08-06 17:09:36 -03:00
Henrique Dias	438c4f4d0c	using mongoexport for s3 archive	2015-08-06 15:46:44 -03:00