201 commits

Author	SHA1	Message	Date
Brian Gough	692e8c657c	Revert to the default lock timeout now we have write barriers ... Revert "increase lock timeouts for archiving" This reverts commit 9eee1b383772adf058130d6e5eab409f57ce03cd.	2015-09-24 08:53:09 +01:00
Brian Gough	2ab1778dd9	move default value of lastVersion into function body	2015-09-23 16:31:33 +01:00
Brian Gough	bdf1d267f0	fix tests	2015-09-23 15:34:36 +01:00
Brian Gough	dc0044020f	only archive entries older than the current update ... to avoid a stale version of the current update ever being pulled back from S3	2015-09-23 14:33:40 +01:00
Brian Gough	696a866b67	pause the stream of ops, not the download ... the download is buffered in the lineStream so a lot comes out even after pausing the S3 download.	2015-09-23 13:38:57 +01:00
Brian Gough	847a553344	prevent double archiving by checking if any inS3 field is already present	2015-09-23 13:29:32 +01:00
Brian Gough	e49f260507	allow rollback/locking by setting inS3:false when starting the archive process	2015-09-23 13:28:07 +01:00
Brian Gough	551e8334cf	compressedUpdates are now never inserted with inS3 ... it is now always added later, and a new update is forced for any addition to an archived update	2015-09-23 13:25:10 +01:00
Brian Gough	d6b827426c	support forcing new compressed update in popLastCompressedUpdate ... callback with a null update, passing the version as an additional argument	2015-09-23 13:22:38 +01:00
Brian Gough	dc35ef5cda	fix tests for archiving	2015-09-21 13:24:06 +01:00
Brian Gough	a10dc4f898	Merge pull request #6 from heukirne/s3-archive ... Add S3 archive track changes feature	2015-09-21 11:25:06 +01:00
Brian Gough	0e627c92d8	avoid clobbering global _ in loop	2015-09-18 16:26:05 +01:00
Henrique Dias	aa66c5ee8c	improve size function	2015-09-17 10:41:53 -03:00
Henrique Dias	3f712c452a	add size bulk limit	2015-09-17 09:23:13 -03:00
Henrique Dias	01cbfd5338	fix minor issues in acceptance test	2015-09-16 19:33:23 -03:00
Henrique Dias	a4575f178d	fix unit test	2015-09-16 18:39:07 -03:00
Brian Gough	12c5098b48	fix lock in test	2015-09-16 16:50:36 +01:00
Brian Gough	7af5050370	add lock to unarchive doc	2015-09-16 16:18:36 +01:00
Brian Gough	18f06a3daf	increase lock timeouts for archiving	2015-09-16 16:09:38 +01:00
Brian Gough	b4ffa7d57e	share the document lock between archiving and packing	2015-09-16 16:03:55 +01:00
Brian Gough	9d39012b49	add error handler to each stage of download pipeline	2015-09-16 16:00:37 +01:00
Brian Gough	d9085a5e5e	add error handler to each stage of upload pipeline	2015-09-16 16:00:25 +01:00
Brian Gough	1c1b1d9595	log the case where there are no entries in the document history	2015-09-16 15:34:30 +01:00
Brian Gough	82d0f4fce8	make unarchive more responsive by downloading documents in parallel ... unarchive is triggered interactively so we should try to make it reasonably fast	2015-09-16 15:33:59 +01:00
Brian Gough	dfa0036507	pause stream while writing to mongo	2015-09-16 15:32:36 +01:00
Brian Gough	70200a9cf1	only log document ids, not document content ... avoid filling the log with large documents	2015-09-16 15:31:43 +01:00
Brian Gough	d3dff28bea	Merge remote-tracking branch 'origin/master' into heukirne-s3-archive	2015-09-15 15:19:43 +01:00
Brian Gough	092f98d3ad	suppress error in normal shutdown case	2015-09-12 11:07:54 +01:00
Shane Kilkelly	39f528bcbc	Add a test to check that users are summarised properly even when a null user occurs earlier in the update list.	2015-09-11 14:12:01 +01:00
Shane Kilkelly	eab8b4b6c8	Null safe access of id property, needed as user can be null.	2015-09-11 14:07:06 +01:00
Shane Kilkelly	0ad374556d	Add a comment for clarity.	2015-09-10 16:43:40 +01:00
Shane Kilkelly	97326308fa	Update the Acceptance tests to include a case where a user doesn't exist.	2015-09-10 15:40:43 +01:00
Shane Kilkelly	8387383cb4	In _summarizeUpdates, allow null users through. ... A null value represents a deleted or otherwise missing user record.	2015-09-10 14:32:47 +01:00
Shane Kilkelly	810bddb2cb	Log a message when the web api produces a 404 response.	2015-09-10 14:32:35 +01:00
Shane Kilkelly	575bdc62ec	Add a test for when the user can't be found.	2015-09-10 14:32:18 +01:00
Shane Kilkelly	522786d45e	Produce a null value, rather than crashing when the user info service returns 404.	2015-09-09 15:48:22 +01:00
Henry Oswald	18d817ee0a	added some missing error handling	2015-09-08 16:33:45 +01:00
Henry Oswald	17b0d99a65	rework the archiveDocChangesWithLock function ... make it a bit more readable for me, struggle to trust indentation based calls in coffeescript	2015-09-08 16:26:01 +01:00
Henry Oswald	0b3ebcff06	remove if statments checking if s3 is a backend ... if its not enable then it can crash. In prod it should always be there or not used at all	2015-09-08 16:23:15 +01:00
Henrique Dias	c5a8a249c6	add unarchive acceptance tests	2015-09-03 08:36:32 -03:00
Henrique Dias	da9e7dc7e1	init archive acceptance tests	2015-09-02 18:47:34 -03:00
Henrique Dias	7de103af68	fix unit scope error	2015-09-02 17:00:32 -03:00
Henrique Dias	d2b1243701	split MongoAWS files	2015-09-02 15:45:29 -03:00
Henrique Dias	0c16fbed88	add more unit tests	2015-09-02 15:39:19 -03:00
Henrique Dias	1abcea1a66	add some unit test	2015-08-31 18:13:18 -03:00
Henrique Dias	efff026a79	handle easier propagation	2015-08-25 16:52:28 -03:00
Henrique Dias	f910e63e90	fix null case	2015-08-24 12:22:17 -03:00
Henrique Dias	fcbe4aa925	fix inS3 propagation	2015-08-24 12:19:19 -03:00
Henrique Dias	1ccba422c8	remove unused function	2015-08-24 10:55:27 -03:00
Henrique Dias	98ce03b2f2	replace docs collection to DocstoreHandler	2015-08-24 10:38:31 -03:00