541 commits

Author	SHA1	Message	Date
Henrique Dias	ae047ecf76	init s3 feature	2015-08-06 10:00:09 -03:00
Brian Gough	775f5ebbe1	add configurable limit, delay and timeout to /pack via query string	2015-06-05 13:38:47 +01:00
Brian Gough	76d1350593	fix unit tests ... packManager now uses LockManager, which tries to talk to redis and needs to be replaced by a stub in the unit tests.	2015-06-05 10:59:46 +01:00
Brian Gough	23d2518ebb	added a /pack endpoint to fork a pack worker	2015-06-04 16:36:56 +01:00
Brian Gough	289616d3db	added a /doc/:doc_id/pack endpoint	2015-06-04 16:23:49 +01:00
Brian Gough	3f2e4b0c11	move pack script functionality into PackManager	2015-06-04 16:21:40 +01:00
Brian Gough	8668bb9be1	use docHistoryStats to drive the packing script	2015-06-03 10:36:31 +01:00
Brian Gough	27a3511b37	update docHistoryStats after packing ops	2015-06-03 10:36:07 +01:00
Brian Gough	19d812734e	make PackManager parameters configurable	2015-06-03 10:35:36 +01:00
Brian Gough	2521c8f751	added npm packages for pack.coffee script	2015-05-26 14:14:51 +01:00
Brian Gough	66bca8d05c	include the current date in the updates to docHistoryStats	2015-05-26 11:00:55 +01:00
Brian Gough	1811ac2145	added support for cleaning old expired ops in packs	2015-05-22 15:40:02 +01:00
Brian Gough	5c4afd5303	add docHistoryStats collection to keep track of updates to docs	2015-05-22 15:40:01 +01:00
Brian Gough	78f0bdbae3	fix name of temporary parameter to match other methods	2015-05-22 15:40:01 +01:00
Brian Gough	adc2866a7d	add check to exclude temporary ops from packs	2015-05-22 15:40:01 +01:00
Brian Gough	ffeb1cccb6	move pack migration code into PackManager	2015-05-22 15:40:01 +01:00
Brian Gough	8d3e0b2353	added consistency check to pack migration script	2015-05-21 15:46:29 +01:00
Henry Oswald	b37d0dd08a	make startup message consistent	2015-04-30 15:07:11 +01:00
James Allen	24319f6d50	Release version 0.1.4	2015-03-20 15:27:34 +00:00
James Allen	79b9c48efd	Release version 0.1.3	2015-02-26 11:35:14 +00:00
Brian Gough	34fa398cf6	increase pack migrate delay to 2 seconds	2015-02-23 17:00:54 +00:00
Brian Gough	09e24413e6	fix pack migration to run sequentially not in parallel	2015-02-23 17:00:43 +00:00
Brian Gough	ff7860b6dc	added a time delay to the pack migration script	2015-02-23 14:10:09 +00:00
Brian Gough	b19f3835d5	added checkpointing and logging to pack script	2015-02-23 12:13:38 +00:00
Brian Gough	4c0eea9916	return an error if trying to pop the last update from a pack	2015-02-17 14:53:50 +00:00
Brian Gough	24701219ff	clean up comments and remove console.logs	2015-02-17 13:47:55 +00:00
Brian Gough	76cbf32e6a	cleanup ... remove unnecessary comments, whitespace and unused modules	2015-02-17 13:41:31 +00:00
Brian Gough	f69f6452e9	Merge remote-tracking branch 'origin/master' into support-packs	2015-02-17 13:38:30 +00:00
Brian Gough	448a7b7f36	rename MongoPackManager to PackManager	2015-02-17 11:14:13 +00:00
Brian Gough	92e67511d5	sort by doc_id to ensure consistent order for all changes with the same timestamp	2015-02-16 16:44:06 +00:00
Brian Gough	7d6811559c	test pack for getDocUpdates with time	2015-02-16 16:43:27 +00:00
Brian Gough	e55f2a0ff7	test pack for getDocUpdates with version	2015-02-16 15:59:58 +00:00
Brian Gough	f5a9184b57	update the tests to work with packs	2015-02-16 15:38:40 +00:00
Brian Gough	fa1e890282	move update deduplication to filterAndLimit method	2015-02-16 15:37:47 +00:00
Brian Gough	0516e9429d	ensure document results are in version order	2015-02-16 13:43:27 +00:00
Brian Gough	8615b7a086	fix project results order, sort by time	2015-02-16 13:43:11 +00:00
Brian Gough	3ab8238013	update mongojs version to allow use of bulk updates	2015-02-13 17:01:55 +00:00
Brian Gough	18b34d1bcb	clean up packing script	2015-02-13 17:01:37 +00:00
Brian Gough	cafda57710	command-line packing script	2015-02-13 16:23:36 +00:00
Brian Gough	4f36ccd519	moved pack code into MongoPackManager	2015-02-13 16:18:15 +00:00
Brian Gough	682e8e8ab6	add a sparse mongo index for finding packs	2015-02-10 16:54:58 +00:00
Brian Gough	1d7f0919a4	fix mongo logic for project search with packs	2015-02-10 16:54:45 +00:00
James Allen	2d25d54572	Release version 0.1.2	2015-02-10 13:18:12 +00:00
Brian Gough	7fdce8fc48	split pack handling code into separate parts for docs and projects ... need to debug projects	2015-02-09 16:53:44 +00:00
Brian Gough	d8dc56b031	don't try to getLastCompressedUpdates from packs	2015-02-09 16:53:05 +00:00
Brian Gough	f50f091fc6	added comments	2015-02-06 16:59:09 +00:00
Brian Gough	e47476369a	clean up docHistory pack decoding	2015-02-06 15:04:46 +00:00
Brian Gough	8e810bab14	report error for inconsistent history results	2015-02-05 16:37:06 +00:00
Brian Gough	715b0df4a1	prototype for expanding packs	2015-02-05 16:36:41 +00:00
Henry Oswald	3328acc82b	bump sharelatex-redis	2014-10-07 08:33:00 +01:00
Henry Oswald	66bba0dec9	bump redis-sharelatex	2014-09-29 11:41:09 +01:00
Henry Oswald	3f4452e6af	don't run acceptence tests on travis	2014-09-29 11:33:40 +01:00
Henry Oswald	1452eb4572	fixed tests	2014-09-26 17:51:41 +01:00
Henry Oswald	9891bc37d8	Merge branch 'master' of https://github.com/sharelatex/track-changes-sharelatex	2014-09-26 17:41:27 +01:00
Henry Oswald	a0bb289fd3	works with sentinal v1	2014-09-26 17:21:33 +01:00
James Allen	f0e5b914a6	Release version 0.1.0	2014-08-19 14:17:11 +01:00
James Allen	e7b4359c62	Lock down module versions	2014-08-19 14:15:44 +01:00
James Allen	556afc8776	Rename settings to defaults	2014-06-06 13:00:28 +01:00
James Allen	eb7bcc6922	Remove old history deletion method	2014-05-29 15:37:16 +01:00
James Allen	cc962e1c44	Add indexes in background	2014-05-16 17:00:30 +01:00
James Allen	6a371c267f	Fix indexes	2014-05-16 16:41:40 +01:00
James Allen	29ad81c134	Use expiresAt rather than tempCreatedAt	2014-05-16 16:41:14 +01:00
James Allen	8b0b79bc32	Use TTL index to auto-delete updates after a week when versioning is not enabled	2014-05-16 15:59:12 +01:00
James Allen	c835528dea	Add in metrics	2014-05-09 12:44:13 +01:00
James Allen	24343a38cd	Add error handling of mismatched versions back in	2014-05-01 10:26:01 +01:00
James Allen	eebfa6499e	Be more forgiving with version mismatches	2014-04-15 08:02:49 +01:00
James Allen	953081c39d	Delete updates that are older than a week, unless versioning or preserveHistory is enabled	2014-03-28 16:01:34 +00:00
James Allen	9f1efe6dca	Add in getProjectDetails to WebApiManager	2014-03-28 13:05:16 +00:00
James Allen	22e3aac5b0	Don't trip up over empty ops	2014-03-25 11:40:48 +00:00
James Allen	bd4bb3d3cf	Add in project flushing end point	2014-03-21 15:57:17 +00:00
James Allen	8cae726838	Backport project_id onto doc updates that don't have one yet	2014-03-21 14:40:51 +00:00
James Allen	2cd85fefec	Flush all docs in a project before getting updates	2014-03-21 13:48:14 +00:00
James Allen	cd9cb51027	Delete doc_id from DocsWithHistoryOps set when processing	2014-03-21 13:17:58 +00:00
James Allen	6aaa7ba8d5	Group toV and fromV by the doc they correspond to	2014-03-20 13:37:23 +00:00
James Allen	5595910122	Return a summary of the entire project changes	2014-03-20 12:10:04 +00:00
James Allen	b7de4c9c6c	Implement project update getter methods	2014-03-19 17:44:16 +00:00
James Allen	962fc18329	Store project_id alongside each change	2014-03-19 16:40:55 +00:00
James Allen	fec648666c	Add in project_id to flushing HTTP endpoint	2014-03-19 16:14:17 +00:00
James Allen	d32b0ee12f	Fetch updates up to the limit, even after summarising	2014-03-18 18:09:25 +00:00
James Allen	81811d7cc5	Move update summarizing logic into UpdatesManager	2014-03-18 11:41:48 +00:00
James Allen	22a806a200	Fix issues with consuming delete updates beyond the end of the diff	2014-03-17 14:54:02 +00:00
James Allen	540aaf0672	Add in some null checks	2014-03-13 14:30:01 +00:00
James Allen	9977a418c3	Compare diffs with user.id, not user_id	2014-03-12 14:04:33 +00:00
James Allen	575afae048	Compress adjacent inserts and deletes into one diff item	2014-03-12 13:27:18 +00:00
James Allen	7fa88f6b41	Summarise updates when returning them	2014-03-11 17:40:07 +00:00
James Allen	4ecb17b16e	Put multiple ops in one update	2014-03-11 15:24:38 +00:00
James Allen	064bdc3eea	Take user id in from request header and pass to doc updater	2014-03-11 13:01:07 +00:00
James Allen	3d1d962501	Don't try to fetch user when id doesn't exist	2014-03-11 11:45:25 +00:00
James Allen	83b2aa3082	add in restore end point	2014-03-10 16:58:39 +00:00
James Allen	7eb8699b93	Refactor diff manager	2014-03-10 16:03:03 +00:00
James Allen	866084ee6b	Pop last version, not last timestamp	2014-03-07 14:02:16 +00:00
James Allen	b45db3aa2b	Return user details like email and name in updates	2014-03-06 18:04:00 +00:00
James Allen	5fc139db00	Return version numbers of updates	2014-03-06 11:00:49 +00:00
James Allen	aadce232a1	Use version numbers for sorting and querying, not dates	2014-03-06 10:45:51 +00:00
James Allen	2e1307bd82	Doc is always one version ahead of latest op version	2014-03-05 16:31:38 +00:00
James Allen	3660253fd4	Add in /updates end point to get updates	2014-03-05 15:59:40 +00:00
James Allen	a46963a349	Refactor arguments to MongoManager.getUpdatesBetweenDates	2014-03-05 15:06:46 +00:00
James Allen	188d620ce1	Log out and put the last update back when an error occurs	2014-03-05 13:22:38 +00:00
James Allen	e4d8cc7a11	Add in acceptance tests for getting a diff	2014-03-04 15:27:03 +00:00
James Allen	8b71d222d4	Create DiffManager.getDiff	2014-03-04 14:05:17 +00:00
James Allen	1d1dcdfa2f	Add in methods for retrieving updates and doc version	2014-03-04 13:02:48 +00:00
James Allen	20d70859aa	Create buildDiff function	2014-03-03 17:39:59 +00:00
James Allen	8d044b7f13	Fix acceptance tests	2014-03-03 13:36:05 +00:00
James Allen	ba2c4768d0	Use port 3015 by default	2014-03-03 13:31:10 +00:00
James Allen	c15df4c2aa	Add in /status end point	2014-03-03 13:14:01 +00:00
James Allen	e0402692cf	Add in diff generating functions	2014-03-01 13:31:34 +00:00
James Allen	76cdd5cf98	Delete only the updates that were processed, not a full batch	2014-03-01 11:42:31 +00:00
James Allen	d1e18d82aa	Add simple benchmark script	2014-02-26 17:35:10 +00:00
James Allen	3570e8f408	Use grunt forever to run service daemonized for acceptance testing on travis	2014-02-26 16:44:13 +00:00
James Allen	53db11466b	Fix YAML	2014-02-26 16:39:58 +00:00
James Allen	cef17a7fbc	Experiment with acceptance tests in Travis	2014-02-26 16:36:53 +00:00
James Allen	6e9485c0c4	Add README and LICENSE	2014-02-26 16:29:14 +00:00
James Allen	1e7fc3dc29	Add Travis CI	2014-02-26 16:27:09 +00:00
James Allen	834f41eea1	Remove Rakefile	2014-02-26 16:17:26 +00:00
James Allen	ad18c1d5a5	Add Gruntfile	2014-02-26 16:17:15 +00:00
James Allen	a21f8a8004	Add acceptance test for batch processing	2014-02-26 12:44:13 +00:00
James Allen	80af34895b	Refactor and improve acceptance tests	2014-02-26 12:38:47 +00:00
James Allen	d27872c9bd	Get acceptance tests running	2014-02-26 12:11:45 +00:00
James Allen	45fe6978af	Add in /doc/<doc-id>/flush endpoint	2014-02-26 11:34:56 +00:00
James Allen	65360a9a2b	Continue to process raw updates until redis is empty	2014-02-26 11:18:26 +00:00
James Allen	8405a37c2c	Add a lock around processing updates	2014-02-26 10:55:20 +00:00
James Allen	f33a3bde3e	Create processUncompressedUpdates method	2014-02-25 16:48:42 +00:00
James Allen	34d3847fe4	Add Redis Manager for fetching and deleting uncompressed ops	2014-02-25 16:27:14 +00:00
James Allen	8ae9bcd60f	Ensure that updates are compressed in continuous incrementing order	2014-02-25 12:28:15 +00:00
James Allen	de783bf5b0	Pull out MongoManager module and add tests	2014-02-24 17:43:27 +00:00
James Allen	d040e7c410	Add acceptance tests for multiple appends	2014-01-27 18:20:38 +00:00
James Allen	d4295c2023	Create HttpController for HTTP requestS	2014-01-27 18:09:37 +00:00
James Allen	5dea123b13	Get append end point working with a simple acceptance test	2014-01-27 17:51:09 +00:00
James Allen	533b8e59a3	Big refactor to use better names and separation of concerns	2014-01-27 16:26:58 +00:00
James Allen	8a0aa55c91	Refactor to take doc ops from an HTTP request from doc updater	2013-09-03 17:17:27 +01:00
James Allen	dab2781e87	Do two passes better	2013-08-23 14:58:00 +01:00
James Allen	dbee4a57fb	Compress doc ops in two passes	2013-08-23 14:35:13 +01:00
James Allen	bef05c31f4	No need to print the entire history each time	2013-08-09 13:54:30 +01:00
James Allen	7a3b78da43	Return when calling callback	2013-08-09 13:51:02 +01:00
James Allen	07f078b527	Don't insert a null update if there are no updates or raw updates	2013-08-09 13:46:03 +01:00
James Allen	fe503010aa	Pop and set history in batches for speed	2013-08-09 13:27:35 +01:00
James Allen	3ab45c2a35	Handle if two updates cancel each other	2013-08-09 12:36:17 +01:00
James Allen	54c7959647	Match if both user ids are null/undefined	2013-08-09 12:26:23 +01:00
James Allen	64eb794c02	Compress all docs using compressHistory script	2013-08-09 11:36:19 +01:00
James Allen	c5e7f14ba1	Factor methods out into ConversionManager	2013-08-09 11:10:19 +01:00
James Allen	7e96933cf2	Initial commit of script for compressing one docs history	2013-08-08 16:42:14 +01:00