316 commits

Author	SHA1	Message	Date
Michael Walker	3028fb9c3d	Only flush project updates when crossing the threshold	2018-01-24 11:57:03 +00:00
Brian Gough	41f15c4fa8	fix unit test	2017-12-14 14:46:27 +00:00
Brian Gough	947e18eccf	Merge branch 'bg-limit-ops-in-lock'	2017-12-14 14:26:44 +00:00
James Allen	6abf9c227d	Calculate and send doc length before each update	2017-12-11 17:26:32 +00:00
Hayden Faulds	d0e56e5054	add acceptance tests for entity additions	2017-11-10 15:01:37 +00:00
Hayden Faulds	a3420b1236	version entity additions	2017-11-10 14:54:56 +00:00
Hayden Faulds	944e633bac	add acceptance test for entity renaming	2017-11-08 09:20:38 +00:00
Hayden Faulds	7e86afe55e	version file renames	2017-11-08 09:20:38 +00:00
Hayden Faulds	6d571e6d23	version document renames	2017-11-08 09:20:37 +00:00
Hayden Faulds	5b7be93c79	Merge pull request #2 from sharelatex/hof-doc-pathname ... Add metadata to updates for history	2017-10-25 13:13:00 +01:00
Brian Gough	e590543e42	Merge pull request #4 from sharelatex/bg-fix-listen-in-acceptance-tests ... exit if mock servers fail to start	2017-10-25 09:13:48 +01:00
Hayden Faulds	d1f0c4ceae	add acceptance tests for project history API	2017-10-24 09:06:23 +01:00
Hayden Faulds	04ecd1e7ee	set pathname in acceptance tests	2017-10-23 15:32:25 +01:00
Hayden Faulds	be41a1614d	decorate ops with pathname and doc_length	2017-10-23 15:32:25 +01:00
Hayden Faulds	962520fca8	flush track-changes and project-history in HistoryManager	2017-10-23 15:32:25 +01:00
Hayden Faulds	d003aef31c	conditionally enqueue history updates for project	2017-10-23 15:32:25 +01:00
Hayden Faulds	c27df0bfef	split apart multi and rclient in tests	2017-10-23 14:24:23 +01:00
Hayden Faulds	81f998afe9	return pathname from DocumentManager.getDoc	2017-10-23 14:23:21 +01:00
Hayden Faulds	748315aadc	handle pathname in DocumentManager.getDoc	2017-10-23 14:23:21 +01:00
Hayden Faulds	cbdace7386	store pathname in Redis cache	2017-10-23 14:23:21 +01:00
Hayden Faulds	05b93a629a	return pathname from PersistenceManager	2017-10-23 14:17:14 +01:00
Brian Gough	02d3d1bd17	fix unit tests ... timekeeper must be called before SandboxedModule.require	2017-10-23 14:02:53 +01:00
Brian Gough	7c2a28c161	fix use of timekeeper in unit tests	2017-10-23 13:33:14 +01:00
Brian Gough	448f131011	exit if mock servers fail to start	2017-10-20 14:56:12 +01:00
Brian Gough	7cbb3e7af8	change getProjectDocs endpoint from GET to POST ... also note that it flushes docs if they are considered old (i.e. not recently flushed)	2017-10-11 15:30:29 +01:00
Brian Gough	8d3f82360c	update DocumentManager.getDoc signature ... keep alreadyLoaded flag at the end for consistency	2017-10-11 14:25:33 +01:00
Brian Gough	2bbbf3c005	add unflushed time to doc in redis	2017-10-06 12:34:28 +01:00
Brian Gough	ad745f133d	Merge pull request #59 from sharelatex/bg-reset-project-state ... add endpoint to clear project state	2017-09-13 10:39:47 +01:00
Brian Gough	2047bd5516	fix broken unit test	2017-09-11 09:44:05 +01:00
Brian Gough	f34c12fdc7	add endpoint to clear project state	2017-09-08 15:50:26 +01:00
Brian Gough	9d931a3fa7	fix broken unit tests	2017-08-18 12:08:39 +01:00
Brian Gough	230c93766d	fix broken RedisManager unit test	2017-08-18 11:59:31 +01:00
Brian Gough	5971a19084	added acceptance tests	2017-08-11 16:55:31 +01:00
Brian Gough	f6be68eb08	added missing unit tests file	2017-08-11 16:49:16 +01:00
Brian Gough	f696ccb0d9	add unit tests for getProjectDocs	2017-08-10 16:41:28 +01:00
Brian Gough	ca7d9dce85	fix unit tests	2017-08-08 09:40:39 +01:00
Shane Kilkelly	c3465cf3d4	Reduce retry delay to 10ms, and refactor	2017-07-18 15:28:18 +01:00
Shane Kilkelly	dc8d08c030	Update acceptance tests to reflect retry logic	2017-07-17 11:13:21 +01:00
Shane Kilkelly	0d46c3e2b3	WIP: auto-retry web requests	2017-07-17 10:08:21 +01:00
Brian Gough	d2730c8d6b	unit tests for locking timeouts	2017-07-12 10:48:32 +01:00
Brian Gough	963e513057	add a timeout for getPreviousDocOps ... it uses several redis operations and this makes it prone to subsequent timeouts if getDoc succeeds but is slow	2017-06-23 15:50:21 +01:00
James Allen	7d22e4422c	Merge pull request #49 from sharelatex/bg-add-redis-timeout ... add redis timeout	2017-06-13 14:46:32 +01:00
James Allen	770b53f6af	Add missing mocks to speed up tests and clean up output	2017-06-08 16:43:27 +01:00
Brian Gough	1ef258c878	add a timeout on the redis getDoc request	2017-06-07 11:34:42 +01:00
Brian Gough	729216c9b1	add unit tests for DocsIn check	2017-05-31 16:08:45 +01:00
Brian Gough	bed64d26c8	check if doc is missing from DocsIn set	2017-05-31 15:33:59 +01:00
Brian Gough	891ffda3bf	put a rate limit on worker calls	2017-05-26 15:08:35 +01:00
Brian Gough	b15d2ef796	Revert "fail safely if doc cannot be loaded"	2017-05-15 16:18:40 +01:00
Paulo Jorge Reis	2c3f0aa6b4	Merge pull request #34 from sharelatex/pr-bulk-actions ... Add methods to bulk accept changes.	2017-05-15 15:14:33 +01:00
Paulo Reis	05d7d1b8c6	Merge branch 'master' into pr-bulk-actions	2017-05-15 10:35:16 +01:00
Paulo Reis	36d16a76e0	Unit test accept changes in the ranges manager.	2017-05-12 14:42:40 +01:00
Brian Gough	2ee40d0748	ensure document is in redis before consuming ops	2017-05-12 14:22:18 +01:00
Brian Gough	d4a8d88750	put a limit on the number of ops per iteration	2017-05-12 13:15:59 +01:00
Paulo Reis	d53266ea2d	Update web API stub.	2017-05-11 12:00:34 +01:00
Paulo Reis	f690ef078a	Update tests; add new ones for multiple changes.	2017-05-09 16:16:25 +01:00
Brian Gough	36407ac726	rename HistoryManager pushUncompressedHistoryOps	2017-05-09 12:02:27 +01:00
Brian Gough	7ce6285e3d	increase flush threshold to 100 ops	2017-05-09 10:44:26 +01:00
Brian Gough	fdf5e8e0b8	get history ops length directly from redis update	2017-05-09 10:34:31 +01:00
Brian Gough	2d158b03d7	rename pushUncompressedHistoryOps	2017-05-09 09:32:56 +01:00
Paulo Reis	144804e516	Update tests with new methods.	2017-05-08 16:34:17 +01:00
Brian Gough	e2f70aca1a	fix tests for redis cluster	2017-05-08 16:02:49 +01:00
Brian Gough	79d8fced49	make history update more atomic	2017-05-08 15:58:29 +01:00
James Allen	5f93640077	Add scripts for testing cluster failover scenarios	2017-05-04 11:14:17 +01:00
James Allen	4104ca4889	Add in separate redis config for the lock and fix a few web -> realtime	2017-05-03 16:27:32 +01:00
James Allen	5c3661d6d9	Merge remote-tracking branch 'origin/bg-redis-cluster-refactor' into ja-redis-cluster-refactor	2017-05-02 15:39:54 +01:00
James Allen	ed493d8ad3	Rename web -> realtime for consistency with realtime	2017-05-02 15:38:33 +01:00
Brian Gough	64aef0b55a	fix acceptance test to work with redis cluster too	2017-04-27 10:42:43 +01:00
Brian Gough	dc77bc207d	change acceptance test to use redis history client	2017-04-24 16:31:23 +01:00
James Allen	c5449ae282	Split out redis config for real-time and track-changes into separate cluster-compatible configs	2017-04-13 17:00:42 +01:00
James Allen	f21208e841	Use new redis-sharelatex instead of RedisBackend for cluster abstraction	2017-04-12 14:53:03 +01:00
Brian Gough	c11618b475	improve unlock error handling	2017-03-30 15:31:34 +01:00
James Allen	0245bfd031	Support an incoming undo flag for history restores	2017-03-21 11:20:38 +00:00
James Allen	a3a5524778	Fix broken unit test from merge	2017-03-20 10:37:02 +00:00
James Allen	5499a67d78	Add in a consistency check after applying updates that ranges still match	2017-03-15 14:12:06 +00:00
James Allen	a3a9a726d8	Update unit tests	2017-03-06 12:01:18 +00:00
James Allen	f544814dda	Add in an absolute size limit on the ranges JSON object	2017-02-27 14:34:20 +01:00
James Allen	dd45a8dd7d	Improve logging around hashes and null byte errors	2017-02-27 10:06:24 +01:00
James Allen	60f3393c44	Do a naive check for null bytes after JSON.stringify on doc lines	2017-02-27 09:55:04 +01:00
Brian Gough	168bab6960	make hash error logging fine grained	2017-02-22 14:32:51 +00:00
Brian Gough	19e7522d3f	make logging of hash errors optional	2017-02-22 13:46:34 +00:00
Brian Gough	8d1b11f37a	Merge branch 'bg-verify-writes-II'	2017-02-21 16:03:48 +00:00
Brian Gough	5eb339e525	fix utf8 encoding in sha1 hash	2017-02-21 16:03:06 +00:00
Brian Gough	6b808522dc	Merge pull request #18 from sharelatex/bg-verify-writes ... merge redis hash check	2017-02-20 15:55:19 +00:00
Brian Gough	590f8e7ced	Merge pull request #15 from sharelatex/bg-verify-writes ... store sha1 hash of docLines in redis	2017-02-20 15:53:56 +00:00
Brian Gough	62165ddeab	add unit tests	2017-02-20 15:33:19 +00:00
Brian Gough	c57d3ce31c	compute hash on write in redis server	2017-02-20 14:09:09 +00:00
Brian Gough	a3c127e469	added unit tests	2017-02-20 13:53:25 +00:00
Shane Kilkelly	37172a2115	Debug Acceptance Tests: Try without npm rebuild	2017-02-20 09:22:54 +00:00
Shane Kilkelly	28536bfe73	try running mochatest directly without compiling	2017-02-17 16:14:13 +00:00
Brian Gough	bd70aaa76c	add sha1 hash support on writes	2017-02-14 16:11:43 +00:00
James Allen	8725aeab03	Allow deleting of comment ranges	2017-01-24 15:57:11 +01:00
James Allen	5fed2424d0	Remove unused redis package reference	2017-01-16 13:05:05 +01:00
James Allen	0706feb26b	Add max limit on number of comments and changes per doc	2017-01-10 16:58:11 +01:00
James Allen	be19532a1d	Add HTTP end point for accepting changes	2017-01-09 14:41:18 +01:00
James Allen	65f4360738	Consolidate HttpController tests into one file	2017-01-09 14:34:10 +01:00
James Allen	7cac2f7d76	Generate deterministic range ids based on seed	2017-01-09 10:46:58 +01:00
James Allen	2c7029cc50	Revert "Support a {dr:...} op for deleting ranges" ... This reverts commit 24c58e5ad430e0240533cc1e5c21122859fe8dc9.	2017-01-09 09:24:19 +01:00
James Allen	0f13cb3aa7	Support a {dr:...} op for deleting ranges	2017-01-06 16:58:51 +01:00
James Allen	59a06cd798	Accept comments with thread id as an op type	2016-12-13 15:51:47 +00:00
James Allen	47b19818ff	Add in new comment op type	2016-12-12 17:53:43 +00:00
James Allen	e3fee1a1d1	Rename 'track changes entries' -> 'ranges'	2016-12-08 12:31:43 +00:00
James Allen	3ea2e07993	Add tests for fetching and flushing track changes entries to persistence layer	2016-12-02 12:01:23 +00:00
James Allen	418405e8b9	Return track changes entries in HTTP request	2016-12-02 11:37:27 +00:00
James Allen	4fadd75ef3	Track changes based on flag on op, not global setting	2016-12-02 11:04:21 +00:00
James Allen	f43355b74d	Fix UpdateManager tests	2016-12-01 18:19:47 +00:00
James Allen	ce93a76e7f	Fix ShareJsUpdateManager tests	2016-12-01 18:11:03 +00:00
James Allen	e7ff05e792	Fix UpdateManager tests	2016-12-01 18:06:33 +00:00
James Allen	889f5fdf9f	Fix ShareJsDB tests	2016-12-01 17:14:40 +00:00
James Allen	b6c93c718d	Update TrackChangesManagerTests -> HistoryManagerTests	2016-12-01 16:50:55 +00:00
James Allen	d878dd5758	Fix RedisManagerTests	2016-12-01 16:49:53 +00:00
James Allen	9ee913be39	Update PersistenceManagerTests	2016-12-01 16:40:15 +00:00
James Allen	fb39e37fe0	Update DocumentManager tests	2016-12-01 16:27:40 +00:00
James Allen	d36f1a86f1	Merge branch 'master' into ja-track-changes ... Conflicts: app/coffee/PersistenceManager.coffee	2016-12-01 16:06:13 +00:00
James Allen	efab68e6b2	Add in guards on bad data from web API	2016-11-29 17:13:16 +00:00
James Allen	546d9e9cee	Remove left over test	2016-11-29 17:13:07 +00:00
James Allen	2852043a6d	Don't store doc version in Mongo directly, instead use docstore	2016-11-29 17:06:23 +00:00
James Allen	f7ba0946be	More acceptance test fixes	2016-11-29 15:09:11 +00:00
James Allen	2df5c08389	Update acceptance tests to use redis-sharelatex	2016-11-29 15:05:02 +00:00
James Allen	db8b4bf991	Update acceptance test script	2016-11-29 14:57:05 +00:00
James Allen	e739e86c48	Get basic ChangeTracker hooked up. WIP	2016-11-28 10:14:42 +00:00
James Allen	5ce15c4d60	Move check of zero length op array up a level	2016-09-12 10:37:41 +01:00
James Allen	993aab7a78	Don't try to redis rpush with no arguments	2016-09-12 10:37:32 +01:00
James Allen	2d82d56f93	Process updates one at a time and write into redis after each one	2016-09-09 11:01:14 +01:00
James Allen	c85fd7ef41	Merge branch 'master' into ja-consistency-refactor	2016-09-07 11:08:04 +01:00
Brian Gough	9bc7594226	clean up redis query	2016-09-05 10:31:54 +01:00
James Allen	8779f3f686	Only write DocOps atomically with version and lines, after all docs are applied	2016-08-23 16:00:46 +01:00
James Allen	87f3e5e809	Remove old migration code	2016-08-23 11:03:37 +01:00
Shane Kilkelly	39cc9cda6d	Remove docker-related config. test:acceptance:docker now depends on acceptance-test-runner-sharelatex.	2016-07-18 11:09:04 +01:00
Shane Kilkelly	4c0dc5a0ef	Increase timeout for test.	2016-07-08 16:05:36 +01:00
Shane Kilkelly	0a5f95f22d	Move docker-entrypoint.sh into the test/acceptance/scripts directory	2016-07-08 15:52:02 +01:00
Shane Kilkelly	32d06b805c	run only the acceptance tests inside container, rather than compile and test.	2016-07-08 15:15:41 +01:00
Shane Kilkelly	b16b34d7c6	only execute app inside container, instead of compile and execute.	2016-07-08 14:50:30 +01:00
Shane Kilkelly	6d9aecae4c	capture exit code of the test run.	2016-07-08 14:31:43 +01:00
Shane Kilkelly	879482a955	grunt scripts to run the docker acceptance tests	2016-07-08 12:09:01 +01:00
Shane Kilkelly	bca8f6c376	working docker image which can run the acceptance tests	2016-07-08 10:51:22 +01:00
Shane Kilkelly	c38d903f92	Basic docker file.	2016-07-07 16:07:14 +01:00
James Allen	1c62a1c5a1	Don't propagate error on secondary timeout	2016-07-07 09:46:15 +01:00
James Allen	59883023ca	Ignore different order of results from smembers	2016-07-06 14:24:27 +01:00
James Allen	89f90c1b04	Timeout secondary requests if they take longer than 200ms	2016-07-06 12:25:36 +01:00
James Allen	508a95c19b	Use the main redis instance for locks	2016-07-06 11:50:02 +01:00
James Allen	dfd45bd23c	Add timers to time how long each redis request takes	2016-07-04 11:14:23 +01:00
James Allen	fde334acc4	Create script that will ensure data is migrate between redis backends	2016-06-29 12:57:56 +01:00
James Allen	dbe03e2757	Track metrics of when backends match or disagree	2016-06-28 16:49:15 +01:00
James Allen	e04c946ecd	Fix unit tests	2016-06-23 18:04:26 +01:00
James Allen	da89ff7172	Add in external health check rather than internal	2016-06-23 18:00:03 +01:00
James Allen	414ab5d6a9	Refactor TCP redis monitoring to be more explicit	2016-06-23 15:59:25 +01:00
James Allen	8ef03c3d2f	Add in application layer monitoring of the health of each cluster node	2016-06-23 15:38:51 +01:00
James Allen	27a74d6b71	Improve consistency of stress test script	2016-06-21 14:31:20 +01:00
James Allen	392beac57b	Improve error reporting and retry ops in stress test script	2016-06-20 15:01:39 +01:00
James Allen	7915c2ab6c	Tweak stress test to work on server	2016-06-20 13:56:46 +01:00