254 commits

Author	SHA1	Message	Date
Brian Gough	fd1425d83f	include a unique id in every message published to redis	2019-03-21 12:10:15 +00:00
Brian Gough	8c5d74faef	use explicit json content-type to avoid security issues with text/html	2019-02-12 16:45:11 +00:00
Henry Oswald	ecaef6485b	revert the removal of realtime keyspace	2019-02-07 15:27:51 +00:00
Henry Oswald	4e1a2c787c	Revert "turn down logging, use logger.info for less important data" ... This reverts commit c5f91428e3c7702fbbd3ffd1ef7a772d513f33f2.	2019-02-06 15:29:22 +00:00
Christopher Hoskin	1217d8a80a	Merge branch 'master' into csh-ho-docker-issue-1338-bulk-upgrade	2019-01-04 09:18:40 +00:00
Henry Oswald	2505be283a	turn down logging, use logger.info for less important data	2018-12-13 12:46:10 +00:00
Henry Oswald	25e1ad7491	remove redis.realtime ... real time pushes data into the document updater redis instance	2018-12-05 16:17:41 +00:00
Brian Gough	aae1352519	ensure that project history is flushed when the project is deleted	2018-10-30 12:03:44 +00:00
Brian Gough	fdcb806518	set a timestamp for the first entry in the projectHistory:Ops queue	2018-07-24 09:17:31 +01:00
Brian Gough	e471730efb	Revert "Merge pull request #33 from sharelatex/bg-add-timestamp-marker-to-project-history-queue" ... This reverts commit 8ec398b133aa75c45384dd7ceb630cd743f4a15c, reversing changes made to e3a6c934cf53fd245d7d0df737463cc90f954885.	2018-07-23 16:05:04 +01:00
Brian Gough	6237577f87	set a timestamp for the first entry in the projectHistory:Ops queue	2018-07-20 10:43:31 +01:00
Hayden Faulds	62a10f25f2	move tk call to before sandboxed module call	2018-06-05 13:54:30 +01:00
Hayden Faulds	af92ca70a1	coerce projectHistoryId to integer after reading from Redis	2018-04-23 15:19:06 +01:00
Hayden Faulds	1b63141e49	Revert "Revert "Add projectHistoryId to updates""	2018-04-23 12:08:04 +01:00
Hayden Faulds	3cc20d6311	Revert "Add projectHistoryId to updates"	2018-04-23 11:41:29 +01:00
Hayden Faulds	25c0be4eaa	add projectHistryId to project structure updates	2018-04-13 15:22:05 +01:00
Hayden Faulds	a220794d32	add projectHistoryId to resync updates	2018-04-13 15:22:05 +01:00
Hayden Faulds	fb1852a593	cache projectHistoryId with doc in Redis	2018-04-13 15:22:05 +01:00
Brian Gough	3385d2640a	fix structure ordering bug	2018-03-15 16:27:10 +00:00
Brian Gough	b3887fd984	update unit tests for incoming project versions	2018-03-15 16:24:11 +00:00
Brian Gough	0642e3c8c9	support project version on incoming requests	2018-03-15 16:15:10 +00:00
Hayden Faulds	4a5731440a	Merge pull request #21 from sharelatex/hof-history-sync ... resync project history endpoint	2018-03-14 10:40:59 +00:00
Hayden Faulds	1226f96fc3	extract ProjectHistoryRedisManager	2018-03-09 15:36:38 +00:00
Hayden Faulds	a72d228354	add unit tests	2018-03-07 13:29:53 +00:00
James Allen	939eaa2d4b	Don't allow a document to be loaded without a pathname	2018-03-02 10:02:49 +00:00
Hayden Faulds	5a11332aa3	remove unecessary method wrapping	2018-02-22 10:17:59 +00:00
Hayden Faulds	ea0dd9700b	reduce the number of times we flush project history	2018-02-22 10:01:05 +00:00
James Allen	3d5740fd7d	Merge pull request #18 from sharelatex/msw-flush-project-ops ... Flush project structure changes every 500	2018-02-16 10:30:00 +00:00
James Allen	657da70d45	Fix unit test namespacing from mocha upgrade	2018-02-15 16:28:58 +00:00
James Allen	f6c79f3203	Fix and extend unit tests	2018-02-15 15:55:12 +00:00
Michael Walker	241d1b27d5	Remove _shouldFlushHistoryOps wrapper	2018-01-31 11:27:40 +00:00
Michael Walker	3028fb9c3d	Only flush project updates when crossing the threshold	2018-01-24 11:57:03 +00:00
Brian Gough	41f15c4fa8	fix unit test	2017-12-14 14:46:27 +00:00
Brian Gough	947e18eccf	Merge branch 'bg-limit-ops-in-lock'	2017-12-14 14:26:44 +00:00
James Allen	6abf9c227d	Calculate and send doc length before each update	2017-12-11 17:26:32 +00:00
Hayden Faulds	a3420b1236	version entity additions	2017-11-10 14:54:56 +00:00
Hayden Faulds	944e633bac	add acceptance test for entity renaming	2017-11-08 09:20:38 +00:00
Hayden Faulds	7e86afe55e	version file renames	2017-11-08 09:20:38 +00:00
Hayden Faulds	6d571e6d23	version document renames	2017-11-08 09:20:37 +00:00
Hayden Faulds	be41a1614d	decorate ops with pathname and doc_length	2017-10-23 15:32:25 +01:00
Hayden Faulds	962520fca8	flush track-changes and project-history in HistoryManager	2017-10-23 15:32:25 +01:00
Hayden Faulds	d003aef31c	conditionally enqueue history updates for project	2017-10-23 15:32:25 +01:00
Hayden Faulds	c27df0bfef	split apart multi and rclient in tests	2017-10-23 14:24:23 +01:00
Hayden Faulds	81f998afe9	return pathname from DocumentManager.getDoc	2017-10-23 14:23:21 +01:00
Hayden Faulds	748315aadc	handle pathname in DocumentManager.getDoc	2017-10-23 14:23:21 +01:00
Hayden Faulds	cbdace7386	store pathname in Redis cache	2017-10-23 14:23:21 +01:00
Hayden Faulds	05b93a629a	return pathname from PersistenceManager	2017-10-23 14:17:14 +01:00
Brian Gough	02d3d1bd17	fix unit tests ... timekeeper must be called before SandboxedModule.require	2017-10-23 14:02:53 +01:00
Brian Gough	7c2a28c161	fix use of timekeeper in unit tests	2017-10-23 13:33:14 +01:00
Brian Gough	7cbb3e7af8	change getProjectDocs endpoint from GET to POST ... also note that it flushes docs if they are considered old (i.e. not recently flushed)	2017-10-11 15:30:29 +01:00
Brian Gough	8d3f82360c	update DocumentManager.getDoc signature ... keep alreadyLoaded flag at the end for consistency	2017-10-11 14:25:33 +01:00
Brian Gough	2bbbf3c005	add unflushed time to doc in redis	2017-10-06 12:34:28 +01:00
Brian Gough	ad745f133d	Merge pull request #59 from sharelatex/bg-reset-project-state ... add endpoint to clear project state	2017-09-13 10:39:47 +01:00
Brian Gough	2047bd5516	fix broken unit test	2017-09-11 09:44:05 +01:00
Brian Gough	f34c12fdc7	add endpoint to clear project state	2017-09-08 15:50:26 +01:00
Brian Gough	9d931a3fa7	fix broken unit tests	2017-08-18 12:08:39 +01:00
Brian Gough	230c93766d	fix broken RedisManager unit test	2017-08-18 11:59:31 +01:00
Brian Gough	f6be68eb08	added missing unit tests file	2017-08-11 16:49:16 +01:00
Brian Gough	f696ccb0d9	add unit tests for getProjectDocs	2017-08-10 16:41:28 +01:00
Brian Gough	ca7d9dce85	fix unit tests	2017-08-08 09:40:39 +01:00
Shane Kilkelly	c3465cf3d4	Reduce retry delay to 10ms, and refactor	2017-07-18 15:28:18 +01:00
Shane Kilkelly	0d46c3e2b3	WIP: auto-retry web requests	2017-07-17 10:08:21 +01:00
Brian Gough	d2730c8d6b	unit tests for locking timeouts	2017-07-12 10:48:32 +01:00
Brian Gough	963e513057	add a timeout for getPreviousDocOps ... it uses several redis operations and this makes it prone to subsequent timeouts if getDoc succeeds but is slow	2017-06-23 15:50:21 +01:00
James Allen	7d22e4422c	Merge pull request #49 from sharelatex/bg-add-redis-timeout ... add redis timeout	2017-06-13 14:46:32 +01:00
James Allen	770b53f6af	Add missing mocks to speed up tests and clean up output	2017-06-08 16:43:27 +01:00
Brian Gough	1ef258c878	add a timeout on the redis getDoc request	2017-06-07 11:34:42 +01:00
Brian Gough	729216c9b1	add unit tests for DocsIn check	2017-05-31 16:08:45 +01:00
Brian Gough	bed64d26c8	check if doc is missing from DocsIn set	2017-05-31 15:33:59 +01:00
Brian Gough	891ffda3bf	put a rate limit on worker calls	2017-05-26 15:08:35 +01:00
Brian Gough	b15d2ef796	Revert "fail safely if doc cannot be loaded"	2017-05-15 16:18:40 +01:00
Paulo Jorge Reis	2c3f0aa6b4	Merge pull request #34 from sharelatex/pr-bulk-actions ... Add methods to bulk accept changes.	2017-05-15 15:14:33 +01:00
Paulo Reis	05d7d1b8c6	Merge branch 'master' into pr-bulk-actions	2017-05-15 10:35:16 +01:00
Paulo Reis	36d16a76e0	Unit test accept changes in the ranges manager.	2017-05-12 14:42:40 +01:00
Brian Gough	2ee40d0748	ensure document is in redis before consuming ops	2017-05-12 14:22:18 +01:00
Brian Gough	d4a8d88750	put a limit on the number of ops per iteration	2017-05-12 13:15:59 +01:00
Paulo Reis	f690ef078a	Update tests; add new ones for multiple changes.	2017-05-09 16:16:25 +01:00
Brian Gough	36407ac726	rename HistoryManager pushUncompressedHistoryOps	2017-05-09 12:02:27 +01:00
Brian Gough	fdf5e8e0b8	get history ops length directly from redis update	2017-05-09 10:34:31 +01:00
Brian Gough	2d158b03d7	rename pushUncompressedHistoryOps	2017-05-09 09:32:56 +01:00
Paulo Reis	144804e516	Update tests with new methods.	2017-05-08 16:34:17 +01:00
Brian Gough	e2f70aca1a	fix tests for redis cluster	2017-05-08 16:02:49 +01:00
Brian Gough	79d8fced49	make history update more atomic	2017-05-08 15:58:29 +01:00
James Allen	4104ca4889	Add in separate redis config for the lock and fix a few web -> realtime	2017-05-03 16:27:32 +01:00
James Allen	ed493d8ad3	Rename web -> realtime for consistency with realtime	2017-05-02 15:38:33 +01:00
James Allen	c5449ae282	Split out redis config for real-time and track-changes into separate cluster-compatible configs	2017-04-13 17:00:42 +01:00
James Allen	f21208e841	Use new redis-sharelatex instead of RedisBackend for cluster abstraction	2017-04-12 14:53:03 +01:00
Brian Gough	c11618b475	improve unlock error handling	2017-03-30 15:31:34 +01:00
James Allen	0245bfd031	Support an incoming undo flag for history restores	2017-03-21 11:20:38 +00:00
James Allen	a3a5524778	Fix broken unit test from merge	2017-03-20 10:37:02 +00:00
James Allen	5499a67d78	Add in a consistency check after applying updates that ranges still match	2017-03-15 14:12:06 +00:00
James Allen	a3a9a726d8	Update unit tests	2017-03-06 12:01:18 +00:00
James Allen	f544814dda	Add in an absolute size limit on the ranges JSON object	2017-02-27 14:34:20 +01:00
James Allen	dd45a8dd7d	Improve logging around hashes and null byte errors	2017-02-27 10:06:24 +01:00
James Allen	60f3393c44	Do a naive check for null bytes after JSON.stringify on doc lines	2017-02-27 09:55:04 +01:00
Brian Gough	168bab6960	make hash error logging fine grained	2017-02-22 14:32:51 +00:00
Brian Gough	19e7522d3f	make logging of hash errors optional	2017-02-22 13:46:34 +00:00
Brian Gough	5eb339e525	fix utf8 encoding in sha1 hash	2017-02-21 16:03:06 +00:00
Brian Gough	62165ddeab	add unit tests	2017-02-20 15:33:19 +00:00
Brian Gough	c57d3ce31c	compute hash on write in redis server	2017-02-20 14:09:09 +00:00