github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-21 20:47:08 -05:00
Code Issues Projects Releases Packages Wiki Activity
23781 commits 3 branches 7 tags 206 MiB
Commit graph

1039 commits

Author SHA1 Message Date
Brian Gough
ca7d9dce85 fix unit tests 2017-08-08 09:40:39 +01:00
Brian Gough
9f3ec72f81 switch to single get/set method for getProjectDocs 
if project state hasn't changed, return the docs. Otherwise set the hash
and return a 409 Conflict response.
 2017-08-07 14:43:28 +01:00
Brian Gough
f5f516a910 delete clsi state when deleting doc (if used) 2017-08-03 15:04:12 +01:00
Brian Gough
860537146b avoid logging doclines in getProjectDocs 2017-08-03 15:04:12 +01:00
Brian Gough
80be5adc0d change filter to exclude in getProjectDocs 2017-08-03 15:04:06 +01:00
Brian Gough
54e02cd895 update docupdater endpoint to /project/id/docs 2017-08-03 14:41:34 +01:00
Brian Gough
0109585092 provide endpoint for current project docs in redis 2017-08-02 13:42:10 +01:00
Shane Kilkelly
c3465cf3d4 Reduce retry delay to 10ms, and refactor 2017-07-18 15:28:18 +01:00
Shane Kilkelly
dc8d08c030 Update acceptance tests to reflect retry logic 2017-07-17 11:13:21 +01:00
Shane Kilkelly
0d46c3e2b3 WIP: auto-retry web requests 2017-07-17 10:08:21 +01:00
Brian Gough
c21cc013f3 use null instead of err in callbacks on success 2017-07-12 10:48:32 +01:00
Brian Gough
d2730c8d6b unit tests for locking timeouts 2017-07-12 10:48:32 +01:00
Brian Gough
bb0dc4b4d5 release the lock if it took too long to acquire it 2017-07-12 10:48:32 +01:00
Brian Gough
54c0fc1180 put a 5 second timeout in getting redis lock 2017-07-11 14:28:58 +01:00
Brian Gough
ffea31290c return total time from the profile .end() method 2017-07-11 14:28:58 +01:00
Brian Gough
41c2bd7fcd Merge pull request #53 from sharelatex/bg-timeout-on-slow-prev-ops 
add a timeout for getPreviousDocOps
 2017-06-27 13:26:02 +01:00
Brian Gough
59ace9d5c5 avoid long lines in timeout check 2017-06-26 16:36:05 +01:00
Brian Gough
8e1e14c9ca update to redis-sharelatex 1.0.3 
adds keepalive support
 2017-06-26 13:49:44 +01:00
Brian Gough
963e513057 add a timeout for getPreviousDocOps 
it uses several redis operations and this makes it prone to subsequent
timeouts if getDoc succeeds but is slow
 2017-06-23 15:50:21 +01:00
James Allen
7b6966899b Add script to expire existing DocOps lists 2017-06-14 12:09:34 +01:00
James Allen
54f128289f Merge pull request #50 from sharelatex/bg-metric-for-unlock-errors 
add metric for unlock errors
 2017-06-13 14:46:40 +01:00
James Allen
7d22e4422c Merge pull request #49 from sharelatex/bg-add-redis-timeout 
add redis timeout
 2017-06-13 14:46:32 +01:00
James Allen
30f57a7bfc Merge pull request #51 from sharelatex/ja-expires-ordering 
Ensure expires is called after the key is created
 2017-06-13 14:05:22 +01:00
James Allen
770b53f6af Add missing mocks to speed up tests and clean up output 2017-06-08 16:43:27 +01:00
James Allen
a3d7260619 Ensure expires is called after the key is created 2017-06-08 16:43:06 +01:00
Brian Gough
2e5d57cd99 don't increase rate limit when tasks are failing 2017-06-07 11:47:28 +01:00
Brian Gough
1ef258c878 add a timeout on the redis getDoc request 2017-06-07 11:34:42 +01:00
Brian Gough
673f4228ce add metric for unlock errors 
sentry does not record them reliably, due to rate limiting
 2017-06-05 16:29:58 +01:00
Brian Gough
729216c9b1 add unit tests for DocsIn check 2017-05-31 16:08:45 +01:00
Brian Gough
12e8eaa9b6 fix bug in doclines check 2017-05-31 16:08:33 +01:00
Brian Gough
bed64d26c8 check if doc is missing from DocsIn set 2017-05-31 15:33:59 +01:00
Brian Gough
fa089dc28d avoid graphite drift in active worker count 2017-05-30 14:37:01 +01:00
Brian Gough
891ffda3bf put a rate limit on worker calls 2017-05-26 15:08:35 +01:00
Brian Gough
c43f97e8e3 add metric for active workers 2017-05-24 12:12:15 +01:00
Brian Gough
a3947c587f fix bug, jsonOps is array not string 2017-05-22 15:20:28 +01:00
Brian Gough
54ad45c870 add null byte check for ranges 2017-05-22 15:00:05 +01:00
Brian Gough
a13f055d62 add null byte check for ops 2017-05-22 15:00:00 +01:00
Brian Gough
78828ebbc5 fine grained logging for cpu usage in applyUpdate 2017-05-19 16:00:16 +01:00
Brian Gough
a550078071 replace verbose logging by slow query log 2017-05-18 15:04:12 +01:00
Brian Gough
95596061d4 clarify calculation of process.hrtime 2017-05-18 15:02:54 +01:00
Brian Gough
e00f4dde8e log profilers start and end times for reference 2017-05-18 15:02:44 +01:00
Brian Gough
e11f64f83a increase profiling cutoff to 1s 2017-05-18 15:02:08 +01:00
Brian Gough
b6efb051e1 add profiling 2017-05-18 11:17:08 +01:00
Brian Gough
387ecac6cb log for each get/try/release lock 2017-05-18 11:17:08 +01:00
Brian Gough
dff6e2c3da bug fix in backoff 2017-05-16 16:31:28 +01:00
Brian Gough
c7d8fbbb8a exponential backoff for lock up to maximum time 2017-05-16 15:22:02 +01:00
Brian Gough
b15d2ef796 Revert "fail safely if doc cannot be loaded" 2017-05-15 16:18:40 +01:00
Paulo Jorge Reis
2c3f0aa6b4 Merge pull request #34 from sharelatex/pr-bulk-actions 
Add methods to bulk accept changes.
 2017-05-15 15:14:33 +01:00
Paulo Reis
ef7fa59257 Update ranges tracker. 2017-05-15 11:11:14 +01:00
Paulo Reis
05d7d1b8c6 Merge branch 'master' into pr-bulk-actions 2017-05-15 10:35:16 +01:00
Paulo Reis
36d16a76e0 Unit test accept changes in the ranges manager. 2017-05-12 14:42:40 +01:00
Brian Gough
2ee40d0748 ensure document is in redis before consuming ops 2017-05-12 14:22:18 +01:00
Brian Gough
be96548199 log number of updates for future debugging 2017-05-12 14:07:59 +01:00
Brian Gough
d4a8d88750 put a limit on the number of ops per iteration 2017-05-12 13:15:59 +01:00
Paulo Reis
d53266ea2d Update web API stub. 2017-05-11 12:00:34 +01:00
Paulo Reis
5677c7ad9a Update range tracker. 2017-05-11 11:31:19 +01:00
Paulo Reis
f690ef078a Update tests; add new ones for multiple changes. 2017-05-09 16:16:25 +01:00
Brian Gough
36407ac726 rename HistoryManager pushUncompressedHistoryOps 2017-05-09 12:02:27 +01:00
Brian Gough
7ce6285e3d increase flush threshold to 100 ops 2017-05-09 10:44:26 +01:00
Brian Gough
fdf5e8e0b8 get history ops length directly from redis update 2017-05-09 10:34:31 +01:00
Brian Gough
2d158b03d7 rename pushUncompressedHistoryOps 2017-05-09 09:32:56 +01:00
Paulo Reis
144804e516 Update tests with new methods. 2017-05-08 16:34:17 +01:00
Paulo Reis
d7ac57d054 Use new bulk method. 2017-05-08 16:08:42 +01:00
Paulo Reis
70341e4e80 Add bulk method from range tracker. 2017-05-08 16:08:34 +01:00
Brian Gough
e2f70aca1a fix tests for redis cluster 2017-05-08 16:02:49 +01:00
Brian Gough
79d8fced49 make history update more atomic 2017-05-08 15:58:29 +01:00
Paulo Reis
e96e6d6a29 Remove unused method. 2017-05-05 15:17:19 +01:00
Paulo Reis
9b8a2c3030 Consolidate methods to accept changes. 2017-05-05 15:12:06 +01:00
James Allen
87a5e30a0b Bump redis-sharelatex version 2017-05-05 13:37:58 +01:00
James Allen
7456238a71 Bump redis-sharelatex version 2017-05-04 15:42:10 +01:00
Paulo Reis
d530ee9501 Add methods to bulk accept changes. 2017-05-04 15:32:54 +01:00
James Allen
5f93640077 Add scripts for testing cluster failover scenarios 2017-05-04 11:14:17 +01:00
James Allen
4104ca4889 Add in separate redis config for the lock and fix a few web -> realtime 2017-05-03 16:27:32 +01:00
James Allen
5c3661d6d9 Merge remote-tracking branch 'origin/bg-redis-cluster-refactor' into ja-redis-cluster-refactor 2017-05-02 15:39:54 +01:00
James Allen
ed493d8ad3 Rename web -> realtime for consistency with realtime 2017-05-02 15:38:33 +01:00
Brian Gough
64aef0b55a fix acceptance test to work with redis cluster too 2017-04-27 10:42:43 +01:00
Brian Gough
dc77bc207d change acceptance test to use redis history client 2017-04-24 16:31:23 +01:00
James Allen
c5449ae282 Split out redis config for real-time and track-changes into separate cluster-compatible configs 2017-04-13 17:00:42 +01:00
James Allen
f21208e841 Use new redis-sharelatex instead of RedisBackend for cluster abstraction 2017-04-12 14:53:03 +01:00
Brian Gough
5583764d05 fix callback prototype in getLock 2017-03-30 15:31:47 +01:00
Brian Gough
c11618b475 improve unlock error handling 2017-03-30 15:31:34 +01:00
Brian Gough
b2b4bc44df upgrade logger-sharelatex to v1.5.6 2017-03-30 14:16:35 +01:00
Brian Gough
527c02a00b log op versions pushed to track changes 2017-03-30 11:20:50 +01:00
Shane Kilkelly
64658d0034 Add a .nvmrc file 2017-03-27 14:50:09 +01:00
James Allen
0245bfd031 Support an incoming undo flag for history restores 2017-03-21 11:20:38 +00:00
James Allen
a3a5524778 Fix broken unit test from merge 2017-03-20 10:37:02 +00:00
James Allen
199079a5b7 Merge branch 'master' into ja-cut-and-paste-comments 2017-03-20 10:29:01 +00:00
James Allen
9c5299ec7c Update RangeTracker to support upserting and moving comments for cut and paste 2017-03-16 15:49:41 +00:00
James Allen
5499a67d78 Add in a consistency check after applying updates that ranges still match 2017-03-15 14:12:06 +00:00
James Allen
b9e4492001 Wrap RangeTracker in try catch 2017-03-06 19:05:11 +00:00
James Allen
a3a9a726d8 Update unit tests 2017-03-06 12:01:18 +00:00
James Allen
d8135749d2 Merge branch 'master' into ja-increase-limits-on-changes 
Conflicts:
	app/coffee/RedisManager.coffee
 2017-03-06 11:57:40 +00:00
James Allen
30294dafb1 Merge pull request #23 from sharelatex/bg-upgrade-raven-logging 
upgrade to logger-sharelatex 1.5.3
 2017-03-06 11:53:47 +00:00
James Allen
269433f250 Merge pull request #25 from sharelatex/bg-log-doclines-on-hash-mismatch 
log doclines on hash mismatch
 2017-03-06 11:53:24 +00:00
Brian Gough
501d907299 upgrade to logger-sharelatex 1.5.4 2017-03-03 16:08:14 +00:00
Brian Gough
d086e0b61b log doclines on hash mismatch 2017-03-03 15:57:44 +00:00
Brian Gough
3f13263ecf upgrade to logger-sharelatex 1.5.3 2017-03-03 15:32:11 +00:00
Brian Gough
b186a01c04 don't log errors from redis backend 
this also picks up errors from RedisManager like "doc ops range is not
loaded in redis"
 2017-03-03 15:27:42 +00:00
James Allen
d56bb55953 Revert PR #19 2017-03-01 16:49:46 +00:00
James Allen
9cfe651930 Merge pull request #19 from sharelatex/ja-track-changes 
Don't consume partial parts of delete when inserting a change
 2017-03-01 10:20:06 +01:00
James Allen
f544814dda Add in an absolute size limit on the ranges JSON object 2017-02-27 14:34:20 +01:00
James Allen
f211c282b2 Merge branch 'master' into ja-increase-limits-on-changes 2017-02-27 13:50:42 +01:00
James Allen
dd45a8dd7d Improve logging around hashes and null byte errors 2017-02-27 10:06:24 +01:00
James Allen
60f3393c44 Do a naive check for null bytes after JSON.stringify on doc lines 2017-02-27 09:55:04 +01:00
James Allen
fd3dee0af5 Increase changes count limit to 2000 2017-02-24 14:58:47 +01:00
James Allen
80284e1b01 Only cancel deletes with inserts on undo and reject 2017-02-24 14:21:06 +01:00
James Allen
0420929e2a Reduce the data logged when updating ranges 2017-02-23 13:57:47 +01:00
James Allen
e1481df76a Don't consume partial parts of delete when inserting a change 2017-02-23 11:39:19 +01:00
Brian Gough
49271d4ee4 update logger to 1.5.2 - rate limiting on sentry 2017-02-23 10:15:56 +00:00
James Allen
5ab903169a evert "update logger to 1.5.2 - rate limiting on sentry" 
This reverts commit ba1c4a84959cd2eb33d7d624239d3e621ebae1fd.
 2017-02-22 18:26:42 +01:00
James Allen
d6b5cc6389 Increase changes limit to 1200 2017-02-22 18:11:55 +01:00
Brian Gough
ed59ca32fd update logger to 1.5.2 - rate limiting on sentry 2017-02-22 15:58:40 +00:00
Brian Gough
168bab6960 make hash error logging fine grained 2017-02-22 14:32:51 +00:00
Brian Gough
19e7522d3f make logging of hash errors optional 2017-02-22 13:46:34 +00:00
Brian Gough
8d1b11f37a Merge branch 'bg-verify-writes-II' 2017-02-21 16:03:48 +00:00
Brian Gough
5eb339e525 fix utf8 encoding in sha1 hash 2017-02-21 16:03:06 +00:00
Brian Gough
6b808522dc Merge pull request #18 from sharelatex/bg-verify-writes 
merge redis hash check
 2017-02-20 15:55:19 +00:00
Brian Gough
590f8e7ced Merge pull request #15 from sharelatex/bg-verify-writes 
store sha1 hash of docLines in redis
 2017-02-20 15:53:56 +00:00
Brian Gough
62165ddeab add unit tests 2017-02-20 15:33:19 +00:00
Brian Gough
c57d3ce31c compute hash on write in redis server 2017-02-20 14:09:09 +00:00
Brian Gough
a3c127e469 added unit tests 2017-02-20 13:53:25 +00:00
Shane Kilkelly
37172a2115 Debug Acceptance Tests: Try without npm rebuild 2017-02-20 09:22:54 +00:00
Shane Kilkelly
28536bfe73 try running mochatest directly without compiling 2017-02-17 16:14:13 +00:00
James Allen
660752dfa6 Increase change limit to 1000 2017-02-17 15:32:44 +01:00
James Allen
89537927b5 Update Ranges Tracker 2017-02-17 09:24:19 +01:00
James Allen
de7f760ea4 Update Ranges Tracker 2017-02-16 17:09:08 +01:00
Brian Gough
b5a4458b68 check sha1 hash value only if present 2017-02-15 14:12:36 +00:00
Brian Gough
bd70aaa76c add sha1 hash support on writes 2017-02-14 16:11:43 +00:00
Brian Gough
a33d4f505b log request object on error 2017-02-14 15:20:05 +00:00
Brian Gough
8d1aa56689 enable sentry for docupdater 2017-02-14 14:34:28 +00:00
James Allen
8725aeab03 Allow deleting of comment ranges 2017-01-24 15:57:11 +01:00
James Allen
dc976fd9e9 Use thread id for comment id 2017-01-23 09:45:46 +01:00
James Allen
0705242a55 Increase max op age to 80 2017-01-17 11:45:10 +01:00
James Allen
5fed2424d0 Remove unused redis package reference 2017-01-16 13:05:05 +01:00
James Allen
0706feb26b Add max limit on number of comments and changes per doc 2017-01-10 16:58:11 +01:00
James Allen
a3d5971d54 Update RangesTRacker 2017-01-10 11:59:09 +01:00
James Allen
540d0f7ec7 Make sure comment ids are unique 2017-01-10 11:55:38 +01:00
James Allen
be19532a1d Add HTTP end point for accepting changes 2017-01-09 14:41:18 +01:00
James Allen
65f4360738 Consolidate HttpController tests into one file 2017-01-09 14:34:10 +01:00
James Allen
593e7260d4 Update RangesTracker 2017-01-09 10:52:06 +01:00
James Allen
7cac2f7d76 Generate deterministic range ids based on seed 2017-01-09 10:46:58 +01:00
James Allen
2c7029cc50 Revert "Support a {dr:...} op for deleting ranges" 
This reverts commit 24c58e5ad430e0240533cc1e5c21122859fe8dc9.
 2017-01-09 09:24:19 +01:00
James Allen
0f13cb3aa7 Support a {dr:...} op for deleting ranges 2017-01-06 16:58:51 +01:00
James Allen
59a06cd798 Accept comments with thread id as an op type 2016-12-13 15:51:47 +00:00
James Allen
47b19818ff Add in new comment op type 2016-12-12 17:53:43 +00:00
James Allen
e3fee1a1d1 Rename 'track changes entries' -> 'ranges' 2016-12-08 12:31:43 +00:00
James Allen
3ea2e07993 Add tests for fetching and flushing track changes entries to persistence layer 2016-12-02 12:01:23 +00:00
James Allen
418405e8b9 Return track changes entries in HTTP request 2016-12-02 11:37:27 +00:00
James Allen
4fadd75ef3 Track changes based on flag on op, not global setting 2016-12-02 11:04:21 +00:00
James Allen
f43355b74d Fix UpdateManager tests 2016-12-01 18:19:47 +00:00
James Allen
ce93a76e7f Fix ShareJsUpdateManager tests 2016-12-01 18:11:03 +00:00
James Allen
e7ff05e792 Fix UpdateManager tests 2016-12-01 18:06:33 +00:00
James Allen
889f5fdf9f Fix ShareJsDB tests 2016-12-01 17:14:40 +00:00
James Allen
b6c93c718d Update TrackChangesManagerTests -> HistoryManagerTests 2016-12-01 16:50:55 +00:00
James Allen
d878dd5758 Fix RedisManagerTests 2016-12-01 16:49:53 +00:00
James Allen
9ee913be39 Update PersistenceManagerTests 2016-12-01 16:40:15 +00:00
James Allen
fb39e37fe0 Update DocumentManager tests 2016-12-01 16:27:40 +00:00
James Allen
d36f1a86f1 Merge branch 'master' into ja-track-changes 
Conflicts:
	app/coffee/PersistenceManager.coffee
 2016-12-01 16:06:13 +00:00
James Allen
efab68e6b2 Add in guards on bad data from web API 2016-11-29 17:13:16 +00:00
James Allen
546d9e9cee Remove left over test 2016-11-29 17:13:07 +00:00
James Allen
2852043a6d Don't store doc version in Mongo directly, instead use docstore 2016-11-29 17:06:23 +00:00
James Allen
77b1d0ea21 Fix up package versions 2016-11-29 15:29:22 +00:00
James Allen
f7ba0946be More acceptance test fixes 2016-11-29 15:09:11 +00:00
James Allen
2df5c08389 Update acceptance tests to use redis-sharelatex 2016-11-29 15:05:02 +00:00
James Allen
db8b4bf991 Update acceptance test script 2016-11-29 14:57:05 +00:00
James Allen
e739e86c48 Get basic ChangeTracker hooked up. WIP 2016-11-28 10:14:42 +00:00
James Allen
5ce15c4d60 Move check of zero length op array up a level 2016-09-12 10:37:41 +01:00
James Allen
993aab7a78 Don't try to redis rpush with no arguments 2016-09-12 10:37:32 +01:00
James Allen
6ca48523d9 mapSeries -> eachSeries 2016-09-09 15:28:27 +01:00
James Allen
2d82d56f93 Process updates one at a time and write into redis after each one 2016-09-09 11:01:14 +01:00
James Allen
c85fd7ef41 Merge branch 'master' into ja-consistency-refactor 2016-09-07 11:08:04 +01:00
Brian Gough
8d14f8f7f0 add route parameters 2016-09-05 10:32:11 +01:00
Brian Gough
9bc7594226 clean up redis query 2016-09-05 10:31:54 +01:00
James Allen
8779f3f686 Only write DocOps atomically with version and lines, after all docs are applied 2016-08-23 16:00:46 +01:00
James Allen
87f3e5e809 Remove old migration code 2016-08-23 11:03:37 +01:00
James Allen
cb62b005f3 Add in event loop monitoring 2016-08-23 09:53:17 +01:00
Shane Kilkelly
39cc9cda6d Remove docker-related config. test:acceptance:docker now depends on acceptance-test-runner-sharelatex. 2016-07-18 11:09:04 +01:00
Shane Kilkelly
02b7b7baf2 Merge branch 'master' of github.com:sharelatex/document-updater-sharelatex 2016-07-18 10:36:12 +01:00
Shane Kilkelly
c446b20375 Add alternative test:acceptance:docker command, using separate docker repo 2016-07-18 10:36:03 +01:00
James Allen
a1bac9719a Update secondary timeout for multi commands too 2016-07-12 11:45:10 +01:00
James Allen
d0c54f1be6 Increase secondary timeout to 600ms 2016-07-12 11:11:55 +01:00
Shane Kilkelly
4c0dc5a0ef Increase timeout for test. 2016-07-08 16:05:36 +01:00
Shane Kilkelly
0a5f95f22d Move docker-entrypoint.sh into the test/acceptance/scripts directory 2016-07-08 15:52:02 +01:00
Shane Kilkelly
32d06b805c run only the acceptance tests inside container, rather than compile and test. 2016-07-08 15:15:41 +01:00
Shane Kilkelly
b16b34d7c6 only execute app inside container, instead of compile and execute. 2016-07-08 14:50:30 +01:00
Shane Kilkelly
6d9aecae4c capture exit code of the test run. 2016-07-08 14:31:43 +01:00
Shane Kilkelly
ea41453442 fix docker build command. 2016-07-08 13:44:43 +01:00
Shane Kilkelly
28fda02d2d Merge branch 'master' into sk-dockerize-tests 2016-07-08 13:37:04 +01:00
Shane Kilkelly
879482a955 grunt scripts to run the docker acceptance tests 2016-07-08 12:09:01 +01:00
James Allen
4f34cb7363 Only set keys in migration if they exist 2016-07-08 11:45:36 +01:00
James Allen
bd392f5682 Comment out cluster config (again...) 2016-07-08 10:54:56 +01:00
Shane Kilkelly
bca8f6c376 working docker image which can run the acceptance tests 2016-07-08 10:51:22 +01:00
James Allen
8b090c0a60 Don't call sadd with blank args 2016-07-07 16:41:38 +01:00
James Allen
9bd7c0017d Allow migrate script to process multi docs at once 2016-07-07 16:29:01 +01:00
Shane Kilkelly
c38d903f92 Basic docker file. 2016-07-07 16:07:14 +01:00
James Allen
6bc78ccf77 Add in some null checks on comparing backend results 2016-07-07 11:24:33 +01:00
James Allen
1c62a1c5a1 Don't propagate error on secondary timeout 2016-07-07 09:46:15 +01:00
James Allen
edf1e1ab49 Initialize primaryResult to an array, not null 2016-07-06 15:02:05 +01:00
James Allen
59883023ca Ignore different order of results from smembers 2016-07-06 14:24:27 +01:00
James Allen
89f90c1b04 Timeout secondary requests if they take longer than 200ms 2016-07-06 12:25:36 +01:00
James Allen
508a95c19b Use the main redis instance for locks 2016-07-06 11:50:02 +01:00
James Allen
e912ccc562 Wait for both backends to return so that they are always in sync 2016-07-05 16:07:47 +01:00
James Allen
dfd45bd23c Add timers to time how long each redis request takes 2016-07-04 11:14:23 +01:00
James Allen
149351aa49 Remove debugging log lines 2016-06-29 21:18:13 +01:00
James Allen
fde334acc4 Create script that will ensure data is migrate between redis backends 2016-06-29 12:57:56 +01:00
James Allen
dbe03e2757 Track metrics of when backends match or disagree 2016-06-28 16:49:15 +01:00
James Allen
0b9e85ea50 Update ioredis version 2016-06-28 15:47:06 +01:00
James Allen
e04c946ecd Fix unit tests 2016-06-23 18:04:26 +01:00
James Allen
da89ff7172 Add in external health check rather than internal 2016-06-23 18:00:03 +01:00
James Allen
414ab5d6a9 Refactor TCP redis monitoring to be more explicit 2016-06-23 15:59:25 +01:00
James Allen
8ef03c3d2f Add in application layer monitoring of the health of each cluster node 2016-06-23 15:38:51 +01:00
James Allen
27a74d6b71 Improve consistency of stress test script 2016-06-21 14:31:20 +01:00
James Allen
392beac57b Improve error reporting and retry ops in stress test script 2016-06-20 15:01:39 +01:00
James Allen
7915c2ab6c Tweak stress test to work on server 2016-06-20 13:56:46 +01:00
James Allen
39d920682a Only pass through keys to redis that are explicitly set in the config 2016-06-17 14:11:22 +01:00
James Allen
40b7da8edb Support endpoints and masterName in redis config 2016-06-17 14:05:54 +01:00
James Allen
e5cf856ddf Add in stress test script 2016-06-17 12:17:43 +01:00
James Allen
bc00aab7b1 Explicitly separate calls to web and docupdater redis instances 2016-06-17 12:17:22 +01:00
James Allen
9ff3026807 Merge branch 'master' into ja_redis_cluster 2016-06-09 10:00:58 +01:00
James Allen
48a92b28e5 Don't run redis commands in parallel for easier consistency reasoning 2016-06-09 09:54:13 +01:00
James Allen
c823e06912 Don't run redis commands in parallel for easier consistency reasoning 2016-06-08 16:42:09 +01:00
James Allen
c6605ed5f0 Fix misapplied arguments to multi commands 2016-06-08 16:41:58 +01:00
James Allen
437e885812 Lock down to specific async version 2016-06-08 16:21:56 +01:00
James Allen
b4936f62af Check that return values from different redis backends match 2016-06-08 12:18:37 +01:00
James Allen
ef43e2b325 Configure backend database as redis cluster 2016-06-07 18:38:32 +01:00
James Allen
1db6f8f159 Pull out rclient into RedisBackend that supports sending requests to multiple rclients 2016-06-07 17:58:18 +01:00
James Allen
e38e883081 Clean up and consolidate RedisManager tests for easy refactoring later 2016-06-01 12:22:47 +01:00
James Allen
ea8f4a6d7a Remove unused getDocOpsLength method 2016-06-01 11:58:15 +01:00
James Allen
8004e022fe Fix async version and fix affected unit tests 2016-06-01 11:49:24 +01:00
James Allen
05b09a447e Ensure that all multi call keys will hash to the same node in cluster 2016-06-01 11:40:12 +01:00
James Allen
f707783aba Remove old methods of triggering doc updates 
Remove the old pub/sub listener which is no longer used.

Also remove the DocsWithPendingUpdates set, which used to track
docs waiting to be updated. This was necessary incase messages were
missed on the pub/sub channel, so we knew which docs still had pending
updates. However, now we use the BLPOP queue, these updates just sit in
the queue until a consumer comes back to continue consuming them.
 2016-06-01 11:28:23 +01:00
James Allen
b2e58266ae Remove unused total doc counting redis commands 2016-06-01 11:03:39 +01:00
James Allen
32ce44f9c6 Remove unused gzipping logic 2016-06-01 10:53:50 +01:00
James Allen
1d7cbd8a78 Fix unit tests 2016-05-31 13:32:15 +01:00
James Allen
11667e8250 Return semantic error if the requested ops are not available when getting a document 2016-05-31 13:24:19 +01:00
Shane Kilkelly
048fd19418 Add one to size of line. 
To account for newline characters in the original document
 2016-05-12 09:26:50 +01:00
Shane Kilkelly
82d5a7fafd check total size of lines, rather than content-length 2016-05-11 15:55:21 +01:00
Shane Kilkelly
210a61112b reject payloads larger than 2MB for setDoc 2016-04-29 15:08:21 +01:00
James Allen
4ab4e27724 Include COUNT in lock and make RND per-process incase randomBytes blocks 2016-04-13 16:27:53 +01:00
James Allen
d15a22be26 Remove misleading log line about lock value 2016-04-13 12:20:53 +01:00
James Allen
945c728db2 Use signed locks so only the locking party can remove their lock 2016-04-13 11:59:56 +01:00
James Allen
6c79ab4321 Don't let HTTP calls take longer than 5 seconds since we're inside a 30 second lock 2016-04-12 17:10:39 +01:00
James Allen
3c7c318ea0 Clean up and speed up unit tests by making sure requires are mocked 2016-01-20 17:36:06 +00:00
James Allen
af5d01e440 Flush track changes when unloading data from redis 2016-01-20 15:05:31 +00:00
James Allen
184c9031cd Delete doc from redis after a full set if it wasn't in redis before 2016-01-20 14:31:25 +00:00
Brian Gough
e8f09f3357 added comment about flushing to track changes 2015-12-01 16:03:05 +00:00
Brian Gough
6e97521971 Revert "Merge pull request #4 from sharelatex/flush-ops-from-redis" 
This reverts commit 84a9ad8b67b860a92271cf7bb669cf21cc0b5b86, reversing
changes made to a6d4649f4f2db06862e8cbc5e132a073ce8c20b2.
 2015-12-01 15:58:40 +00:00
Brian Gough
13e22e1802 Revert "only flush to track-changes when ops are queued" 
This reverts commit 0baa8f989481ca263fa71e90af21f13b29c29504.
 2015-12-01 15:48:49 +00:00
Brian Gough
64f5d5526e Revert "update tests" 
This reverts commit 501e891760fd5c5bd0e6ffc75f866fd565561f31.
 2015-12-01 15:48:31 +00:00
Brian Gough
4bf90afe0e update tests 2015-11-30 15:22:21 +00:00
Brian Gough
7a9577e081 only flush to track-changes when ops are queued 2015-11-30 15:17:11 +00:00
Brian Gough
529a061567 Merge pull request #4 from sharelatex/flush-ops-from-redis 
push ops to track changes when document is flushed
 2015-11-30 10:46:21 +00:00
James Allen
2589e2d417 Gracefully return when an op has already been submitted 
It is not a fatal error if an op has already been submitted. We just
need to send an ack back to the client that submitted it and continue.
If we detect a duplicate op, set dup: true on the op and pass it back
to real-time for distributing. The dup: true flag will ensure it only
gets acknowledged to the submitting client, not everyone.
 2015-11-19 10:54:28 +00:00
James Allen
e73890bfc2 Error if update would make document too long 2015-11-06 12:52:03 +00:00
James Allen
03e9d7390f Add in mongo health check end point 2015-08-31 15:57:26 +01:00
James Allen
d27d2b90a2 Merge pull request #5 from sharelatex/utf16_fix 
Replace UTF-16 surrogate characters with 'replacement character'
 2015-06-12 10:18:34 +01:00
James Allen
333591d087 Extra null check 2015-06-12 10:16:33 +01:00
James Allen
a3847d21d5 Replace UTF-16 surrogate characters with 'replacement character' 
In Javascript, characters are 16-bits wide. It does not understand surrogates as characters.

From Wikipedia (http://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane):
"The High Surrogates (U+D800–U+DBFF) and Low Surrogate (U+DC00–U+DFFF) codes are reserved
for encoding non-BMP characters in UTF-16 by using a pair of 16-bit codes: one High Surrogate
and one Low Surrogate. A single surrogate code point will never be assigned a character.""

The main offender seems to be \uD835 as a stand alone character, which would be the first
16-bit character of a blackboard bold character (http://www.fileformat.info/info/unicode/char/1d400/index.htm).
Something must be going on client side that is screwing up the encoding and splitting the
two 16-bit characters so that \uD835 is standalone.
 2015-06-12 10:14:35 +01:00
Brian Gough
e7e82fc89d update port to match default track changes configuration 2015-05-28 16:43:58 +01:00
Brian
60db8bf5de flush ops when document is flushed 2015-05-28 14:48:44 +01:00
Brian Gough
85a4bf0da6 add tests calling zlib.gzip directly (for debugging) 2015-05-19 14:39:47 +01:00
Brian Gough
85eab2e967 modify DispatchManager test to allow for slow shutdown 2015-05-19 14:23:32 +01:00
Brian Gough
57f6919485 modify LockManager test to avoid dependence on timing 2015-05-18 09:03:51 +01:00
Henry Oswald
d862227314 make startup message consistent 2015-04-30 16:01:22 +01:00
Brian Gough
ea92fe0e9e Merge pull request #3 from sharelatex/gzip-large-docs 
Gzip large docs
 2015-03-31 10:58:18 +01:00
Brian Gough
6cdf5615fc remove old unused functions 2015-03-31 10:24:09 +01:00
Brian Gough
36f60d5bce enforce minimum size of 1k for compression 2015-03-31 10:07:39 +01:00
Brian Gough
9bb08d7ba5 add ZipManager comments 2015-03-31 10:07:11 +01:00
Brian Gough
e61beed92f suppress logging in ZipManager tests 2015-03-30 16:58:00 +01:00
Brian Gough
495af5d568 remove console.logs from tests 2015-03-30 16:56:07 +01:00
Brian Gough
c8c12e8b41 fix error in ZipManager writesEnabled setting 2015-03-30 16:41:45 +01:00
Brian Gough
8e8ee5b3da fix tests 2015-03-30 16:41:23 +01:00
Brian Gough
86505047a3 added unit test (work in progress) 2015-03-27 17:04:38 +00:00
Brian Gough
03564b2138 fix variable zip.minsize to match config name zip.minSize 2015-03-27 17:00:30 +00:00
Brian Gough
66fa170ac8 disable compression by default 2015-03-27 15:32:13 +00:00
Brian Gough
27d466aa85 added acceptance test for redis compression 2015-03-27 13:17:38 +00:00
Brian Gough
6bffa4d9e0 don't log docLines when document removed from redis 
they can now be binary gzipped data which messes up the logs
 2015-03-25 16:54:36 +00:00
Brian Gough
e3d73d4456 add gzip support for large documents to reduce load on redis 2015-03-25 16:53:20 +00:00
James Allen
d12341da1d Release version 0.1.4 2015-03-20 14:21:34 +00:00
James Allen
4942038a23 Release version 0.1.3 2015-02-26 11:23:45 +00:00
James Allen
5cf819cd44 Release version 0.1.2 2015-02-10 13:17:14 +00:00
Henry Oswald
e23b0f0076 bump redis package 2014-11-20 12:53:41 +00:00
Henry Oswald
db0322e054 added redis health check 2014-11-20 11:40:52 +00:00
Henry Oswald
4f56d8d4e8 Revert "Use new 'robust' subscription scheme" 
This reverts commit dd4b938365d0f6196206fd7c70056c34477d13d5
 2014-11-19 16:54:02 +00:00
Henry Oswald
566bf72bfd Revert "lock down the redis version" 
This reverts commit 8c942ca7149afff241c968f66669b17c7e36ca4e. and dd4b938365d0f6196206fd7c70056c34477d13d5
 2014-11-19 16:53:50 +00:00
Henry Oswald
5ae364e6c1 lock down the redis version 2014-11-19 16:48:43 +00:00
James Allen
0eab2d99b6 Use new 'robust' subscription scheme 2014-11-19 12:54:47 +00:00
James Allen
c822bb6ea4 Add in missed error catching 2014-11-19 12:51:19 +00:00
Henry Oswald
c137e05bfd fixed acceptence tests 2014-10-07 12:31:16 +01:00
Henry Oswald
5ff8983528 implemeneted redis-sharelatex 2014-10-07 12:08:36 +01:00
Henry Oswald
9c85e9b848 moved back to not include sentinal while we upgrade to 0.10.x node 2014-09-29 12:07:32 +01:00
Henry Oswald
fd6c2b5616 bump redis-sharelatex 2014-09-29 11:41:26 +01:00
Henry Oswald
228045667c increament redis-sharelatex 2014-09-28 20:35:05 +01:00
Henry Oswald
7b1b6d27f3 fixed missing new redis depenency 2014-09-28 20:14:14 +01:00
Henry Oswald
d7894c4ea7 fixed tests 2014-09-26 17:57:15 +01:00
Henry Oswald
672dd3de1b works with sentinal v1 2014-09-26 17:38:59 +01:00
Henry Oswald
57fc710be6 tried downgrading forever 2014-09-09 12:12:04 +01:00
Henry Oswald
8b7d92b149 Merge branch 'master' of https://github.com/sharelatex/document-updater-sharelatex 2014-09-04 12:40:34 +01:00
Henry Oswald
114a9ba4d6 bumbed forever up to same version as docstore as npm is complaining 2014-09-04 12:40:17 +01:00
James Allen
a4a0eef153 Release version 0.1.0 2014-08-19 14:07:14 +01:00
James Allen
3ffe96f548 Lock down module versions 2014-08-19 14:06:56 +01:00
James Allen
48c5f66af3 Remove acceptance test from travis 2014-08-15 11:06:31 +01:00
James Allen
9da118c809 Process updates in the background so that workers only act as dispatchers 2014-08-11 14:16:05 +01:00
James Allen
773e17df66 Send worker idle waiting times to statsd 2014-08-07 13:16:11 +01:00
James Allen
cab5509076 Listen on queue via BLPOP rather than pub/sub 2014-08-07 11:45:53 +01:00
James Allen
946f451155 Move to default settings file 2014-05-15 16:22:06 +01:00
James Allen
ffd10a8439 Don't send or get versioning info to/from web 
The version number is only used by the doc updater and so can be cleanly
encapsulated in a collection that only the doc updater knows about. The version
is already stored under docOps, so continue to tore it there.
 2014-05-15 11:13:16 +01:00
James Allen
e25fb7e435 Keeping writing and reading updates from Mongo to cover the deploy period 2014-05-14 16:04:25 +01:00
James Allen
bdfe018cad Log out warning when using Mongo for version still 2014-05-14 14:16:27 +01:00
James Allen
0199f2e129 Use version from web, with fallback to old mongo collection 2014-05-14 13:28:17 +01:00
James Allen
8973969224 Merge branch 'master' into remove_mongo_doc_ops 
Conflicts:
	app/coffee/DocOpsManager.coffee
	test/acceptance/coffee/ApplyingUpdatesToADocTests.coffee
	test/acceptance/coffee/FlushingDocsTests.coffee
	test/unit/coffee/DocOpsManager/DocOpsManagerTests.coffee
	test/unit/coffee/RedisManager/prependDocOpsTests.coffee
	test/unit/coffee/RedisManager/pushDocOpTests.coffee
 2014-05-14 12:39:40 +01:00
James Allen
6011ce4783 Use new metrics module 2014-05-08 09:28:13 +01:00
James Allen
f511ebd4b6 Exit cleanly on SIGINT et al 2014-05-07 10:05:07 +01:00
James Allen
d1434f7646 Increase redis lock expiry time to 30 seconds 2014-05-07 09:48:29 +01:00
James Allen
c2ebaaa338 Split lines on Windows line endings too 2014-04-10 12:44:46 +01:00
James Allen
375427bf5e Remove extraneous logging 2014-03-21 13:15:42 +00:00
James Allen
c0be3ef37b Put doc_ids with history changes into project level set 2014-03-21 12:41:05 +00:00
James Allen
2d28f1903f Flush to the track changes api using the project id as well 2014-03-19 15:56:44 +00:00
James Allen
4f878e000b Allow source and user_id to be included when setting a document 2014-03-11 12:47:26 +00:00
James Allen
5d45e191f3 Don't crash when logging out error 2014-03-04 12:39:02 +00:00
James Allen
86195ce7c3 Add in load throttling based on a redis key 2014-02-28 19:09:29 +00:00
James Allen
3d70f9126e Flush track changes api every 50 updates 2014-02-28 18:29:05 +00:00
James Allen
77c5a27e12 Set up acceptance tests in TravisCI 2014-02-26 16:54:35 +00:00
James Allen
f3192da87f Tell track changes api to flush doc when flushing doc to mongo 2014-02-26 15:56:52 +00:00
James Allen
dfd3ec993b Ensure version is still returned from redis 2014-02-26 14:49:52 +00:00
James Allen
b13f70eadb Push ops into uncompressedHistoryOps list 2014-02-24 16:52:12 +00:00
James Allen
6f19f46d96 Create method for pushing uncompressed ops into redis 2014-02-24 16:40:29 +00:00
James Allen
9cf883e59f Merge branch 'master' into remove_mongo_doc_ops 2014-02-24 16:17:30 +00:00
James Allen
7a977ca597 Don't store ops in Mongo, just keep them in redis with a timeout 2014-02-24 16:13:36 +00:00
James Allen
86e42ce1de Update README.md 2014-02-24 14:43:33 +00:00
James Allen
55edef6d3d Update .travis.yml 2014-02-24 14:37:38 +00:00
James Allen
1a7d2643c0 Create .travis.yml 2014-02-24 14:24:44 +00:00
goodbest
45437c6006 change repo URL from git+ssh:// to git+https:// 2014-02-21 23:10:26 +08:00
James Allen
53929197dd Create LICENSE 2014-02-18 17:23:36 +00:00
James Allen
b45a9968b3 Create README.md 2014-02-18 17:23:12 +00:00
James Allen
8e8d25e185 Point custom npm modules to github 2014-02-17 15:08:46 +00:00
James Allen
414b669a2c Point every redis connection to the new config 2014-02-13 10:36:26 +00:00
James Allen
a5d816272e Remove unused DocumentUpdater.js file 2014-02-12 10:52:53 +00:00
James Allen
e1a7d4f24a Initial open sourcing 2014-02-12 10:40:42 +00:00