github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-21 20:47:08 -05:00
Code Issues Projects Releases Packages Wiki Activity
22317 commits 3 branches 7 tags 206 MiB
Commit graph

515 commits

Author SHA1 Message Date
Brian Gough
16fb297043 Revert "skip hash check when non-BMP characters replaced" 2019-06-27 11:39:45 +01:00
Brian Gough
f37860599d skip hash check when non-BMP characters replaced 2019-06-25 16:36:10 +01:00
Henry Oswald
fdef197271 Merge branch 'master' into ho-detailed-flush-status 2019-06-13 14:33:22 +01:00
Henry Oswald
d9a737f97c return failed and succesfully flushed projects when flushing everything 2019-06-13 14:21:38 +01:00
Brian Gough
e8dd1aae9c Merge pull request #70 from overleaf/bg-metric-for-invalid-hash 
add metric for invalid hash and other sharejs errors
 2019-06-12 13:50:34 +01:00
Brian Gough
d50b93df2f add metric for invalid hash and other sharejs errors 2019-06-11 16:48:06 +01:00
Brian Gough
966478cac4 fix hash check to use 'v' field instead of version 2019-06-11 14:11:46 +01:00
Brian Gough
e95059f98e handle non-urgent flushes in background 2019-06-03 10:01:10 +01:00
Brian Gough
0bbfa7de27 Merge branch 'master' into bg-downgrade-delete-component-error 2019-05-08 09:07:02 +01:00
Brian Gough
27a8248196 convert "Delete component" errors into warnings 2019-05-07 16:55:17 +01:00
Henry Oswald
daca83a057 add dryRun option to flush all projects 2019-05-02 16:54:22 +01:00
Henry Oswald
d5d1736a5e adds /flush_all_projects project 2019-05-02 16:43:35 +01:00
Tim Alby
c1c23e4bee record last author id on document flush 
This is a multi-steps process:
* get a update's `user_id` from the metadata
* store the `user_id` (`lastUpdatedBy`) and current date (`lastUpdatedAt`) for
  the document in Redis on every updates
* fetch `lastUpdatedAt` and `lastUpdatedBy` from Redis on document flush
* send the data to web to be persisted in Mongo
 2019-05-02 11:10:02 +01:00
Brian Gough
68e7b9c4e9 Merge pull request #48 from sharelatex/bg-check-incoming-hash 
check incoming hash when present
 2019-04-29 10:15:44 +01:00
James Allen
52f3596e53 Review feedback 2019-04-16 11:05:17 +01:00
James Allen
3d76f4b9bf Record a snapshot to mongo when a doc's comments/changes get collapsed 2019-04-11 13:27:46 +01:00
Brian Gough
3c635c8d98 check version before it is modified by applyOp 2019-04-09 09:20:48 +01:00
Brian Gough
cc1f3fce5b check incoming hash when present 2019-04-08 14:12:18 +01:00
Brian Gough
fd1425d83f include a unique id in every message published to redis 2019-03-21 12:10:15 +00:00
Brian Gough
8c5d74faef use explicit json content-type to avoid security issues with text/html 2019-02-12 16:45:11 +00:00
Henry Oswald
3bc4cb492a added log line 2019-02-07 16:30:53 +00:00
Henry Oswald
ecaef6485b revert the removal of realtime keyspace 2019-02-07 15:27:51 +00:00
Henry Oswald
4e1a2c787c Revert "turn down logging, use logger.info for less important data" 
This reverts commit c5f91428e3c7702fbbd3ffd1ef7a772d513f33f2.
 2019-02-06 15:29:22 +00:00
Christopher Hoskin
1217d8a80a Merge branch 'master' into csh-ho-docker-issue-1338-bulk-upgrade 2019-01-04 09:18:40 +00:00
Henry Oswald
2505be283a turn down logging, use logger.info for less important data 2018-12-13 12:46:10 +00:00
Henry Oswald
25e1ad7491 remove redis.realtime 
real time pushes data into the document updater redis instance
 2018-12-05 16:17:41 +00:00
Brian Gough
aae1352519 ensure that project history is flushed when the project is deleted 2018-10-30 12:03:44 +00:00
Brian Gough
537bd9a0c0 Merge pull request #36 from sharelatex/bg-limit-resync-requests-to-web 
limit resync requests to web (connects to overleaf/sharelatex#775)
 2018-08-16 12:25:16 +01:00
Brian Gough
910b27357d add more logging to resync operations 2018-08-16 11:14:11 +01:00
Brian Gough
aa013f0bee limit parallel resync doc requests to web 2018-08-16 11:13:11 +01:00
Brian Gough
fdcb806518 set a timestamp for the first entry in the projectHistory:Ops queue 2018-07-24 09:17:31 +01:00
Brian Gough
e471730efb Revert "Merge pull request #33 from sharelatex/bg-add-timestamp-marker-to-project-history-queue" 
This reverts commit 8ec398b133aa75c45384dd7ceb630cd743f4a15c, reversing
changes made to e3a6c934cf53fd245d7d0df737463cc90f954885.
 2018-07-23 16:05:04 +01:00
Brian Gough
6237577f87 set a timestamp for the first entry in the projectHistory:Ops queue 2018-07-20 10:43:31 +01:00
Brian Gough
7e526395fb fix check for "delete component" message 2018-04-30 09:56:01 +01:00
Brian Gough
37f431f148 also block "delete component" messages from sentry 2018-04-27 16:03:46 +01:00
Brian Gough
64670f272a stop OpRangeNotAvailable errors going to sentry 2018-04-27 15:45:28 +01:00
Hayden Faulds
af92ca70a1 coerce projectHistoryId to integer after reading from Redis 2018-04-23 15:19:06 +01:00
Hayden Faulds
1b63141e49 Revert "Revert "Add projectHistoryId to updates"" 2018-04-23 12:08:04 +01:00
Hayden Faulds
3cc20d6311 Revert "Add projectHistoryId to updates" 2018-04-23 11:41:29 +01:00
Hayden Faulds
25c0be4eaa add projectHistryId to project structure updates 2018-04-13 15:22:05 +01:00
Hayden Faulds
a220794d32 add projectHistoryId to resync updates 2018-04-13 15:22:05 +01:00
Hayden Faulds
fb1852a593 cache projectHistoryId with doc in Redis 2018-04-13 15:22:05 +01:00
Brian Gough
dd0f8b880a change update to projectUpdate in project related methods 2018-03-16 10:54:12 +00:00
Brian Gough
3385d2640a fix structure ordering bug 2018-03-15 16:27:10 +00:00
Brian Gough
0642e3c8c9 support project version on incoming requests 2018-03-15 16:15:10 +00:00
Hayden Faulds
4a5731440a Merge pull request #21 from sharelatex/hof-history-sync 
resync project history endpoint
 2018-03-14 10:40:59 +00:00
Hayden Faulds
1226f96fc3 extract ProjectHistoryRedisManager 2018-03-09 15:36:38 +00:00
Hayden Faulds
fc6ef6ea7a improve logging 2018-03-09 12:21:48 +00:00
Hayden Faulds
088ab6e7ba rename queueResyncDocContents -> queueResyncDocContent 2018-03-09 12:21:37 +00:00
Hayden Faulds
64d1fe13bc additional logging for project history resync 2018-03-07 16:24:22 +00:00
Hayden Faulds
152a25e96d rename resyncProject -> resyncProjectHistory 2018-03-07 13:29:42 +00:00
Hayden Faulds
d46217be52 queue doc content sync updates 2018-03-02 15:02:27 +00:00
Hayden Faulds
52d6b710bd add ability to resync project structure 2018-03-02 15:01:53 +00:00
James Allen
939eaa2d4b Don't allow a document to be loaded without a pathname 2018-03-02 10:02:49 +00:00
Hayden Faulds
5a11332aa3 remove unecessary method wrapping 2018-02-22 10:17:59 +00:00
Hayden Faulds
0f87ae1f74 simplify comments 2018-02-22 10:16:29 +00:00
Hayden Faulds
ea0dd9700b reduce the number of times we flush project history 2018-02-22 10:01:05 +00:00
Michael Walker
241d1b27d5 Remove _shouldFlushHistoryOps wrapper 2018-01-31 11:27:40 +00:00
Michael Walker
d2865f0650 Correctly count project_ops_length 2018-01-26 14:41:28 +00:00
Michael Walker
3028fb9c3d Only flush project updates when crossing the threshold 2018-01-24 11:57:03 +00:00
Michael Walker
24c74db0dc Flush project changes when queue hits limit 2018-01-23 11:18:20 +00:00
Michael Walker
3d313ebc53 Add method to get number of queued project updates 2018-01-23 11:10:54 +00:00
James Allen
ccf6cb1a18 Provide hosts as environment settings, add npm run start script and update coffeescript 2017-12-29 08:13:16 +00:00
Brian Gough
0b583dd4e6 fix log message 2017-12-20 13:27:10 +00:00
Brian Gough
a8b14552c0 allow flush to succeed when doc is already deleted 2017-12-19 12:27:57 +00:00
Brian Gough
947e18eccf Merge branch 'bg-limit-ops-in-lock' 2017-12-14 14:26:44 +00:00
James Allen
6abf9c227d Calculate and send doc length before each update 2017-12-11 17:26:32 +00:00
Hayden Faulds
bd45fcb5f8 Merge pull request #12 from sharelatex/hof-version-entity-creation 
version entity creation; Connects to sharelatex/web-sharelatex-internal#104
 2017-11-24 10:02:51 +00:00
Brian Gough
fb572670cd Merge pull request #11 from sharelatex/bg-use-separate-flush-count-for-projects 
use a separate parameter for doc and project flush
 2017-11-21 13:54:03 +00:00
Brian Gough
0bb9f147aa fix whitespace 2017-11-13 11:56:08 +00:00
Brian Gough
a4bbf5cf52 use a separate parameter for doc and project flush 2017-11-13 11:53:39 +00:00
Hayden Faulds
a3420b1236 version entity additions 2017-11-10 14:54:56 +00:00
Hayden Faulds
944e633bac add acceptance test for entity renaming 2017-11-08 09:20:38 +00:00
Hayden Faulds
7e86afe55e version file renames 2017-11-08 09:20:38 +00:00
Hayden Faulds
6d571e6d23 version document renames 2017-11-08 09:20:37 +00:00
Hayden Faulds
b8052e7612 allow settings.redis.project_history to be undefined 2017-10-25 13:56:38 +01:00
Hayden Faulds
5b7be93c79 Merge pull request #2 from sharelatex/hof-doc-pathname 
Add metadata to updates for history
 2017-10-25 13:13:00 +01:00
Brian Gough
20655b6eeb remove unused requires of async 2017-10-23 15:59:32 +01:00
Hayden Faulds
be41a1614d decorate ops with pathname and doc_length 2017-10-23 15:32:25 +01:00
Hayden Faulds
962520fca8 flush track-changes and project-history in HistoryManager 2017-10-23 15:32:25 +01:00
Hayden Faulds
d003aef31c conditionally enqueue history updates for project 2017-10-23 15:32:25 +01:00
Hayden Faulds
81f998afe9 return pathname from DocumentManager.getDoc 2017-10-23 14:23:21 +01:00
Hayden Faulds
748315aadc handle pathname in DocumentManager.getDoc 2017-10-23 14:23:21 +01:00
Hayden Faulds
cbdace7386 store pathname in Redis cache 2017-10-23 14:23:21 +01:00
Hayden Faulds
05b93a629a return pathname from PersistenceManager 2017-10-23 14:17:14 +01:00
Brian Gough
7cbb3e7af8 change getProjectDocs endpoint from GET to POST 
also note that it flushes docs if they are considered old (i.e. not
recently flushed)
 2017-10-11 15:30:29 +01:00
Brian Gough
8d3f82360c update DocumentManager.getDoc signature 
keep alreadyLoaded flag at the end for consistency
 2017-10-11 14:25:33 +01:00
Brian Gough
2bbbf3c005 add unflushed time to doc in redis 2017-10-06 12:34:28 +01:00
Brian Gough
ad745f133d Merge pull request #59 from sharelatex/bg-reset-project-state 
add endpoint to clear project state
 2017-09-13 10:39:47 +01:00
Brian Gough
f34c12fdc7 add endpoint to clear project state 2017-09-08 15:50:26 +01:00
Brian Gough
215a939d4e add comment about log.error for excludeVersions 2017-09-08 13:43:22 +01:00
Brian Gough
1e54b4c72c log all errors in getProjectDocs 2017-09-07 14:12:48 +01:00
Brian Gough
8372911a1b return doclines as array from getProjectDocs 
for compatibility with getDoc
 2017-08-11 16:56:30 +01:00
Brian Gough
f696ccb0d9 add unit tests for getProjectDocs 2017-08-10 16:41:28 +01:00
Brian Gough
1eb80936f2 fix log line 2017-08-10 14:57:27 +01:00
Brian Gough
47bebf9638 no need to support optional keys.projectState 2017-08-09 16:47:09 +01:00
Brian Gough
bd6b5b2c59 use v instead of rev in getProjectDocs 2017-08-09 15:30:53 +01:00
Brian Gough
9f3ec72f81 switch to single get/set method for getProjectDocs 
if project state hasn't changed, return the docs. Otherwise set the hash
and return a 409 Conflict response.
 2017-08-07 14:43:28 +01:00
Brian Gough
f5f516a910 delete clsi state when deleting doc (if used) 2017-08-03 15:04:12 +01:00
Brian Gough
860537146b avoid logging doclines in getProjectDocs 2017-08-03 15:04:12 +01:00
Brian Gough
80be5adc0d change filter to exclude in getProjectDocs 2017-08-03 15:04:06 +01:00
Brian Gough
0109585092 provide endpoint for current project docs in redis 2017-08-02 13:42:10 +01:00
Shane Kilkelly
c3465cf3d4 Reduce retry delay to 10ms, and refactor 2017-07-18 15:28:18 +01:00
Shane Kilkelly
0d46c3e2b3 WIP: auto-retry web requests 2017-07-17 10:08:21 +01:00
Brian Gough
c21cc013f3 use null instead of err in callbacks on success 2017-07-12 10:48:32 +01:00
Brian Gough
bb0dc4b4d5 release the lock if it took too long to acquire it 2017-07-12 10:48:32 +01:00
Brian Gough
54c0fc1180 put a 5 second timeout in getting redis lock 2017-07-11 14:28:58 +01:00
Brian Gough
ffea31290c return total time from the profile .end() method 2017-07-11 14:28:58 +01:00
Brian Gough
59ace9d5c5 avoid long lines in timeout check 2017-06-26 16:36:05 +01:00
Brian Gough
963e513057 add a timeout for getPreviousDocOps 
it uses several redis operations and this makes it prone to subsequent
timeouts if getDoc succeeds but is slow
 2017-06-23 15:50:21 +01:00
James Allen
54f128289f Merge pull request #50 from sharelatex/bg-metric-for-unlock-errors 
add metric for unlock errors
 2017-06-13 14:46:40 +01:00
James Allen
7d22e4422c Merge pull request #49 from sharelatex/bg-add-redis-timeout 
add redis timeout
 2017-06-13 14:46:32 +01:00
James Allen
a3d7260619 Ensure expires is called after the key is created 2017-06-08 16:43:06 +01:00
Brian Gough
2e5d57cd99 don't increase rate limit when tasks are failing 2017-06-07 11:47:28 +01:00
Brian Gough
1ef258c878 add a timeout on the redis getDoc request 2017-06-07 11:34:42 +01:00
Brian Gough
673f4228ce add metric for unlock errors 
sentry does not record them reliably, due to rate limiting
 2017-06-05 16:29:58 +01:00
Brian Gough
12e8eaa9b6 fix bug in doclines check 2017-05-31 16:08:33 +01:00
Brian Gough
bed64d26c8 check if doc is missing from DocsIn set 2017-05-31 15:33:59 +01:00
Brian Gough
fa089dc28d avoid graphite drift in active worker count 2017-05-30 14:37:01 +01:00
Brian Gough
891ffda3bf put a rate limit on worker calls 2017-05-26 15:08:35 +01:00
Brian Gough
c43f97e8e3 add metric for active workers 2017-05-24 12:12:15 +01:00
Brian Gough
a3947c587f fix bug, jsonOps is array not string 2017-05-22 15:20:28 +01:00
Brian Gough
54ad45c870 add null byte check for ranges 2017-05-22 15:00:05 +01:00
Brian Gough
a13f055d62 add null byte check for ops 2017-05-22 15:00:00 +01:00
Brian Gough
78828ebbc5 fine grained logging for cpu usage in applyUpdate 2017-05-19 16:00:16 +01:00
Brian Gough
a550078071 replace verbose logging by slow query log 2017-05-18 15:04:12 +01:00
Brian Gough
95596061d4 clarify calculation of process.hrtime 2017-05-18 15:02:54 +01:00
Brian Gough
e00f4dde8e log profilers start and end times for reference 2017-05-18 15:02:44 +01:00
Brian Gough
e11f64f83a increase profiling cutoff to 1s 2017-05-18 15:02:08 +01:00
Brian Gough
b6efb051e1 add profiling 2017-05-18 11:17:08 +01:00
Brian Gough
387ecac6cb log for each get/try/release lock 2017-05-18 11:17:08 +01:00
Brian Gough
dff6e2c3da bug fix in backoff 2017-05-16 16:31:28 +01:00
Brian Gough
c7d8fbbb8a exponential backoff for lock up to maximum time 2017-05-16 15:22:02 +01:00
Brian Gough
b15d2ef796 Revert "fail safely if doc cannot be loaded" 2017-05-15 16:18:40 +01:00
Paulo Jorge Reis
2c3f0aa6b4 Merge pull request #34 from sharelatex/pr-bulk-actions 
Add methods to bulk accept changes.
 2017-05-15 15:14:33 +01:00
Paulo Reis
ef7fa59257 Update ranges tracker. 2017-05-15 11:11:14 +01:00
Paulo Reis
05d7d1b8c6 Merge branch 'master' into pr-bulk-actions 2017-05-15 10:35:16 +01:00
Brian Gough
2ee40d0748 ensure document is in redis before consuming ops 2017-05-12 14:22:18 +01:00
Brian Gough
be96548199 log number of updates for future debugging 2017-05-12 14:07:59 +01:00
Brian Gough
d4a8d88750 put a limit on the number of ops per iteration 2017-05-12 13:15:59 +01:00
Paulo Reis
5677c7ad9a Update range tracker. 2017-05-11 11:31:19 +01:00
Brian Gough
36407ac726 rename HistoryManager pushUncompressedHistoryOps 2017-05-09 12:02:27 +01:00
Brian Gough
7ce6285e3d increase flush threshold to 100 ops 2017-05-09 10:44:26 +01:00
Brian Gough
fdf5e8e0b8 get history ops length directly from redis update 2017-05-09 10:34:31 +01:00
Brian Gough
2d158b03d7 rename pushUncompressedHistoryOps 2017-05-09 09:32:56 +01:00
Paulo Reis
144804e516 Update tests with new methods. 2017-05-08 16:34:17 +01:00
Paulo Reis
d7ac57d054 Use new bulk method. 2017-05-08 16:08:42 +01:00
Paulo Reis
70341e4e80 Add bulk method from range tracker. 2017-05-08 16:08:34 +01:00
Brian Gough
79d8fced49 make history update more atomic 2017-05-08 15:58:29 +01:00
Paulo Reis
e96e6d6a29 Remove unused method. 2017-05-05 15:17:19 +01:00
Paulo Reis
9b8a2c3030 Consolidate methods to accept changes. 2017-05-05 15:12:06 +01:00
Paulo Reis
d530ee9501 Add methods to bulk accept changes. 2017-05-04 15:32:54 +01:00
James Allen
4104ca4889 Add in separate redis config for the lock and fix a few web -> realtime 2017-05-03 16:27:32 +01:00
James Allen
ed493d8ad3 Rename web -> realtime for consistency with realtime 2017-05-02 15:38:33 +01:00
James Allen
c5449ae282 Split out redis config for real-time and track-changes into separate cluster-compatible configs 2017-04-13 17:00:42 +01:00
James Allen
f21208e841 Use new redis-sharelatex instead of RedisBackend for cluster abstraction 2017-04-12 14:53:03 +01:00
Brian Gough
5583764d05 fix callback prototype in getLock 2017-03-30 15:31:47 +01:00
Brian Gough
c11618b475 improve unlock error handling 2017-03-30 15:31:34 +01:00
Brian Gough
527c02a00b log op versions pushed to track changes 2017-03-30 11:20:50 +01:00
James Allen
0245bfd031 Support an incoming undo flag for history restores 2017-03-21 11:20:38 +00:00
James Allen
199079a5b7 Merge branch 'master' into ja-cut-and-paste-comments 2017-03-20 10:29:01 +00:00
James Allen
9c5299ec7c Update RangeTracker to support upserting and moving comments for cut and paste 2017-03-16 15:49:41 +00:00
James Allen
5499a67d78 Add in a consistency check after applying updates that ranges still match 2017-03-15 14:12:06 +00:00
James Allen
b9e4492001 Wrap RangeTracker in try catch 2017-03-06 19:05:11 +00:00
James Allen
d8135749d2 Merge branch 'master' into ja-increase-limits-on-changes 
Conflicts:
	app/coffee/RedisManager.coffee
 2017-03-06 11:57:40 +00:00
James Allen
269433f250 Merge pull request #25 from sharelatex/bg-log-doclines-on-hash-mismatch 
log doclines on hash mismatch
 2017-03-06 11:53:24 +00:00
Brian Gough
d086e0b61b log doclines on hash mismatch 2017-03-03 15:57:44 +00:00
Brian Gough
b186a01c04 don't log errors from redis backend 
this also picks up errors from RedisManager like "doc ops range is not
loaded in redis"
 2017-03-03 15:27:42 +00:00
James Allen
d56bb55953 Revert PR #19 2017-03-01 16:49:46 +00:00
James Allen
9cfe651930 Merge pull request #19 from sharelatex/ja-track-changes 
Don't consume partial parts of delete when inserting a change
 2017-03-01 10:20:06 +01:00
James Allen
f544814dda Add in an absolute size limit on the ranges JSON object 2017-02-27 14:34:20 +01:00
James Allen
f211c282b2 Merge branch 'master' into ja-increase-limits-on-changes 2017-02-27 13:50:42 +01:00
James Allen
dd45a8dd7d Improve logging around hashes and null byte errors 2017-02-27 10:06:24 +01:00
James Allen
60f3393c44 Do a naive check for null bytes after JSON.stringify on doc lines 2017-02-27 09:55:04 +01:00
James Allen
fd3dee0af5 Increase changes count limit to 2000 2017-02-24 14:58:47 +01:00
James Allen
80284e1b01 Only cancel deletes with inserts on undo and reject 2017-02-24 14:21:06 +01:00
James Allen
0420929e2a Reduce the data logged when updating ranges 2017-02-23 13:57:47 +01:00
James Allen
e1481df76a Don't consume partial parts of delete when inserting a change 2017-02-23 11:39:19 +01:00
James Allen
d6b5cc6389 Increase changes limit to 1200 2017-02-22 18:11:55 +01:00
Brian Gough
168bab6960 make hash error logging fine grained 2017-02-22 14:32:51 +00:00
Brian Gough
19e7522d3f make logging of hash errors optional 2017-02-22 13:46:34 +00:00
Brian Gough
8d1b11f37a Merge branch 'bg-verify-writes-II' 2017-02-21 16:03:48 +00:00
Brian Gough
5eb339e525 fix utf8 encoding in sha1 hash 2017-02-21 16:03:06 +00:00
Brian Gough
6b808522dc Merge pull request #18 from sharelatex/bg-verify-writes 
merge redis hash check
 2017-02-20 15:55:19 +00:00
Brian Gough
590f8e7ced Merge pull request #15 from sharelatex/bg-verify-writes 
store sha1 hash of docLines in redis
 2017-02-20 15:53:56 +00:00
Brian Gough
c57d3ce31c compute hash on write in redis server 2017-02-20 14:09:09 +00:00
James Allen
660752dfa6 Increase change limit to 1000 2017-02-17 15:32:44 +01:00
James Allen
89537927b5 Update Ranges Tracker 2017-02-17 09:24:19 +01:00
James Allen
de7f760ea4 Update Ranges Tracker 2017-02-16 17:09:08 +01:00
Brian Gough
b5a4458b68 check sha1 hash value only if present 2017-02-15 14:12:36 +00:00
Brian Gough
bd70aaa76c add sha1 hash support on writes 2017-02-14 16:11:43 +00:00
James Allen
8725aeab03 Allow deleting of comment ranges 2017-01-24 15:57:11 +01:00
James Allen
dc976fd9e9 Use thread id for comment id 2017-01-23 09:45:46 +01:00
James Allen
0705242a55 Increase max op age to 80 2017-01-17 11:45:10 +01:00
James Allen
0706feb26b Add max limit on number of comments and changes per doc 2017-01-10 16:58:11 +01:00
James Allen
a3d5971d54 Update RangesTRacker 2017-01-10 11:59:09 +01:00
James Allen
540d0f7ec7 Make sure comment ids are unique 2017-01-10 11:55:38 +01:00
James Allen
be19532a1d Add HTTP end point for accepting changes 2017-01-09 14:41:18 +01:00
James Allen
593e7260d4 Update RangesTracker 2017-01-09 10:52:06 +01:00
James Allen
7cac2f7d76 Generate deterministic range ids based on seed 2017-01-09 10:46:58 +01:00
James Allen
2c7029cc50 Revert "Support a {dr:...} op for deleting ranges" 
This reverts commit 24c58e5ad430e0240533cc1e5c21122859fe8dc9.
 2017-01-09 09:24:19 +01:00
James Allen
0f13cb3aa7 Support a {dr:...} op for deleting ranges 2017-01-06 16:58:51 +01:00
James Allen
59a06cd798 Accept comments with thread id as an op type 2016-12-13 15:51:47 +00:00
James Allen
47b19818ff Add in new comment op type 2016-12-12 17:53:43 +00:00
James Allen
e3fee1a1d1 Rename 'track changes entries' -> 'ranges' 2016-12-08 12:31:43 +00:00
James Allen
3ea2e07993 Add tests for fetching and flushing track changes entries to persistence layer 2016-12-02 12:01:23 +00:00
James Allen
418405e8b9 Return track changes entries in HTTP request 2016-12-02 11:37:27 +00:00
James Allen
4fadd75ef3 Track changes based on flag on op, not global setting 2016-12-02 11:04:21 +00:00
James Allen
889f5fdf9f Fix ShareJsDB tests 2016-12-01 17:14:40 +00:00
James Allen
9ee913be39 Update PersistenceManagerTests 2016-12-01 16:40:15 +00:00
James Allen
d36f1a86f1 Merge branch 'master' into ja-track-changes 
Conflicts:
	app/coffee/PersistenceManager.coffee
 2016-12-01 16:06:13 +00:00
James Allen
efab68e6b2 Add in guards on bad data from web API 2016-11-29 17:13:16 +00:00
James Allen
2852043a6d Don't store doc version in Mongo directly, instead use docstore 2016-11-29 17:06:23 +00:00
James Allen
e739e86c48 Get basic ChangeTracker hooked up. WIP 2016-11-28 10:14:42 +00:00
James Allen
5ce15c4d60 Move check of zero length op array up a level 2016-09-12 10:37:41 +01:00
James Allen
993aab7a78 Don't try to redis rpush with no arguments 2016-09-12 10:37:32 +01:00
James Allen
6ca48523d9 mapSeries -> eachSeries 2016-09-09 15:28:27 +01:00
James Allen
2d82d56f93 Process updates one at a time and write into redis after each one 2016-09-09 11:01:14 +01:00
James Allen
c85fd7ef41 Merge branch 'master' into ja-consistency-refactor 2016-09-07 11:08:04 +01:00
Brian Gough
9bc7594226 clean up redis query 2016-09-05 10:31:54 +01:00
James Allen
8779f3f686 Only write DocOps atomically with version and lines, after all docs are applied 2016-08-23 16:00:46 +01:00
James Allen
87f3e5e809 Remove old migration code 2016-08-23 11:03:37 +01:00
James Allen
a1bac9719a Update secondary timeout for multi commands too 2016-07-12 11:45:10 +01:00
James Allen
d0c54f1be6 Increase secondary timeout to 600ms 2016-07-12 11:11:55 +01:00
James Allen
4f34cb7363 Only set keys in migration if they exist 2016-07-08 11:45:36 +01:00
James Allen
8b090c0a60 Don't call sadd with blank args 2016-07-07 16:41:38 +01:00
James Allen
6bc78ccf77 Add in some null checks on comparing backend results 2016-07-07 11:24:33 +01:00
James Allen
1c62a1c5a1 Don't propagate error on secondary timeout 2016-07-07 09:46:15 +01:00
James Allen
edf1e1ab49 Initialize primaryResult to an array, not null 2016-07-06 15:02:05 +01:00
James Allen
59883023ca Ignore different order of results from smembers 2016-07-06 14:24:27 +01:00
James Allen
89f90c1b04 Timeout secondary requests if they take longer than 200ms 2016-07-06 12:25:36 +01:00
James Allen
508a95c19b Use the main redis instance for locks 2016-07-06 11:50:02 +01:00
James Allen
e912ccc562 Wait for both backends to return so that they are always in sync 2016-07-05 16:07:47 +01:00
James Allen
dfd45bd23c Add timers to time how long each redis request takes 2016-07-04 11:14:23 +01:00
James Allen
149351aa49 Remove debugging log lines 2016-06-29 21:18:13 +01:00
James Allen
fde334acc4 Create script that will ensure data is migrate between redis backends 2016-06-29 12:57:56 +01:00
James Allen
dbe03e2757 Track metrics of when backends match or disagree 2016-06-28 16:49:15 +01:00
James Allen
da89ff7172 Add in external health check rather than internal 2016-06-23 18:00:03 +01:00
James Allen
414ab5d6a9 Refactor TCP redis monitoring to be more explicit 2016-06-23 15:59:25 +01:00
James Allen
8ef03c3d2f Add in application layer monitoring of the health of each cluster node 2016-06-23 15:38:51 +01:00
James Allen
39d920682a Only pass through keys to redis that are explicitly set in the config 2016-06-17 14:11:22 +01:00
James Allen
40b7da8edb Support endpoints and masterName in redis config 2016-06-17 14:05:54 +01:00
James Allen
bc00aab7b1 Explicitly separate calls to web and docupdater redis instances 2016-06-17 12:17:22 +01:00
James Allen
9ff3026807 Merge branch 'master' into ja_redis_cluster 2016-06-09 10:00:58 +01:00
James Allen
48a92b28e5 Don't run redis commands in parallel for easier consistency reasoning 2016-06-09 09:54:13 +01:00
James Allen
c823e06912 Don't run redis commands in parallel for easier consistency reasoning 2016-06-08 16:42:09 +01:00
James Allen
c6605ed5f0 Fix misapplied arguments to multi commands 2016-06-08 16:41:58 +01:00
James Allen
b4936f62af Check that return values from different redis backends match 2016-06-08 12:18:37 +01:00
James Allen
ef43e2b325 Configure backend database as redis cluster 2016-06-07 18:38:32 +01:00
James Allen
1db6f8f159 Pull out rclient into RedisBackend that supports sending requests to multiple rclients 2016-06-07 17:58:18 +01:00