Commit graph

291 commits

Author SHA1 Message Date
Hayden Faulds
7e86afe55e version file renames 2017-11-08 09:20:38 +00:00
Hayden Faulds
6d571e6d23 version document renames 2017-11-08 09:20:37 +00:00
Hayden Faulds
b8052e7612 allow settings.redis.project_history to be undefined 2017-10-25 13:56:38 +01:00
Hayden Faulds
5b7be93c79 Merge pull request #2 from sharelatex/hof-doc-pathname
Add metadata to updates for history
2017-10-25 13:13:00 +01:00
Brian Gough
20655b6eeb remove unused requires of async 2017-10-23 15:59:32 +01:00
Hayden Faulds
be41a1614d decorate ops with pathname and doc_length 2017-10-23 15:32:25 +01:00
Hayden Faulds
962520fca8 flush track-changes and project-history in HistoryManager 2017-10-23 15:32:25 +01:00
Hayden Faulds
d003aef31c conditionally enqueue history updates for project 2017-10-23 15:32:25 +01:00
Hayden Faulds
81f998afe9 return pathname from DocumentManager.getDoc 2017-10-23 14:23:21 +01:00
Hayden Faulds
748315aadc handle pathname in DocumentManager.getDoc 2017-10-23 14:23:21 +01:00
Hayden Faulds
cbdace7386 store pathname in Redis cache 2017-10-23 14:23:21 +01:00
Hayden Faulds
05b93a629a return pathname from PersistenceManager 2017-10-23 14:17:14 +01:00
Brian Gough
7cbb3e7af8 change getProjectDocs endpoint from GET to POST
also note that it flushes docs if they are considered old (i.e. not
recently flushed)
2017-10-11 15:30:29 +01:00
Brian Gough
8d3f82360c update DocumentManager.getDoc signature
keep alreadyLoaded flag at the end for consistency
2017-10-11 14:25:33 +01:00
Brian Gough
2bbbf3c005 add unflushed time to doc in redis 2017-10-06 12:34:28 +01:00
Brian Gough
ad745f133d Merge pull request #59 from sharelatex/bg-reset-project-state
add endpoint to clear project state
2017-09-13 10:39:47 +01:00
Brian Gough
f34c12fdc7 add endpoint to clear project state 2017-09-08 15:50:26 +01:00
Brian Gough
215a939d4e add comment about log.error for excludeVersions 2017-09-08 13:43:22 +01:00
Brian Gough
1e54b4c72c log all errors in getProjectDocs 2017-09-07 14:12:48 +01:00
Brian Gough
8372911a1b return doclines as array from getProjectDocs
for compatibility with getDoc
2017-08-11 16:56:30 +01:00
Brian Gough
f696ccb0d9 add unit tests for getProjectDocs 2017-08-10 16:41:28 +01:00
Brian Gough
1eb80936f2 fix log line 2017-08-10 14:57:27 +01:00
Brian Gough
47bebf9638 no need to support optional keys.projectState 2017-08-09 16:47:09 +01:00
Brian Gough
bd6b5b2c59 use v instead of rev in getProjectDocs 2017-08-09 15:30:53 +01:00
Brian Gough
9f3ec72f81 switch to single get/set method for getProjectDocs
if project state hasn't changed, return the docs. Otherwise set the hash
and return a 409 Conflict response.
2017-08-07 14:43:28 +01:00
Brian Gough
f5f516a910 delete clsi state when deleting doc (if used) 2017-08-03 15:04:12 +01:00
Brian Gough
860537146b avoid logging doclines in getProjectDocs 2017-08-03 15:04:12 +01:00
Brian Gough
80be5adc0d change filter to exclude in getProjectDocs 2017-08-03 15:04:06 +01:00
Brian Gough
0109585092 provide endpoint for current project docs in redis 2017-08-02 13:42:10 +01:00
Shane Kilkelly
c3465cf3d4 Reduce retry delay to 10ms, and refactor 2017-07-18 15:28:18 +01:00
Shane Kilkelly
0d46c3e2b3 WIP: auto-retry web requests 2017-07-17 10:08:21 +01:00
Brian Gough
c21cc013f3 use null instead of err in callbacks on success 2017-07-12 10:48:32 +01:00
Brian Gough
bb0dc4b4d5 release the lock if it took too long to acquire it 2017-07-12 10:48:32 +01:00
Brian Gough
54c0fc1180 put a 5 second timeout in getting redis lock 2017-07-11 14:28:58 +01:00
Brian Gough
ffea31290c return total time from the profile .end() method 2017-07-11 14:28:58 +01:00
Brian Gough
59ace9d5c5 avoid long lines in timeout check 2017-06-26 16:36:05 +01:00
Brian Gough
963e513057 add a timeout for getPreviousDocOps
it uses several redis operations and this makes it prone to subsequent
timeouts if getDoc succeeds but is slow
2017-06-23 15:50:21 +01:00
James Allen
54f128289f Merge pull request #50 from sharelatex/bg-metric-for-unlock-errors
add metric for unlock errors
2017-06-13 14:46:40 +01:00
James Allen
7d22e4422c Merge pull request #49 from sharelatex/bg-add-redis-timeout
add redis timeout
2017-06-13 14:46:32 +01:00
James Allen
a3d7260619 Ensure expires is called after the key is created 2017-06-08 16:43:06 +01:00
Brian Gough
2e5d57cd99 don't increase rate limit when tasks are failing 2017-06-07 11:47:28 +01:00
Brian Gough
1ef258c878 add a timeout on the redis getDoc request 2017-06-07 11:34:42 +01:00
Brian Gough
673f4228ce add metric for unlock errors
sentry does not record them reliably, due to rate limiting
2017-06-05 16:29:58 +01:00
Brian Gough
12e8eaa9b6 fix bug in doclines check 2017-05-31 16:08:33 +01:00
Brian Gough
bed64d26c8 check if doc is missing from DocsIn set 2017-05-31 15:33:59 +01:00
Brian Gough
fa089dc28d avoid graphite drift in active worker count 2017-05-30 14:37:01 +01:00
Brian Gough
891ffda3bf put a rate limit on worker calls 2017-05-26 15:08:35 +01:00
Brian Gough
c43f97e8e3 add metric for active workers 2017-05-24 12:12:15 +01:00
Brian Gough
a3947c587f fix bug, jsonOps is array not string 2017-05-22 15:20:28 +01:00
Brian Gough
54ad45c870 add null byte check for ranges 2017-05-22 15:00:05 +01:00
Brian Gough
a13f055d62 add null byte check for ops 2017-05-22 15:00:00 +01:00
Brian Gough
78828ebbc5 fine grained logging for cpu usage in applyUpdate 2017-05-19 16:00:16 +01:00
Brian Gough
a550078071 replace verbose logging by slow query log 2017-05-18 15:04:12 +01:00
Brian Gough
95596061d4 clarify calculation of process.hrtime 2017-05-18 15:02:54 +01:00
Brian Gough
e00f4dde8e log profilers start and end times for reference 2017-05-18 15:02:44 +01:00
Brian Gough
e11f64f83a increase profiling cutoff to 1s 2017-05-18 15:02:08 +01:00
Brian Gough
b6efb051e1 add profiling 2017-05-18 11:17:08 +01:00
Brian Gough
387ecac6cb log for each get/try/release lock 2017-05-18 11:17:08 +01:00
Brian Gough
dff6e2c3da bug fix in backoff 2017-05-16 16:31:28 +01:00
Brian Gough
c7d8fbbb8a exponential backoff for lock up to maximum time 2017-05-16 15:22:02 +01:00
Brian Gough
b15d2ef796 Revert "fail safely if doc cannot be loaded" 2017-05-15 16:18:40 +01:00
Paulo Jorge Reis
2c3f0aa6b4 Merge pull request #34 from sharelatex/pr-bulk-actions
Add methods to bulk accept changes.
2017-05-15 15:14:33 +01:00
Paulo Reis
ef7fa59257 Update ranges tracker. 2017-05-15 11:11:14 +01:00
Paulo Reis
05d7d1b8c6 Merge branch 'master' into pr-bulk-actions 2017-05-15 10:35:16 +01:00
Brian Gough
2ee40d0748 ensure document is in redis before consuming ops 2017-05-12 14:22:18 +01:00
Brian Gough
be96548199 log number of updates for future debugging 2017-05-12 14:07:59 +01:00
Brian Gough
d4a8d88750 put a limit on the number of ops per iteration 2017-05-12 13:15:59 +01:00
Paulo Reis
5677c7ad9a Update range tracker. 2017-05-11 11:31:19 +01:00
Brian Gough
36407ac726 rename HistoryManager pushUncompressedHistoryOps 2017-05-09 12:02:27 +01:00
Brian Gough
7ce6285e3d increase flush threshold to 100 ops 2017-05-09 10:44:26 +01:00
Brian Gough
fdf5e8e0b8 get history ops length directly from redis update 2017-05-09 10:34:31 +01:00
Brian Gough
2d158b03d7 rename pushUncompressedHistoryOps 2017-05-09 09:32:56 +01:00
Paulo Reis
144804e516 Update tests with new methods. 2017-05-08 16:34:17 +01:00
Paulo Reis
d7ac57d054 Use new bulk method. 2017-05-08 16:08:42 +01:00
Paulo Reis
70341e4e80 Add bulk method from range tracker. 2017-05-08 16:08:34 +01:00
Brian Gough
79d8fced49 make history update more atomic 2017-05-08 15:58:29 +01:00
Paulo Reis
e96e6d6a29 Remove unused method. 2017-05-05 15:17:19 +01:00
Paulo Reis
9b8a2c3030 Consolidate methods to accept changes. 2017-05-05 15:12:06 +01:00
Paulo Reis
d530ee9501 Add methods to bulk accept changes. 2017-05-04 15:32:54 +01:00
James Allen
4104ca4889 Add in separate redis config for the lock and fix a few web -> realtime 2017-05-03 16:27:32 +01:00
James Allen
ed493d8ad3 Rename web -> realtime for consistency with realtime 2017-05-02 15:38:33 +01:00
James Allen
c5449ae282 Split out redis config for real-time and track-changes into separate cluster-compatible configs 2017-04-13 17:00:42 +01:00
James Allen
f21208e841 Use new redis-sharelatex instead of RedisBackend for cluster abstraction 2017-04-12 14:53:03 +01:00
Brian Gough
5583764d05 fix callback prototype in getLock 2017-03-30 15:31:47 +01:00
Brian Gough
c11618b475 improve unlock error handling 2017-03-30 15:31:34 +01:00
Brian Gough
527c02a00b log op versions pushed to track changes 2017-03-30 11:20:50 +01:00
James Allen
0245bfd031 Support an incoming undo flag for history restores 2017-03-21 11:20:38 +00:00
James Allen
199079a5b7 Merge branch 'master' into ja-cut-and-paste-comments 2017-03-20 10:29:01 +00:00
James Allen
9c5299ec7c Update RangeTracker to support upserting and moving comments for cut and paste 2017-03-16 15:49:41 +00:00
James Allen
5499a67d78 Add in a consistency check after applying updates that ranges still match 2017-03-15 14:12:06 +00:00
James Allen
b9e4492001 Wrap RangeTracker in try catch 2017-03-06 19:05:11 +00:00
James Allen
d8135749d2 Merge branch 'master' into ja-increase-limits-on-changes
Conflicts:
	app/coffee/RedisManager.coffee
2017-03-06 11:57:40 +00:00
James Allen
269433f250 Merge pull request #25 from sharelatex/bg-log-doclines-on-hash-mismatch
log doclines on hash mismatch
2017-03-06 11:53:24 +00:00
Brian Gough
d086e0b61b log doclines on hash mismatch 2017-03-03 15:57:44 +00:00
Brian Gough
b186a01c04 don't log errors from redis backend
this also picks up errors from RedisManager like "doc ops range is not
loaded in redis"
2017-03-03 15:27:42 +00:00
James Allen
d56bb55953 Revert PR #19 2017-03-01 16:49:46 +00:00
James Allen
9cfe651930 Merge pull request #19 from sharelatex/ja-track-changes
Don't consume partial parts of delete when inserting a change
2017-03-01 10:20:06 +01:00
James Allen
f544814dda Add in an absolute size limit on the ranges JSON object 2017-02-27 14:34:20 +01:00
James Allen
f211c282b2 Merge branch 'master' into ja-increase-limits-on-changes 2017-02-27 13:50:42 +01:00
James Allen
dd45a8dd7d Improve logging around hashes and null byte errors 2017-02-27 10:06:24 +01:00