Brian Gough
a550078071
replace verbose logging by slow query log
2017-05-18 15:04:12 +01:00
Brian Gough
95596061d4
clarify calculation of process.hrtime
2017-05-18 15:02:54 +01:00
Brian Gough
e00f4dde8e
log profilers start and end times for reference
2017-05-18 15:02:44 +01:00
Brian Gough
e11f64f83a
increase profiling cutoff to 1s
2017-05-18 15:02:08 +01:00
Brian Gough
b6efb051e1
add profiling
2017-05-18 11:17:08 +01:00
Brian Gough
387ecac6cb
log for each get/try/release lock
2017-05-18 11:17:08 +01:00
Brian Gough
dff6e2c3da
bug fix in backoff
2017-05-16 16:31:28 +01:00
Brian Gough
c7d8fbbb8a
exponential backoff for lock up to maximum time
2017-05-16 15:22:02 +01:00
Brian Gough
b15d2ef796
Revert "fail safely if doc cannot be loaded"
2017-05-15 16:18:40 +01:00
Paulo Jorge Reis
2c3f0aa6b4
Merge pull request #34 from sharelatex/pr-bulk-actions
...
Add methods to bulk accept changes.
2017-05-15 15:14:33 +01:00
Paulo Reis
ef7fa59257
Update ranges tracker.
2017-05-15 11:11:14 +01:00
Paulo Reis
05d7d1b8c6
Merge branch 'master' into pr-bulk-actions
2017-05-15 10:35:16 +01:00
Brian Gough
2ee40d0748
ensure document is in redis before consuming ops
2017-05-12 14:22:18 +01:00
Brian Gough
be96548199
log number of updates for future debugging
2017-05-12 14:07:59 +01:00
Brian Gough
d4a8d88750
put a limit on the number of ops per iteration
2017-05-12 13:15:59 +01:00
Paulo Reis
5677c7ad9a
Update range tracker.
2017-05-11 11:31:19 +01:00
Brian Gough
36407ac726
rename HistoryManager pushUncompressedHistoryOps
2017-05-09 12:02:27 +01:00
Brian Gough
7ce6285e3d
increase flush threshold to 100 ops
2017-05-09 10:44:26 +01:00
Brian Gough
fdf5e8e0b8
get history ops length directly from redis update
2017-05-09 10:34:31 +01:00
Brian Gough
2d158b03d7
rename pushUncompressedHistoryOps
2017-05-09 09:32:56 +01:00
Paulo Reis
144804e516
Update tests with new methods.
2017-05-08 16:34:17 +01:00
Paulo Reis
d7ac57d054
Use new bulk method.
2017-05-08 16:08:42 +01:00
Paulo Reis
70341e4e80
Add bulk method from range tracker.
2017-05-08 16:08:34 +01:00
Brian Gough
79d8fced49
make history update more atomic
2017-05-08 15:58:29 +01:00
Paulo Reis
e96e6d6a29
Remove unused method.
2017-05-05 15:17:19 +01:00
Paulo Reis
9b8a2c3030
Consolidate methods to accept changes.
2017-05-05 15:12:06 +01:00
Paulo Reis
d530ee9501
Add methods to bulk accept changes.
2017-05-04 15:32:54 +01:00
James Allen
4104ca4889
Add in separate redis config for the lock and fix a few web -> realtime
2017-05-03 16:27:32 +01:00
James Allen
ed493d8ad3
Rename web -> realtime for consistency with realtime
2017-05-02 15:38:33 +01:00
James Allen
c5449ae282
Split out redis config for real-time and track-changes into separate cluster-compatible configs
2017-04-13 17:00:42 +01:00
James Allen
f21208e841
Use new redis-sharelatex instead of RedisBackend for cluster abstraction
2017-04-12 14:53:03 +01:00
Brian Gough
5583764d05
fix callback prototype in getLock
2017-03-30 15:31:47 +01:00
Brian Gough
c11618b475
improve unlock error handling
2017-03-30 15:31:34 +01:00
Brian Gough
527c02a00b
log op versions pushed to track changes
2017-03-30 11:20:50 +01:00
James Allen
0245bfd031
Support an incoming undo flag for history restores
2017-03-21 11:20:38 +00:00
James Allen
199079a5b7
Merge branch 'master' into ja-cut-and-paste-comments
2017-03-20 10:29:01 +00:00
James Allen
9c5299ec7c
Update RangeTracker to support upserting and moving comments for cut and paste
2017-03-16 15:49:41 +00:00
James Allen
5499a67d78
Add in a consistency check after applying updates that ranges still match
2017-03-15 14:12:06 +00:00
James Allen
b9e4492001
Wrap RangeTracker in try catch
2017-03-06 19:05:11 +00:00
James Allen
d8135749d2
Merge branch 'master' into ja-increase-limits-on-changes
...
Conflicts:
app/coffee/RedisManager.coffee
2017-03-06 11:57:40 +00:00
James Allen
269433f250
Merge pull request #25 from sharelatex/bg-log-doclines-on-hash-mismatch
...
log doclines on hash mismatch
2017-03-06 11:53:24 +00:00
Brian Gough
d086e0b61b
log doclines on hash mismatch
2017-03-03 15:57:44 +00:00
Brian Gough
b186a01c04
don't log errors from redis backend
...
this also picks up errors from RedisManager like "doc ops range is not
loaded in redis"
2017-03-03 15:27:42 +00:00
James Allen
d56bb55953
Revert PR #19
2017-03-01 16:49:46 +00:00
James Allen
9cfe651930
Merge pull request #19 from sharelatex/ja-track-changes
...
Don't consume partial parts of delete when inserting a change
2017-03-01 10:20:06 +01:00
James Allen
f544814dda
Add in an absolute size limit on the ranges JSON object
2017-02-27 14:34:20 +01:00
James Allen
f211c282b2
Merge branch 'master' into ja-increase-limits-on-changes
2017-02-27 13:50:42 +01:00
James Allen
dd45a8dd7d
Improve logging around hashes and null byte errors
2017-02-27 10:06:24 +01:00
James Allen
60f3393c44
Do a naive check for null bytes after JSON.stringify on doc lines
2017-02-27 09:55:04 +01:00
James Allen
fd3dee0af5
Increase changes count limit to 2000
2017-02-24 14:58:47 +01:00
James Allen
80284e1b01
Only cancel deletes with inserts on undo and reject
2017-02-24 14:21:06 +01:00
James Allen
0420929e2a
Reduce the data logged when updating ranges
2017-02-23 13:57:47 +01:00
James Allen
e1481df76a
Don't consume partial parts of delete when inserting a change
2017-02-23 11:39:19 +01:00
James Allen
d6b5cc6389
Increase changes limit to 1200
2017-02-22 18:11:55 +01:00
Brian Gough
168bab6960
make hash error logging fine grained
2017-02-22 14:32:51 +00:00
Brian Gough
19e7522d3f
make logging of hash errors optional
2017-02-22 13:46:34 +00:00
Brian Gough
8d1b11f37a
Merge branch 'bg-verify-writes-II'
2017-02-21 16:03:48 +00:00
Brian Gough
5eb339e525
fix utf8 encoding in sha1 hash
2017-02-21 16:03:06 +00:00
Brian Gough
6b808522dc
Merge pull request #18 from sharelatex/bg-verify-writes
...
merge redis hash check
2017-02-20 15:55:19 +00:00
Brian Gough
590f8e7ced
Merge pull request #15 from sharelatex/bg-verify-writes
...
store sha1 hash of docLines in redis
2017-02-20 15:53:56 +00:00
Brian Gough
c57d3ce31c
compute hash on write in redis server
2017-02-20 14:09:09 +00:00
James Allen
660752dfa6
Increase change limit to 1000
2017-02-17 15:32:44 +01:00
James Allen
89537927b5
Update Ranges Tracker
2017-02-17 09:24:19 +01:00
James Allen
de7f760ea4
Update Ranges Tracker
2017-02-16 17:09:08 +01:00
Brian Gough
b5a4458b68
check sha1 hash value only if present
2017-02-15 14:12:36 +00:00
Brian Gough
bd70aaa76c
add sha1 hash support on writes
2017-02-14 16:11:43 +00:00
James Allen
8725aeab03
Allow deleting of comment ranges
2017-01-24 15:57:11 +01:00
James Allen
dc976fd9e9
Use thread id for comment id
2017-01-23 09:45:46 +01:00
James Allen
0705242a55
Increase max op age to 80
2017-01-17 11:45:10 +01:00
James Allen
0706feb26b
Add max limit on number of comments and changes per doc
2017-01-10 16:58:11 +01:00
James Allen
a3d5971d54
Update RangesTRacker
2017-01-10 11:59:09 +01:00
James Allen
540d0f7ec7
Make sure comment ids are unique
2017-01-10 11:55:38 +01:00
James Allen
be19532a1d
Add HTTP end point for accepting changes
2017-01-09 14:41:18 +01:00
James Allen
593e7260d4
Update RangesTracker
2017-01-09 10:52:06 +01:00
James Allen
7cac2f7d76
Generate deterministic range ids based on seed
2017-01-09 10:46:58 +01:00
James Allen
2c7029cc50
Revert "Support a {dr:...} op for deleting ranges"
...
This reverts commit 24c58e5ad430e0240533cc1e5c21122859fe8dc9.
2017-01-09 09:24:19 +01:00
James Allen
0f13cb3aa7
Support a {dr:...} op for deleting ranges
2017-01-06 16:58:51 +01:00
James Allen
59a06cd798
Accept comments with thread id as an op type
2016-12-13 15:51:47 +00:00
James Allen
47b19818ff
Add in new comment op type
2016-12-12 17:53:43 +00:00
James Allen
e3fee1a1d1
Rename 'track changes entries' -> 'ranges'
2016-12-08 12:31:43 +00:00
James Allen
3ea2e07993
Add tests for fetching and flushing track changes entries to persistence layer
2016-12-02 12:01:23 +00:00
James Allen
418405e8b9
Return track changes entries in HTTP request
2016-12-02 11:37:27 +00:00
James Allen
4fadd75ef3
Track changes based on flag on op, not global setting
2016-12-02 11:04:21 +00:00
James Allen
889f5fdf9f
Fix ShareJsDB tests
2016-12-01 17:14:40 +00:00
James Allen
9ee913be39
Update PersistenceManagerTests
2016-12-01 16:40:15 +00:00
James Allen
d36f1a86f1
Merge branch 'master' into ja-track-changes
...
Conflicts:
app/coffee/PersistenceManager.coffee
2016-12-01 16:06:13 +00:00
James Allen
efab68e6b2
Add in guards on bad data from web API
2016-11-29 17:13:16 +00:00
James Allen
2852043a6d
Don't store doc version in Mongo directly, instead use docstore
2016-11-29 17:06:23 +00:00
James Allen
e739e86c48
Get basic ChangeTracker hooked up. WIP
2016-11-28 10:14:42 +00:00
James Allen
5ce15c4d60
Move check of zero length op array up a level
2016-09-12 10:37:41 +01:00
James Allen
993aab7a78
Don't try to redis rpush with no arguments
2016-09-12 10:37:32 +01:00
James Allen
6ca48523d9
mapSeries -> eachSeries
2016-09-09 15:28:27 +01:00
James Allen
2d82d56f93
Process updates one at a time and write into redis after each one
2016-09-09 11:01:14 +01:00
James Allen
c85fd7ef41
Merge branch 'master' into ja-consistency-refactor
2016-09-07 11:08:04 +01:00
Brian Gough
9bc7594226
clean up redis query
2016-09-05 10:31:54 +01:00
James Allen
8779f3f686
Only write DocOps atomically with version and lines, after all docs are applied
2016-08-23 16:00:46 +01:00
James Allen
87f3e5e809
Remove old migration code
2016-08-23 11:03:37 +01:00
James Allen
a1bac9719a
Update secondary timeout for multi commands too
2016-07-12 11:45:10 +01:00
James Allen
d0c54f1be6
Increase secondary timeout to 600ms
2016-07-12 11:11:55 +01:00
James Allen
4f34cb7363
Only set keys in migration if they exist
2016-07-08 11:45:36 +01:00
James Allen
8b090c0a60
Don't call sadd with blank args
2016-07-07 16:41:38 +01:00
James Allen
6bc78ccf77
Add in some null checks on comparing backend results
2016-07-07 11:24:33 +01:00
James Allen
1c62a1c5a1
Don't propagate error on secondary timeout
2016-07-07 09:46:15 +01:00
James Allen
edf1e1ab49
Initialize primaryResult to an array, not null
2016-07-06 15:02:05 +01:00
James Allen
59883023ca
Ignore different order of results from smembers
2016-07-06 14:24:27 +01:00
James Allen
89f90c1b04
Timeout secondary requests if they take longer than 200ms
2016-07-06 12:25:36 +01:00
James Allen
508a95c19b
Use the main redis instance for locks
2016-07-06 11:50:02 +01:00
James Allen
e912ccc562
Wait for both backends to return so that they are always in sync
2016-07-05 16:07:47 +01:00
James Allen
dfd45bd23c
Add timers to time how long each redis request takes
2016-07-04 11:14:23 +01:00
James Allen
149351aa49
Remove debugging log lines
2016-06-29 21:18:13 +01:00
James Allen
fde334acc4
Create script that will ensure data is migrate between redis backends
2016-06-29 12:57:56 +01:00
James Allen
dbe03e2757
Track metrics of when backends match or disagree
2016-06-28 16:49:15 +01:00
James Allen
da89ff7172
Add in external health check rather than internal
2016-06-23 18:00:03 +01:00
James Allen
414ab5d6a9
Refactor TCP redis monitoring to be more explicit
2016-06-23 15:59:25 +01:00
James Allen
8ef03c3d2f
Add in application layer monitoring of the health of each cluster node
2016-06-23 15:38:51 +01:00
James Allen
39d920682a
Only pass through keys to redis that are explicitly set in the config
2016-06-17 14:11:22 +01:00
James Allen
40b7da8edb
Support endpoints and masterName in redis config
2016-06-17 14:05:54 +01:00
James Allen
bc00aab7b1
Explicitly separate calls to web and docupdater redis instances
2016-06-17 12:17:22 +01:00
James Allen
9ff3026807
Merge branch 'master' into ja_redis_cluster
2016-06-09 10:00:58 +01:00
James Allen
48a92b28e5
Don't run redis commands in parallel for easier consistency reasoning
2016-06-09 09:54:13 +01:00
James Allen
c823e06912
Don't run redis commands in parallel for easier consistency reasoning
2016-06-08 16:42:09 +01:00
James Allen
c6605ed5f0
Fix misapplied arguments to multi commands
2016-06-08 16:41:58 +01:00
James Allen
b4936f62af
Check that return values from different redis backends match
2016-06-08 12:18:37 +01:00
James Allen
ef43e2b325
Configure backend database as redis cluster
2016-06-07 18:38:32 +01:00
James Allen
1db6f8f159
Pull out rclient into RedisBackend that supports sending requests to multiple rclients
2016-06-07 17:58:18 +01:00
James Allen
ea8f4a6d7a
Remove unused getDocOpsLength method
2016-06-01 11:58:15 +01:00
James Allen
05b09a447e
Ensure that all multi call keys will hash to the same node in cluster
2016-06-01 11:40:12 +01:00
James Allen
f707783aba
Remove old methods of triggering doc updates
...
Remove the old pub/sub listener which is no longer used.
Also remove the DocsWithPendingUpdates set, which used to track
docs waiting to be updated. This was necessary incase messages were
missed on the pub/sub channel, so we knew which docs still had pending
updates. However, now we use the BLPOP queue, these updates just sit in
the queue until a consumer comes back to continue consuming them.
2016-06-01 11:28:23 +01:00
James Allen
b2e58266ae
Remove unused total doc counting redis commands
2016-06-01 11:03:39 +01:00
James Allen
32ce44f9c6
Remove unused gzipping logic
2016-06-01 10:53:50 +01:00
James Allen
11667e8250
Return semantic error if the requested ops are not available when getting a document
2016-05-31 13:24:19 +01:00
Shane Kilkelly
048fd19418
Add one to size of line.
...
To account for newline characters in the original document
2016-05-12 09:26:50 +01:00
Shane Kilkelly
82d5a7fafd
check total size of lines, rather than content-length
2016-05-11 15:55:21 +01:00
Shane Kilkelly
210a61112b
reject payloads larger than 2MB for setDoc
2016-04-29 15:08:21 +01:00
James Allen
4ab4e27724
Include COUNT in lock and make RND per-process incase randomBytes blocks
2016-04-13 16:27:53 +01:00
James Allen
d15a22be26
Remove misleading log line about lock value
2016-04-13 12:20:53 +01:00
James Allen
945c728db2
Use signed locks so only the locking party can remove their lock
2016-04-13 11:59:56 +01:00
James Allen
6c79ab4321
Don't let HTTP calls take longer than 5 seconds since we're inside a 30 second lock
2016-04-12 17:10:39 +01:00
James Allen
af5d01e440
Flush track changes when unloading data from redis
2016-01-20 15:05:31 +00:00
James Allen
184c9031cd
Delete doc from redis after a full set if it wasn't in redis before
2016-01-20 14:31:25 +00:00
Brian Gough
e8f09f3357
added comment about flushing to track changes
2015-12-01 16:03:05 +00:00
Brian Gough
6e97521971
Revert "Merge pull request #4 from sharelatex/flush-ops-from-redis"
...
This reverts commit 84a9ad8b67b860a92271cf7bb669cf21cc0b5b86, reversing
changes made to a6d4649f4f2db06862e8cbc5e132a073ce8c20b2.
2015-12-01 15:58:40 +00:00
Brian Gough
13e22e1802
Revert "only flush to track-changes when ops are queued"
...
This reverts commit 0baa8f989481ca263fa71e90af21f13b29c29504.
2015-12-01 15:48:49 +00:00
Brian Gough
7a9577e081
only flush to track-changes when ops are queued
2015-11-30 15:17:11 +00:00
Brian Gough
529a061567
Merge pull request #4 from sharelatex/flush-ops-from-redis
...
push ops to track changes when document is flushed
2015-11-30 10:46:21 +00:00
James Allen
2589e2d417
Gracefully return when an op has already been submitted
...
It is not a fatal error if an op has already been submitted. We just
need to send an ack back to the client that submitted it and continue.
If we detect a duplicate op, set dup: true on the op and pass it back
to real-time for distributing. The dup: true flag will ensure it only
gets acknowledged to the submitting client, not everyone.
2015-11-19 10:54:28 +00:00
James Allen
e73890bfc2
Error if update would make document too long
2015-11-06 12:52:03 +00:00
James Allen
03e9d7390f
Add in mongo health check end point
2015-08-31 15:57:26 +01:00
James Allen
d27d2b90a2
Merge pull request #5 from sharelatex/utf16_fix
...
Replace UTF-16 surrogate characters with 'replacement character'
2015-06-12 10:18:34 +01:00
James Allen
333591d087
Extra null check
2015-06-12 10:16:33 +01:00
James Allen
a3847d21d5
Replace UTF-16 surrogate characters with 'replacement character'
...
In Javascript, characters are 16-bits wide. It does not understand surrogates as characters.
From Wikipedia (http://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane ):
"The High Surrogates (U+D800–U+DBFF) and Low Surrogate (U+DC00–U+DFFF) codes are reserved
for encoding non-BMP characters in UTF-16 by using a pair of 16-bit codes: one High Surrogate
and one Low Surrogate. A single surrogate code point will never be assigned a character.""
The main offender seems to be \uD835 as a stand alone character, which would be the first
16-bit character of a blackboard bold character (http://www.fileformat.info/info/unicode/char/1d400/index.htm ).
Something must be going on client side that is screwing up the encoding and splitting the
two 16-bit characters so that \uD835 is standalone.
2015-06-12 10:14:35 +01:00
Brian
60db8bf5de
flush ops when document is flushed
2015-05-28 14:48:44 +01:00
Brian Gough
6cdf5615fc
remove old unused functions
2015-03-31 10:24:09 +01:00
Brian Gough
36f60d5bce
enforce minimum size of 1k for compression
2015-03-31 10:07:39 +01:00
Brian Gough
9bb08d7ba5
add ZipManager comments
2015-03-31 10:07:11 +01:00
Brian Gough
c8c12e8b41
fix error in ZipManager writesEnabled setting
2015-03-30 16:41:45 +01:00
Brian Gough
03564b2138
fix variable zip.minsize to match config name zip.minSize
2015-03-27 17:00:30 +00:00
Brian Gough
66fa170ac8
disable compression by default
2015-03-27 15:32:13 +00:00
Brian Gough
6bffa4d9e0
don't log docLines when document removed from redis
...
they can now be binary gzipped data which messes up the logs
2015-03-25 16:54:36 +00:00
Brian Gough
e3d73d4456
add gzip support for large documents to reduce load on redis
2015-03-25 16:53:20 +00:00
James Allen
c822bb6ea4
Add in missed error catching
2014-11-19 12:51:19 +00:00
Henry Oswald
5ff8983528
implemeneted redis-sharelatex
2014-10-07 12:08:36 +01:00
Henry Oswald
9c85e9b848
moved back to not include sentinal while we upgrade to 0.10.x node
2014-09-29 12:07:32 +01:00
Henry Oswald
672dd3de1b
works with sentinal v1
2014-09-26 17:38:59 +01:00
James Allen
9da118c809
Process updates in the background so that workers only act as dispatchers
2014-08-11 14:16:05 +01:00
James Allen
773e17df66
Send worker idle waiting times to statsd
2014-08-07 13:16:11 +01:00
James Allen
cab5509076
Listen on queue via BLPOP rather than pub/sub
2014-08-07 11:45:53 +01:00
James Allen
ffd10a8439
Don't send or get versioning info to/from web
...
The version number is only used by the doc updater and so can be cleanly
encapsulated in a collection that only the doc updater knows about. The version
is already stored under docOps, so continue to tore it there.
2014-05-15 11:13:16 +01:00
James Allen
e25fb7e435
Keeping writing and reading updates from Mongo to cover the deploy period
2014-05-14 16:04:25 +01:00
James Allen
bdfe018cad
Log out warning when using Mongo for version still
2014-05-14 14:16:27 +01:00
James Allen
0199f2e129
Use version from web, with fallback to old mongo collection
2014-05-14 13:28:17 +01:00
James Allen
8973969224
Merge branch 'master' into remove_mongo_doc_ops
...
Conflicts:
app/coffee/DocOpsManager.coffee
test/acceptance/coffee/ApplyingUpdatesToADocTests.coffee
test/acceptance/coffee/FlushingDocsTests.coffee
test/unit/coffee/DocOpsManager/DocOpsManagerTests.coffee
test/unit/coffee/RedisManager/prependDocOpsTests.coffee
test/unit/coffee/RedisManager/pushDocOpTests.coffee
2014-05-14 12:39:40 +01:00
James Allen
6011ce4783
Use new metrics module
2014-05-08 09:28:13 +01:00
James Allen
d1434f7646
Increase redis lock expiry time to 30 seconds
2014-05-07 09:48:29 +01:00
James Allen
c2ebaaa338
Split lines on Windows line endings too
2014-04-10 12:44:46 +01:00
James Allen
375427bf5e
Remove extraneous logging
2014-03-21 13:15:42 +00:00
James Allen
c0be3ef37b
Put doc_ids with history changes into project level set
2014-03-21 12:41:05 +00:00
James Allen
2d28f1903f
Flush to the track changes api using the project id as well
2014-03-19 15:56:44 +00:00
James Allen
4f878e000b
Allow source and user_id to be included when setting a document
2014-03-11 12:47:26 +00:00
James Allen
5d45e191f3
Don't crash when logging out error
2014-03-04 12:39:02 +00:00
James Allen
86195ce7c3
Add in load throttling based on a redis key
2014-02-28 19:09:29 +00:00
James Allen
3d70f9126e
Flush track changes api every 50 updates
2014-02-28 18:29:05 +00:00
James Allen
f3192da87f
Tell track changes api to flush doc when flushing doc to mongo
2014-02-26 15:56:52 +00:00
James Allen
dfd3ec993b
Ensure version is still returned from redis
2014-02-26 14:49:52 +00:00
James Allen
b13f70eadb
Push ops into uncompressedHistoryOps list
2014-02-24 16:52:12 +00:00
James Allen
6f19f46d96
Create method for pushing uncompressed ops into redis
2014-02-24 16:40:29 +00:00
James Allen
7a977ca597
Don't store ops in Mongo, just keep them in redis with a timeout
2014-02-24 16:13:36 +00:00
James Allen
414b669a2c
Point every redis connection to the new config
2014-02-13 10:36:26 +00:00
James Allen
a5d816272e
Remove unused DocumentUpdater.js file
2014-02-12 10:52:53 +00:00
James Allen
e1a7d4f24a
Initial open sourcing
2014-02-12 10:40:42 +00:00
