James Allen
e7ff05e792
Fix UpdateManager tests
2016-12-01 18:06:33 +00:00
James Allen
889f5fdf9f
Fix ShareJsDB tests
2016-12-01 17:14:40 +00:00
James Allen
b6c93c718d
Update TrackChangesManagerTests -> HistoryManagerTests
2016-12-01 16:50:55 +00:00
James Allen
d878dd5758
Fix RedisManagerTests
2016-12-01 16:49:53 +00:00
James Allen
9ee913be39
Update PersistenceManagerTests
2016-12-01 16:40:15 +00:00
James Allen
fb39e37fe0
Update DocumentManager tests
2016-12-01 16:27:40 +00:00
James Allen
d36f1a86f1
Merge branch 'master' into ja-track-changes
...
Conflicts:
app/coffee/PersistenceManager.coffee
2016-12-01 16:06:13 +00:00
James Allen
efab68e6b2
Add in guards on bad data from web API
2016-11-29 17:13:16 +00:00
James Allen
546d9e9cee
Remove left over test
2016-11-29 17:13:07 +00:00
James Allen
2852043a6d
Don't store doc version in Mongo directly, instead use docstore
2016-11-29 17:06:23 +00:00
James Allen
f7ba0946be
More acceptance test fixes
2016-11-29 15:09:11 +00:00
James Allen
2df5c08389
Update acceptance tests to use redis-sharelatex
2016-11-29 15:05:02 +00:00
James Allen
db8b4bf991
Update acceptance test script
2016-11-29 14:57:05 +00:00
James Allen
e739e86c48
Get basic ChangeTracker hooked up. WIP
2016-11-28 10:14:42 +00:00
James Allen
5ce15c4d60
Move check of zero length op array up a level
2016-09-12 10:37:41 +01:00
James Allen
993aab7a78
Don't try to redis rpush with no arguments
2016-09-12 10:37:32 +01:00
James Allen
2d82d56f93
Process updates one at a time and write into redis after each one
2016-09-09 11:01:14 +01:00
James Allen
c85fd7ef41
Merge branch 'master' into ja-consistency-refactor
2016-09-07 11:08:04 +01:00
Brian Gough
9bc7594226
clean up redis query
2016-09-05 10:31:54 +01:00
James Allen
8779f3f686
Only write DocOps atomically with version and lines, after all docs are applied
2016-08-23 16:00:46 +01:00
James Allen
87f3e5e809
Remove old migration code
2016-08-23 11:03:37 +01:00
Shane Kilkelly
39cc9cda6d
Remove docker-related config. test:acceptance:docker now depends on acceptance-test-runner-sharelatex.
2016-07-18 11:09:04 +01:00
Shane Kilkelly
4c0dc5a0ef
Increase timeout for test.
2016-07-08 16:05:36 +01:00
Shane Kilkelly
0a5f95f22d
Move docker-entrypoint.sh into the test/acceptance/scripts directory
2016-07-08 15:52:02 +01:00
Shane Kilkelly
32d06b805c
run only the acceptance tests inside container, rather than compile and test.
2016-07-08 15:15:41 +01:00
Shane Kilkelly
b16b34d7c6
only execute app inside container, instead of compile and execute.
2016-07-08 14:50:30 +01:00
Shane Kilkelly
6d9aecae4c
capture exit code of the test run.
2016-07-08 14:31:43 +01:00
Shane Kilkelly
879482a955
grunt scripts to run the docker acceptance tests
2016-07-08 12:09:01 +01:00
Shane Kilkelly
bca8f6c376
working docker image which can run the acceptance tests
2016-07-08 10:51:22 +01:00
Shane Kilkelly
c38d903f92
Basic docker file.
2016-07-07 16:07:14 +01:00
James Allen
1c62a1c5a1
Don't propagate error on secondary timeout
2016-07-07 09:46:15 +01:00
James Allen
59883023ca
Ignore different order of results from smembers
2016-07-06 14:24:27 +01:00
James Allen
89f90c1b04
Timeout secondary requests if they take longer than 200ms
2016-07-06 12:25:36 +01:00
James Allen
508a95c19b
Use the main redis instance for locks
2016-07-06 11:50:02 +01:00
James Allen
dfd45bd23c
Add timers to time how long each redis request takes
2016-07-04 11:14:23 +01:00
James Allen
fde334acc4
Create script that will ensure data is migrate between redis backends
2016-06-29 12:57:56 +01:00
James Allen
dbe03e2757
Track metrics of when backends match or disagree
2016-06-28 16:49:15 +01:00
James Allen
e04c946ecd
Fix unit tests
2016-06-23 18:04:26 +01:00
James Allen
da89ff7172
Add in external health check rather than internal
2016-06-23 18:00:03 +01:00
James Allen
414ab5d6a9
Refactor TCP redis monitoring to be more explicit
2016-06-23 15:59:25 +01:00
James Allen
8ef03c3d2f
Add in application layer monitoring of the health of each cluster node
2016-06-23 15:38:51 +01:00
James Allen
27a74d6b71
Improve consistency of stress test script
2016-06-21 14:31:20 +01:00
James Allen
392beac57b
Improve error reporting and retry ops in stress test script
2016-06-20 15:01:39 +01:00
James Allen
7915c2ab6c
Tweak stress test to work on server
2016-06-20 13:56:46 +01:00
James Allen
e5cf856ddf
Add in stress test script
2016-06-17 12:17:43 +01:00
James Allen
bc00aab7b1
Explicitly separate calls to web and docupdater redis instances
2016-06-17 12:17:22 +01:00
James Allen
9ff3026807
Merge branch 'master' into ja_redis_cluster
2016-06-09 10:00:58 +01:00
James Allen
48a92b28e5
Don't run redis commands in parallel for easier consistency reasoning
2016-06-09 09:54:13 +01:00
James Allen
c823e06912
Don't run redis commands in parallel for easier consistency reasoning
2016-06-08 16:42:09 +01:00
James Allen
b4936f62af
Check that return values from different redis backends match
2016-06-08 12:18:37 +01:00
James Allen
1db6f8f159
Pull out rclient into RedisBackend that supports sending requests to multiple rclients
2016-06-07 17:58:18 +01:00
James Allen
e38e883081
Clean up and consolidate RedisManager tests for easy refactoring later
2016-06-01 12:22:47 +01:00
James Allen
8004e022fe
Fix async version and fix affected unit tests
2016-06-01 11:49:24 +01:00
James Allen
05b09a447e
Ensure that all multi call keys will hash to the same node in cluster
2016-06-01 11:40:12 +01:00
James Allen
f707783aba
Remove old methods of triggering doc updates
...
Remove the old pub/sub listener which is no longer used.
Also remove the DocsWithPendingUpdates set, which used to track
docs waiting to be updated. This was necessary incase messages were
missed on the pub/sub channel, so we knew which docs still had pending
updates. However, now we use the BLPOP queue, these updates just sit in
the queue until a consumer comes back to continue consuming them.
2016-06-01 11:28:23 +01:00
James Allen
b2e58266ae
Remove unused total doc counting redis commands
2016-06-01 11:03:39 +01:00
James Allen
32ce44f9c6
Remove unused gzipping logic
2016-06-01 10:53:50 +01:00
James Allen
1d7cbd8a78
Fix unit tests
2016-05-31 13:32:15 +01:00
James Allen
11667e8250
Return semantic error if the requested ops are not available when getting a document
2016-05-31 13:24:19 +01:00
Shane Kilkelly
048fd19418
Add one to size of line.
...
To account for newline characters in the original document
2016-05-12 09:26:50 +01:00
Shane Kilkelly
82d5a7fafd
check total size of lines, rather than content-length
2016-05-11 15:55:21 +01:00
Shane Kilkelly
210a61112b
reject payloads larger than 2MB for setDoc
2016-04-29 15:08:21 +01:00
James Allen
945c728db2
Use signed locks so only the locking party can remove their lock
2016-04-13 11:59:56 +01:00
James Allen
6c79ab4321
Don't let HTTP calls take longer than 5 seconds since we're inside a 30 second lock
2016-04-12 17:10:39 +01:00
James Allen
3c7c318ea0
Clean up and speed up unit tests by making sure requires are mocked
2016-01-20 17:36:06 +00:00
James Allen
af5d01e440
Flush track changes when unloading data from redis
2016-01-20 15:05:31 +00:00
James Allen
184c9031cd
Delete doc from redis after a full set if it wasn't in redis before
2016-01-20 14:31:25 +00:00
Brian Gough
6e97521971
Revert "Merge pull request #4 from sharelatex/flush-ops-from-redis"
...
This reverts commit 84a9ad8b67b860a92271cf7bb669cf21cc0b5b86, reversing
changes made to a6d4649f4f2db06862e8cbc5e132a073ce8c20b2.
2015-12-01 15:58:40 +00:00
Brian Gough
64f5d5526e
Revert "update tests"
...
This reverts commit 501e891760fd5c5bd0e6ffc75f866fd565561f31.
2015-12-01 15:48:31 +00:00
Brian Gough
4bf90afe0e
update tests
2015-11-30 15:22:21 +00:00
Brian Gough
529a061567
Merge pull request #4 from sharelatex/flush-ops-from-redis
...
push ops to track changes when document is flushed
2015-11-30 10:46:21 +00:00
James Allen
03e9d7390f
Add in mongo health check end point
2015-08-31 15:57:26 +01:00
James Allen
d27d2b90a2
Merge pull request #5 from sharelatex/utf16_fix
...
Replace UTF-16 surrogate characters with 'replacement character'
2015-06-12 10:18:34 +01:00
James Allen
a3847d21d5
Replace UTF-16 surrogate characters with 'replacement character'
...
In Javascript, characters are 16-bits wide. It does not understand surrogates as characters.
From Wikipedia (http://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane ):
"The High Surrogates (U+D800–U+DBFF) and Low Surrogate (U+DC00–U+DFFF) codes are reserved
for encoding non-BMP characters in UTF-16 by using a pair of 16-bit codes: one High Surrogate
and one Low Surrogate. A single surrogate code point will never be assigned a character.""
The main offender seems to be \uD835 as a stand alone character, which would be the first
16-bit character of a blackboard bold character (http://www.fileformat.info/info/unicode/char/1d400/index.htm ).
Something must be going on client side that is screwing up the encoding and splitting the
two 16-bit characters so that \uD835 is standalone.
2015-06-12 10:14:35 +01:00
Brian
60db8bf5de
flush ops when document is flushed
2015-05-28 14:48:44 +01:00
Brian Gough
85a4bf0da6
add tests calling zlib.gzip directly (for debugging)
2015-05-19 14:39:47 +01:00
Brian Gough
85eab2e967
modify DispatchManager test to allow for slow shutdown
2015-05-19 14:23:32 +01:00
Brian Gough
57f6919485
modify LockManager test to avoid dependence on timing
2015-05-18 09:03:51 +01:00
Brian Gough
e61beed92f
suppress logging in ZipManager tests
2015-03-30 16:58:00 +01:00
Brian Gough
495af5d568
remove console.logs from tests
2015-03-30 16:56:07 +01:00
Brian Gough
8e8ee5b3da
fix tests
2015-03-30 16:41:23 +01:00
Brian Gough
86505047a3
added unit test (work in progress)
2015-03-27 17:04:38 +00:00
Brian Gough
27d466aa85
added acceptance test for redis compression
2015-03-27 13:17:38 +00:00
Brian Gough
e3d73d4456
add gzip support for large documents to reduce load on redis
2015-03-25 16:53:20 +00:00
Henry Oswald
c137e05bfd
fixed acceptence tests
2014-10-07 12:31:16 +01:00
Henry Oswald
5ff8983528
implemeneted redis-sharelatex
2014-10-07 12:08:36 +01:00
Henry Oswald
9c85e9b848
moved back to not include sentinal while we upgrade to 0.10.x node
2014-09-29 12:07:32 +01:00
Henry Oswald
7b1b6d27f3
fixed missing new redis depenency
2014-09-28 20:14:14 +01:00
Henry Oswald
d7894c4ea7
fixed tests
2014-09-26 17:57:15 +01:00
James Allen
9da118c809
Process updates in the background so that workers only act as dispatchers
2014-08-11 14:16:05 +01:00
James Allen
cab5509076
Listen on queue via BLPOP rather than pub/sub
2014-08-07 11:45:53 +01:00
James Allen
ffd10a8439
Don't send or get versioning info to/from web
...
The version number is only used by the doc updater and so can be cleanly
encapsulated in a collection that only the doc updater knows about. The version
is already stored under docOps, so continue to tore it there.
2014-05-15 11:13:16 +01:00
James Allen
e25fb7e435
Keeping writing and reading updates from Mongo to cover the deploy period
2014-05-14 16:04:25 +01:00
James Allen
bdfe018cad
Log out warning when using Mongo for version still
2014-05-14 14:16:27 +01:00
James Allen
0199f2e129
Use version from web, with fallback to old mongo collection
2014-05-14 13:28:17 +01:00
James Allen
8973969224
Merge branch 'master' into remove_mongo_doc_ops
...
Conflicts:
app/coffee/DocOpsManager.coffee
test/acceptance/coffee/ApplyingUpdatesToADocTests.coffee
test/acceptance/coffee/FlushingDocsTests.coffee
test/unit/coffee/DocOpsManager/DocOpsManagerTests.coffee
test/unit/coffee/RedisManager/prependDocOpsTests.coffee
test/unit/coffee/RedisManager/pushDocOpTests.coffee
2014-05-14 12:39:40 +01:00
James Allen
d1434f7646
Increase redis lock expiry time to 30 seconds
2014-05-07 09:48:29 +01:00
James Allen
c2ebaaa338
Split lines on Windows line endings too
2014-04-10 12:44:46 +01:00
James Allen
c0be3ef37b
Put doc_ids with history changes into project level set
2014-03-21 12:41:05 +00:00
James Allen
2d28f1903f
Flush to the track changes api using the project id as well
2014-03-19 15:56:44 +00:00
