James Allen
bc00aab7b1
Explicitly separate calls to web and docupdater redis instances
2016-06-17 12:17:22 +01:00
James Allen
9ff3026807
Merge branch 'master' into ja_redis_cluster
2016-06-09 10:00:58 +01:00
James Allen
48a92b28e5
Don't run redis commands in parallel for easier consistency reasoning
2016-06-09 09:54:13 +01:00
James Allen
c823e06912
Don't run redis commands in parallel for easier consistency reasoning
2016-06-08 16:42:09 +01:00
James Allen
c6605ed5f0
Fix misapplied arguments to multi commands
2016-06-08 16:41:58 +01:00
James Allen
437e885812
Lock down to specific async version
2016-06-08 16:21:56 +01:00
James Allen
b4936f62af
Check that return values from different redis backends match
2016-06-08 12:18:37 +01:00
James Allen
ef43e2b325
Configure backend database as redis cluster
2016-06-07 18:38:32 +01:00
James Allen
1db6f8f159
Pull out rclient into RedisBackend that supports sending requests to multiple rclients
2016-06-07 17:58:18 +01:00
James Allen
e38e883081
Clean up and consolidate RedisManager tests for easy refactoring later
2016-06-01 12:22:47 +01:00
James Allen
ea8f4a6d7a
Remove unused getDocOpsLength method
2016-06-01 11:58:15 +01:00
James Allen
8004e022fe
Fix async version and fix affected unit tests
2016-06-01 11:49:24 +01:00
James Allen
05b09a447e
Ensure that all multi call keys will hash to the same node in cluster
2016-06-01 11:40:12 +01:00
James Allen
f707783aba
Remove old methods of triggering doc updates
...
Remove the old pub/sub listener which is no longer used.
Also remove the DocsWithPendingUpdates set, which used to track
docs waiting to be updated. This was necessary incase messages were
missed on the pub/sub channel, so we knew which docs still had pending
updates. However, now we use the BLPOP queue, these updates just sit in
the queue until a consumer comes back to continue consuming them.
2016-06-01 11:28:23 +01:00
James Allen
b2e58266ae
Remove unused total doc counting redis commands
2016-06-01 11:03:39 +01:00
James Allen
32ce44f9c6
Remove unused gzipping logic
2016-06-01 10:53:50 +01:00
James Allen
1d7cbd8a78
Fix unit tests
2016-05-31 13:32:15 +01:00
James Allen
11667e8250
Return semantic error if the requested ops are not available when getting a document
2016-05-31 13:24:19 +01:00
Shane Kilkelly
048fd19418
Add one to size of line.
...
To account for newline characters in the original document
2016-05-12 09:26:50 +01:00
Shane Kilkelly
82d5a7fafd
check total size of lines, rather than content-length
2016-05-11 15:55:21 +01:00
Shane Kilkelly
210a61112b
reject payloads larger than 2MB for setDoc
2016-04-29 15:08:21 +01:00
James Allen
4ab4e27724
Include COUNT in lock and make RND per-process incase randomBytes blocks
2016-04-13 16:27:53 +01:00
James Allen
d15a22be26
Remove misleading log line about lock value
2016-04-13 12:20:53 +01:00
James Allen
945c728db2
Use signed locks so only the locking party can remove their lock
2016-04-13 11:59:56 +01:00
James Allen
6c79ab4321
Don't let HTTP calls take longer than 5 seconds since we're inside a 30 second lock
2016-04-12 17:10:39 +01:00
James Allen
3c7c318ea0
Clean up and speed up unit tests by making sure requires are mocked
2016-01-20 17:36:06 +00:00
James Allen
af5d01e440
Flush track changes when unloading data from redis
2016-01-20 15:05:31 +00:00
James Allen
184c9031cd
Delete doc from redis after a full set if it wasn't in redis before
2016-01-20 14:31:25 +00:00
Brian Gough
e8f09f3357
added comment about flushing to track changes
2015-12-01 16:03:05 +00:00
Brian Gough
6e97521971
Revert "Merge pull request #4 from sharelatex/flush-ops-from-redis"
...
This reverts commit 84a9ad8b67b860a92271cf7bb669cf21cc0b5b86, reversing
changes made to a6d4649f4f2db06862e8cbc5e132a073ce8c20b2.
2015-12-01 15:58:40 +00:00
Brian Gough
13e22e1802
Revert "only flush to track-changes when ops are queued"
...
This reverts commit 0baa8f989481ca263fa71e90af21f13b29c29504.
2015-12-01 15:48:49 +00:00
Brian Gough
64f5d5526e
Revert "update tests"
...
This reverts commit 501e891760fd5c5bd0e6ffc75f866fd565561f31.
2015-12-01 15:48:31 +00:00
Brian Gough
4bf90afe0e
update tests
2015-11-30 15:22:21 +00:00
Brian Gough
7a9577e081
only flush to track-changes when ops are queued
2015-11-30 15:17:11 +00:00
Brian Gough
529a061567
Merge pull request #4 from sharelatex/flush-ops-from-redis
...
push ops to track changes when document is flushed
2015-11-30 10:46:21 +00:00
James Allen
2589e2d417
Gracefully return when an op has already been submitted
...
It is not a fatal error if an op has already been submitted. We just
need to send an ack back to the client that submitted it and continue.
If we detect a duplicate op, set dup: true on the op and pass it back
to real-time for distributing. The dup: true flag will ensure it only
gets acknowledged to the submitting client, not everyone.
2015-11-19 10:54:28 +00:00
James Allen
e73890bfc2
Error if update would make document too long
2015-11-06 12:52:03 +00:00
James Allen
03e9d7390f
Add in mongo health check end point
2015-08-31 15:57:26 +01:00
James Allen
d27d2b90a2
Merge pull request #5 from sharelatex/utf16_fix
...
Replace UTF-16 surrogate characters with 'replacement character'
2015-06-12 10:18:34 +01:00
James Allen
333591d087
Extra null check
2015-06-12 10:16:33 +01:00
James Allen
a3847d21d5
Replace UTF-16 surrogate characters with 'replacement character'
...
In Javascript, characters are 16-bits wide. It does not understand surrogates as characters.
From Wikipedia (http://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane ):
"The High Surrogates (U+D800–U+DBFF) and Low Surrogate (U+DC00–U+DFFF) codes are reserved
for encoding non-BMP characters in UTF-16 by using a pair of 16-bit codes: one High Surrogate
and one Low Surrogate. A single surrogate code point will never be assigned a character.""
The main offender seems to be \uD835 as a stand alone character, which would be the first
16-bit character of a blackboard bold character (http://www.fileformat.info/info/unicode/char/1d400/index.htm ).
Something must be going on client side that is screwing up the encoding and splitting the
two 16-bit characters so that \uD835 is standalone.
2015-06-12 10:14:35 +01:00
Brian Gough
e7e82fc89d
update port to match default track changes configuration
2015-05-28 16:43:58 +01:00
Brian
60db8bf5de
flush ops when document is flushed
2015-05-28 14:48:44 +01:00
Brian Gough
85a4bf0da6
add tests calling zlib.gzip directly (for debugging)
2015-05-19 14:39:47 +01:00
Brian Gough
85eab2e967
modify DispatchManager test to allow for slow shutdown
2015-05-19 14:23:32 +01:00
Brian Gough
57f6919485
modify LockManager test to avoid dependence on timing
2015-05-18 09:03:51 +01:00
Henry Oswald
d862227314
make startup message consistent
2015-04-30 16:01:22 +01:00
Brian Gough
ea92fe0e9e
Merge pull request #3 from sharelatex/gzip-large-docs
...
Gzip large docs
2015-03-31 10:58:18 +01:00
Brian Gough
6cdf5615fc
remove old unused functions
2015-03-31 10:24:09 +01:00
Brian Gough
36f60d5bce
enforce minimum size of 1k for compression
2015-03-31 10:07:39 +01:00
