Commit graph

115 commits

Author SHA1 Message Date
James Allen
437e885812 Lock down to specific async version 2016-06-08 16:21:56 +01:00
James Allen
e38e883081 Clean up and consolidate RedisManager tests for easy refactoring later 2016-06-01 12:22:47 +01:00
James Allen
ea8f4a6d7a Remove unused getDocOpsLength method 2016-06-01 11:58:15 +01:00
James Allen
8004e022fe Fix async version and fix affected unit tests 2016-06-01 11:49:24 +01:00
James Allen
05b09a447e Ensure that all multi call keys will hash to the same node in cluster 2016-06-01 11:40:12 +01:00
James Allen
f707783aba Remove old methods of triggering doc updates
Remove the old pub/sub listener which is no longer used.

Also remove the DocsWithPendingUpdates set, which used to track
docs waiting to be updated. This was necessary incase messages were
missed on the pub/sub channel, so we knew which docs still had pending
updates. However, now we use the BLPOP queue, these updates just sit in
the queue until a consumer comes back to continue consuming them.
2016-06-01 11:28:23 +01:00
James Allen
b2e58266ae Remove unused total doc counting redis commands 2016-06-01 11:03:39 +01:00
James Allen
32ce44f9c6 Remove unused gzipping logic 2016-06-01 10:53:50 +01:00
James Allen
1d7cbd8a78 Fix unit tests 2016-05-31 13:32:15 +01:00
James Allen
11667e8250 Return semantic error if the requested ops are not available when getting a document 2016-05-31 13:24:19 +01:00
Shane Kilkelly
048fd19418 Add one to size of line.
To account for newline characters in the original document
2016-05-12 09:26:50 +01:00
Shane Kilkelly
82d5a7fafd check total size of lines, rather than content-length 2016-05-11 15:55:21 +01:00
Shane Kilkelly
210a61112b reject payloads larger than 2MB for setDoc 2016-04-29 15:08:21 +01:00
James Allen
4ab4e27724 Include COUNT in lock and make RND per-process incase randomBytes blocks 2016-04-13 16:27:53 +01:00
James Allen
d15a22be26 Remove misleading log line about lock value 2016-04-13 12:20:53 +01:00
James Allen
945c728db2 Use signed locks so only the locking party can remove their lock 2016-04-13 11:59:56 +01:00
James Allen
6c79ab4321 Don't let HTTP calls take longer than 5 seconds since we're inside a 30 second lock 2016-04-12 17:10:39 +01:00
James Allen
3c7c318ea0 Clean up and speed up unit tests by making sure requires are mocked 2016-01-20 17:36:06 +00:00
James Allen
af5d01e440 Flush track changes when unloading data from redis 2016-01-20 15:05:31 +00:00
James Allen
184c9031cd Delete doc from redis after a full set if it wasn't in redis before 2016-01-20 14:31:25 +00:00
Brian Gough
e8f09f3357 added comment about flushing to track changes 2015-12-01 16:03:05 +00:00
Brian Gough
6e97521971 Revert "Merge pull request #4 from sharelatex/flush-ops-from-redis"
This reverts commit 84a9ad8b67b860a92271cf7bb669cf21cc0b5b86, reversing
changes made to a6d4649f4f2db06862e8cbc5e132a073ce8c20b2.
2015-12-01 15:58:40 +00:00
Brian Gough
13e22e1802 Revert "only flush to track-changes when ops are queued"
This reverts commit 0baa8f989481ca263fa71e90af21f13b29c29504.
2015-12-01 15:48:49 +00:00
Brian Gough
64f5d5526e Revert "update tests"
This reverts commit 501e891760fd5c5bd0e6ffc75f866fd565561f31.
2015-12-01 15:48:31 +00:00
Brian Gough
4bf90afe0e update tests 2015-11-30 15:22:21 +00:00
Brian Gough
7a9577e081 only flush to track-changes when ops are queued 2015-11-30 15:17:11 +00:00
Brian Gough
529a061567 Merge pull request #4 from sharelatex/flush-ops-from-redis
push ops to track changes when document is flushed
2015-11-30 10:46:21 +00:00
James Allen
2589e2d417 Gracefully return when an op has already been submitted
It is not a fatal error if an op has already been submitted. We just
need to send an ack back to the client that submitted it and continue.
If we detect a duplicate op, set dup: true on the op and pass it back
to real-time for distributing. The dup: true flag will ensure it only
gets acknowledged to the submitting client, not everyone.
2015-11-19 10:54:28 +00:00
James Allen
e73890bfc2 Error if update would make document too long 2015-11-06 12:52:03 +00:00
James Allen
03e9d7390f Add in mongo health check end point 2015-08-31 15:57:26 +01:00
James Allen
d27d2b90a2 Merge pull request #5 from sharelatex/utf16_fix
Replace UTF-16 surrogate characters with 'replacement character'
2015-06-12 10:18:34 +01:00
James Allen
333591d087 Extra null check 2015-06-12 10:16:33 +01:00
James Allen
a3847d21d5 Replace UTF-16 surrogate characters with 'replacement character'
In Javascript, characters are 16-bits wide. It does not understand surrogates as characters.

From Wikipedia (http://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane):
"The High Surrogates (U+D800–U+DBFF) and Low Surrogate (U+DC00–U+DFFF) codes are reserved
for encoding non-BMP characters in UTF-16 by using a pair of 16-bit codes: one High Surrogate
and one Low Surrogate. A single surrogate code point will never be assigned a character.""

The main offender seems to be \uD835 as a stand alone character, which would be the first
16-bit character of a blackboard bold character (http://www.fileformat.info/info/unicode/char/1d400/index.htm).
Something must be going on client side that is screwing up the encoding and splitting the
two 16-bit characters so that \uD835 is standalone.
2015-06-12 10:14:35 +01:00
Brian Gough
e7e82fc89d update port to match default track changes configuration 2015-05-28 16:43:58 +01:00
Brian
60db8bf5de flush ops when document is flushed 2015-05-28 14:48:44 +01:00
Brian Gough
85a4bf0da6 add tests calling zlib.gzip directly (for debugging) 2015-05-19 14:39:47 +01:00
Brian Gough
85eab2e967 modify DispatchManager test to allow for slow shutdown 2015-05-19 14:23:32 +01:00
Brian Gough
57f6919485 modify LockManager test to avoid dependence on timing 2015-05-18 09:03:51 +01:00
Henry Oswald
d862227314 make startup message consistent 2015-04-30 16:01:22 +01:00
Brian Gough
ea92fe0e9e Merge pull request #3 from sharelatex/gzip-large-docs
Gzip large docs
2015-03-31 10:58:18 +01:00
Brian Gough
6cdf5615fc remove old unused functions 2015-03-31 10:24:09 +01:00
Brian Gough
36f60d5bce enforce minimum size of 1k for compression 2015-03-31 10:07:39 +01:00
Brian Gough
9bb08d7ba5 add ZipManager comments 2015-03-31 10:07:11 +01:00
Brian Gough
e61beed92f suppress logging in ZipManager tests 2015-03-30 16:58:00 +01:00
Brian Gough
495af5d568 remove console.logs from tests 2015-03-30 16:56:07 +01:00
Brian Gough
c8c12e8b41 fix error in ZipManager writesEnabled setting 2015-03-30 16:41:45 +01:00
Brian Gough
8e8ee5b3da fix tests 2015-03-30 16:41:23 +01:00
Brian Gough
86505047a3 added unit test (work in progress) 2015-03-27 17:04:38 +00:00
Brian Gough
03564b2138 fix variable zip.minsize to match config name zip.minSize 2015-03-27 17:00:30 +00:00
Brian Gough
66fa170ac8 disable compression by default 2015-03-27 15:32:13 +00:00