Brian Gough
7a9577e081
only flush to track-changes when ops are queued
2015-11-30 15:17:11 +00:00
Brian Gough
529a061567
Merge pull request #4 from sharelatex/flush-ops-from-redis
...
push ops to track changes when document is flushed
2015-11-30 10:46:21 +00:00
James Allen
2589e2d417
Gracefully return when an op has already been submitted
...
It is not a fatal error if an op has already been submitted. We just
need to send an ack back to the client that submitted it and continue.
If we detect a duplicate op, set dup: true on the op and pass it back
to real-time for distributing. The dup: true flag will ensure it only
gets acknowledged to the submitting client, not everyone.
2015-11-19 10:54:28 +00:00
James Allen
e73890bfc2
Error if update would make document too long
2015-11-06 12:52:03 +00:00
James Allen
03e9d7390f
Add in mongo health check end point
2015-08-31 15:57:26 +01:00
James Allen
d27d2b90a2
Merge pull request #5 from sharelatex/utf16_fix
...
Replace UTF-16 surrogate characters with 'replacement character'
2015-06-12 10:18:34 +01:00
James Allen
333591d087
Extra null check
2015-06-12 10:16:33 +01:00
James Allen
a3847d21d5
Replace UTF-16 surrogate characters with 'replacement character'
...
In Javascript, characters are 16-bits wide. It does not understand surrogates as characters.
From Wikipedia (http://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane ):
"The High Surrogates (U+D800–U+DBFF) and Low Surrogate (U+DC00–U+DFFF) codes are reserved
for encoding non-BMP characters in UTF-16 by using a pair of 16-bit codes: one High Surrogate
and one Low Surrogate. A single surrogate code point will never be assigned a character.""
The main offender seems to be \uD835 as a stand alone character, which would be the first
16-bit character of a blackboard bold character (http://www.fileformat.info/info/unicode/char/1d400/index.htm ).
Something must be going on client side that is screwing up the encoding and splitting the
two 16-bit characters so that \uD835 is standalone.
2015-06-12 10:14:35 +01:00
Brian
60db8bf5de
flush ops when document is flushed
2015-05-28 14:48:44 +01:00
Brian Gough
6cdf5615fc
remove old unused functions
2015-03-31 10:24:09 +01:00
Brian Gough
36f60d5bce
enforce minimum size of 1k for compression
2015-03-31 10:07:39 +01:00
Brian Gough
9bb08d7ba5
add ZipManager comments
2015-03-31 10:07:11 +01:00
Brian Gough
c8c12e8b41
fix error in ZipManager writesEnabled setting
2015-03-30 16:41:45 +01:00
Brian Gough
03564b2138
fix variable zip.minsize to match config name zip.minSize
2015-03-27 17:00:30 +00:00
Brian Gough
66fa170ac8
disable compression by default
2015-03-27 15:32:13 +00:00
Brian Gough
6bffa4d9e0
don't log docLines when document removed from redis
...
they can now be binary gzipped data which messes up the logs
2015-03-25 16:54:36 +00:00
Brian Gough
e3d73d4456
add gzip support for large documents to reduce load on redis
2015-03-25 16:53:20 +00:00
James Allen
c822bb6ea4
Add in missed error catching
2014-11-19 12:51:19 +00:00
Henry Oswald
5ff8983528
implemeneted redis-sharelatex
2014-10-07 12:08:36 +01:00
Henry Oswald
9c85e9b848
moved back to not include sentinal while we upgrade to 0.10.x node
2014-09-29 12:07:32 +01:00
Henry Oswald
672dd3de1b
works with sentinal v1
2014-09-26 17:38:59 +01:00
James Allen
9da118c809
Process updates in the background so that workers only act as dispatchers
2014-08-11 14:16:05 +01:00
James Allen
773e17df66
Send worker idle waiting times to statsd
2014-08-07 13:16:11 +01:00
James Allen
cab5509076
Listen on queue via BLPOP rather than pub/sub
2014-08-07 11:45:53 +01:00
James Allen
ffd10a8439
Don't send or get versioning info to/from web
...
The version number is only used by the doc updater and so can be cleanly
encapsulated in a collection that only the doc updater knows about. The version
is already stored under docOps, so continue to tore it there.
2014-05-15 11:13:16 +01:00
James Allen
e25fb7e435
Keeping writing and reading updates from Mongo to cover the deploy period
2014-05-14 16:04:25 +01:00
James Allen
bdfe018cad
Log out warning when using Mongo for version still
2014-05-14 14:16:27 +01:00
James Allen
0199f2e129
Use version from web, with fallback to old mongo collection
2014-05-14 13:28:17 +01:00
James Allen
8973969224
Merge branch 'master' into remove_mongo_doc_ops
...
Conflicts:
app/coffee/DocOpsManager.coffee
test/acceptance/coffee/ApplyingUpdatesToADocTests.coffee
test/acceptance/coffee/FlushingDocsTests.coffee
test/unit/coffee/DocOpsManager/DocOpsManagerTests.coffee
test/unit/coffee/RedisManager/prependDocOpsTests.coffee
test/unit/coffee/RedisManager/pushDocOpTests.coffee
2014-05-14 12:39:40 +01:00
James Allen
6011ce4783
Use new metrics module
2014-05-08 09:28:13 +01:00
James Allen
d1434f7646
Increase redis lock expiry time to 30 seconds
2014-05-07 09:48:29 +01:00
James Allen
c2ebaaa338
Split lines on Windows line endings too
2014-04-10 12:44:46 +01:00
James Allen
375427bf5e
Remove extraneous logging
2014-03-21 13:15:42 +00:00
James Allen
c0be3ef37b
Put doc_ids with history changes into project level set
2014-03-21 12:41:05 +00:00
James Allen
2d28f1903f
Flush to the track changes api using the project id as well
2014-03-19 15:56:44 +00:00
James Allen
4f878e000b
Allow source and user_id to be included when setting a document
2014-03-11 12:47:26 +00:00
James Allen
5d45e191f3
Don't crash when logging out error
2014-03-04 12:39:02 +00:00
James Allen
86195ce7c3
Add in load throttling based on a redis key
2014-02-28 19:09:29 +00:00
James Allen
3d70f9126e
Flush track changes api every 50 updates
2014-02-28 18:29:05 +00:00
James Allen
f3192da87f
Tell track changes api to flush doc when flushing doc to mongo
2014-02-26 15:56:52 +00:00
James Allen
dfd3ec993b
Ensure version is still returned from redis
2014-02-26 14:49:52 +00:00
James Allen
b13f70eadb
Push ops into uncompressedHistoryOps list
2014-02-24 16:52:12 +00:00
James Allen
6f19f46d96
Create method for pushing uncompressed ops into redis
2014-02-24 16:40:29 +00:00
James Allen
7a977ca597
Don't store ops in Mongo, just keep them in redis with a timeout
2014-02-24 16:13:36 +00:00
James Allen
414b669a2c
Point every redis connection to the new config
2014-02-13 10:36:26 +00:00
James Allen
a5d816272e
Remove unused DocumentUpdater.js file
2014-02-12 10:52:53 +00:00
James Allen
e1a7d4f24a
Initial open sourcing
2014-02-12 10:40:42 +00:00
