github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-21 20:47:08 -05:00
Code Issues Projects Releases Packages Wiki Activity
336 commits 3 branches 7 tags 206 MiB
Commit graph

327 commits

Author SHA1 Message Date
Brian Gough
a23ddf31c0 allow packing of temporary ops 2016-01-29 12:36:03 +00:00
Brian Gough
612079ef8a started adding more pack tests and packing of temp ops 2016-01-28 16:56:32 +00:00
Brian Gough
77cafa36af support continuing from last packed doc 2016-01-28 16:40:20 +00:00
Brian - Work
666a07e5ba move lock check into HealthChecker 
to avoid dependency of HttpController on LockManager in unit tests
 2016-01-27 16:04:55 +00:00
Brian - Work
64545d6540 improve packing script 2016-01-27 15:48:36 +00:00
Brian Gough
199d2aaa92 script to pack existing docs 2016-01-27 15:14:23 +00:00
Brian Gough
ba0de92081 improve logging by avoiding string escapes 2016-01-26 14:54:06 +00:00
Brian Gough
b44a7b9aa6 reject very large ops 2016-01-26 14:52:40 +00:00
Brian Gough
b7a4c72f9c avoid compressing updates if the result would be too big 2016-01-26 12:23:21 +00:00
Brian Gough
ed0aaa189d add test for non-overlapping insert-delete case 2016-01-26 12:13:43 +00:00
Brian Gough
b3ddd839e6 add logging of raw updates 2016-01-26 11:28:02 +00:00
Brian Gough
29c7c5e249 enable packs by default for new docs 2016-01-25 09:55:55 +00:00
Brian Gough
d10123d3c4 include n parameter when packing 2016-01-25 09:45:25 +00:00
Brian Gough
9b2cd11cd4 don't try to append to packs when using the old op code 2016-01-22 10:45:24 +00:00
Brian Gough
84ace7f4c7 use packs only for temporary ops 2016-01-20 14:22:13 +00:00
Brian Gough
ae61d1261e added tests for pack updates 2016-01-19 15:58:09 +00:00
Brian Gough
61d22f027a added test for inserting as pack 2016-01-19 15:58:09 +00:00
Brian Gough
399a8f0a29 update tests to assume packs are created 2016-01-19 15:58:09 +00:00
Brian Gough
78b3412ca8 decrease delay when packing 2016-01-19 15:58:09 +00:00
Brian Gough
679a81564e respect mongo 3 limit of 1000 bulk operations 2016-01-19 15:58:09 +00:00
Brian Gough
f592611cac always create a new pack, never keep as op 2016-01-19 15:58:09 +00:00
Brian Gough
c6be12f3d5 set v_end on pack creation 2016-01-19 15:58:09 +00:00
Brian - Work
f64969c784 added comment about query memory usage for toArray 2016-01-19 15:58:09 +00:00
Brian Gough
0532a4daaa use compound index to replace separate index for packs 2016-01-19 15:56:09 +00:00
Brian Gough
0ba00a9eb7 expire temporary packs and roll over to a new pack each day 2016-01-19 15:56:09 +00:00
Brian Gough
5e830cbbdb put all new ops into packs 2016-01-19 15:56:09 +00:00
Brian Gough
dc564fd5d0 archiving document history now sends all changes to s3 2016-01-15 15:54:46 +00:00
Brian Gough
5153ed8217 make peekLastUpdate alway return lastVersion when available 2016-01-15 15:54:44 +00:00
Brian Gough
933b6c50e7 add memory logger to track changes 2016-01-14 16:24:13 +00:00
Brian Gough
8e53d66079 log the key for lock timeouts 2016-01-12 10:47:15 +00:00
Brian Gough
6199532d08 increase logging on s3 operations 2016-01-12 10:36:00 +00:00
Brian Gough
ca1f1dc944 handle exception in parsing retrieved json from aws 2016-01-12 09:26:29 +00:00
Brian Gough
b8862ca5af switch to node-byline module to avoid buffering problem with readline-stream 
for lines > 64k the readline-stream module is affected by
https://github.com/jahewson/node-byline/issues/30 which is fixed in
node-byline (readline-stream was an earlier fork of the byline module)
 2016-01-11 16:51:35 +00:00
Brian Gough
cb109a27a6 allow PackWorker to shut down cleanly 2016-01-06 09:43:10 +00:00
Brian Gough
ffe30962c9 add a close() method to LockManager to allow clean shutdown 2016-01-06 09:34:39 +00:00
Brian Gough
05163837cb add sentry error reporting to PackWorker 2016-01-05 16:00:52 +00:00
Brian Gough
ee8931bc34 log the request to sentry when an error occurs 2016-01-05 15:41:06 +00:00
Brian Gough
bad923e073 fix acceptance tests for mongojs 1.x driver 2016-01-05 12:10:57 +00:00
Brian Gough
6754bdca1c log timestamp in human-readable form for inconsistent ops 2016-01-05 11:30:24 +00:00
Brian Gough
e1aa436286 respect mongo bulk operations limit of 1000 operations 2016-01-05 11:13:13 +00:00
Brian Gough
bb7153c6c1 workaround for mongojs db.close issue 
https://github.com/mafintosh/mongojs/issues/224
 2015-12-22 15:36:15 +00:00
Brian Gough
d3583b4ef6 respect limit of 1000 ops in bulk operation with mongojs 1.x 2015-12-22 14:38:04 +00:00
Brian Gough
c7b4062412 remove unsupported options argument in count() method of mongojs 1.x 2015-12-22 14:20:34 +00:00
Brian Gough
d49997d9f3 fix usage of BSON module 2015-12-21 16:56:49 +00:00
Brian Gough
b7de6f2f71 don't try to compress updates across point of broken history 2015-12-21 13:52:26 +00:00
Brian Gough
4a6374efe8 fix read order when retrieving diffs 2015-12-18 12:38:42 +00:00
Brian Gough
911883a566 upgrade logger-sharelatex 2015-12-18 10:38:12 +00:00
Brian Gough
9f69c95192 Merge branch 'upgrade-mongojs' 2015-12-17 16:31:04 +00:00
Brian Gough
4a82dfe618 add setting trackchanges.continueOnError to allow recovery from missing ops 2015-12-17 16:28:02 +00:00
Brian Gough
b84a9e6e91 upgrade mongojs 2015-12-17 14:11:44 +00:00
Brian Gough
54d1036e37 skip ops marked as broken in database 2015-12-09 15:13:37 +00:00
Brian Gough
2a7c33d7ca added /check endpoint for documents 2015-12-09 14:57:04 +00:00
Brian Gough
c51d249221 remove explicit sentry release number 
use environment variable SENTRY_RELEASE instead
 2015-12-08 10:44:59 +00:00
Brian Gough
6483233f48 added /oops endpoint for generating dummy exception 2015-12-07 16:17:17 +00:00
Brian Gough
e7af602d32 add sentry logging 2015-12-07 15:52:21 +00:00
Brian Gough
23c43b8042 skip any broken ops when viewing history diffs 2015-12-04 15:17:28 +00:00
Brian Gough
be2136de7c fix update-in-place bug for array ops 2015-12-04 15:17:28 +00:00
Brian Gough
11be8c3733 increase timeout in lock manager test 
on heavily loaded machines 20ms isn't long enough to consistently get
multiple calls to the lock manager
 2015-11-27 14:25:25 +00:00
Brian Gough
3842f0d1cc Merge pull request #9 from sharelatex/only-delete-applied-ops 
Only delete applied ops
 2015-11-27 12:45:51 +00:00
Brian Gough
8ebc069ddb modify last compressed op in place 2015-11-26 16:17:18 +00:00
Brian Gough
3432d9e91a added comments for redis delete 2015-11-26 15:16:54 +00:00
Brian Gough
e65549099c only delete the applied ops from redis 2015-11-25 16:01:07 +00:00
Brian Gough
992857d6a2 added redis write check to healthcheck 2015-10-29 10:52:23 +00:00
Brian Gough
c44d5b1b3d added healthcheck 2015-10-19 10:59:39 +01:00
Brian Gough
ad144371d0 gracefully handle updates marked as broken 
set update.broken == true to allow the user to view history without a
crash
 2015-10-16 11:24:50 +01:00
Brian Gough
8961e23954 enhance LockManager to avoid accidental unlocking 2015-10-14 14:42:17 +01:00
Brian Gough
b6dae59655 fix callback logic in compressAndSaveRawUpdates 2015-10-08 16:39:13 +01:00
Brian Gough
8226bf3be4 increase lock time to 5 minutes 2015-10-08 16:11:39 +01:00
Brian Gough
add6a68fe1 add missing callback in compressAndSaveRawUpdates 2015-10-08 10:53:25 +01:00
James Allen
1a4b8f4269 API/service layout deprecation warning 2015-10-07 13:44:40 +01:00
Henry Oswald
ef69729dfd change archive and unarchive to post's 2015-10-01 15:55:07 +01:00
Henry Oswald
6052bf6a7e change test to 19 not 20 as a stub in mongo is left behind 2015-10-01 15:54:46 +01:00
James Allen
2a03591030 Stub out noisy/slow logger-sharelatex and mongojs modules in tests 2015-09-25 13:46:20 +01:00
James Allen
23dfe68cb8 Don't error when rewinding and insert op which is beyond the length of the document. 
ShareJS will accept an op where p > content.length when applied,
and it applies as though p == content.length. However, the op is
passed to us with the original p > content.length. Detect if that
is the case with this op, and shift p back appropriately to match
ShareJS if so.
 2015-09-25 13:44:44 +01:00
Brian Gough
92e0b0f04c add logging to each stage of archiving 2015-09-24 09:10:06 +01:00
Brian Gough
e683b0275a bug fix for clear archive in progress flag 2015-09-24 09:09:49 +01:00
Brian Gough
692e8c657c Revert to the default lock timeout now we have write barriers 
Revert "increase lock timeouts for archiving"

This reverts commit 9eee1b383772adf058130d6e5eab409f57ce03cd.
 2015-09-24 08:53:09 +01:00
Brian Gough
2ab1778dd9 move default value of lastVersion into function body 2015-09-23 16:31:33 +01:00
Brian Gough
bdf1d267f0 fix tests 2015-09-23 15:34:36 +01:00
Brian Gough
dc0044020f only archive entries older than the current update 
to avoid a stale version of the current update ever being pulled back
from S3
 2015-09-23 14:33:40 +01:00
Brian Gough
696a866b67 pause the stream of ops, not the download 
the download is buffered in the lineStream so a lot comes out even
after pausing the S3 download.
 2015-09-23 13:38:57 +01:00
Brian Gough
847a553344 prevent double archiving by checking if any inS3 field is already present 2015-09-23 13:29:32 +01:00
Brian Gough
e49f260507 allow rollback/locking by setting inS3:false when starting the archive process 2015-09-23 13:28:07 +01:00
Brian Gough
551e8334cf compressedUpdates are now never inserted with inS3 
it is now always added later, and a new update is forced for any
addition to an archived update
 2015-09-23 13:25:10 +01:00
Brian Gough
d6b827426c support forcing new compressed update in popLastCompressedUpdate 
callback with a null update, passing the version as an additional
argument
 2015-09-23 13:22:38 +01:00
Brian Gough
dc35ef5cda fix tests for archiving 2015-09-21 13:24:06 +01:00
Brian Gough
a10dc4f898 Merge pull request #6 from heukirne/s3-archive 
Add S3 archive track changes feature
 2015-09-21 11:25:06 +01:00
Brian Gough
0e627c92d8 avoid clobbering global _ in loop 2015-09-18 16:26:05 +01:00
Henrique Dias
aa66c5ee8c improve size function 2015-09-17 10:41:53 -03:00
Henrique Dias
3f712c452a add size bulk limit 2015-09-17 09:23:13 -03:00
Henrique Dias
01cbfd5338 fix minor issues in acceptance test 2015-09-16 19:33:23 -03:00
Henrique Dias
a4575f178d fix unit test 2015-09-16 18:39:07 -03:00
Brian Gough
12c5098b48 fix lock in test 2015-09-16 16:50:36 +01:00
Brian Gough
7af5050370 add lock to unarchive doc 2015-09-16 16:18:36 +01:00
Brian Gough
18f06a3daf increase lock timeouts for archiving 2015-09-16 16:09:38 +01:00
Brian Gough
b4ffa7d57e share the document lock between archiving and packing 2015-09-16 16:03:55 +01:00
Brian Gough
9d39012b49 add error handler to each stage of download pipeline 2015-09-16 16:00:37 +01:00
Brian Gough
d9085a5e5e add error handler to each stage of upload pipeline 2015-09-16 16:00:25 +01:00
Brian Gough
1c1b1d9595 log the case where there are no entries in the document history 2015-09-16 15:34:30 +01:00
Brian Gough
82d0f4fce8 make unarchive more responsive by downloading documents in parallel 
unarchive is triggered interactively so we should try to make it
reasonably fast
 2015-09-16 15:33:59 +01:00