github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-09-16 02:52:31 -04:00
Code Issues Projects Releases Packages Wiki Activity
381 commits 3 branches 7 tags 221 MiB
Commit graph

367 commits

Author SHA1 Message Date
Brian Gough
e292de5eb0 fix to avoid ever appending permanent changes to expiring packs 2016-04-06 17:00:16 +01:00
Brian Gough
8b7bdd345b consider all packs for archiving 2016-04-06 15:17:31 +01:00
Brian Gough
ef47337c78 remove additional fields 2016-04-06 15:17:20 +01:00
Brian Gough
0b9a0730c0 mark temporary packs with a last_checked date in the far future 
they do not need to be checked for archiving
 2016-04-06 14:29:49 +01:00
Brian Gough
08fc151eee avoid unnecessary call to insert packs into index 2016-04-06 14:29:21 +01:00
Brian Gough
719e0291aa consider all packs for processing 
to allow finalisation of old head packs
 2016-04-06 14:27:44 +01:00
Brian Gough
79baa99634 clean up logging 2016-04-06 14:26:54 +01:00
Brian Gough
6ab75795a2 archive head packs after sufficient time 2016-04-06 13:30:09 +01:00
Brian Gough
7a0c2900ab add error handler in tests 2016-04-06 13:29:19 +01:00
Brian Gough
6e18d49736 support archiving from list of project_ids/doc_ids 2016-04-04 17:00:19 +01:00
Brian Gough
6a2ecbac06 upgrade sentry logging 2016-04-04 16:30:17 +01:00
Brian Gough
537a1189e7 update mongodb-core version 2016-03-24 16:10:27 +00:00
Brian Gough
31348141d8 increase logging for discarded updates and version mismatch 2016-03-24 11:55:29 +00:00
Brian Gough
181cebecef avoid call to fetch packs unnecessarily 2016-03-24 11:55:29 +00:00
Brian Gough
98683de3ae temporarily disable ttl behaviour 
allow existing packs without temporary flag to expire
 2016-03-24 11:38:59 +00:00
Brian Gough
3f388fb0ac only change ttl on cached packs, not temporary ones 
temporary = without versioning feature enabled
cached = permanent versioned retrieved from s3
 2016-03-24 11:38:09 +00:00
Brian Gough
8d900013d9 record whether a pack is temporary in the pack itself 
using the expiresAt field no longer determines if the pack is
temporary because archived packs have an expiresAt field added when
they are retrieved from s3
 2016-03-24 11:02:58 +00:00
Brian Gough
9543c21dd7 added npm-shrinkwrap.json to avoid bug in mongodb-core 2016-03-11 09:43:07 +00:00
Brian Gough
10932eb4a9 remove debug logs 2016-03-10 15:20:32 +00:00
Brian Gough
5ea7a31ad9 update archiving acceptance tests 2016-03-10 15:15:57 +00:00
Brian Gough
98738d1344 fix for acceptance test 2016-03-10 15:15:29 +00:00
Brian Gough
f01bf99682 acceptance tests - work in progress 2016-03-09 16:56:49 +00:00
Brian Gough
e2e8292590 update tests 2016-03-09 15:24:28 +00:00
Brian Gough
6d43273f04 working on packmanager tests 2016-03-09 14:44:59 +00:00
Brian Gough
f6367e21b8 give separate error for archive in progress vs completed 2016-03-09 14:44:59 +00:00
Brian Gough
7350ab531d exclude already cached packs from archival 2016-03-09 14:44:59 +00:00
Brian Gough
28b184e0ca fix incorrect use of _.union (argument must be array) 2016-03-09 14:44:59 +00:00
Brian Gough
8922b97bd7 avoid duplicate filling of UserInfo in getDocUpdates 2016-03-09 14:44:59 +00:00
Brian Gough
bfc9494cc1 update UpdatesManager tests 2016-03-09 14:44:59 +00:00
Brian Gough
4e63c50a75 update MongoManager tests 2016-03-09 14:44:59 +00:00
Brian Gough
b0d2e22c58 update config file for changes s3 settings 2016-03-09 13:28:22 +00:00
Brian Gough
7e6ea2793b remove startup dependency on s3 settings 2016-03-09 13:28:02 +00:00
Brian Gough
b780df6801 removed unused tests 2016-03-04 15:47:59 +00:00
Brian Gough
4d58c14573 updated MongoAWS tests 2016-03-04 15:47:47 +00:00
Brian Gough
1419d20b1f fix indentation 2016-03-04 15:43:32 +00:00
Brian Gough
3175f6d3a6 handle case where index does not exist 2016-03-03 14:36:16 +00:00
Henry Oswald
e8b3fb5be6 added more logging to failed health checks 2016-03-03 10:50:55 +00:00
Brian Gough
795f717bab added index definitions 2016-03-01 11:38:23 +00:00
Brian Gough
ebd5628e53 temporary workaround to get tests passing 
temporary workaround to get tests passing
 2016-03-01 10:10:12 +00:00
Brian Gough
3d9dfeccc3 remove pack worker 
remove the op-specific code

remove tests for ops, now only packing

remove unused packing code

work in progress

store index for completed packs only

support archiving and unarchiving of individual packs

remove support for archiving whole document history

split out ArchiveManager, IndexManager

remove old DocArchive code

remove docHistoryStats collection

comment about archiving

added method to look at index when last pack has been archived

added start of iterator for project results

use a proper iterator

added heap module

getting it working

increase pack size since bulk operations no longer needed

remove unused MongoAWSexternal

cleanup

added doc iterator

remove old query code

added missing files

cleanup

clean upclean up

started adding pack worker for archiving

work in progress

work in progress

getting pack worker working

updating worker

getting packworker working

added lock

use correct key name for track changes aws access

use correct key name for track changes aws access

always send back users array

fix up comparison of retrieved objects

handle op ids inside packs

log when s3 download completes

comments

cleanup, remove finalisation ideacleanup, remove finalisation idea

remove logging
 2016-03-01 10:10:02 +00:00
Brian Gough
a23ddf31c0 allow packing of temporary ops 2016-01-29 12:36:03 +00:00
Brian Gough
612079ef8a started adding more pack tests and packing of temp ops 2016-01-28 16:56:32 +00:00
Brian Gough
77cafa36af support continuing from last packed doc 2016-01-28 16:40:20 +00:00
Brian - Work
666a07e5ba move lock check into HealthChecker 
to avoid dependency of HttpController on LockManager in unit tests
 2016-01-27 16:04:55 +00:00
Brian - Work
64545d6540 improve packing script 2016-01-27 15:48:36 +00:00
Brian Gough
199d2aaa92 script to pack existing docs 2016-01-27 15:14:23 +00:00
Brian Gough
ba0de92081 improve logging by avoiding string escapes 2016-01-26 14:54:06 +00:00
Brian Gough
b44a7b9aa6 reject very large ops 2016-01-26 14:52:40 +00:00
Brian Gough
b7a4c72f9c avoid compressing updates if the result would be too big 2016-01-26 12:23:21 +00:00
Brian Gough
ed0aaa189d add test for non-overlapping insert-delete case 2016-01-26 12:13:43 +00:00
Brian Gough
b3ddd839e6 add logging of raw updates 2016-01-26 11:28:02 +00:00
Brian Gough
29c7c5e249 enable packs by default for new docs 2016-01-25 09:55:55 +00:00
Brian Gough
d10123d3c4 include n parameter when packing 2016-01-25 09:45:25 +00:00
Brian Gough
9b2cd11cd4 don't try to append to packs when using the old op code 2016-01-22 10:45:24 +00:00
Brian Gough
84ace7f4c7 use packs only for temporary ops 2016-01-20 14:22:13 +00:00
Brian Gough
ae61d1261e added tests for pack updates 2016-01-19 15:58:09 +00:00
Brian Gough
61d22f027a added test for inserting as pack 2016-01-19 15:58:09 +00:00
Brian Gough
399a8f0a29 update tests to assume packs are created 2016-01-19 15:58:09 +00:00
Brian Gough
78b3412ca8 decrease delay when packing 2016-01-19 15:58:09 +00:00
Brian Gough
679a81564e respect mongo 3 limit of 1000 bulk operations 2016-01-19 15:58:09 +00:00
Brian Gough
f592611cac always create a new pack, never keep as op 2016-01-19 15:58:09 +00:00
Brian Gough
c6be12f3d5 set v_end on pack creation 2016-01-19 15:58:09 +00:00
Brian - Work
f64969c784 added comment about query memory usage for toArray 2016-01-19 15:58:09 +00:00
Brian Gough
0532a4daaa use compound index to replace separate index for packs 2016-01-19 15:56:09 +00:00
Brian Gough
0ba00a9eb7 expire temporary packs and roll over to a new pack each day 2016-01-19 15:56:09 +00:00
Brian Gough
5e830cbbdb put all new ops into packs 2016-01-19 15:56:09 +00:00
Brian Gough
dc564fd5d0 archiving document history now sends all changes to s3 2016-01-15 15:54:46 +00:00
Brian Gough
5153ed8217 make peekLastUpdate alway return lastVersion when available 2016-01-15 15:54:44 +00:00
Brian Gough
933b6c50e7 add memory logger to track changes 2016-01-14 16:24:13 +00:00
Brian Gough
8e53d66079 log the key for lock timeouts 2016-01-12 10:47:15 +00:00
Brian Gough
6199532d08 increase logging on s3 operations 2016-01-12 10:36:00 +00:00
Brian Gough
ca1f1dc944 handle exception in parsing retrieved json from aws 2016-01-12 09:26:29 +00:00
Brian Gough
b8862ca5af switch to node-byline module to avoid buffering problem with readline-stream 
for lines > 64k the readline-stream module is affected by
https://github.com/jahewson/node-byline/issues/30 which is fixed in
node-byline (readline-stream was an earlier fork of the byline module)
 2016-01-11 16:51:35 +00:00
Brian Gough
cb109a27a6 allow PackWorker to shut down cleanly 2016-01-06 09:43:10 +00:00
Brian Gough
ffe30962c9 add a close() method to LockManager to allow clean shutdown 2016-01-06 09:34:39 +00:00
Brian Gough
05163837cb add sentry error reporting to PackWorker 2016-01-05 16:00:52 +00:00
Brian Gough
ee8931bc34 log the request to sentry when an error occurs 2016-01-05 15:41:06 +00:00
Brian Gough
bad923e073 fix acceptance tests for mongojs 1.x driver 2016-01-05 12:10:57 +00:00
Brian Gough
6754bdca1c log timestamp in human-readable form for inconsistent ops 2016-01-05 11:30:24 +00:00
Brian Gough
e1aa436286 respect mongo bulk operations limit of 1000 operations 2016-01-05 11:13:13 +00:00
Brian Gough
bb7153c6c1 workaround for mongojs db.close issue 
https://github.com/mafintosh/mongojs/issues/224
 2015-12-22 15:36:15 +00:00
Brian Gough
d3583b4ef6 respect limit of 1000 ops in bulk operation with mongojs 1.x 2015-12-22 14:38:04 +00:00
Brian Gough
c7b4062412 remove unsupported options argument in count() method of mongojs 1.x 2015-12-22 14:20:34 +00:00
Brian Gough
d49997d9f3 fix usage of BSON module 2015-12-21 16:56:49 +00:00
Brian Gough
b7de6f2f71 don't try to compress updates across point of broken history 2015-12-21 13:52:26 +00:00
Brian Gough
4a6374efe8 fix read order when retrieving diffs 2015-12-18 12:38:42 +00:00
Brian Gough
911883a566 upgrade logger-sharelatex 2015-12-18 10:38:12 +00:00
Brian Gough
9f69c95192 Merge branch 'upgrade-mongojs' 2015-12-17 16:31:04 +00:00
Brian Gough
4a82dfe618 add setting trackchanges.continueOnError to allow recovery from missing ops 2015-12-17 16:28:02 +00:00
Brian Gough
b84a9e6e91 upgrade mongojs 2015-12-17 14:11:44 +00:00
Brian Gough
54d1036e37 skip ops marked as broken in database 2015-12-09 15:13:37 +00:00
Brian Gough
2a7c33d7ca added /check endpoint for documents 2015-12-09 14:57:04 +00:00
Brian Gough
c51d249221 remove explicit sentry release number 
use environment variable SENTRY_RELEASE instead
 2015-12-08 10:44:59 +00:00
Brian Gough
6483233f48 added /oops endpoint for generating dummy exception 2015-12-07 16:17:17 +00:00
Brian Gough
e7af602d32 add sentry logging 2015-12-07 15:52:21 +00:00
Brian Gough
23c43b8042 skip any broken ops when viewing history diffs 2015-12-04 15:17:28 +00:00
Brian Gough
be2136de7c fix update-in-place bug for array ops 2015-12-04 15:17:28 +00:00
Brian Gough
11be8c3733 increase timeout in lock manager test 
on heavily loaded machines 20ms isn't long enough to consistently get
multiple calls to the lock manager
 2015-11-27 14:25:25 +00:00
Brian Gough
3842f0d1cc Merge pull request #9 from sharelatex/only-delete-applied-ops 
Only delete applied ops
 2015-11-27 12:45:51 +00:00
Brian Gough
8ebc069ddb modify last compressed op in place 2015-11-26 16:17:18 +00:00