github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-10-31 21:21:03 -04:00
Code Issues Projects Releases Packages Wiki Activity
417 commits 3 branches 7 tags 199 MiB
Commit graph

397 commits

Author SHA1 Message Date
Shane Kilkelly
c30e672549 Merge branch 'master' into sk-upgrade-metrics 2017-03-16 15:17:57 +00:00
Shane Kilkelly
c303a1a386 wip: upgrade metrics 2017-03-16 15:17:38 +00:00
Brian Gough
ac5d59211d revert random TTL in favour of delay in archiving 
there could be some issues with newer packs expiring before older ones
 2017-01-31 15:07:58 +00:00
Henry Oswald
ad5af5f4dd ceil not round 2017-01-26 15:14:20 +00:00
Henry Oswald
1d356f93ec added random up to 30 mins delay to the ttl of mongo objects 2017-01-26 13:07:54 +00:00
James Allen
dadc548f81 Ignore comment updates 2017-01-12 10:04:50 +01:00
James Allen
694be95e66 Swap start_ts and end_ts comparison to correctly break chunks after 5 minutes 2016-10-11 11:03:52 +01:00
James Allen
a197c0d148 Merge pull request #18 from sharelatex/ja-split-on-deletes 
Split update summary on big deletes
 2016-10-10 08:46:46 +01:00
James Allen
40ed6fee46 Split update summary on big deletes 2016-10-04 15:13:04 +01:00
Brian Gough
92e36023e5 Merge pull request #17 from sharelatex/ja-consistency-tweaks 
Retry rewind if doc and update versions don't match
 2016-09-30 15:14:27 +01:00
James Allen
f8865e616d Update ensureIndices to reflect reality 2016-09-30 14:34:38 +01:00
James Allen
3c8aeb1262 Log number of retries 2016-09-30 13:38:47 +01:00
James Allen
a7f44bcd01 Add missing return on callback 2016-09-30 13:36:31 +01:00
James Allen
deced1aa15 Fix unit tests with logger.warn stub 2016-09-30 11:41:49 +01:00
James Allen
bddd1fda7d Retry rewind if doc and update versions don't match 2016-09-30 11:36:47 +01:00
James Allen
cce93b95a5 Fetch updates before doc content when building a diff 2016-09-22 11:19:05 +01:00
James Allen
05a048db9a Add in a little more logging 2016-09-22 11:19:05 +01:00
James Allen
8dda44f880 Speed up unit tests by only generating big fixtures once up front 2016-09-16 11:51:41 +01:00
James Allen
dfe26262ec Return a No-op if diff returns nothing 2016-09-16 11:50:44 +01:00
James Allen
80375ae2dd Run a diff against big delete - insert changes which are likely copy-pastes 2016-09-16 11:33:36 +01:00
Brian Gough
6e5eadfa86 include a timeout on WebApiManager requests 2016-04-13 16:42:36 +01:00
Brian Gough
a55b72871f don't let s3 errors stop archive worker 2016-04-13 14:39:11 +01:00
Brian Gough
b343be844e added metrics to pack worker for archiving 2016-04-08 10:29:04 +01:00
Brian Gough
76fe194815 add a metric for append-pack 2016-04-07 15:56:11 +01:00
Brian Gough
25365789b4 use latest sharelatex-metrics package 2016-04-07 15:53:10 +01:00
Brian Gough
78100e40c8 add missing metrics file 2016-04-07 15:33:20 +01:00
Brian Gough
1a1fa8798d log attempted update when throwing error in DiffGenerator 2016-04-07 15:16:50 +01:00
Brian Gough
6db310bf6b add insert/archive/unarchive metrics 2016-04-07 15:16:50 +01:00
Brian Gough
d0e08039da don't modify expiry for temporary packs 2016-04-07 15:16:50 +01:00
Brian Gough
fd49601716 preserve existing history when user upgrades 2016-04-07 15:16:38 +01:00
Brian Gough
e292de5eb0 fix to avoid ever appending permanent changes to expiring packs 2016-04-06 17:00:16 +01:00
Brian Gough
8b7bdd345b consider all packs for archiving 2016-04-06 15:17:31 +01:00
Brian Gough
ef47337c78 remove additional fields 2016-04-06 15:17:20 +01:00
Brian Gough
0b9a0730c0 mark temporary packs with a last_checked date in the far future 
they do not need to be checked for archiving
 2016-04-06 14:29:49 +01:00
Brian Gough
08fc151eee avoid unnecessary call to insert packs into index 2016-04-06 14:29:21 +01:00
Brian Gough
719e0291aa consider all packs for processing 
to allow finalisation of old head packs
 2016-04-06 14:27:44 +01:00
Brian Gough
79baa99634 clean up logging 2016-04-06 14:26:54 +01:00
Brian Gough
6ab75795a2 archive head packs after sufficient time 2016-04-06 13:30:09 +01:00
Brian Gough
7a0c2900ab add error handler in tests 2016-04-06 13:29:19 +01:00
Brian Gough
6e18d49736 support archiving from list of project_ids/doc_ids 2016-04-04 17:00:19 +01:00
Brian Gough
6a2ecbac06 upgrade sentry logging 2016-04-04 16:30:17 +01:00
Brian Gough
537a1189e7 update mongodb-core version 2016-03-24 16:10:27 +00:00
Brian Gough
31348141d8 increase logging for discarded updates and version mismatch 2016-03-24 11:55:29 +00:00
Brian Gough
181cebecef avoid call to fetch packs unnecessarily 2016-03-24 11:55:29 +00:00
Brian Gough
98683de3ae temporarily disable ttl behaviour 
allow existing packs without temporary flag to expire
 2016-03-24 11:38:59 +00:00
Brian Gough
3f388fb0ac only change ttl on cached packs, not temporary ones 
temporary = without versioning feature enabled
cached = permanent versioned retrieved from s3
 2016-03-24 11:38:09 +00:00
Brian Gough
8d900013d9 record whether a pack is temporary in the pack itself 
using the expiresAt field no longer determines if the pack is
temporary because archived packs have an expiresAt field added when
they are retrieved from s3
 2016-03-24 11:02:58 +00:00
Brian Gough
9543c21dd7 added npm-shrinkwrap.json to avoid bug in mongodb-core 2016-03-11 09:43:07 +00:00
Brian Gough
10932eb4a9 remove debug logs 2016-03-10 15:20:32 +00:00
Brian Gough
5ea7a31ad9 update archiving acceptance tests 2016-03-10 15:15:57 +00:00
Brian Gough
98738d1344 fix for acceptance test 2016-03-10 15:15:29 +00:00
Brian Gough
f01bf99682 acceptance tests - work in progress 2016-03-09 16:56:49 +00:00
Brian Gough
e2e8292590 update tests 2016-03-09 15:24:28 +00:00
Brian Gough
6d43273f04 working on packmanager tests 2016-03-09 14:44:59 +00:00
Brian Gough
f6367e21b8 give separate error for archive in progress vs completed 2016-03-09 14:44:59 +00:00
Brian Gough
7350ab531d exclude already cached packs from archival 2016-03-09 14:44:59 +00:00
Brian Gough
28b184e0ca fix incorrect use of _.union (argument must be array) 2016-03-09 14:44:59 +00:00
Brian Gough
8922b97bd7 avoid duplicate filling of UserInfo in getDocUpdates 2016-03-09 14:44:59 +00:00
Brian Gough
bfc9494cc1 update UpdatesManager tests 2016-03-09 14:44:59 +00:00
Brian Gough
4e63c50a75 update MongoManager tests 2016-03-09 14:44:59 +00:00
Brian Gough
b0d2e22c58 update config file for changes s3 settings 2016-03-09 13:28:22 +00:00
Brian Gough
7e6ea2793b remove startup dependency on s3 settings 2016-03-09 13:28:02 +00:00
Brian Gough
b780df6801 removed unused tests 2016-03-04 15:47:59 +00:00
Brian Gough
4d58c14573 updated MongoAWS tests 2016-03-04 15:47:47 +00:00
Brian Gough
1419d20b1f fix indentation 2016-03-04 15:43:32 +00:00
Brian Gough
3175f6d3a6 handle case where index does not exist 2016-03-03 14:36:16 +00:00
Henry Oswald
e8b3fb5be6 added more logging to failed health checks 2016-03-03 10:50:55 +00:00
Brian Gough
795f717bab added index definitions 2016-03-01 11:38:23 +00:00
Brian Gough
ebd5628e53 temporary workaround to get tests passing 
temporary workaround to get tests passing
 2016-03-01 10:10:12 +00:00
Brian Gough
3d9dfeccc3 remove pack worker 
remove the op-specific code

remove tests for ops, now only packing

remove unused packing code

work in progress

store index for completed packs only

support archiving and unarchiving of individual packs

remove support for archiving whole document history

split out ArchiveManager, IndexManager

remove old DocArchive code

remove docHistoryStats collection

comment about archiving

added method to look at index when last pack has been archived

added start of iterator for project results

use a proper iterator

added heap module

getting it working

increase pack size since bulk operations no longer needed

remove unused MongoAWSexternal

cleanup

added doc iterator

remove old query code

added missing files

cleanup

clean upclean up

started adding pack worker for archiving

work in progress

work in progress

getting pack worker working

updating worker

getting packworker working

added lock

use correct key name for track changes aws access

use correct key name for track changes aws access

always send back users array

fix up comparison of retrieved objects

handle op ids inside packs

log when s3 download completes

comments

cleanup, remove finalisation ideacleanup, remove finalisation idea

remove logging
 2016-03-01 10:10:02 +00:00
Brian Gough
a23ddf31c0 allow packing of temporary ops 2016-01-29 12:36:03 +00:00
Brian Gough
612079ef8a started adding more pack tests and packing of temp ops 2016-01-28 16:56:32 +00:00
Brian Gough
77cafa36af support continuing from last packed doc 2016-01-28 16:40:20 +00:00
Brian - Work
666a07e5ba move lock check into HealthChecker 
to avoid dependency of HttpController on LockManager in unit tests
 2016-01-27 16:04:55 +00:00
Brian - Work
64545d6540 improve packing script 2016-01-27 15:48:36 +00:00
Brian Gough
199d2aaa92 script to pack existing docs 2016-01-27 15:14:23 +00:00
Brian Gough
ba0de92081 improve logging by avoiding string escapes 2016-01-26 14:54:06 +00:00
Brian Gough
b44a7b9aa6 reject very large ops 2016-01-26 14:52:40 +00:00
Brian Gough
b7a4c72f9c avoid compressing updates if the result would be too big 2016-01-26 12:23:21 +00:00
Brian Gough
ed0aaa189d add test for non-overlapping insert-delete case 2016-01-26 12:13:43 +00:00
Brian Gough
b3ddd839e6 add logging of raw updates 2016-01-26 11:28:02 +00:00
Brian Gough
29c7c5e249 enable packs by default for new docs 2016-01-25 09:55:55 +00:00
Brian Gough
d10123d3c4 include n parameter when packing 2016-01-25 09:45:25 +00:00
Brian Gough
9b2cd11cd4 don't try to append to packs when using the old op code 2016-01-22 10:45:24 +00:00
Brian Gough
84ace7f4c7 use packs only for temporary ops 2016-01-20 14:22:13 +00:00
Brian Gough
ae61d1261e added tests for pack updates 2016-01-19 15:58:09 +00:00
Brian Gough
61d22f027a added test for inserting as pack 2016-01-19 15:58:09 +00:00
Brian Gough
399a8f0a29 update tests to assume packs are created 2016-01-19 15:58:09 +00:00
Brian Gough
78b3412ca8 decrease delay when packing 2016-01-19 15:58:09 +00:00
Brian Gough
679a81564e respect mongo 3 limit of 1000 bulk operations 2016-01-19 15:58:09 +00:00
Brian Gough
f592611cac always create a new pack, never keep as op 2016-01-19 15:58:09 +00:00
Brian Gough
c6be12f3d5 set v_end on pack creation 2016-01-19 15:58:09 +00:00
Brian - Work
f64969c784 added comment about query memory usage for toArray 2016-01-19 15:58:09 +00:00
Brian Gough
0532a4daaa use compound index to replace separate index for packs 2016-01-19 15:56:09 +00:00
Brian Gough
0ba00a9eb7 expire temporary packs and roll over to a new pack each day 2016-01-19 15:56:09 +00:00
Brian Gough
5e830cbbdb put all new ops into packs 2016-01-19 15:56:09 +00:00
Brian Gough
dc564fd5d0 archiving document history now sends all changes to s3 2016-01-15 15:54:46 +00:00
Brian Gough
5153ed8217 make peekLastUpdate alway return lastVersion when available 2016-01-15 15:54:44 +00:00
Brian Gough
933b6c50e7 add memory logger to track changes 2016-01-14 16:24:13 +00:00
Brian Gough
8e53d66079 log the key for lock timeouts 2016-01-12 10:47:15 +00:00