github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-29 20:53:39 -05:00
Code Issues Projects Releases Packages Wiki Activity
19763 commits 3 branches 7 tags 206 MiB
Commit graph

638 commits

Author SHA1 Message Date
Brian Gough
9eb36f9692 improve log message for flushing 2017-04-20 11:55:32 +01:00
Brian Gough
43f51355b2 add limit parameter to flushAll 2017-04-20 11:01:46 +01:00
Brian Gough
976f19f763 remove spurious call to getAllDocIdsWithHistoryOps 2017-04-20 10:14:44 +01:00
Brian Gough
70ce3a5271 Merge pull request #29 from sharelatex/bg-simple-flush-all 
simple flush for all projects
 2017-04-19 16:32:09 +01:00
Brian Gough
b5b61b98d0 avoid blocking when fetching redis keys 
use scan instead of keys method
 2017-04-19 15:39:33 +01:00
Brian Gough
9ce6d77cca add check for dangling updates 2017-04-13 11:31:45 +01:00
Brian Gough
1ec9f87542 update timeouts to allow for 5 second retry delay 
now using requestretry module, which has a five second delay between
retries. Need to increase the timeout on the health checks to allow for
this, as the timeout is currently shorter than the retry delay.
 2017-04-12 16:47:34 +01:00
Brian Gough
d9ed026d91 simple flush for all projects 
does not work with redis cluster, only single redis
 2017-04-12 16:34:28 +01:00
Shane Kilkelly
96f5802462 Merge branch 'sk-upgrade-metrics' 2017-04-04 16:16:37 +01:00
Shane Kilkelly
9a9f0f5c4d Remove the Metrics module, use metrics-sharelatex 2017-04-03 15:17:29 +01:00
Brian Gough
5d6d33966b Merge pull request #27 from sharelatex/bg-retry-web-api-request 
use requestretry to work around web api timeouts
 2017-04-03 14:05:07 +01:00
Brian Gough
7237467cb4 use requestretry to work around web api timeouts 2017-04-03 11:06:50 +01:00
Brian Gough
b0b05d1e1b improve error handling of releaseLock 
ported from docupdater
 2017-03-31 15:17:13 +01:00
Shane Kilkelly
b10f3adf2b mock out metrics in tests 2017-03-31 11:33:25 +01:00
Shane Kilkelly
9a84798293 Upgrade to latest metrics 2017-03-31 11:32:59 +01:00
Brian Gough
8bb3dd07a4 continue when ops are out of order 2017-03-30 13:37:11 +01:00
Brian Gough
d29141d583 return error when out-of-order ops detected 2017-03-30 11:48:26 +01:00
Brian Gough
70bb95fb36 update logger-sharelatex to v1.5.6 2017-03-30 09:41:42 +01:00
Brian Gough
28d2ec93b8 Revert "Revert "Merge pull request #23 from sharelatex/bg-move-lock"" 
This reverts commit 85bc45099f047aa8dfa6d189f02f4b4327c9d602.
 2017-03-28 12:52:49 +01:00
Shane Kilkelly
5288814bd6 Merge branch 'master' of github.com:sharelatex/track-changes-sharelatex 2017-03-27 14:37:47 +01:00
Shane Kilkelly
715ccbe227 Add a .nvmrc file 2017-03-27 14:37:32 +01:00
Brian Gough
e745cce9f5 Revert "Revert "added v8-profiler to shrinkwrap"" 
This reverts commit 642641ff7b97034346d494101124bee616927ade.
 2017-03-27 13:26:55 +01:00
Brian Gough
0130d3513d Revert "Revert "Merge pull request #24 from sharelatex/bg-add-profile"" 
This reverts commit a58cd0d787b51634caf694f9d858e638d27178b2.
 2017-03-27 13:25:59 +01:00
Brian Gough
8bd3dcca75 Revert "Merge pull request #24 from sharelatex/bg-add-profile" 
This reverts commit 8f507dcc9fbcd51072047c82a443137aaac0f377, reversing
changes made to 70abd7a155b396e3b44ce21ff8d36be6ad03a060.
 2017-03-24 09:17:38 +00:00
Brian Gough
a73c8ba7b8 Revert "Merge pull request #23 from sharelatex/bg-move-lock" 
This reverts commit 72e6756415e390a29fa784529453f926e5144358, reversing
changes made to 8f507dcc9fbcd51072047c82a443137aaac0f377.
 2017-03-24 09:17:11 +00:00
Brian Gough
6ada7842a9 Revert "added v8-profiler to shrinkwrap" 
This reverts commit 62d481b31ebed2a1714cdfcb037a9c1a9c053756.
 2017-03-24 09:12:28 +00:00
Brian Gough
f1a66df4ad added v8-profiler to shrinkwrap 2017-03-23 17:01:01 +00:00
Brian Gough
7802bcc911 Revert "update shrinkwrap file for new npm package" 
This reverts commit 68283e2650420c1cedc957b039b52766a8d4ce3c.
 2017-03-23 16:57:48 +00:00
Brian Gough
a4cca1f948 update shrinkwrap file for new npm package 2017-03-23 14:21:39 +00:00
Brian Gough
e028de115e Merge pull request #23 from sharelatex/bg-move-lock 
move lock inside web http calls
 2017-03-23 13:42:52 +00:00
Brian Gough
b897afeb1c Merge pull request #24 from sharelatex/bg-add-profile 
add /profile endpoint
 2017-03-23 13:39:12 +00:00
Brian Gough
bc7815f7fc remove old processUncompressedUpdates method 
replace with new per doc method
 2017-03-22 16:59:52 +00:00
Brian Gough
b1c0ebbaae add withLock to processUncompressedUpdatesForDoc 2017-03-22 16:16:04 +00:00
Brian Gough
0dd668416d increase request timeout now it is outside lock 2017-03-22 16:02:50 +00:00
Brian Gough
04b82a49ef add /profile endpoint 2017-03-22 15:38:12 +00:00
Brian Gough
6795820933 move lock inside web http calls 2017-03-21 16:49:23 +00:00
Shane Kilkelly
d520c78ae3 Update metrics version to 1.7.0 2017-03-21 13:45:15 +00:00
James Allen
d60445adfc Pass undo flag to doc updater when restoring from history 2017-03-21 11:32:53 +00:00
Shane Kilkelly
7fbc08d68a Add logger stub to unit test 2017-03-21 09:27:47 +00:00
Shane Kilkelly
118511ae4a Remove commented-out code 2017-03-20 09:39:51 +00:00
Shane Kilkelly
fa7ee739bb Update to new metrics api 2017-03-17 14:58:35 +00:00
Shane Kilkelly
c30e672549 Merge branch 'master' into sk-upgrade-metrics 2017-03-16 15:17:57 +00:00
Shane Kilkelly
c303a1a386 wip: upgrade metrics 2017-03-16 15:17:38 +00:00
Brian Gough
ac5d59211d revert random TTL in favour of delay in archiving 
there could be some issues with newer packs expiring before older ones
 2017-01-31 15:07:58 +00:00
Henry Oswald
ad5af5f4dd ceil not round 2017-01-26 15:14:20 +00:00
Henry Oswald
1d356f93ec added random up to 30 mins delay to the ttl of mongo objects 2017-01-26 13:07:54 +00:00
James Allen
dadc548f81 Ignore comment updates 2017-01-12 10:04:50 +01:00
James Allen
694be95e66 Swap start_ts and end_ts comparison to correctly break chunks after 5 minutes 2016-10-11 11:03:52 +01:00
James Allen
a197c0d148 Merge pull request #18 from sharelatex/ja-split-on-deletes 
Split update summary on big deletes
 2016-10-10 08:46:46 +01:00
James Allen
40ed6fee46 Split update summary on big deletes 2016-10-04 15:13:04 +01:00
Brian Gough
92e36023e5 Merge pull request #17 from sharelatex/ja-consistency-tweaks 
Retry rewind if doc and update versions don't match
 2016-09-30 15:14:27 +01:00
James Allen
f8865e616d Update ensureIndices to reflect reality 2016-09-30 14:34:38 +01:00
James Allen
3c8aeb1262 Log number of retries 2016-09-30 13:38:47 +01:00
James Allen
a7f44bcd01 Add missing return on callback 2016-09-30 13:36:31 +01:00
James Allen
deced1aa15 Fix unit tests with logger.warn stub 2016-09-30 11:41:49 +01:00
James Allen
bddd1fda7d Retry rewind if doc and update versions don't match 2016-09-30 11:36:47 +01:00
James Allen
cce93b95a5 Fetch updates before doc content when building a diff 2016-09-22 11:19:05 +01:00
James Allen
05a048db9a Add in a little more logging 2016-09-22 11:19:05 +01:00
James Allen
8dda44f880 Speed up unit tests by only generating big fixtures once up front 2016-09-16 11:51:41 +01:00
James Allen
dfe26262ec Return a No-op if diff returns nothing 2016-09-16 11:50:44 +01:00
James Allen
80375ae2dd Run a diff against big delete - insert changes which are likely copy-pastes 2016-09-16 11:33:36 +01:00
Brian Gough
6e5eadfa86 include a timeout on WebApiManager requests 2016-04-13 16:42:36 +01:00
Brian Gough
a55b72871f don't let s3 errors stop archive worker 2016-04-13 14:39:11 +01:00
Brian Gough
b343be844e added metrics to pack worker for archiving 2016-04-08 10:29:04 +01:00
Brian Gough
76fe194815 add a metric for append-pack 2016-04-07 15:56:11 +01:00
Brian Gough
25365789b4 use latest sharelatex-metrics package 2016-04-07 15:53:10 +01:00
Brian Gough
78100e40c8 add missing metrics file 2016-04-07 15:33:20 +01:00
Brian Gough
1a1fa8798d log attempted update when throwing error in DiffGenerator 2016-04-07 15:16:50 +01:00
Brian Gough
6db310bf6b add insert/archive/unarchive metrics 2016-04-07 15:16:50 +01:00
Brian Gough
d0e08039da don't modify expiry for temporary packs 2016-04-07 15:16:50 +01:00
Brian Gough
fd49601716 preserve existing history when user upgrades 2016-04-07 15:16:38 +01:00
Brian Gough
e292de5eb0 fix to avoid ever appending permanent changes to expiring packs 2016-04-06 17:00:16 +01:00
Brian Gough
8b7bdd345b consider all packs for archiving 2016-04-06 15:17:31 +01:00
Brian Gough
ef47337c78 remove additional fields 2016-04-06 15:17:20 +01:00
Brian Gough
0b9a0730c0 mark temporary packs with a last_checked date in the far future 
they do not need to be checked for archiving
 2016-04-06 14:29:49 +01:00
Brian Gough
08fc151eee avoid unnecessary call to insert packs into index 2016-04-06 14:29:21 +01:00
Brian Gough
719e0291aa consider all packs for processing 
to allow finalisation of old head packs
 2016-04-06 14:27:44 +01:00
Brian Gough
79baa99634 clean up logging 2016-04-06 14:26:54 +01:00
Brian Gough
6ab75795a2 archive head packs after sufficient time 2016-04-06 13:30:09 +01:00
Brian Gough
7a0c2900ab add error handler in tests 2016-04-06 13:29:19 +01:00
Brian Gough
6e18d49736 support archiving from list of project_ids/doc_ids 2016-04-04 17:00:19 +01:00
Brian Gough
6a2ecbac06 upgrade sentry logging 2016-04-04 16:30:17 +01:00
Brian Gough
537a1189e7 update mongodb-core version 2016-03-24 16:10:27 +00:00
Brian Gough
31348141d8 increase logging for discarded updates and version mismatch 2016-03-24 11:55:29 +00:00
Brian Gough
181cebecef avoid call to fetch packs unnecessarily 2016-03-24 11:55:29 +00:00
Brian Gough
98683de3ae temporarily disable ttl behaviour 
allow existing packs without temporary flag to expire
 2016-03-24 11:38:59 +00:00
Brian Gough
3f388fb0ac only change ttl on cached packs, not temporary ones 
temporary = without versioning feature enabled
cached = permanent versioned retrieved from s3
 2016-03-24 11:38:09 +00:00
Brian Gough
8d900013d9 record whether a pack is temporary in the pack itself 
using the expiresAt field no longer determines if the pack is
temporary because archived packs have an expiresAt field added when
they are retrieved from s3
 2016-03-24 11:02:58 +00:00
Brian Gough
9543c21dd7 added npm-shrinkwrap.json to avoid bug in mongodb-core 2016-03-11 09:43:07 +00:00
Brian Gough
10932eb4a9 remove debug logs 2016-03-10 15:20:32 +00:00
Brian Gough
5ea7a31ad9 update archiving acceptance tests 2016-03-10 15:15:57 +00:00
Brian Gough
98738d1344 fix for acceptance test 2016-03-10 15:15:29 +00:00
Brian Gough
f01bf99682 acceptance tests - work in progress 2016-03-09 16:56:49 +00:00
Brian Gough
e2e8292590 update tests 2016-03-09 15:24:28 +00:00
Brian Gough
6d43273f04 working on packmanager tests 2016-03-09 14:44:59 +00:00
Brian Gough
f6367e21b8 give separate error for archive in progress vs completed 2016-03-09 14:44:59 +00:00
Brian Gough
7350ab531d exclude already cached packs from archival 2016-03-09 14:44:59 +00:00
Brian Gough
28b184e0ca fix incorrect use of _.union (argument must be array) 2016-03-09 14:44:59 +00:00
Brian Gough
8922b97bd7 avoid duplicate filling of UserInfo in getDocUpdates 2016-03-09 14:44:59 +00:00
Brian Gough
bfc9494cc1 update UpdatesManager tests 2016-03-09 14:44:59 +00:00
Brian Gough
4e63c50a75 update MongoManager tests 2016-03-09 14:44:59 +00:00
Brian Gough
b0d2e22c58 update config file for changes s3 settings 2016-03-09 13:28:22 +00:00
Brian Gough
7e6ea2793b remove startup dependency on s3 settings 2016-03-09 13:28:02 +00:00
Brian Gough
b780df6801 removed unused tests 2016-03-04 15:47:59 +00:00
Brian Gough
4d58c14573 updated MongoAWS tests 2016-03-04 15:47:47 +00:00
Brian Gough
1419d20b1f fix indentation 2016-03-04 15:43:32 +00:00
Brian Gough
3175f6d3a6 handle case where index does not exist 2016-03-03 14:36:16 +00:00
Henry Oswald
e8b3fb5be6 added more logging to failed health checks 2016-03-03 10:50:55 +00:00
Brian Gough
795f717bab added index definitions 2016-03-01 11:38:23 +00:00
Brian Gough
ebd5628e53 temporary workaround to get tests passing 
temporary workaround to get tests passing
 2016-03-01 10:10:12 +00:00
Brian Gough
3d9dfeccc3 remove pack worker 
remove the op-specific code

remove tests for ops, now only packing

remove unused packing code

work in progress

store index for completed packs only

support archiving and unarchiving of individual packs

remove support for archiving whole document history

split out ArchiveManager, IndexManager

remove old DocArchive code

remove docHistoryStats collection

comment about archiving

added method to look at index when last pack has been archived

added start of iterator for project results

use a proper iterator

added heap module

getting it working

increase pack size since bulk operations no longer needed

remove unused MongoAWSexternal

cleanup

added doc iterator

remove old query code

added missing files

cleanup

clean upclean up

started adding pack worker for archiving

work in progress

work in progress

getting pack worker working

updating worker

getting packworker working

added lock

use correct key name for track changes aws access

use correct key name for track changes aws access

always send back users array

fix up comparison of retrieved objects

handle op ids inside packs

log when s3 download completes

comments

cleanup, remove finalisation ideacleanup, remove finalisation idea

remove logging
 2016-03-01 10:10:02 +00:00
Brian Gough
a23ddf31c0 allow packing of temporary ops 2016-01-29 12:36:03 +00:00
Brian Gough
612079ef8a started adding more pack tests and packing of temp ops 2016-01-28 16:56:32 +00:00
Brian Gough
77cafa36af support continuing from last packed doc 2016-01-28 16:40:20 +00:00
Brian - Work
666a07e5ba move lock check into HealthChecker 
to avoid dependency of HttpController on LockManager in unit tests
 2016-01-27 16:04:55 +00:00
Brian - Work
64545d6540 improve packing script 2016-01-27 15:48:36 +00:00
Brian Gough
199d2aaa92 script to pack existing docs 2016-01-27 15:14:23 +00:00
Brian Gough
ba0de92081 improve logging by avoiding string escapes 2016-01-26 14:54:06 +00:00
Brian Gough
b44a7b9aa6 reject very large ops 2016-01-26 14:52:40 +00:00
Brian Gough
b7a4c72f9c avoid compressing updates if the result would be too big 2016-01-26 12:23:21 +00:00
Brian Gough
ed0aaa189d add test for non-overlapping insert-delete case 2016-01-26 12:13:43 +00:00
Brian Gough
b3ddd839e6 add logging of raw updates 2016-01-26 11:28:02 +00:00
Brian Gough
29c7c5e249 enable packs by default for new docs 2016-01-25 09:55:55 +00:00
Brian Gough
d10123d3c4 include n parameter when packing 2016-01-25 09:45:25 +00:00
Brian Gough
9b2cd11cd4 don't try to append to packs when using the old op code 2016-01-22 10:45:24 +00:00
Brian Gough
84ace7f4c7 use packs only for temporary ops 2016-01-20 14:22:13 +00:00
Brian Gough
ae61d1261e added tests for pack updates 2016-01-19 15:58:09 +00:00
Brian Gough
61d22f027a added test for inserting as pack 2016-01-19 15:58:09 +00:00
Brian Gough
399a8f0a29 update tests to assume packs are created 2016-01-19 15:58:09 +00:00
Brian Gough
78b3412ca8 decrease delay when packing 2016-01-19 15:58:09 +00:00
Brian Gough
679a81564e respect mongo 3 limit of 1000 bulk operations 2016-01-19 15:58:09 +00:00
Brian Gough
f592611cac always create a new pack, never keep as op 2016-01-19 15:58:09 +00:00
Brian Gough
c6be12f3d5 set v_end on pack creation 2016-01-19 15:58:09 +00:00
Brian - Work
f64969c784 added comment about query memory usage for toArray 2016-01-19 15:58:09 +00:00
Brian Gough
0532a4daaa use compound index to replace separate index for packs 2016-01-19 15:56:09 +00:00
Brian Gough
0ba00a9eb7 expire temporary packs and roll over to a new pack each day 2016-01-19 15:56:09 +00:00
Brian Gough
5e830cbbdb put all new ops into packs 2016-01-19 15:56:09 +00:00
Brian Gough
dc564fd5d0 archiving document history now sends all changes to s3 2016-01-15 15:54:46 +00:00
Brian Gough
5153ed8217 make peekLastUpdate alway return lastVersion when available 2016-01-15 15:54:44 +00:00
Brian Gough
933b6c50e7 add memory logger to track changes 2016-01-14 16:24:13 +00:00
Brian Gough
8e53d66079 log the key for lock timeouts 2016-01-12 10:47:15 +00:00
Brian Gough
6199532d08 increase logging on s3 operations 2016-01-12 10:36:00 +00:00
Brian Gough
ca1f1dc944 handle exception in parsing retrieved json from aws 2016-01-12 09:26:29 +00:00
Brian Gough
b8862ca5af switch to node-byline module to avoid buffering problem with readline-stream 
for lines > 64k the readline-stream module is affected by
https://github.com/jahewson/node-byline/issues/30 which is fixed in
node-byline (readline-stream was an earlier fork of the byline module)
 2016-01-11 16:51:35 +00:00
Brian Gough
cb109a27a6 allow PackWorker to shut down cleanly 2016-01-06 09:43:10 +00:00
Brian Gough
ffe30962c9 add a close() method to LockManager to allow clean shutdown 2016-01-06 09:34:39 +00:00
Brian Gough
05163837cb add sentry error reporting to PackWorker 2016-01-05 16:00:52 +00:00
Brian Gough
ee8931bc34 log the request to sentry when an error occurs 2016-01-05 15:41:06 +00:00
Brian Gough
bad923e073 fix acceptance tests for mongojs 1.x driver 2016-01-05 12:10:57 +00:00
Brian Gough
6754bdca1c log timestamp in human-readable form for inconsistent ops 2016-01-05 11:30:24 +00:00
Brian Gough
e1aa436286 respect mongo bulk operations limit of 1000 operations 2016-01-05 11:13:13 +00:00
Brian Gough
bb7153c6c1 workaround for mongojs db.close issue 
https://github.com/mafintosh/mongojs/issues/224
 2015-12-22 15:36:15 +00:00
Brian Gough
d3583b4ef6 respect limit of 1000 ops in bulk operation with mongojs 1.x 2015-12-22 14:38:04 +00:00
Brian Gough
c7b4062412 remove unsupported options argument in count() method of mongojs 1.x 2015-12-22 14:20:34 +00:00
Brian Gough
d49997d9f3 fix usage of BSON module 2015-12-21 16:56:49 +00:00
Brian Gough
b7de6f2f71 don't try to compress updates across point of broken history 2015-12-21 13:52:26 +00:00
Brian Gough
4a6374efe8 fix read order when retrieving diffs 2015-12-18 12:38:42 +00:00
Brian Gough
911883a566 upgrade logger-sharelatex 2015-12-18 10:38:12 +00:00
Brian Gough
9f69c95192 Merge branch 'upgrade-mongojs' 2015-12-17 16:31:04 +00:00
Brian Gough
4a82dfe618 add setting trackchanges.continueOnError to allow recovery from missing ops 2015-12-17 16:28:02 +00:00
Brian Gough
b84a9e6e91 upgrade mongojs 2015-12-17 14:11:44 +00:00
Brian Gough
54d1036e37 skip ops marked as broken in database 2015-12-09 15:13:37 +00:00
Brian Gough
2a7c33d7ca added /check endpoint for documents 2015-12-09 14:57:04 +00:00
Brian Gough
c51d249221 remove explicit sentry release number 
use environment variable SENTRY_RELEASE instead
 2015-12-08 10:44:59 +00:00
Brian Gough
6483233f48 added /oops endpoint for generating dummy exception 2015-12-07 16:17:17 +00:00
Brian Gough
e7af602d32 add sentry logging 2015-12-07 15:52:21 +00:00
Brian Gough
23c43b8042 skip any broken ops when viewing history diffs 2015-12-04 15:17:28 +00:00
Brian Gough
be2136de7c fix update-in-place bug for array ops 2015-12-04 15:17:28 +00:00
Brian Gough
11be8c3733 increase timeout in lock manager test 
on heavily loaded machines 20ms isn't long enough to consistently get
multiple calls to the lock manager
 2015-11-27 14:25:25 +00:00
Brian Gough
3842f0d1cc Merge pull request #9 from sharelatex/only-delete-applied-ops 
Only delete applied ops
 2015-11-27 12:45:51 +00:00
Brian Gough
8ebc069ddb modify last compressed op in place 2015-11-26 16:17:18 +00:00
Brian Gough
3432d9e91a added comments for redis delete 2015-11-26 15:16:54 +00:00
Brian Gough
e65549099c only delete the applied ops from redis 2015-11-25 16:01:07 +00:00
Brian Gough
992857d6a2 added redis write check to healthcheck 2015-10-29 10:52:23 +00:00
Brian Gough
c44d5b1b3d added healthcheck 2015-10-19 10:59:39 +01:00
Brian Gough
ad144371d0 gracefully handle updates marked as broken 
set update.broken == true to allow the user to view history without a
crash
 2015-10-16 11:24:50 +01:00
Brian Gough
8961e23954 enhance LockManager to avoid accidental unlocking 2015-10-14 14:42:17 +01:00
Brian Gough
b6dae59655 fix callback logic in compressAndSaveRawUpdates 2015-10-08 16:39:13 +01:00
Brian Gough
8226bf3be4 increase lock time to 5 minutes 2015-10-08 16:11:39 +01:00
Brian Gough
add6a68fe1 add missing callback in compressAndSaveRawUpdates 2015-10-08 10:53:25 +01:00
James Allen
1a4b8f4269 API/service layout deprecation warning 2015-10-07 13:44:40 +01:00
Henry Oswald
ef69729dfd change archive and unarchive to post's 2015-10-01 15:55:07 +01:00
Henry Oswald
6052bf6a7e change test to 19 not 20 as a stub in mongo is left behind 2015-10-01 15:54:46 +01:00
James Allen
2a03591030 Stub out noisy/slow logger-sharelatex and mongojs modules in tests 2015-09-25 13:46:20 +01:00
James Allen
23dfe68cb8 Don't error when rewinding and insert op which is beyond the length of the document. 
ShareJS will accept an op where p > content.length when applied,
and it applies as though p == content.length. However, the op is
passed to us with the original p > content.length. Detect if that
is the case with this op, and shift p back appropriately to match
ShareJS if so.
 2015-09-25 13:44:44 +01:00
Brian Gough
92e0b0f04c add logging to each stage of archiving 2015-09-24 09:10:06 +01:00
Brian Gough
e683b0275a bug fix for clear archive in progress flag 2015-09-24 09:09:49 +01:00
Brian Gough
692e8c657c Revert to the default lock timeout now we have write barriers 
Revert "increase lock timeouts for archiving"

This reverts commit 9eee1b383772adf058130d6e5eab409f57ce03cd.
 2015-09-24 08:53:09 +01:00
Brian Gough
2ab1778dd9 move default value of lastVersion into function body 2015-09-23 16:31:33 +01:00
Brian Gough
bdf1d267f0 fix tests 2015-09-23 15:34:36 +01:00
Brian Gough
dc0044020f only archive entries older than the current update 
to avoid a stale version of the current update ever being pulled back
from S3
 2015-09-23 14:33:40 +01:00
Brian Gough
696a866b67 pause the stream of ops, not the download 
the download is buffered in the lineStream so a lot comes out even
after pausing the S3 download.
 2015-09-23 13:38:57 +01:00
Brian Gough
847a553344 prevent double archiving by checking if any inS3 field is already present 2015-09-23 13:29:32 +01:00
Brian Gough
e49f260507 allow rollback/locking by setting inS3:false when starting the archive process 2015-09-23 13:28:07 +01:00
Brian Gough
551e8334cf compressedUpdates are now never inserted with inS3 
it is now always added later, and a new update is forced for any
addition to an archived update
 2015-09-23 13:25:10 +01:00
Brian Gough
d6b827426c support forcing new compressed update in popLastCompressedUpdate 
callback with a null update, passing the version as an additional
argument
 2015-09-23 13:22:38 +01:00
Brian Gough
dc35ef5cda fix tests for archiving 2015-09-21 13:24:06 +01:00
Brian Gough
a10dc4f898 Merge pull request #6 from heukirne/s3-archive 
Add S3 archive track changes feature
 2015-09-21 11:25:06 +01:00
Brian Gough
0e627c92d8 avoid clobbering global _ in loop 2015-09-18 16:26:05 +01:00
Henrique Dias
aa66c5ee8c improve size function 2015-09-17 10:41:53 -03:00
Henrique Dias
3f712c452a add size bulk limit 2015-09-17 09:23:13 -03:00
Henrique Dias
01cbfd5338 fix minor issues in acceptance test 2015-09-16 19:33:23 -03:00
Henrique Dias
a4575f178d fix unit test 2015-09-16 18:39:07 -03:00
Brian Gough
12c5098b48 fix lock in test 2015-09-16 16:50:36 +01:00
Brian Gough
7af5050370 add lock to unarchive doc 2015-09-16 16:18:36 +01:00
Brian Gough
18f06a3daf increase lock timeouts for archiving 2015-09-16 16:09:38 +01:00
Brian Gough
b4ffa7d57e share the document lock between archiving and packing 2015-09-16 16:03:55 +01:00
Brian Gough
9d39012b49 add error handler to each stage of download pipeline 2015-09-16 16:00:37 +01:00
Brian Gough
d9085a5e5e add error handler to each stage of upload pipeline 2015-09-16 16:00:25 +01:00
Brian Gough
1c1b1d9595 log the case where there are no entries in the document history 2015-09-16 15:34:30 +01:00
Brian Gough
82d0f4fce8 make unarchive more responsive by downloading documents in parallel 
unarchive is triggered interactively so we should try to make it
reasonably fast
 2015-09-16 15:33:59 +01:00
Brian Gough
dfa0036507 pause stream while writing to mongo 2015-09-16 15:32:36 +01:00
Brian Gough
70200a9cf1 only log document ids, not document content 
avoid filling the log with large documents
 2015-09-16 15:31:43 +01:00
Brian Gough
d3dff28bea Merge remote-tracking branch 'origin/master' into heukirne-s3-archive 2015-09-15 15:19:43 +01:00
Brian Gough
092f98d3ad suppress error in normal shutdown case 2015-09-12 11:07:54 +01:00
Shane Kilkelly
39f528bcbc Add a test to check that users are summarised properly even when a null user occurs earlier in the update list. 2015-09-11 14:12:01 +01:00
Shane Kilkelly
eab8b4b6c8 Null safe access of id property, needed as user can be null. 2015-09-11 14:07:06 +01:00
Shane Kilkelly
0ad374556d Add a comment for clarity. 2015-09-10 16:43:40 +01:00
Shane Kilkelly
97326308fa Update the Acceptance tests to include a case where a user doesn't exist. 2015-09-10 15:40:43 +01:00
Shane Kilkelly
8387383cb4 In _summarizeUpdates, allow null users through. 
A null value represents a deleted or otherwise missing user record.
 2015-09-10 14:32:47 +01:00
Shane Kilkelly
810bddb2cb Log a message when the web api produces a 404 response. 2015-09-10 14:32:35 +01:00
Shane Kilkelly
575bdc62ec Add a test for when the user can't be found. 2015-09-10 14:32:18 +01:00
Shane Kilkelly
522786d45e Produce a null value, rather than crashing when the user info service returns 404. 2015-09-09 15:48:22 +01:00
Henry Oswald
18d817ee0a added some missing error handling 2015-09-08 16:33:45 +01:00
Henry Oswald
17b0d99a65 rework the archiveDocChangesWithLock function 
make it a bit more readable for me, struggle to trust indentation
based calls in coffeescript
 2015-09-08 16:26:01 +01:00
Henry Oswald
0b3ebcff06 remove if statments checking if s3 is a backend 
if its not enable then it can crash. In prod it should always be there
or not used at all
 2015-09-08 16:23:15 +01:00
Henrique Dias
c5a8a249c6 add unarchive acceptance tests 2015-09-03 08:36:32 -03:00
Henrique Dias
da9e7dc7e1 init archive acceptance tests 2015-09-02 18:47:34 -03:00
Henrique Dias
7de103af68 fix unit scope error 2015-09-02 17:00:32 -03:00
Henrique Dias
d2b1243701 split MongoAWS files 2015-09-02 15:45:29 -03:00
Henrique Dias
0c16fbed88 add more unit tests 2015-09-02 15:39:19 -03:00
Henrique Dias
1abcea1a66 add some unit test 2015-08-31 18:13:18 -03:00
Henrique Dias
efff026a79 handle easier propagation 2015-08-25 16:52:28 -03:00
Henrique Dias
f910e63e90 fix null case 2015-08-24 12:22:17 -03:00
Henrique Dias
fcbe4aa925 fix inS3 propagation 2015-08-24 12:19:19 -03:00
Henrique Dias
1ccba422c8 remove unused function 2015-08-24 10:55:27 -03:00
Henrique Dias
98ce03b2f2 replace docs collection to DocstoreHandler 2015-08-24 10:38:31 -03:00
Henrique Dias
04ec45529f restore updates from S3 when exists 
fix: avoid rearchiving
 2015-08-18 17:11:19 -03:00
Henrique Dias
20c3e15f93 fix bulk insert limit 2015-08-14 19:58:38 -03:00
Henrique Dias
26c8048729 change mongo stream method (still have a bug in bulk insert limit) 2015-08-14 19:19:54 -03:00
Henrique Dias
fd4afb3574 Archive changes, care about: versioin, expiresAt and Lock 2015-08-14 15:07:16 -03:00
Henrique Dias
6bc9c9010a handle auto unarchive track changes 2015-08-09 19:52:32 -03:00
Henrique Dias
3bc5380468 handle inS3 flag 2015-08-09 17:50:15 -03:00
Henrique Dias
daa42bcea0 change s3Stream lib 2015-08-09 15:47:47 -03:00
Henrique Dias
bca48ac117 add unarchive doc track from s3 2015-08-06 17:09:36 -03:00
Henrique Dias
438c4f4d0c using mongoexport for s3 archive 2015-08-06 15:46:44 -03:00
Henrique Dias
028fe2fa03 archive docChanges list to s3 2015-08-06 11:11:43 -03:00
Henrique Dias
ae047ecf76 init s3 feature 2015-08-06 10:00:09 -03:00
Brian Gough
775f5ebbe1 add configurable limit, delay and timeout to /pack via query string 2015-06-05 13:38:47 +01:00
Brian Gough
76d1350593 fix unit tests 
packManager now uses LockManager, which tries to talk to redis and
needs to be replaced by a stub in the unit tests.
 2015-06-05 10:59:46 +01:00
Brian Gough
23d2518ebb added a /pack endpoint to fork a pack worker 2015-06-04 16:36:56 +01:00
Brian Gough
289616d3db added a /doc/:doc_id/pack endpoint 2015-06-04 16:23:49 +01:00
Brian Gough
3f2e4b0c11 move pack script functionality into PackManager 2015-06-04 16:21:40 +01:00
Brian Gough
8668bb9be1 use docHistoryStats to drive the packing script 2015-06-03 10:36:31 +01:00
Brian Gough
27a3511b37 update docHistoryStats after packing ops 2015-06-03 10:36:07 +01:00
Brian Gough
19d812734e make PackManager parameters configurable 2015-06-03 10:35:36 +01:00
Brian Gough
2521c8f751 added npm packages for pack.coffee script 2015-05-26 14:14:51 +01:00
Brian Gough
66bca8d05c include the current date in the updates to docHistoryStats 2015-05-26 11:00:55 +01:00
Brian Gough
1811ac2145 added support for cleaning old expired ops in packs 2015-05-22 15:40:02 +01:00
Brian Gough
5c4afd5303 add docHistoryStats collection to keep track of updates to docs 2015-05-22 15:40:01 +01:00
Brian Gough
78f0bdbae3 fix name of temporary parameter to match other methods 2015-05-22 15:40:01 +01:00
Brian Gough
adc2866a7d add check to exclude temporary ops from packs 2015-05-22 15:40:01 +01:00
Brian Gough
ffeb1cccb6 move pack migration code into PackManager 2015-05-22 15:40:01 +01:00
Brian Gough
8d3e0b2353 added consistency check to pack migration script 2015-05-21 15:46:29 +01:00
Henry Oswald
b37d0dd08a make startup message consistent 2015-04-30 15:07:11 +01:00
James Allen
24319f6d50 Release version 0.1.4 2015-03-20 15:27:34 +00:00
James Allen
79b9c48efd Release version 0.1.3 2015-02-26 11:35:14 +00:00
Brian Gough
34fa398cf6 increase pack migrate delay to 2 seconds 2015-02-23 17:00:54 +00:00
Brian Gough
09e24413e6 fix pack migration to run sequentially not in parallel 2015-02-23 17:00:43 +00:00
Brian Gough
ff7860b6dc added a time delay to the pack migration script 2015-02-23 14:10:09 +00:00
Brian Gough
b19f3835d5 added checkpointing and logging to pack script 2015-02-23 12:13:38 +00:00
Brian Gough
4c0eea9916 return an error if trying to pop the last update from a pack 2015-02-17 14:53:50 +00:00
Brian Gough
24701219ff clean up comments and remove console.logs 2015-02-17 13:47:55 +00:00
Brian Gough
76cbf32e6a cleanup 
remove unnecessary comments, whitespace and unused modules
 2015-02-17 13:41:31 +00:00
Brian Gough
f69f6452e9 Merge remote-tracking branch 'origin/master' into support-packs 2015-02-17 13:38:30 +00:00
Brian Gough
448a7b7f36 rename MongoPackManager to PackManager 2015-02-17 11:14:13 +00:00
Brian Gough
92e67511d5 sort by doc_id to ensure consistent order for all changes with the same timestamp 2015-02-16 16:44:06 +00:00
Brian Gough
7d6811559c test pack for getDocUpdates with time 2015-02-16 16:43:27 +00:00
Brian Gough
e55f2a0ff7 test pack for getDocUpdates with version 2015-02-16 15:59:58 +00:00
Brian Gough
f5a9184b57 update the tests to work with packs 2015-02-16 15:38:40 +00:00
Brian Gough
fa1e890282 move update deduplication to filterAndLimit method 2015-02-16 15:37:47 +00:00
Brian Gough
0516e9429d ensure document results are in version order 2015-02-16 13:43:27 +00:00
Brian Gough
8615b7a086 fix project results order, sort by time 2015-02-16 13:43:11 +00:00
Brian Gough
3ab8238013 update mongojs version to allow use of bulk updates 2015-02-13 17:01:55 +00:00
Brian Gough
18b34d1bcb clean up packing script 2015-02-13 17:01:37 +00:00
Brian Gough
cafda57710 command-line packing script 2015-02-13 16:23:36 +00:00
Brian Gough
4f36ccd519 moved pack code into MongoPackManager 2015-02-13 16:18:15 +00:00
Brian Gough
682e8e8ab6 add a sparse mongo index for finding packs 2015-02-10 16:54:58 +00:00
Brian Gough
1d7f0919a4 fix mongo logic for project search with packs 2015-02-10 16:54:45 +00:00
James Allen
2d25d54572 Release version 0.1.2 2015-02-10 13:18:12 +00:00
Brian Gough
7fdce8fc48 split pack handling code into separate parts for docs and projects 
need to debug projects
 2015-02-09 16:53:44 +00:00
Brian Gough
d8dc56b031 don't try to getLastCompressedUpdates from packs 2015-02-09 16:53:05 +00:00
Brian Gough
f50f091fc6 added comments 2015-02-06 16:59:09 +00:00
Brian Gough
e47476369a clean up docHistory pack decoding 2015-02-06 15:04:46 +00:00
Brian Gough
8e810bab14 report error for inconsistent history results 2015-02-05 16:37:06 +00:00
Brian Gough
715b0df4a1 prototype for expanding packs 2015-02-05 16:36:41 +00:00
Henry Oswald
3328acc82b bump sharelatex-redis 2014-10-07 08:33:00 +01:00
Henry Oswald
66bba0dec9 bump redis-sharelatex 2014-09-29 11:41:09 +01:00
Henry Oswald
3f4452e6af don't run acceptence tests on travis 2014-09-29 11:33:40 +01:00
Henry Oswald
1452eb4572 fixed tests 2014-09-26 17:51:41 +01:00