Commit graph

346 commits

Author SHA1 Message Date
Brian Gough
2781d9fd80 keep track of processed/total docs in archiving 2017-05-09 14:00:16 +01:00
Brian Gough
c26bccd390 make arguments handling for packworker more robust 2017-05-09 14:00:16 +01:00
Brian Gough
cac7556ad5 only check packs for archiving once each week 2017-05-09 14:00:15 +01:00
Brian Gough
c6e83c6cb8 allow archiving to exit properly on hard timeout 2017-05-09 14:00:15 +01:00
Brian Gough
9936710a1a replace error with warning for archiving timeout 2017-05-09 14:00:15 +01:00
Brian Gough
b82567ef79 support scan operations on redis cluster 2017-05-05 11:30:55 +01:00
Brian Gough
534bd8f7ad add flushing code with error check for cluster 2017-05-02 14:19:49 +01:00
Brian Gough
3b5a07d530 fix update to work with redis cluster
the items do not have the same tag, so we must delete the entry from the
docsWithHistoryOps set outside the multi.
2017-04-26 16:25:34 +01:00
Brian Gough
3045becbb6 use redis-sharelatex with redis cluster config 2017-04-24 14:23:32 +01:00
Brian Gough
df38893538 remove old single redis flushing code 2017-04-24 11:36:31 +01:00
Brian Gough
2a68b81cc1 randomise order of flush 2017-04-20 14:32:23 +01:00
Brian Gough
9eb36f9692 improve log message for flushing 2017-04-20 11:55:32 +01:00
Brian Gough
43f51355b2 add limit parameter to flushAll 2017-04-20 11:01:46 +01:00
Brian Gough
976f19f763 remove spurious call to getAllDocIdsWithHistoryOps 2017-04-20 10:14:44 +01:00
Brian Gough
70ce3a5271 Merge pull request #29 from sharelatex/bg-simple-flush-all
simple flush for all projects
2017-04-19 16:32:09 +01:00
Brian Gough
b5b61b98d0 avoid blocking when fetching redis keys
use scan instead of keys method
2017-04-19 15:39:33 +01:00
Brian Gough
9ce6d77cca add check for dangling updates 2017-04-13 11:31:45 +01:00
Brian Gough
1ec9f87542 update timeouts to allow for 5 second retry delay
now using requestretry module, which has a five second delay between
retries. Need to increase the timeout on the health checks to allow for
this, as the timeout is currently shorter than the retry delay.
2017-04-12 16:47:34 +01:00
Brian Gough
d9ed026d91 simple flush for all projects
does not work with redis cluster, only single redis
2017-04-12 16:34:28 +01:00
Shane Kilkelly
96f5802462 Merge branch 'sk-upgrade-metrics' 2017-04-04 16:16:37 +01:00
Shane Kilkelly
9a9f0f5c4d Remove the Metrics module, use metrics-sharelatex 2017-04-03 15:17:29 +01:00
Brian Gough
5d6d33966b Merge pull request #27 from sharelatex/bg-retry-web-api-request
use requestretry to work around web api timeouts
2017-04-03 14:05:07 +01:00
Brian Gough
7237467cb4 use requestretry to work around web api timeouts 2017-04-03 11:06:50 +01:00
Brian Gough
b0b05d1e1b improve error handling of releaseLock
ported from docupdater
2017-03-31 15:17:13 +01:00
Brian Gough
8bb3dd07a4 continue when ops are out of order 2017-03-30 13:37:11 +01:00
Brian Gough
d29141d583 return error when out-of-order ops detected 2017-03-30 11:48:26 +01:00
Brian Gough
28d2ec93b8 Revert "Revert "Merge pull request #23 from sharelatex/bg-move-lock""
This reverts commit 85bc45099f047aa8dfa6d189f02f4b4327c9d602.
2017-03-28 12:52:49 +01:00
Brian Gough
a73c8ba7b8 Revert "Merge pull request #23 from sharelatex/bg-move-lock"
This reverts commit 72e6756415e390a29fa784529453f926e5144358, reversing
changes made to 8f507dcc9fbcd51072047c82a443137aaac0f377.
2017-03-24 09:17:11 +00:00
Brian Gough
e028de115e Merge pull request #23 from sharelatex/bg-move-lock
move lock inside web http calls
2017-03-23 13:42:52 +00:00
Brian Gough
bc7815f7fc remove old processUncompressedUpdates method
replace with new per doc method
2017-03-22 16:59:52 +00:00
Brian Gough
b1c0ebbaae add withLock to processUncompressedUpdatesForDoc 2017-03-22 16:16:04 +00:00
Brian Gough
0dd668416d increase request timeout now it is outside lock 2017-03-22 16:02:50 +00:00
Brian Gough
6795820933 move lock inside web http calls 2017-03-21 16:49:23 +00:00
James Allen
d60445adfc Pass undo flag to doc updater when restoring from history 2017-03-21 11:32:53 +00:00
Shane Kilkelly
fa7ee739bb Update to new metrics api 2017-03-17 14:58:35 +00:00
Shane Kilkelly
c30e672549 Merge branch 'master' into sk-upgrade-metrics 2017-03-16 15:17:57 +00:00
Shane Kilkelly
c303a1a386 wip: upgrade metrics 2017-03-16 15:17:38 +00:00
Brian Gough
ac5d59211d revert random TTL in favour of delay in archiving
there could be some issues with newer packs expiring before older ones
2017-01-31 15:07:58 +00:00
Henry Oswald
ad5af5f4dd ceil not round 2017-01-26 15:14:20 +00:00
Henry Oswald
1d356f93ec added random up to 30 mins delay to the ttl of mongo objects 2017-01-26 13:07:54 +00:00
James Allen
dadc548f81 Ignore comment updates 2017-01-12 10:04:50 +01:00
James Allen
694be95e66 Swap start_ts and end_ts comparison to correctly break chunks after 5 minutes 2016-10-11 11:03:52 +01:00
James Allen
a197c0d148 Merge pull request #18 from sharelatex/ja-split-on-deletes
Split update summary on big deletes
2016-10-10 08:46:46 +01:00
James Allen
40ed6fee46 Split update summary on big deletes 2016-10-04 15:13:04 +01:00
Brian Gough
92e36023e5 Merge pull request #17 from sharelatex/ja-consistency-tweaks
Retry rewind if doc and update versions don't match
2016-09-30 15:14:27 +01:00
James Allen
f8865e616d Update ensureIndices to reflect reality 2016-09-30 14:34:38 +01:00
James Allen
3c8aeb1262 Log number of retries 2016-09-30 13:38:47 +01:00
James Allen
a7f44bcd01 Add missing return on callback 2016-09-30 13:36:31 +01:00
James Allen
bddd1fda7d Retry rewind if doc and update versions don't match 2016-09-30 11:36:47 +01:00
James Allen
cce93b95a5 Fetch updates before doc content when building a diff 2016-09-22 11:19:05 +01:00
James Allen
05a048db9a Add in a little more logging 2016-09-22 11:19:05 +01:00
James Allen
dfe26262ec Return a No-op if diff returns nothing 2016-09-16 11:50:44 +01:00
James Allen
80375ae2dd Run a diff against big delete - insert changes which are likely copy-pastes 2016-09-16 11:33:36 +01:00
Brian Gough
6e5eadfa86 include a timeout on WebApiManager requests 2016-04-13 16:42:36 +01:00
Brian Gough
a55b72871f don't let s3 errors stop archive worker 2016-04-13 14:39:11 +01:00
Brian Gough
b343be844e added metrics to pack worker for archiving 2016-04-08 10:29:04 +01:00
Brian Gough
76fe194815 add a metric for append-pack 2016-04-07 15:56:11 +01:00
Brian Gough
78100e40c8 add missing metrics file 2016-04-07 15:33:20 +01:00
Brian Gough
1a1fa8798d log attempted update when throwing error in DiffGenerator 2016-04-07 15:16:50 +01:00
Brian Gough
6db310bf6b add insert/archive/unarchive metrics 2016-04-07 15:16:50 +01:00
Brian Gough
d0e08039da don't modify expiry for temporary packs 2016-04-07 15:16:50 +01:00
Brian Gough
fd49601716 preserve existing history when user upgrades 2016-04-07 15:16:38 +01:00
Brian Gough
e292de5eb0 fix to avoid ever appending permanent changes to expiring packs 2016-04-06 17:00:16 +01:00
Brian Gough
8b7bdd345b consider all packs for archiving 2016-04-06 15:17:31 +01:00
Brian Gough
ef47337c78 remove additional fields 2016-04-06 15:17:20 +01:00
Brian Gough
0b9a0730c0 mark temporary packs with a last_checked date in the far future
they do not need to be checked for archiving
2016-04-06 14:29:49 +01:00
Brian Gough
08fc151eee avoid unnecessary call to insert packs into index 2016-04-06 14:29:21 +01:00
Brian Gough
719e0291aa consider all packs for processing
to allow finalisation of old head packs
2016-04-06 14:27:44 +01:00
Brian Gough
79baa99634 clean up logging 2016-04-06 14:26:54 +01:00
Brian Gough
6ab75795a2 archive head packs after sufficient time 2016-04-06 13:30:09 +01:00
Brian Gough
6e18d49736 support archiving from list of project_ids/doc_ids 2016-04-04 17:00:19 +01:00
Brian Gough
31348141d8 increase logging for discarded updates and version mismatch 2016-03-24 11:55:29 +00:00
Brian Gough
181cebecef avoid call to fetch packs unnecessarily 2016-03-24 11:55:29 +00:00
Brian Gough
98683de3ae temporarily disable ttl behaviour
allow existing packs without temporary flag to expire
2016-03-24 11:38:59 +00:00
Brian Gough
3f388fb0ac only change ttl on cached packs, not temporary ones
temporary = without versioning feature enabled
cached = permanent versioned retrieved from s3
2016-03-24 11:38:09 +00:00
Brian Gough
8d900013d9 record whether a pack is temporary in the pack itself
using the expiresAt field no longer determines if the pack is
temporary because archived packs have an expiresAt field added when
they are retrieved from s3
2016-03-24 11:02:58 +00:00
Brian Gough
98738d1344 fix for acceptance test 2016-03-10 15:15:29 +00:00
Brian Gough
f01bf99682 acceptance tests - work in progress 2016-03-09 16:56:49 +00:00
Brian Gough
f6367e21b8 give separate error for archive in progress vs completed 2016-03-09 14:44:59 +00:00
Brian Gough
7350ab531d exclude already cached packs from archival 2016-03-09 14:44:59 +00:00
Brian Gough
28b184e0ca fix incorrect use of _.union (argument must be array) 2016-03-09 14:44:59 +00:00
Brian Gough
8922b97bd7 avoid duplicate filling of UserInfo in getDocUpdates 2016-03-09 14:44:59 +00:00
Brian Gough
7e6ea2793b remove startup dependency on s3 settings 2016-03-09 13:28:02 +00:00
Brian Gough
1419d20b1f fix indentation 2016-03-04 15:43:32 +00:00
Brian Gough
3175f6d3a6 handle case where index does not exist 2016-03-03 14:36:16 +00:00
Henry Oswald
e8b3fb5be6 added more logging to failed health checks 2016-03-03 10:50:55 +00:00
Brian Gough
795f717bab added index definitions 2016-03-01 11:38:23 +00:00
Brian Gough
3d9dfeccc3 remove pack worker
remove the op-specific code

remove tests for ops, now only packing

remove unused packing code

work in progress

store index for completed packs only

support archiving and unarchiving of individual packs

remove support for archiving whole document history

split out ArchiveManager, IndexManager

remove old DocArchive code

remove docHistoryStats collection

comment about archiving

added method to look at index when last pack has been archived

added start of iterator for project results

use a proper iterator

added heap module

getting it working

increase pack size since bulk operations no longer needed

remove unused MongoAWSexternal

cleanup

added doc iterator

remove old query code

added missing files

cleanup

clean upclean up

started adding pack worker for archiving

work in progress

work in progress

getting pack worker working

updating worker

getting packworker working

added lock

use correct key name for track changes aws access

use correct key name for track changes aws access

always send back users array

fix up comparison of retrieved objects

handle op ids inside packs

log when s3 download completes

comments

cleanup, remove finalisation ideacleanup, remove finalisation idea

remove logging
2016-03-01 10:10:02 +00:00
Brian Gough
a23ddf31c0 allow packing of temporary ops 2016-01-29 12:36:03 +00:00
Brian Gough
77cafa36af support continuing from last packed doc 2016-01-28 16:40:20 +00:00
Brian - Work
666a07e5ba move lock check into HealthChecker
to avoid dependency of HttpController on LockManager in unit tests
2016-01-27 16:04:55 +00:00
Brian Gough
199d2aaa92 script to pack existing docs 2016-01-27 15:14:23 +00:00
Brian Gough
b44a7b9aa6 reject very large ops 2016-01-26 14:52:40 +00:00
Brian Gough
b7a4c72f9c avoid compressing updates if the result would be too big 2016-01-26 12:23:21 +00:00
Brian Gough
ed0aaa189d add test for non-overlapping insert-delete case 2016-01-26 12:13:43 +00:00
Brian Gough
b3ddd839e6 add logging of raw updates 2016-01-26 11:28:02 +00:00
Brian Gough
29c7c5e249 enable packs by default for new docs 2016-01-25 09:55:55 +00:00
Brian Gough
d10123d3c4 include n parameter when packing 2016-01-25 09:45:25 +00:00
Brian Gough
9b2cd11cd4 don't try to append to packs when using the old op code 2016-01-22 10:45:24 +00:00
Brian Gough
84ace7f4c7 use packs only for temporary ops 2016-01-20 14:22:13 +00:00