Brian Gough
2781d9fd80
keep track of processed/total docs in archiving
2017-05-09 14:00:16 +01:00
Brian Gough
c26bccd390
make arguments handling for packworker more robust
2017-05-09 14:00:16 +01:00
Brian Gough
cac7556ad5
only check packs for archiving once each week
2017-05-09 14:00:15 +01:00
Brian Gough
c6e83c6cb8
allow archiving to exit properly on hard timeout
2017-05-09 14:00:15 +01:00
Brian Gough
9936710a1a
replace error with warning for archiving timeout
2017-05-09 14:00:15 +01:00
Brian Gough
b82567ef79
support scan operations on redis cluster
2017-05-05 11:30:55 +01:00
Brian Gough
534bd8f7ad
add flushing code with error check for cluster
2017-05-02 14:19:49 +01:00
Brian Gough
3b5a07d530
fix update to work with redis cluster
...
the items do not have the same tag, so we must delete the entry from the
docsWithHistoryOps set outside the multi.
2017-04-26 16:25:34 +01:00
Brian Gough
3045becbb6
use redis-sharelatex with redis cluster config
2017-04-24 14:23:32 +01:00
Brian Gough
df38893538
remove old single redis flushing code
2017-04-24 11:36:31 +01:00
Brian Gough
2a68b81cc1
randomise order of flush
2017-04-20 14:32:23 +01:00
Brian Gough
9eb36f9692
improve log message for flushing
2017-04-20 11:55:32 +01:00
Brian Gough
43f51355b2
add limit parameter to flushAll
2017-04-20 11:01:46 +01:00
Brian Gough
976f19f763
remove spurious call to getAllDocIdsWithHistoryOps
2017-04-20 10:14:44 +01:00
Brian Gough
70ce3a5271
Merge pull request #29 from sharelatex/bg-simple-flush-all
...
simple flush for all projects
2017-04-19 16:32:09 +01:00
Brian Gough
b5b61b98d0
avoid blocking when fetching redis keys
...
use scan instead of keys method
2017-04-19 15:39:33 +01:00
Brian Gough
9ce6d77cca
add check for dangling updates
2017-04-13 11:31:45 +01:00
Brian Gough
1ec9f87542
update timeouts to allow for 5 second retry delay
...
now using requestretry module, which has a five second delay between
retries. Need to increase the timeout on the health checks to allow for
this, as the timeout is currently shorter than the retry delay.
2017-04-12 16:47:34 +01:00
Brian Gough
d9ed026d91
simple flush for all projects
...
does not work with redis cluster, only single redis
2017-04-12 16:34:28 +01:00
Shane Kilkelly
96f5802462
Merge branch 'sk-upgrade-metrics'
2017-04-04 16:16:37 +01:00
Shane Kilkelly
9a9f0f5c4d
Remove the Metrics module, use metrics-sharelatex
2017-04-03 15:17:29 +01:00
Brian Gough
5d6d33966b
Merge pull request #27 from sharelatex/bg-retry-web-api-request
...
use requestretry to work around web api timeouts
2017-04-03 14:05:07 +01:00
Brian Gough
7237467cb4
use requestretry to work around web api timeouts
2017-04-03 11:06:50 +01:00
Brian Gough
b0b05d1e1b
improve error handling of releaseLock
...
ported from docupdater
2017-03-31 15:17:13 +01:00
Brian Gough
8bb3dd07a4
continue when ops are out of order
2017-03-30 13:37:11 +01:00
Brian Gough
d29141d583
return error when out-of-order ops detected
2017-03-30 11:48:26 +01:00
Brian Gough
28d2ec93b8
Revert "Revert "Merge pull request #23 from sharelatex/bg-move-lock""
...
This reverts commit 85bc45099f047aa8dfa6d189f02f4b4327c9d602.
2017-03-28 12:52:49 +01:00
Brian Gough
a73c8ba7b8
Revert "Merge pull request #23 from sharelatex/bg-move-lock"
...
This reverts commit 72e6756415e390a29fa784529453f926e5144358, reversing
changes made to 8f507dcc9fbcd51072047c82a443137aaac0f377.
2017-03-24 09:17:11 +00:00
Brian Gough
e028de115e
Merge pull request #23 from sharelatex/bg-move-lock
...
move lock inside web http calls
2017-03-23 13:42:52 +00:00
Brian Gough
bc7815f7fc
remove old processUncompressedUpdates method
...
replace with new per doc method
2017-03-22 16:59:52 +00:00
Brian Gough
b1c0ebbaae
add withLock to processUncompressedUpdatesForDoc
2017-03-22 16:16:04 +00:00
Brian Gough
0dd668416d
increase request timeout now it is outside lock
2017-03-22 16:02:50 +00:00
Brian Gough
6795820933
move lock inside web http calls
2017-03-21 16:49:23 +00:00
James Allen
d60445adfc
Pass undo flag to doc updater when restoring from history
2017-03-21 11:32:53 +00:00
Shane Kilkelly
fa7ee739bb
Update to new metrics api
2017-03-17 14:58:35 +00:00
Shane Kilkelly
c30e672549
Merge branch 'master' into sk-upgrade-metrics
2017-03-16 15:17:57 +00:00
Shane Kilkelly
c303a1a386
wip: upgrade metrics
2017-03-16 15:17:38 +00:00
Brian Gough
ac5d59211d
revert random TTL in favour of delay in archiving
...
there could be some issues with newer packs expiring before older ones
2017-01-31 15:07:58 +00:00
Henry Oswald
ad5af5f4dd
ceil not round
2017-01-26 15:14:20 +00:00
Henry Oswald
1d356f93ec
added random up to 30 mins delay to the ttl of mongo objects
2017-01-26 13:07:54 +00:00
James Allen
dadc548f81
Ignore comment updates
2017-01-12 10:04:50 +01:00
James Allen
694be95e66
Swap start_ts and end_ts comparison to correctly break chunks after 5 minutes
2016-10-11 11:03:52 +01:00
James Allen
a197c0d148
Merge pull request #18 from sharelatex/ja-split-on-deletes
...
Split update summary on big deletes
2016-10-10 08:46:46 +01:00
James Allen
40ed6fee46
Split update summary on big deletes
2016-10-04 15:13:04 +01:00
Brian Gough
92e36023e5
Merge pull request #17 from sharelatex/ja-consistency-tweaks
...
Retry rewind if doc and update versions don't match
2016-09-30 15:14:27 +01:00
James Allen
f8865e616d
Update ensureIndices to reflect reality
2016-09-30 14:34:38 +01:00
James Allen
3c8aeb1262
Log number of retries
2016-09-30 13:38:47 +01:00
James Allen
a7f44bcd01
Add missing return on callback
2016-09-30 13:36:31 +01:00
James Allen
bddd1fda7d
Retry rewind if doc and update versions don't match
2016-09-30 11:36:47 +01:00
James Allen
cce93b95a5
Fetch updates before doc content when building a diff
2016-09-22 11:19:05 +01:00
James Allen
05a048db9a
Add in a little more logging
2016-09-22 11:19:05 +01:00
James Allen
dfe26262ec
Return a No-op if diff returns nothing
2016-09-16 11:50:44 +01:00
James Allen
80375ae2dd
Run a diff against big delete - insert changes which are likely copy-pastes
2016-09-16 11:33:36 +01:00
Brian Gough
6e5eadfa86
include a timeout on WebApiManager requests
2016-04-13 16:42:36 +01:00
Brian Gough
a55b72871f
don't let s3 errors stop archive worker
2016-04-13 14:39:11 +01:00
Brian Gough
b343be844e
added metrics to pack worker for archiving
2016-04-08 10:29:04 +01:00
Brian Gough
76fe194815
add a metric for append-pack
2016-04-07 15:56:11 +01:00
Brian Gough
78100e40c8
add missing metrics file
2016-04-07 15:33:20 +01:00
Brian Gough
1a1fa8798d
log attempted update when throwing error in DiffGenerator
2016-04-07 15:16:50 +01:00
Brian Gough
6db310bf6b
add insert/archive/unarchive metrics
2016-04-07 15:16:50 +01:00
Brian Gough
d0e08039da
don't modify expiry for temporary packs
2016-04-07 15:16:50 +01:00
Brian Gough
fd49601716
preserve existing history when user upgrades
2016-04-07 15:16:38 +01:00
Brian Gough
e292de5eb0
fix to avoid ever appending permanent changes to expiring packs
2016-04-06 17:00:16 +01:00
Brian Gough
8b7bdd345b
consider all packs for archiving
2016-04-06 15:17:31 +01:00
Brian Gough
ef47337c78
remove additional fields
2016-04-06 15:17:20 +01:00
Brian Gough
0b9a0730c0
mark temporary packs with a last_checked date in the far future
...
they do not need to be checked for archiving
2016-04-06 14:29:49 +01:00
Brian Gough
08fc151eee
avoid unnecessary call to insert packs into index
2016-04-06 14:29:21 +01:00
Brian Gough
719e0291aa
consider all packs for processing
...
to allow finalisation of old head packs
2016-04-06 14:27:44 +01:00
Brian Gough
79baa99634
clean up logging
2016-04-06 14:26:54 +01:00
Brian Gough
6ab75795a2
archive head packs after sufficient time
2016-04-06 13:30:09 +01:00
Brian Gough
6e18d49736
support archiving from list of project_ids/doc_ids
2016-04-04 17:00:19 +01:00
Brian Gough
31348141d8
increase logging for discarded updates and version mismatch
2016-03-24 11:55:29 +00:00
Brian Gough
181cebecef
avoid call to fetch packs unnecessarily
2016-03-24 11:55:29 +00:00
Brian Gough
98683de3ae
temporarily disable ttl behaviour
...
allow existing packs without temporary flag to expire
2016-03-24 11:38:59 +00:00
Brian Gough
3f388fb0ac
only change ttl on cached packs, not temporary ones
...
temporary = without versioning feature enabled
cached = permanent versioned retrieved from s3
2016-03-24 11:38:09 +00:00
Brian Gough
8d900013d9
record whether a pack is temporary in the pack itself
...
using the expiresAt field no longer determines if the pack is
temporary because archived packs have an expiresAt field added when
they are retrieved from s3
2016-03-24 11:02:58 +00:00
Brian Gough
98738d1344
fix for acceptance test
2016-03-10 15:15:29 +00:00
Brian Gough
f01bf99682
acceptance tests - work in progress
2016-03-09 16:56:49 +00:00
Brian Gough
f6367e21b8
give separate error for archive in progress vs completed
2016-03-09 14:44:59 +00:00
Brian Gough
7350ab531d
exclude already cached packs from archival
2016-03-09 14:44:59 +00:00
Brian Gough
28b184e0ca
fix incorrect use of _.union (argument must be array)
2016-03-09 14:44:59 +00:00
Brian Gough
8922b97bd7
avoid duplicate filling of UserInfo in getDocUpdates
2016-03-09 14:44:59 +00:00
Brian Gough
7e6ea2793b
remove startup dependency on s3 settings
2016-03-09 13:28:02 +00:00
Brian Gough
1419d20b1f
fix indentation
2016-03-04 15:43:32 +00:00
Brian Gough
3175f6d3a6
handle case where index does not exist
2016-03-03 14:36:16 +00:00
Henry Oswald
e8b3fb5be6
added more logging to failed health checks
2016-03-03 10:50:55 +00:00
Brian Gough
795f717bab
added index definitions
2016-03-01 11:38:23 +00:00
Brian Gough
3d9dfeccc3
remove pack worker
...
remove the op-specific code
remove tests for ops, now only packing
remove unused packing code
work in progress
store index for completed packs only
support archiving and unarchiving of individual packs
remove support for archiving whole document history
split out ArchiveManager, IndexManager
remove old DocArchive code
remove docHistoryStats collection
comment about archiving
added method to look at index when last pack has been archived
added start of iterator for project results
use a proper iterator
added heap module
getting it working
increase pack size since bulk operations no longer needed
remove unused MongoAWSexternal
cleanup
added doc iterator
remove old query code
added missing files
cleanup
clean upclean up
started adding pack worker for archiving
work in progress
work in progress
getting pack worker working
updating worker
getting packworker working
added lock
use correct key name for track changes aws access
use correct key name for track changes aws access
always send back users array
fix up comparison of retrieved objects
handle op ids inside packs
log when s3 download completes
comments
cleanup, remove finalisation ideacleanup, remove finalisation idea
remove logging
2016-03-01 10:10:02 +00:00
Brian Gough
a23ddf31c0
allow packing of temporary ops
2016-01-29 12:36:03 +00:00
Brian Gough
77cafa36af
support continuing from last packed doc
2016-01-28 16:40:20 +00:00
Brian - Work
666a07e5ba
move lock check into HealthChecker
...
to avoid dependency of HttpController on LockManager in unit tests
2016-01-27 16:04:55 +00:00
Brian Gough
199d2aaa92
script to pack existing docs
2016-01-27 15:14:23 +00:00
Brian Gough
b44a7b9aa6
reject very large ops
2016-01-26 14:52:40 +00:00
Brian Gough
b7a4c72f9c
avoid compressing updates if the result would be too big
2016-01-26 12:23:21 +00:00
Brian Gough
ed0aaa189d
add test for non-overlapping insert-delete case
2016-01-26 12:13:43 +00:00
Brian Gough
b3ddd839e6
add logging of raw updates
2016-01-26 11:28:02 +00:00
Brian Gough
29c7c5e249
enable packs by default for new docs
2016-01-25 09:55:55 +00:00
Brian Gough
d10123d3c4
include n parameter when packing
2016-01-25 09:45:25 +00:00
Brian Gough
9b2cd11cd4
don't try to append to packs when using the old op code
2016-01-22 10:45:24 +00:00
Brian Gough
84ace7f4c7
use packs only for temporary ops
2016-01-20 14:22:13 +00:00
