Brian Gough
9eb36f9692
improve log message for flushing
2017-04-20 11:55:32 +01:00
Brian Gough
43f51355b2
add limit parameter to flushAll
2017-04-20 11:01:46 +01:00
Brian Gough
976f19f763
remove spurious call to getAllDocIdsWithHistoryOps
2017-04-20 10:14:44 +01:00
Brian Gough
70ce3a5271
Merge pull request #29 from sharelatex/bg-simple-flush-all
...
simple flush for all projects
2017-04-19 16:32:09 +01:00
Brian Gough
b5b61b98d0
avoid blocking when fetching redis keys
...
use scan instead of keys method
2017-04-19 15:39:33 +01:00
Brian Gough
9ce6d77cca
add check for dangling updates
2017-04-13 11:31:45 +01:00
Brian Gough
1ec9f87542
update timeouts to allow for 5 second retry delay
...
now using requestretry module, which has a five second delay between
retries. Need to increase the timeout on the health checks to allow for
this, as the timeout is currently shorter than the retry delay.
2017-04-12 16:47:34 +01:00
Brian Gough
d9ed026d91
simple flush for all projects
...
does not work with redis cluster, only single redis
2017-04-12 16:34:28 +01:00
Shane Kilkelly
96f5802462
Merge branch 'sk-upgrade-metrics'
2017-04-04 16:16:37 +01:00
Shane Kilkelly
9a9f0f5c4d
Remove the Metrics module, use metrics-sharelatex
2017-04-03 15:17:29 +01:00
Brian Gough
5d6d33966b
Merge pull request #27 from sharelatex/bg-retry-web-api-request
...
use requestretry to work around web api timeouts
2017-04-03 14:05:07 +01:00
Brian Gough
7237467cb4
use requestretry to work around web api timeouts
2017-04-03 11:06:50 +01:00
Brian Gough
b0b05d1e1b
improve error handling of releaseLock
...
ported from docupdater
2017-03-31 15:17:13 +01:00
Shane Kilkelly
b10f3adf2b
mock out metrics in tests
2017-03-31 11:33:25 +01:00
Shane Kilkelly
9a84798293
Upgrade to latest metrics
2017-03-31 11:32:59 +01:00
Brian Gough
8bb3dd07a4
continue when ops are out of order
2017-03-30 13:37:11 +01:00
Brian Gough
d29141d583
return error when out-of-order ops detected
2017-03-30 11:48:26 +01:00
Brian Gough
70bb95fb36
update logger-sharelatex to v1.5.6
2017-03-30 09:41:42 +01:00
Brian Gough
28d2ec93b8
Revert "Revert "Merge pull request #23 from sharelatex/bg-move-lock""
...
This reverts commit 85bc45099f047aa8dfa6d189f02f4b4327c9d602.
2017-03-28 12:52:49 +01:00
Shane Kilkelly
5288814bd6
Merge branch 'master' of github.com:sharelatex/track-changes-sharelatex
2017-03-27 14:37:47 +01:00
Shane Kilkelly
715ccbe227
Add a .nvmrc file
2017-03-27 14:37:32 +01:00
Brian Gough
e745cce9f5
Revert "Revert "added v8-profiler to shrinkwrap""
...
This reverts commit 642641ff7b97034346d494101124bee616927ade.
2017-03-27 13:26:55 +01:00
Brian Gough
0130d3513d
Revert "Revert "Merge pull request #24 from sharelatex/bg-add-profile""
...
This reverts commit a58cd0d787b51634caf694f9d858e638d27178b2.
2017-03-27 13:25:59 +01:00
Brian Gough
8bd3dcca75
Revert "Merge pull request #24 from sharelatex/bg-add-profile"
...
This reverts commit 8f507dcc9fbcd51072047c82a443137aaac0f377, reversing
changes made to 70abd7a155b396e3b44ce21ff8d36be6ad03a060.
2017-03-24 09:17:38 +00:00
Brian Gough
a73c8ba7b8
Revert "Merge pull request #23 from sharelatex/bg-move-lock"
...
This reverts commit 72e6756415e390a29fa784529453f926e5144358, reversing
changes made to 8f507dcc9fbcd51072047c82a443137aaac0f377.
2017-03-24 09:17:11 +00:00
Brian Gough
6ada7842a9
Revert "added v8-profiler to shrinkwrap"
...
This reverts commit 62d481b31ebed2a1714cdfcb037a9c1a9c053756.
2017-03-24 09:12:28 +00:00
Brian Gough
f1a66df4ad
added v8-profiler to shrinkwrap
2017-03-23 17:01:01 +00:00
Brian Gough
7802bcc911
Revert "update shrinkwrap file for new npm package"
...
This reverts commit 68283e2650420c1cedc957b039b52766a8d4ce3c.
2017-03-23 16:57:48 +00:00
Brian Gough
a4cca1f948
update shrinkwrap file for new npm package
2017-03-23 14:21:39 +00:00
Brian Gough
e028de115e
Merge pull request #23 from sharelatex/bg-move-lock
...
move lock inside web http calls
2017-03-23 13:42:52 +00:00
Brian Gough
b897afeb1c
Merge pull request #24 from sharelatex/bg-add-profile
...
add /profile endpoint
2017-03-23 13:39:12 +00:00
Brian Gough
bc7815f7fc
remove old processUncompressedUpdates method
...
replace with new per doc method
2017-03-22 16:59:52 +00:00
Brian Gough
b1c0ebbaae
add withLock to processUncompressedUpdatesForDoc
2017-03-22 16:16:04 +00:00
Brian Gough
0dd668416d
increase request timeout now it is outside lock
2017-03-22 16:02:50 +00:00
Brian Gough
04b82a49ef
add /profile endpoint
2017-03-22 15:38:12 +00:00
Brian Gough
6795820933
move lock inside web http calls
2017-03-21 16:49:23 +00:00
Shane Kilkelly
d520c78ae3
Update metrics version to 1.7.0
2017-03-21 13:45:15 +00:00
James Allen
d60445adfc
Pass undo flag to doc updater when restoring from history
2017-03-21 11:32:53 +00:00
Shane Kilkelly
7fbc08d68a
Add logger stub to unit test
2017-03-21 09:27:47 +00:00
Shane Kilkelly
118511ae4a
Remove commented-out code
2017-03-20 09:39:51 +00:00
Shane Kilkelly
fa7ee739bb
Update to new metrics api
2017-03-17 14:58:35 +00:00
Shane Kilkelly
c30e672549
Merge branch 'master' into sk-upgrade-metrics
2017-03-16 15:17:57 +00:00
Shane Kilkelly
c303a1a386
wip: upgrade metrics
2017-03-16 15:17:38 +00:00
Brian Gough
ac5d59211d
revert random TTL in favour of delay in archiving
...
there could be some issues with newer packs expiring before older ones
2017-01-31 15:07:58 +00:00
Henry Oswald
ad5af5f4dd
ceil not round
2017-01-26 15:14:20 +00:00
Henry Oswald
1d356f93ec
added random up to 30 mins delay to the ttl of mongo objects
2017-01-26 13:07:54 +00:00
James Allen
dadc548f81
Ignore comment updates
2017-01-12 10:04:50 +01:00
James Allen
694be95e66
Swap start_ts and end_ts comparison to correctly break chunks after 5 minutes
2016-10-11 11:03:52 +01:00
James Allen
a197c0d148
Merge pull request #18 from sharelatex/ja-split-on-deletes
...
Split update summary on big deletes
2016-10-10 08:46:46 +01:00
James Allen
40ed6fee46
Split update summary on big deletes
2016-10-04 15:13:04 +01:00
Brian Gough
92e36023e5
Merge pull request #17 from sharelatex/ja-consistency-tweaks
...
Retry rewind if doc and update versions don't match
2016-09-30 15:14:27 +01:00
James Allen
f8865e616d
Update ensureIndices to reflect reality
2016-09-30 14:34:38 +01:00
James Allen
3c8aeb1262
Log number of retries
2016-09-30 13:38:47 +01:00
James Allen
a7f44bcd01
Add missing return on callback
2016-09-30 13:36:31 +01:00
James Allen
deced1aa15
Fix unit tests with logger.warn stub
2016-09-30 11:41:49 +01:00
James Allen
bddd1fda7d
Retry rewind if doc and update versions don't match
2016-09-30 11:36:47 +01:00
James Allen
cce93b95a5
Fetch updates before doc content when building a diff
2016-09-22 11:19:05 +01:00
James Allen
05a048db9a
Add in a little more logging
2016-09-22 11:19:05 +01:00
James Allen
8dda44f880
Speed up unit tests by only generating big fixtures once up front
2016-09-16 11:51:41 +01:00
James Allen
dfe26262ec
Return a No-op if diff returns nothing
2016-09-16 11:50:44 +01:00
James Allen
80375ae2dd
Run a diff against big delete - insert changes which are likely copy-pastes
2016-09-16 11:33:36 +01:00
Brian Gough
6e5eadfa86
include a timeout on WebApiManager requests
2016-04-13 16:42:36 +01:00
Brian Gough
a55b72871f
don't let s3 errors stop archive worker
2016-04-13 14:39:11 +01:00
Brian Gough
b343be844e
added metrics to pack worker for archiving
2016-04-08 10:29:04 +01:00
Brian Gough
76fe194815
add a metric for append-pack
2016-04-07 15:56:11 +01:00
Brian Gough
25365789b4
use latest sharelatex-metrics package
2016-04-07 15:53:10 +01:00
Brian Gough
78100e40c8
add missing metrics file
2016-04-07 15:33:20 +01:00
Brian Gough
1a1fa8798d
log attempted update when throwing error in DiffGenerator
2016-04-07 15:16:50 +01:00
Brian Gough
6db310bf6b
add insert/archive/unarchive metrics
2016-04-07 15:16:50 +01:00
Brian Gough
d0e08039da
don't modify expiry for temporary packs
2016-04-07 15:16:50 +01:00
Brian Gough
fd49601716
preserve existing history when user upgrades
2016-04-07 15:16:38 +01:00
Brian Gough
e292de5eb0
fix to avoid ever appending permanent changes to expiring packs
2016-04-06 17:00:16 +01:00
Brian Gough
8b7bdd345b
consider all packs for archiving
2016-04-06 15:17:31 +01:00
Brian Gough
ef47337c78
remove additional fields
2016-04-06 15:17:20 +01:00
Brian Gough
0b9a0730c0
mark temporary packs with a last_checked date in the far future
...
they do not need to be checked for archiving
2016-04-06 14:29:49 +01:00
Brian Gough
08fc151eee
avoid unnecessary call to insert packs into index
2016-04-06 14:29:21 +01:00
Brian Gough
719e0291aa
consider all packs for processing
...
to allow finalisation of old head packs
2016-04-06 14:27:44 +01:00
Brian Gough
79baa99634
clean up logging
2016-04-06 14:26:54 +01:00
Brian Gough
6ab75795a2
archive head packs after sufficient time
2016-04-06 13:30:09 +01:00
Brian Gough
7a0c2900ab
add error handler in tests
2016-04-06 13:29:19 +01:00
Brian Gough
6e18d49736
support archiving from list of project_ids/doc_ids
2016-04-04 17:00:19 +01:00
Brian Gough
6a2ecbac06
upgrade sentry logging
2016-04-04 16:30:17 +01:00
Brian Gough
537a1189e7
update mongodb-core version
2016-03-24 16:10:27 +00:00
Brian Gough
31348141d8
increase logging for discarded updates and version mismatch
2016-03-24 11:55:29 +00:00
Brian Gough
181cebecef
avoid call to fetch packs unnecessarily
2016-03-24 11:55:29 +00:00
Brian Gough
98683de3ae
temporarily disable ttl behaviour
...
allow existing packs without temporary flag to expire
2016-03-24 11:38:59 +00:00
Brian Gough
3f388fb0ac
only change ttl on cached packs, not temporary ones
...
temporary = without versioning feature enabled
cached = permanent versioned retrieved from s3
2016-03-24 11:38:09 +00:00
Brian Gough
8d900013d9
record whether a pack is temporary in the pack itself
...
using the expiresAt field no longer determines if the pack is
temporary because archived packs have an expiresAt field added when
they are retrieved from s3
2016-03-24 11:02:58 +00:00
Brian Gough
9543c21dd7
added npm-shrinkwrap.json to avoid bug in mongodb-core
2016-03-11 09:43:07 +00:00
Brian Gough
10932eb4a9
remove debug logs
2016-03-10 15:20:32 +00:00
Brian Gough
5ea7a31ad9
update archiving acceptance tests
2016-03-10 15:15:57 +00:00
Brian Gough
98738d1344
fix for acceptance test
2016-03-10 15:15:29 +00:00
Brian Gough
f01bf99682
acceptance tests - work in progress
2016-03-09 16:56:49 +00:00
Brian Gough
e2e8292590
update tests
2016-03-09 15:24:28 +00:00
Brian Gough
6d43273f04
working on packmanager tests
2016-03-09 14:44:59 +00:00
Brian Gough
f6367e21b8
give separate error for archive in progress vs completed
2016-03-09 14:44:59 +00:00
Brian Gough
7350ab531d
exclude already cached packs from archival
2016-03-09 14:44:59 +00:00
Brian Gough
28b184e0ca
fix incorrect use of _.union (argument must be array)
2016-03-09 14:44:59 +00:00
Brian Gough
8922b97bd7
avoid duplicate filling of UserInfo in getDocUpdates
2016-03-09 14:44:59 +00:00
Brian Gough
bfc9494cc1
update UpdatesManager tests
2016-03-09 14:44:59 +00:00
Brian Gough
4e63c50a75
update MongoManager tests
2016-03-09 14:44:59 +00:00
Brian Gough
b0d2e22c58
update config file for changes s3 settings
2016-03-09 13:28:22 +00:00
Brian Gough
7e6ea2793b
remove startup dependency on s3 settings
2016-03-09 13:28:02 +00:00
Brian Gough
b780df6801
removed unused tests
2016-03-04 15:47:59 +00:00
Brian Gough
4d58c14573
updated MongoAWS tests
2016-03-04 15:47:47 +00:00
Brian Gough
1419d20b1f
fix indentation
2016-03-04 15:43:32 +00:00
Brian Gough
3175f6d3a6
handle case where index does not exist
2016-03-03 14:36:16 +00:00
Henry Oswald
e8b3fb5be6
added more logging to failed health checks
2016-03-03 10:50:55 +00:00
Brian Gough
795f717bab
added index definitions
2016-03-01 11:38:23 +00:00
Brian Gough
ebd5628e53
temporary workaround to get tests passing
...
temporary workaround to get tests passing
2016-03-01 10:10:12 +00:00
Brian Gough
3d9dfeccc3
remove pack worker
...
remove the op-specific code
remove tests for ops, now only packing
remove unused packing code
work in progress
store index for completed packs only
support archiving and unarchiving of individual packs
remove support for archiving whole document history
split out ArchiveManager, IndexManager
remove old DocArchive code
remove docHistoryStats collection
comment about archiving
added method to look at index when last pack has been archived
added start of iterator for project results
use a proper iterator
added heap module
getting it working
increase pack size since bulk operations no longer needed
remove unused MongoAWSexternal
cleanup
added doc iterator
remove old query code
added missing files
cleanup
clean upclean up
started adding pack worker for archiving
work in progress
work in progress
getting pack worker working
updating worker
getting packworker working
added lock
use correct key name for track changes aws access
use correct key name for track changes aws access
always send back users array
fix up comparison of retrieved objects
handle op ids inside packs
log when s3 download completes
comments
cleanup, remove finalisation ideacleanup, remove finalisation idea
remove logging
2016-03-01 10:10:02 +00:00
Brian Gough
a23ddf31c0
allow packing of temporary ops
2016-01-29 12:36:03 +00:00
Brian Gough
612079ef8a
started adding more pack tests and packing of temp ops
2016-01-28 16:56:32 +00:00
Brian Gough
77cafa36af
support continuing from last packed doc
2016-01-28 16:40:20 +00:00
Brian - Work
666a07e5ba
move lock check into HealthChecker
...
to avoid dependency of HttpController on LockManager in unit tests
2016-01-27 16:04:55 +00:00
Brian - Work
64545d6540
improve packing script
2016-01-27 15:48:36 +00:00
Brian Gough
199d2aaa92
script to pack existing docs
2016-01-27 15:14:23 +00:00
Brian Gough
ba0de92081
improve logging by avoiding string escapes
2016-01-26 14:54:06 +00:00
Brian Gough
b44a7b9aa6
reject very large ops
2016-01-26 14:52:40 +00:00
Brian Gough
b7a4c72f9c
avoid compressing updates if the result would be too big
2016-01-26 12:23:21 +00:00
Brian Gough
ed0aaa189d
add test for non-overlapping insert-delete case
2016-01-26 12:13:43 +00:00
Brian Gough
b3ddd839e6
add logging of raw updates
2016-01-26 11:28:02 +00:00
Brian Gough
29c7c5e249
enable packs by default for new docs
2016-01-25 09:55:55 +00:00
Brian Gough
d10123d3c4
include n parameter when packing
2016-01-25 09:45:25 +00:00
Brian Gough
9b2cd11cd4
don't try to append to packs when using the old op code
2016-01-22 10:45:24 +00:00
Brian Gough
84ace7f4c7
use packs only for temporary ops
2016-01-20 14:22:13 +00:00
Brian Gough
ae61d1261e
added tests for pack updates
2016-01-19 15:58:09 +00:00
Brian Gough
61d22f027a
added test for inserting as pack
2016-01-19 15:58:09 +00:00
Brian Gough
399a8f0a29
update tests to assume packs are created
2016-01-19 15:58:09 +00:00
Brian Gough
78b3412ca8
decrease delay when packing
2016-01-19 15:58:09 +00:00
Brian Gough
679a81564e
respect mongo 3 limit of 1000 bulk operations
2016-01-19 15:58:09 +00:00
Brian Gough
f592611cac
always create a new pack, never keep as op
2016-01-19 15:58:09 +00:00
Brian Gough
c6be12f3d5
set v_end on pack creation
2016-01-19 15:58:09 +00:00
Brian - Work
f64969c784
added comment about query memory usage for toArray
2016-01-19 15:58:09 +00:00
Brian Gough
0532a4daaa
use compound index to replace separate index for packs
2016-01-19 15:56:09 +00:00
Brian Gough
0ba00a9eb7
expire temporary packs and roll over to a new pack each day
2016-01-19 15:56:09 +00:00
Brian Gough
5e830cbbdb
put all new ops into packs
2016-01-19 15:56:09 +00:00
Brian Gough
dc564fd5d0
archiving document history now sends all changes to s3
2016-01-15 15:54:46 +00:00
Brian Gough
5153ed8217
make peekLastUpdate alway return lastVersion when available
2016-01-15 15:54:44 +00:00
Brian Gough
933b6c50e7
add memory logger to track changes
2016-01-14 16:24:13 +00:00
Brian Gough
8e53d66079
log the key for lock timeouts
2016-01-12 10:47:15 +00:00
Brian Gough
6199532d08
increase logging on s3 operations
2016-01-12 10:36:00 +00:00
Brian Gough
ca1f1dc944
handle exception in parsing retrieved json from aws
2016-01-12 09:26:29 +00:00
Brian Gough
b8862ca5af
switch to node-byline module to avoid buffering problem with readline-stream
...
for lines > 64k the readline-stream module is affected by
https://github.com/jahewson/node-byline/issues/30 which is fixed in
node-byline (readline-stream was an earlier fork of the byline module)
2016-01-11 16:51:35 +00:00
Brian Gough
cb109a27a6
allow PackWorker to shut down cleanly
2016-01-06 09:43:10 +00:00
Brian Gough
ffe30962c9
add a close() method to LockManager to allow clean shutdown
2016-01-06 09:34:39 +00:00
Brian Gough
05163837cb
add sentry error reporting to PackWorker
2016-01-05 16:00:52 +00:00
Brian Gough
ee8931bc34
log the request to sentry when an error occurs
2016-01-05 15:41:06 +00:00
Brian Gough
bad923e073
fix acceptance tests for mongojs 1.x driver
2016-01-05 12:10:57 +00:00
Brian Gough
6754bdca1c
log timestamp in human-readable form for inconsistent ops
2016-01-05 11:30:24 +00:00
Brian Gough
e1aa436286
respect mongo bulk operations limit of 1000 operations
2016-01-05 11:13:13 +00:00
Brian Gough
bb7153c6c1
workaround for mongojs db.close issue
...
https://github.com/mafintosh/mongojs/issues/224
2015-12-22 15:36:15 +00:00
Brian Gough
d3583b4ef6
respect limit of 1000 ops in bulk operation with mongojs 1.x
2015-12-22 14:38:04 +00:00
Brian Gough
c7b4062412
remove unsupported options argument in count() method of mongojs 1.x
2015-12-22 14:20:34 +00:00
Brian Gough
d49997d9f3
fix usage of BSON module
2015-12-21 16:56:49 +00:00
Brian Gough
b7de6f2f71
don't try to compress updates across point of broken history
2015-12-21 13:52:26 +00:00
Brian Gough
4a6374efe8
fix read order when retrieving diffs
2015-12-18 12:38:42 +00:00
Brian Gough
911883a566
upgrade logger-sharelatex
2015-12-18 10:38:12 +00:00
Brian Gough
9f69c95192
Merge branch 'upgrade-mongojs'
2015-12-17 16:31:04 +00:00
Brian Gough
4a82dfe618
add setting trackchanges.continueOnError to allow recovery from missing ops
2015-12-17 16:28:02 +00:00
Brian Gough
b84a9e6e91
upgrade mongojs
2015-12-17 14:11:44 +00:00
Brian Gough
54d1036e37
skip ops marked as broken in database
2015-12-09 15:13:37 +00:00
Brian Gough
2a7c33d7ca
added /check endpoint for documents
2015-12-09 14:57:04 +00:00
Brian Gough
c51d249221
remove explicit sentry release number
...
use environment variable SENTRY_RELEASE instead
2015-12-08 10:44:59 +00:00
Brian Gough
6483233f48
added /oops endpoint for generating dummy exception
2015-12-07 16:17:17 +00:00
Brian Gough
e7af602d32
add sentry logging
2015-12-07 15:52:21 +00:00
Brian Gough
23c43b8042
skip any broken ops when viewing history diffs
2015-12-04 15:17:28 +00:00
Brian Gough
be2136de7c
fix update-in-place bug for array ops
2015-12-04 15:17:28 +00:00
Brian Gough
11be8c3733
increase timeout in lock manager test
...
on heavily loaded machines 20ms isn't long enough to consistently get
multiple calls to the lock manager
2015-11-27 14:25:25 +00:00
Brian Gough
3842f0d1cc
Merge pull request #9 from sharelatex/only-delete-applied-ops
...
Only delete applied ops
2015-11-27 12:45:51 +00:00
Brian Gough
8ebc069ddb
modify last compressed op in place
2015-11-26 16:17:18 +00:00
Brian Gough
3432d9e91a
added comments for redis delete
2015-11-26 15:16:54 +00:00
Brian Gough
e65549099c
only delete the applied ops from redis
2015-11-25 16:01:07 +00:00
Brian Gough
992857d6a2
added redis write check to healthcheck
2015-10-29 10:52:23 +00:00
Brian Gough
c44d5b1b3d
added healthcheck
2015-10-19 10:59:39 +01:00
Brian Gough
ad144371d0
gracefully handle updates marked as broken
...
set update.broken == true to allow the user to view history without a
crash
2015-10-16 11:24:50 +01:00
Brian Gough
8961e23954
enhance LockManager to avoid accidental unlocking
2015-10-14 14:42:17 +01:00
Brian Gough
b6dae59655
fix callback logic in compressAndSaveRawUpdates
2015-10-08 16:39:13 +01:00
Brian Gough
8226bf3be4
increase lock time to 5 minutes
2015-10-08 16:11:39 +01:00
Brian Gough
add6a68fe1
add missing callback in compressAndSaveRawUpdates
2015-10-08 10:53:25 +01:00
James Allen
1a4b8f4269
API/service layout deprecation warning
2015-10-07 13:44:40 +01:00
Henry Oswald
ef69729dfd
change archive and unarchive to post's
2015-10-01 15:55:07 +01:00
Henry Oswald
6052bf6a7e
change test to 19 not 20 as a stub in mongo is left behind
2015-10-01 15:54:46 +01:00
James Allen
2a03591030
Stub out noisy/slow logger-sharelatex and mongojs modules in tests
2015-09-25 13:46:20 +01:00
James Allen
23dfe68cb8
Don't error when rewinding and insert op which is beyond the length of the document.
...
ShareJS will accept an op where p > content.length when applied,
and it applies as though p == content.length. However, the op is
passed to us with the original p > content.length. Detect if that
is the case with this op, and shift p back appropriately to match
ShareJS if so.
2015-09-25 13:44:44 +01:00
Brian Gough
92e0b0f04c
add logging to each stage of archiving
2015-09-24 09:10:06 +01:00
Brian Gough
e683b0275a
bug fix for clear archive in progress flag
2015-09-24 09:09:49 +01:00
Brian Gough
692e8c657c
Revert to the default lock timeout now we have write barriers
...
Revert "increase lock timeouts for archiving"
This reverts commit 9eee1b383772adf058130d6e5eab409f57ce03cd.
2015-09-24 08:53:09 +01:00
Brian Gough
2ab1778dd9
move default value of lastVersion into function body
2015-09-23 16:31:33 +01:00
Brian Gough
bdf1d267f0
fix tests
2015-09-23 15:34:36 +01:00
Brian Gough
dc0044020f
only archive entries older than the current update
...
to avoid a stale version of the current update ever being pulled back
from S3
2015-09-23 14:33:40 +01:00
Brian Gough
696a866b67
pause the stream of ops, not the download
...
the download is buffered in the lineStream so a lot comes out even
after pausing the S3 download.
2015-09-23 13:38:57 +01:00
Brian Gough
847a553344
prevent double archiving by checking if any inS3 field is already present
2015-09-23 13:29:32 +01:00
Brian Gough
e49f260507
allow rollback/locking by setting inS3:false when starting the archive process
2015-09-23 13:28:07 +01:00
Brian Gough
551e8334cf
compressedUpdates are now never inserted with inS3
...
it is now always added later, and a new update is forced for any
addition to an archived update
2015-09-23 13:25:10 +01:00
Brian Gough
d6b827426c
support forcing new compressed update in popLastCompressedUpdate
...
callback with a null update, passing the version as an additional
argument
2015-09-23 13:22:38 +01:00
Brian Gough
dc35ef5cda
fix tests for archiving
2015-09-21 13:24:06 +01:00
Brian Gough
a10dc4f898
Merge pull request #6 from heukirne/s3-archive
...
Add S3 archive track changes feature
2015-09-21 11:25:06 +01:00
Brian Gough
0e627c92d8
avoid clobbering global _ in loop
2015-09-18 16:26:05 +01:00
Henrique Dias
aa66c5ee8c
improve size function
2015-09-17 10:41:53 -03:00
Henrique Dias
3f712c452a
add size bulk limit
2015-09-17 09:23:13 -03:00
Henrique Dias
01cbfd5338
fix minor issues in acceptance test
2015-09-16 19:33:23 -03:00
Henrique Dias
a4575f178d
fix unit test
2015-09-16 18:39:07 -03:00
Brian Gough
12c5098b48
fix lock in test
2015-09-16 16:50:36 +01:00
Brian Gough
7af5050370
add lock to unarchive doc
2015-09-16 16:18:36 +01:00
Brian Gough
18f06a3daf
increase lock timeouts for archiving
2015-09-16 16:09:38 +01:00
Brian Gough
b4ffa7d57e
share the document lock between archiving and packing
2015-09-16 16:03:55 +01:00
Brian Gough
9d39012b49
add error handler to each stage of download pipeline
2015-09-16 16:00:37 +01:00
Brian Gough
d9085a5e5e
add error handler to each stage of upload pipeline
2015-09-16 16:00:25 +01:00
Brian Gough
1c1b1d9595
log the case where there are no entries in the document history
2015-09-16 15:34:30 +01:00
Brian Gough
82d0f4fce8
make unarchive more responsive by downloading documents in parallel
...
unarchive is triggered interactively so we should try to make it
reasonably fast
2015-09-16 15:33:59 +01:00
Brian Gough
dfa0036507
pause stream while writing to mongo
2015-09-16 15:32:36 +01:00
Brian Gough
70200a9cf1
only log document ids, not document content
...
avoid filling the log with large documents
2015-09-16 15:31:43 +01:00
Brian Gough
d3dff28bea
Merge remote-tracking branch 'origin/master' into heukirne-s3-archive
2015-09-15 15:19:43 +01:00
Brian Gough
092f98d3ad
suppress error in normal shutdown case
2015-09-12 11:07:54 +01:00
Shane Kilkelly
39f528bcbc
Add a test to check that users are summarised properly even when a null user occurs earlier in the update list.
2015-09-11 14:12:01 +01:00
Shane Kilkelly
eab8b4b6c8
Null safe access of id property, needed as user can be null.
2015-09-11 14:07:06 +01:00
Shane Kilkelly
0ad374556d
Add a comment for clarity.
2015-09-10 16:43:40 +01:00
Shane Kilkelly
97326308fa
Update the Acceptance tests to include a case where a user doesn't exist.
2015-09-10 15:40:43 +01:00
Shane Kilkelly
8387383cb4
In _summarizeUpdates, allow null users through.
...
A null value represents a deleted or otherwise missing user record.
2015-09-10 14:32:47 +01:00
Shane Kilkelly
810bddb2cb
Log a message when the web api produces a 404 response.
2015-09-10 14:32:35 +01:00
Shane Kilkelly
575bdc62ec
Add a test for when the user can't be found.
2015-09-10 14:32:18 +01:00
Shane Kilkelly
522786d45e
Produce a null value, rather than crashing when the user info service returns 404.
2015-09-09 15:48:22 +01:00
Henry Oswald
18d817ee0a
added some missing error handling
2015-09-08 16:33:45 +01:00
Henry Oswald
17b0d99a65
rework the archiveDocChangesWithLock function
...
make it a bit more readable for me, struggle to trust indentation
based calls in coffeescript
2015-09-08 16:26:01 +01:00
Henry Oswald
0b3ebcff06
remove if statments checking if s3 is a backend
...
if its not enable then it can crash. In prod it should always be there
or not used at all
2015-09-08 16:23:15 +01:00
Henrique Dias
c5a8a249c6
add unarchive acceptance tests
2015-09-03 08:36:32 -03:00
Henrique Dias
da9e7dc7e1
init archive acceptance tests
2015-09-02 18:47:34 -03:00
Henrique Dias
7de103af68
fix unit scope error
2015-09-02 17:00:32 -03:00
Henrique Dias
d2b1243701
split MongoAWS files
2015-09-02 15:45:29 -03:00
Henrique Dias
0c16fbed88
add more unit tests
2015-09-02 15:39:19 -03:00
Henrique Dias
1abcea1a66
add some unit test
2015-08-31 18:13:18 -03:00
Henrique Dias
efff026a79
handle easier propagation
2015-08-25 16:52:28 -03:00
Henrique Dias
f910e63e90
fix null case
2015-08-24 12:22:17 -03:00
Henrique Dias
fcbe4aa925
fix inS3 propagation
2015-08-24 12:19:19 -03:00
Henrique Dias
1ccba422c8
remove unused function
2015-08-24 10:55:27 -03:00
Henrique Dias
98ce03b2f2
replace docs collection to DocstoreHandler
2015-08-24 10:38:31 -03:00
Henrique Dias
04ec45529f
restore updates from S3 when exists
...
fix: avoid rearchiving
2015-08-18 17:11:19 -03:00
Henrique Dias
20c3e15f93
fix bulk insert limit
2015-08-14 19:58:38 -03:00
Henrique Dias
26c8048729
change mongo stream method (still have a bug in bulk insert limit)
2015-08-14 19:19:54 -03:00
Henrique Dias
fd4afb3574
Archive changes, care about: versioin, expiresAt and Lock
2015-08-14 15:07:16 -03:00
Henrique Dias
6bc9c9010a
handle auto unarchive track changes
2015-08-09 19:52:32 -03:00
Henrique Dias
3bc5380468
handle inS3 flag
2015-08-09 17:50:15 -03:00
Henrique Dias
daa42bcea0
change s3Stream lib
2015-08-09 15:47:47 -03:00
Henrique Dias
bca48ac117
add unarchive doc track from s3
2015-08-06 17:09:36 -03:00
Henrique Dias
438c4f4d0c
using mongoexport for s3 archive
2015-08-06 15:46:44 -03:00
Henrique Dias
028fe2fa03
archive docChanges list to s3
2015-08-06 11:11:43 -03:00
Henrique Dias
ae047ecf76
init s3 feature
2015-08-06 10:00:09 -03:00
Brian Gough
775f5ebbe1
add configurable limit, delay and timeout to /pack via query string
2015-06-05 13:38:47 +01:00
Brian Gough
76d1350593
fix unit tests
...
packManager now uses LockManager, which tries to talk to redis and
needs to be replaced by a stub in the unit tests.
2015-06-05 10:59:46 +01:00
