Brian Gough
b0d2e22c58
update config file for changes s3 settings
2016-03-09 13:28:22 +00:00
Brian Gough
7e6ea2793b
remove startup dependency on s3 settings
2016-03-09 13:28:02 +00:00
Brian Gough
b780df6801
removed unused tests
2016-03-04 15:47:59 +00:00
Brian Gough
4d58c14573
updated MongoAWS tests
2016-03-04 15:47:47 +00:00
Brian Gough
1419d20b1f
fix indentation
2016-03-04 15:43:32 +00:00
Brian Gough
3175f6d3a6
handle case where index does not exist
2016-03-03 14:36:16 +00:00
Henry Oswald
e8b3fb5be6
added more logging to failed health checks
2016-03-03 10:50:55 +00:00
Brian Gough
795f717bab
added index definitions
2016-03-01 11:38:23 +00:00
Brian Gough
ebd5628e53
temporary workaround to get tests passing
...
temporary workaround to get tests passing
2016-03-01 10:10:12 +00:00
Brian Gough
3d9dfeccc3
remove pack worker
...
remove the op-specific code
remove tests for ops, now only packing
remove unused packing code
work in progress
store index for completed packs only
support archiving and unarchiving of individual packs
remove support for archiving whole document history
split out ArchiveManager, IndexManager
remove old DocArchive code
remove docHistoryStats collection
comment about archiving
added method to look at index when last pack has been archived
added start of iterator for project results
use a proper iterator
added heap module
getting it working
increase pack size since bulk operations no longer needed
remove unused MongoAWSexternal
cleanup
added doc iterator
remove old query code
added missing files
cleanup
clean upclean up
started adding pack worker for archiving
work in progress
work in progress
getting pack worker working
updating worker
getting packworker working
added lock
use correct key name for track changes aws access
use correct key name for track changes aws access
always send back users array
fix up comparison of retrieved objects
handle op ids inside packs
log when s3 download completes
comments
cleanup, remove finalisation ideacleanup, remove finalisation idea
remove logging
2016-03-01 10:10:02 +00:00
Brian Gough
a23ddf31c0
allow packing of temporary ops
2016-01-29 12:36:03 +00:00
Brian Gough
612079ef8a
started adding more pack tests and packing of temp ops
2016-01-28 16:56:32 +00:00
Brian Gough
77cafa36af
support continuing from last packed doc
2016-01-28 16:40:20 +00:00
Brian - Work
666a07e5ba
move lock check into HealthChecker
...
to avoid dependency of HttpController on LockManager in unit tests
2016-01-27 16:04:55 +00:00
Brian - Work
64545d6540
improve packing script
2016-01-27 15:48:36 +00:00
Brian Gough
199d2aaa92
script to pack existing docs
2016-01-27 15:14:23 +00:00
Brian Gough
ba0de92081
improve logging by avoiding string escapes
2016-01-26 14:54:06 +00:00
Brian Gough
b44a7b9aa6
reject very large ops
2016-01-26 14:52:40 +00:00
Brian Gough
b7a4c72f9c
avoid compressing updates if the result would be too big
2016-01-26 12:23:21 +00:00
Brian Gough
ed0aaa189d
add test for non-overlapping insert-delete case
2016-01-26 12:13:43 +00:00
Brian Gough
b3ddd839e6
add logging of raw updates
2016-01-26 11:28:02 +00:00
Brian Gough
29c7c5e249
enable packs by default for new docs
2016-01-25 09:55:55 +00:00
Brian Gough
d10123d3c4
include n parameter when packing
2016-01-25 09:45:25 +00:00
Brian Gough
9b2cd11cd4
don't try to append to packs when using the old op code
2016-01-22 10:45:24 +00:00
Brian Gough
84ace7f4c7
use packs only for temporary ops
2016-01-20 14:22:13 +00:00
Brian Gough
ae61d1261e
added tests for pack updates
2016-01-19 15:58:09 +00:00
Brian Gough
61d22f027a
added test for inserting as pack
2016-01-19 15:58:09 +00:00
Brian Gough
399a8f0a29
update tests to assume packs are created
2016-01-19 15:58:09 +00:00
Brian Gough
78b3412ca8
decrease delay when packing
2016-01-19 15:58:09 +00:00
Brian Gough
679a81564e
respect mongo 3 limit of 1000 bulk operations
2016-01-19 15:58:09 +00:00
Brian Gough
f592611cac
always create a new pack, never keep as op
2016-01-19 15:58:09 +00:00
Brian Gough
c6be12f3d5
set v_end on pack creation
2016-01-19 15:58:09 +00:00
Brian - Work
f64969c784
added comment about query memory usage for toArray
2016-01-19 15:58:09 +00:00
Brian Gough
0532a4daaa
use compound index to replace separate index for packs
2016-01-19 15:56:09 +00:00
Brian Gough
0ba00a9eb7
expire temporary packs and roll over to a new pack each day
2016-01-19 15:56:09 +00:00
Brian Gough
5e830cbbdb
put all new ops into packs
2016-01-19 15:56:09 +00:00
Brian Gough
dc564fd5d0
archiving document history now sends all changes to s3
2016-01-15 15:54:46 +00:00
Brian Gough
5153ed8217
make peekLastUpdate alway return lastVersion when available
2016-01-15 15:54:44 +00:00
Brian Gough
933b6c50e7
add memory logger to track changes
2016-01-14 16:24:13 +00:00
Brian Gough
8e53d66079
log the key for lock timeouts
2016-01-12 10:47:15 +00:00
Brian Gough
6199532d08
increase logging on s3 operations
2016-01-12 10:36:00 +00:00
Brian Gough
ca1f1dc944
handle exception in parsing retrieved json from aws
2016-01-12 09:26:29 +00:00
Brian Gough
b8862ca5af
switch to node-byline module to avoid buffering problem with readline-stream
...
for lines > 64k the readline-stream module is affected by
https://github.com/jahewson/node-byline/issues/30 which is fixed in
node-byline (readline-stream was an earlier fork of the byline module)
2016-01-11 16:51:35 +00:00
Brian Gough
cb109a27a6
allow PackWorker to shut down cleanly
2016-01-06 09:43:10 +00:00
Brian Gough
ffe30962c9
add a close() method to LockManager to allow clean shutdown
2016-01-06 09:34:39 +00:00
Brian Gough
05163837cb
add sentry error reporting to PackWorker
2016-01-05 16:00:52 +00:00
Brian Gough
ee8931bc34
log the request to sentry when an error occurs
2016-01-05 15:41:06 +00:00
Brian Gough
bad923e073
fix acceptance tests for mongojs 1.x driver
2016-01-05 12:10:57 +00:00
Brian Gough
6754bdca1c
log timestamp in human-readable form for inconsistent ops
2016-01-05 11:30:24 +00:00
Brian Gough
e1aa436286
respect mongo bulk operations limit of 1000 operations
2016-01-05 11:13:13 +00:00
