Commit graph

67 commits

Author SHA1 Message Date
Brian Gough
d9763eb105 log corrupt updates 2017-05-23 13:55:10 +01:00
James Allen
fb6e028182 Migrate lock to be redis-cluster compatible 2017-05-15 10:34:24 +01:00
Brian Gough
2a68b81cc1 randomise order of flush 2017-04-20 14:32:23 +01:00
Brian Gough
43f51355b2 add limit parameter to flushAll 2017-04-20 11:01:46 +01:00
Brian Gough
976f19f763 remove spurious call to getAllDocIdsWithHistoryOps 2017-04-20 10:14:44 +01:00
Brian Gough
9ce6d77cca add check for dangling updates 2017-04-13 11:31:45 +01:00
Brian Gough
d9ed026d91 simple flush for all projects
does not work with redis cluster, only single redis
2017-04-12 16:34:28 +01:00
Brian Gough
8bb3dd07a4 continue when ops are out of order 2017-03-30 13:37:11 +01:00
Brian Gough
d29141d583 return error when out-of-order ops detected 2017-03-30 11:48:26 +01:00
Brian Gough
28d2ec93b8 Revert "Revert "Merge pull request #23 from sharelatex/bg-move-lock""
This reverts commit 85bc45099f047aa8dfa6d189f02f4b4327c9d602.
2017-03-28 12:52:49 +01:00
James Allen
694be95e66 Swap start_ts and end_ts comparison to correctly break chunks after 5 minutes 2016-10-11 11:03:52 +01:00
James Allen
40ed6fee46 Split update summary on big deletes 2016-10-04 15:13:04 +01:00
Brian Gough
31348141d8 increase logging for discarded updates and version mismatch 2016-03-24 11:55:29 +00:00
Brian Gough
98738d1344 fix for acceptance test 2016-03-10 15:15:29 +00:00
Brian Gough
28b184e0ca fix incorrect use of _.union (argument must be array) 2016-03-09 14:44:59 +00:00
Brian Gough
8922b97bd7 avoid duplicate filling of UserInfo in getDocUpdates 2016-03-09 14:44:59 +00:00
Brian Gough
3d9dfeccc3 remove pack worker
remove the op-specific code

remove tests for ops, now only packing

remove unused packing code

work in progress

store index for completed packs only

support archiving and unarchiving of individual packs

remove support for archiving whole document history

split out ArchiveManager, IndexManager

remove old DocArchive code

remove docHistoryStats collection

comment about archiving

added method to look at index when last pack has been archived

added start of iterator for project results

use a proper iterator

added heap module

getting it working

increase pack size since bulk operations no longer needed

remove unused MongoAWSexternal

cleanup

added doc iterator

remove old query code

added missing files

cleanup

clean upclean up

started adding pack worker for archiving

work in progress

work in progress

getting pack worker working

updating worker

getting packworker working

added lock

use correct key name for track changes aws access

use correct key name for track changes aws access

always send back users array

fix up comparison of retrieved objects

handle op ids inside packs

log when s3 download completes

comments

cleanup, remove finalisation ideacleanup, remove finalisation idea

remove logging
2016-03-01 10:10:02 +00:00
Brian Gough
b44a7b9aa6 reject very large ops 2016-01-26 14:52:40 +00:00
Brian Gough
b3ddd839e6 add logging of raw updates 2016-01-26 11:28:02 +00:00
Brian Gough
29c7c5e249 enable packs by default for new docs 2016-01-25 09:55:55 +00:00
Brian Gough
9b2cd11cd4 don't try to append to packs when using the old op code 2016-01-22 10:45:24 +00:00
Brian Gough
84ace7f4c7 use packs only for temporary ops 2016-01-20 14:22:13 +00:00
Brian Gough
5e830cbbdb put all new ops into packs 2016-01-19 15:56:09 +00:00
Brian Gough
5153ed8217 make peekLastUpdate alway return lastVersion when available 2016-01-15 15:54:44 +00:00
Brian Gough
6754bdca1c log timestamp in human-readable form for inconsistent ops 2016-01-05 11:30:24 +00:00
Brian Gough
b7de6f2f71 don't try to compress updates across point of broken history 2015-12-21 13:52:26 +00:00
Brian Gough
4a82dfe618 add setting trackchanges.continueOnError to allow recovery from missing ops 2015-12-17 16:28:02 +00:00
Brian Gough
be2136de7c fix update-in-place bug for array ops 2015-12-04 15:17:28 +00:00
Brian Gough
3842f0d1cc Merge pull request #9 from sharelatex/only-delete-applied-ops
Only delete applied ops
2015-11-27 12:45:51 +00:00
Brian Gough
8ebc069ddb modify last compressed op in place 2015-11-26 16:17:18 +00:00
Brian Gough
3432d9e91a added comments for redis delete 2015-11-26 15:16:54 +00:00
Brian Gough
e65549099c only delete the applied ops from redis 2015-11-25 16:01:07 +00:00
Brian Gough
b6dae59655 fix callback logic in compressAndSaveRawUpdates 2015-10-08 16:39:13 +01:00
Brian Gough
add6a68fe1 add missing callback in compressAndSaveRawUpdates 2015-10-08 10:53:25 +01:00
Brian Gough
2ab1778dd9 move default value of lastVersion into function body 2015-09-23 16:31:33 +01:00
Brian Gough
e49f260507 allow rollback/locking by setting inS3:false when starting the archive process 2015-09-23 13:28:07 +01:00
Brian Gough
d6b827426c support forcing new compressed update in popLastCompressedUpdate
callback with a null update, passing the version as an additional
argument
2015-09-23 13:22:38 +01:00
Brian Gough
a10dc4f898 Merge pull request #6 from heukirne/s3-archive
Add S3 archive track changes feature
2015-09-21 11:25:06 +01:00
Brian Gough
0e627c92d8 avoid clobbering global _ in loop 2015-09-18 16:26:05 +01:00
Brian Gough
d3dff28bea Merge remote-tracking branch 'origin/master' into heukirne-s3-archive 2015-09-15 15:19:43 +01:00
Shane Kilkelly
eab8b4b6c8 Null safe access of id property, needed as user can be null. 2015-09-11 14:07:06 +01:00
Shane Kilkelly
0ad374556d Add a comment for clarity. 2015-09-10 16:43:40 +01:00
Shane Kilkelly
8387383cb4 In _summarizeUpdates, allow null users through.
A null value represents a deleted or otherwise missing user record.
2015-09-10 14:32:47 +01:00
Henrique Dias
efff026a79 handle easier propagation 2015-08-25 16:52:28 -03:00
Henrique Dias
f910e63e90 fix null case 2015-08-24 12:22:17 -03:00
Henrique Dias
fcbe4aa925 fix inS3 propagation 2015-08-24 12:19:19 -03:00
Henrique Dias
04ec45529f restore updates from S3 when exists
fix: avoid rearchiving
2015-08-18 17:11:19 -03:00
Henrique Dias
6bc9c9010a handle auto unarchive track changes 2015-08-09 19:52:32 -03:00
Brian Gough
8e810bab14 report error for inconsistent history results 2015-02-05 16:37:06 +00:00
James Allen
eb7bcc6922 Remove old history deletion method 2014-05-29 15:37:16 +01:00