Commit graph

205 commits

Author SHA1 Message Date
James Allen
fb6e028182 Migrate lock to be redis-cluster compatible 2017-05-15 10:34:24 +01:00
Brian Gough
0966f694da fix acceptance tests to work with redis cluster 2017-04-27 10:14:44 +01:00
Brian Gough
3045becbb6 use redis-sharelatex with redis cluster config 2017-04-24 14:23:32 +01:00
Shane Kilkelly
96f5802462 Merge branch 'sk-upgrade-metrics' 2017-04-04 16:16:37 +01:00
Shane Kilkelly
9a9f0f5c4d Remove the Metrics module, use metrics-sharelatex 2017-04-03 15:17:29 +01:00
Brian Gough
5d6d33966b Merge pull request #27 from sharelatex/bg-retry-web-api-request
use requestretry to work around web api timeouts
2017-04-03 14:05:07 +01:00
Brian Gough
7237467cb4 use requestretry to work around web api timeouts 2017-04-03 11:06:50 +01:00
Brian Gough
b0b05d1e1b improve error handling of releaseLock
ported from docupdater
2017-03-31 15:17:13 +01:00
Shane Kilkelly
b10f3adf2b mock out metrics in tests 2017-03-31 11:33:25 +01:00
Brian Gough
8bb3dd07a4 continue when ops are out of order 2017-03-30 13:37:11 +01:00
Brian Gough
d29141d583 return error when out-of-order ops detected 2017-03-30 11:48:26 +01:00
Brian Gough
28d2ec93b8 Revert "Revert "Merge pull request #23 from sharelatex/bg-move-lock""
This reverts commit 85bc45099f047aa8dfa6d189f02f4b4327c9d602.
2017-03-28 12:52:49 +01:00
Brian Gough
a73c8ba7b8 Revert "Merge pull request #23 from sharelatex/bg-move-lock"
This reverts commit 72e6756415e390a29fa784529453f926e5144358, reversing
changes made to 8f507dcc9fbcd51072047c82a443137aaac0f377.
2017-03-24 09:17:11 +00:00
Brian Gough
e028de115e Merge pull request #23 from sharelatex/bg-move-lock
move lock inside web http calls
2017-03-23 13:42:52 +00:00
Brian Gough
bc7815f7fc remove old processUncompressedUpdates method
replace with new per doc method
2017-03-22 16:59:52 +00:00
Brian Gough
b1c0ebbaae add withLock to processUncompressedUpdatesForDoc 2017-03-22 16:16:04 +00:00
Brian Gough
6795820933 move lock inside web http calls 2017-03-21 16:49:23 +00:00
James Allen
d60445adfc Pass undo flag to doc updater when restoring from history 2017-03-21 11:32:53 +00:00
Shane Kilkelly
7fbc08d68a Add logger stub to unit test 2017-03-21 09:27:47 +00:00
Shane Kilkelly
fa7ee739bb Update to new metrics api 2017-03-17 14:58:35 +00:00
Brian Gough
ac5d59211d revert random TTL in favour of delay in archiving
there could be some issues with newer packs expiring before older ones
2017-01-31 15:07:58 +00:00
Henry Oswald
1d356f93ec added random up to 30 mins delay to the ttl of mongo objects 2017-01-26 13:07:54 +00:00
James Allen
dadc548f81 Ignore comment updates 2017-01-12 10:04:50 +01:00
James Allen
a197c0d148 Merge pull request #18 from sharelatex/ja-split-on-deletes
Split update summary on big deletes
2016-10-10 08:46:46 +01:00
James Allen
40ed6fee46 Split update summary on big deletes 2016-10-04 15:13:04 +01:00
James Allen
a7f44bcd01 Add missing return on callback 2016-09-30 13:36:31 +01:00
James Allen
deced1aa15 Fix unit tests with logger.warn stub 2016-09-30 11:41:49 +01:00
James Allen
bddd1fda7d Retry rewind if doc and update versions don't match 2016-09-30 11:36:47 +01:00
James Allen
8dda44f880 Speed up unit tests by only generating big fixtures once up front 2016-09-16 11:51:41 +01:00
James Allen
dfe26262ec Return a No-op if diff returns nothing 2016-09-16 11:50:44 +01:00
James Allen
80375ae2dd Run a diff against big delete - insert changes which are likely copy-pastes 2016-09-16 11:33:36 +01:00
Brian Gough
6e5eadfa86 include a timeout on WebApiManager requests 2016-04-13 16:42:36 +01:00
Brian Gough
fd49601716 preserve existing history when user upgrades 2016-04-07 15:16:38 +01:00
Brian Gough
e292de5eb0 fix to avoid ever appending permanent changes to expiring packs 2016-04-06 17:00:16 +01:00
Brian Gough
7a0c2900ab add error handler in tests 2016-04-06 13:29:19 +01:00
Brian Gough
10932eb4a9 remove debug logs 2016-03-10 15:20:32 +00:00
Brian Gough
5ea7a31ad9 update archiving acceptance tests 2016-03-10 15:15:57 +00:00
Brian Gough
f01bf99682 acceptance tests - work in progress 2016-03-09 16:56:49 +00:00
Brian Gough
e2e8292590 update tests 2016-03-09 15:24:28 +00:00
Brian Gough
6d43273f04 working on packmanager tests 2016-03-09 14:44:59 +00:00
Brian Gough
bfc9494cc1 update UpdatesManager tests 2016-03-09 14:44:59 +00:00
Brian Gough
4e63c50a75 update MongoManager tests 2016-03-09 14:44:59 +00:00
Brian Gough
b780df6801 removed unused tests 2016-03-04 15:47:59 +00:00
Brian Gough
4d58c14573 updated MongoAWS tests 2016-03-04 15:47:47 +00:00
Brian Gough
ebd5628e53 temporary workaround to get tests passing
temporary workaround to get tests passing
2016-03-01 10:10:12 +00:00
Brian Gough
3d9dfeccc3 remove pack worker
remove the op-specific code

remove tests for ops, now only packing

remove unused packing code

work in progress

store index for completed packs only

support archiving and unarchiving of individual packs

remove support for archiving whole document history

split out ArchiveManager, IndexManager

remove old DocArchive code

remove docHistoryStats collection

comment about archiving

added method to look at index when last pack has been archived

added start of iterator for project results

use a proper iterator

added heap module

getting it working

increase pack size since bulk operations no longer needed

remove unused MongoAWSexternal

cleanup

added doc iterator

remove old query code

added missing files

cleanup

clean upclean up

started adding pack worker for archiving

work in progress

work in progress

getting pack worker working

updating worker

getting packworker working

added lock

use correct key name for track changes aws access

use correct key name for track changes aws access

always send back users array

fix up comparison of retrieved objects

handle op ids inside packs

log when s3 download completes

comments

cleanup, remove finalisation ideacleanup, remove finalisation idea

remove logging
2016-03-01 10:10:02 +00:00
Brian Gough
a23ddf31c0 allow packing of temporary ops 2016-01-29 12:36:03 +00:00
Brian Gough
612079ef8a started adding more pack tests and packing of temp ops 2016-01-28 16:56:32 +00:00
Brian - Work
666a07e5ba move lock check into HealthChecker
to avoid dependency of HttpController on LockManager in unit tests
2016-01-27 16:04:55 +00:00
Brian Gough
b7a4c72f9c avoid compressing updates if the result would be too big 2016-01-26 12:23:21 +00:00
Brian Gough
ed0aaa189d add test for non-overlapping insert-delete case 2016-01-26 12:13:43 +00:00
Brian Gough
ae61d1261e added tests for pack updates 2016-01-19 15:58:09 +00:00
Brian Gough
61d22f027a added test for inserting as pack 2016-01-19 15:58:09 +00:00
Brian Gough
399a8f0a29 update tests to assume packs are created 2016-01-19 15:58:09 +00:00
Brian Gough
5e830cbbdb put all new ops into packs 2016-01-19 15:56:09 +00:00
Brian Gough
dc564fd5d0 archiving document history now sends all changes to s3 2016-01-15 15:54:46 +00:00
Brian Gough
5153ed8217 make peekLastUpdate alway return lastVersion when available 2016-01-15 15:54:44 +00:00
Brian Gough
8e53d66079 log the key for lock timeouts 2016-01-12 10:47:15 +00:00
Brian Gough
bad923e073 fix acceptance tests for mongojs 1.x driver 2016-01-05 12:10:57 +00:00
Brian Gough
c7b4062412 remove unsupported options argument in count() method of mongojs 1.x 2015-12-22 14:20:34 +00:00
Brian Gough
23c43b8042 skip any broken ops when viewing history diffs 2015-12-04 15:17:28 +00:00
Brian Gough
be2136de7c fix update-in-place bug for array ops 2015-12-04 15:17:28 +00:00
Brian Gough
11be8c3733 increase timeout in lock manager test
on heavily loaded machines 20ms isn't long enough to consistently get
multiple calls to the lock manager
2015-11-27 14:25:25 +00:00
Brian Gough
3842f0d1cc Merge pull request #9 from sharelatex/only-delete-applied-ops
Only delete applied ops
2015-11-27 12:45:51 +00:00
Brian Gough
8ebc069ddb modify last compressed op in place 2015-11-26 16:17:18 +00:00
Brian Gough
e65549099c only delete the applied ops from redis 2015-11-25 16:01:07 +00:00
Brian Gough
8961e23954 enhance LockManager to avoid accidental unlocking 2015-10-14 14:42:17 +01:00
Henry Oswald
ef69729dfd change archive and unarchive to post's 2015-10-01 15:55:07 +01:00
Henry Oswald
6052bf6a7e change test to 19 not 20 as a stub in mongo is left behind 2015-10-01 15:54:46 +01:00
James Allen
2a03591030 Stub out noisy/slow logger-sharelatex and mongojs modules in tests 2015-09-25 13:46:20 +01:00
James Allen
23dfe68cb8 Don't error when rewinding and insert op which is beyond the length of the document.
ShareJS will accept an op where p > content.length when applied,
and it applies as though p == content.length. However, the op is
passed to us with the original p > content.length. Detect if that
is the case with this op, and shift p back appropriately to match
ShareJS if so.
2015-09-25 13:44:44 +01:00
Brian Gough
bdf1d267f0 fix tests 2015-09-23 15:34:36 +01:00
Brian Gough
d6b827426c support forcing new compressed update in popLastCompressedUpdate
callback with a null update, passing the version as an additional
argument
2015-09-23 13:22:38 +01:00
Brian Gough
dc35ef5cda fix tests for archiving 2015-09-21 13:24:06 +01:00
Henrique Dias
aa66c5ee8c improve size function 2015-09-17 10:41:53 -03:00
Henrique Dias
3f712c452a add size bulk limit 2015-09-17 09:23:13 -03:00
Henrique Dias
01cbfd5338 fix minor issues in acceptance test 2015-09-16 19:33:23 -03:00
Henrique Dias
a4575f178d fix unit test 2015-09-16 18:39:07 -03:00
Brian Gough
12c5098b48 fix lock in test 2015-09-16 16:50:36 +01:00
Brian Gough
d3dff28bea Merge remote-tracking branch 'origin/master' into heukirne-s3-archive 2015-09-15 15:19:43 +01:00
Shane Kilkelly
39f528bcbc Add a test to check that users are summarised properly even when a null user occurs earlier in the update list. 2015-09-11 14:12:01 +01:00
Shane Kilkelly
97326308fa Update the Acceptance tests to include a case where a user doesn't exist. 2015-09-10 15:40:43 +01:00
Shane Kilkelly
8387383cb4 In _summarizeUpdates, allow null users through.
A null value represents a deleted or otherwise missing user record.
2015-09-10 14:32:47 +01:00
Shane Kilkelly
575bdc62ec Add a test for when the user can't be found. 2015-09-10 14:32:18 +01:00
Henrique Dias
c5a8a249c6 add unarchive acceptance tests 2015-09-03 08:36:32 -03:00
Henrique Dias
da9e7dc7e1 init archive acceptance tests 2015-09-02 18:47:34 -03:00
Henrique Dias
7de103af68 fix unit scope error 2015-09-02 17:00:32 -03:00
Henrique Dias
0c16fbed88 add more unit tests 2015-09-02 15:39:19 -03:00
Henrique Dias
1abcea1a66 add some unit test 2015-08-31 18:13:18 -03:00
Brian Gough
76d1350593 fix unit tests
packManager now uses LockManager, which tries to talk to redis and
needs to be replaced by a stub in the unit tests.
2015-06-05 10:59:46 +01:00
Brian Gough
7d6811559c test pack for getDocUpdates with time 2015-02-16 16:43:27 +00:00
Brian Gough
e55f2a0ff7 test pack for getDocUpdates with version 2015-02-16 15:59:58 +00:00
Brian Gough
f5a9184b57 update the tests to work with packs 2015-02-16 15:38:40 +00:00
Henry Oswald
1452eb4572 fixed tests 2014-09-26 17:51:41 +01:00
James Allen
eb7bcc6922 Remove old history deletion method 2014-05-29 15:37:16 +01:00
James Allen
29ad81c134 Use expiresAt rather than tempCreatedAt 2014-05-16 16:41:14 +01:00
James Allen
8b0b79bc32 Use TTL index to auto-delete updates after a week when versioning is not enabled 2014-05-16 15:59:12 +01:00
James Allen
24343a38cd Add error handling of mismatched versions back in 2014-05-01 10:26:01 +01:00
James Allen
eebfa6499e Be more forgiving with version mismatches 2014-04-15 08:02:49 +01:00
James Allen
953081c39d Delete updates that are older than a week, unless versioning or preserveHistory is enabled 2014-03-28 16:01:34 +00:00