Brian Gough
b3ddd839e6
add logging of raw updates
2016-01-26 11:28:02 +00:00
Brian Gough
29c7c5e249
enable packs by default for new docs
2016-01-25 09:55:55 +00:00
Brian Gough
d10123d3c4
include n parameter when packing
2016-01-25 09:45:25 +00:00
Brian Gough
9b2cd11cd4
don't try to append to packs when using the old op code
2016-01-22 10:45:24 +00:00
Brian Gough
84ace7f4c7
use packs only for temporary ops
2016-01-20 14:22:13 +00:00
Brian Gough
78b3412ca8
decrease delay when packing
2016-01-19 15:58:09 +00:00
Brian Gough
679a81564e
respect mongo 3 limit of 1000 bulk operations
2016-01-19 15:58:09 +00:00
Brian Gough
f592611cac
always create a new pack, never keep as op
2016-01-19 15:58:09 +00:00
Brian Gough
c6be12f3d5
set v_end on pack creation
2016-01-19 15:58:09 +00:00
Brian - Work
f64969c784
added comment about query memory usage for toArray
2016-01-19 15:58:09 +00:00
Brian Gough
0532a4daaa
use compound index to replace separate index for packs
2016-01-19 15:56:09 +00:00
Brian Gough
0ba00a9eb7
expire temporary packs and roll over to a new pack each day
2016-01-19 15:56:09 +00:00
Brian Gough
5e830cbbdb
put all new ops into packs
2016-01-19 15:56:09 +00:00
Brian Gough
dc564fd5d0
archiving document history now sends all changes to s3
2016-01-15 15:54:46 +00:00
Brian Gough
5153ed8217
make peekLastUpdate alway return lastVersion when available
2016-01-15 15:54:44 +00:00
Brian Gough
8e53d66079
log the key for lock timeouts
2016-01-12 10:47:15 +00:00
Brian Gough
6199532d08
increase logging on s3 operations
2016-01-12 10:36:00 +00:00
Brian Gough
ca1f1dc944
handle exception in parsing retrieved json from aws
2016-01-12 09:26:29 +00:00
Brian Gough
b8862ca5af
switch to node-byline module to avoid buffering problem with readline-stream
...
for lines > 64k the readline-stream module is affected by
https://github.com/jahewson/node-byline/issues/30 which is fixed in
node-byline (readline-stream was an earlier fork of the byline module)
2016-01-11 16:51:35 +00:00
Brian Gough
cb109a27a6
allow PackWorker to shut down cleanly
2016-01-06 09:43:10 +00:00
Brian Gough
ffe30962c9
add a close() method to LockManager to allow clean shutdown
2016-01-06 09:34:39 +00:00
Brian Gough
05163837cb
add sentry error reporting to PackWorker
2016-01-05 16:00:52 +00:00
Brian Gough
6754bdca1c
log timestamp in human-readable form for inconsistent ops
2016-01-05 11:30:24 +00:00
Brian Gough
e1aa436286
respect mongo bulk operations limit of 1000 operations
2016-01-05 11:13:13 +00:00
Brian Gough
bb7153c6c1
workaround for mongojs db.close issue
...
https://github.com/mafintosh/mongojs/issues/224
2015-12-22 15:36:15 +00:00
Brian Gough
d3583b4ef6
respect limit of 1000 ops in bulk operation with mongojs 1.x
2015-12-22 14:38:04 +00:00
Brian Gough
c7b4062412
remove unsupported options argument in count() method of mongojs 1.x
2015-12-22 14:20:34 +00:00
Brian Gough
d49997d9f3
fix usage of BSON module
2015-12-21 16:56:49 +00:00
Brian Gough
b7de6f2f71
don't try to compress updates across point of broken history
2015-12-21 13:52:26 +00:00
Brian Gough
4a6374efe8
fix read order when retrieving diffs
2015-12-18 12:38:42 +00:00
Brian Gough
9f69c95192
Merge branch 'upgrade-mongojs'
2015-12-17 16:31:04 +00:00
Brian Gough
4a82dfe618
add setting trackchanges.continueOnError to allow recovery from missing ops
2015-12-17 16:28:02 +00:00
Brian Gough
b84a9e6e91
upgrade mongojs
2015-12-17 14:11:44 +00:00
Brian Gough
54d1036e37
skip ops marked as broken in database
2015-12-09 15:13:37 +00:00
Brian Gough
2a7c33d7ca
added /check endpoint for documents
2015-12-09 14:57:04 +00:00
Brian Gough
23c43b8042
skip any broken ops when viewing history diffs
2015-12-04 15:17:28 +00:00
Brian Gough
be2136de7c
fix update-in-place bug for array ops
2015-12-04 15:17:28 +00:00
Brian Gough
3842f0d1cc
Merge pull request #9 from sharelatex/only-delete-applied-ops
...
Only delete applied ops
2015-11-27 12:45:51 +00:00
Brian Gough
8ebc069ddb
modify last compressed op in place
2015-11-26 16:17:18 +00:00
Brian Gough
3432d9e91a
added comments for redis delete
2015-11-26 15:16:54 +00:00
Brian Gough
e65549099c
only delete the applied ops from redis
2015-11-25 16:01:07 +00:00
Brian Gough
992857d6a2
added redis write check to healthcheck
2015-10-29 10:52:23 +00:00
Brian Gough
c44d5b1b3d
added healthcheck
2015-10-19 10:59:39 +01:00
Brian Gough
ad144371d0
gracefully handle updates marked as broken
...
set update.broken == true to allow the user to view history without a
crash
2015-10-16 11:24:50 +01:00
Brian Gough
8961e23954
enhance LockManager to avoid accidental unlocking
2015-10-14 14:42:17 +01:00
Brian Gough
b6dae59655
fix callback logic in compressAndSaveRawUpdates
2015-10-08 16:39:13 +01:00
Brian Gough
8226bf3be4
increase lock time to 5 minutes
2015-10-08 16:11:39 +01:00
Brian Gough
add6a68fe1
add missing callback in compressAndSaveRawUpdates
2015-10-08 10:53:25 +01:00
James Allen
1a4b8f4269
API/service layout deprecation warning
2015-10-07 13:44:40 +01:00
James Allen
2a03591030
Stub out noisy/slow logger-sharelatex and mongojs modules in tests
2015-09-25 13:46:20 +01:00
James Allen
23dfe68cb8
Don't error when rewinding and insert op which is beyond the length of the document.
...
ShareJS will accept an op where p > content.length when applied,
and it applies as though p == content.length. However, the op is
passed to us with the original p > content.length. Detect if that
is the case with this op, and shift p back appropriately to match
ShareJS if so.
2015-09-25 13:44:44 +01:00
Brian Gough
92e0b0f04c
add logging to each stage of archiving
2015-09-24 09:10:06 +01:00
Brian Gough
e683b0275a
bug fix for clear archive in progress flag
2015-09-24 09:09:49 +01:00
Brian Gough
692e8c657c
Revert to the default lock timeout now we have write barriers
...
Revert "increase lock timeouts for archiving"
This reverts commit 9eee1b383772adf058130d6e5eab409f57ce03cd.
2015-09-24 08:53:09 +01:00
Brian Gough
2ab1778dd9
move default value of lastVersion into function body
2015-09-23 16:31:33 +01:00
Brian Gough
dc0044020f
only archive entries older than the current update
...
to avoid a stale version of the current update ever being pulled back
from S3
2015-09-23 14:33:40 +01:00
Brian Gough
696a866b67
pause the stream of ops, not the download
...
the download is buffered in the lineStream so a lot comes out even
after pausing the S3 download.
2015-09-23 13:38:57 +01:00
Brian Gough
847a553344
prevent double archiving by checking if any inS3 field is already present
2015-09-23 13:29:32 +01:00
Brian Gough
e49f260507
allow rollback/locking by setting inS3:false when starting the archive process
2015-09-23 13:28:07 +01:00
Brian Gough
551e8334cf
compressedUpdates are now never inserted with inS3
...
it is now always added later, and a new update is forced for any
addition to an archived update
2015-09-23 13:25:10 +01:00
Brian Gough
d6b827426c
support forcing new compressed update in popLastCompressedUpdate
...
callback with a null update, passing the version as an additional
argument
2015-09-23 13:22:38 +01:00
Brian Gough
a10dc4f898
Merge pull request #6 from heukirne/s3-archive
...
Add S3 archive track changes feature
2015-09-21 11:25:06 +01:00
Brian Gough
0e627c92d8
avoid clobbering global _ in loop
2015-09-18 16:26:05 +01:00
Henrique Dias
aa66c5ee8c
improve size function
2015-09-17 10:41:53 -03:00
Henrique Dias
3f712c452a
add size bulk limit
2015-09-17 09:23:13 -03:00
Brian Gough
7af5050370
add lock to unarchive doc
2015-09-16 16:18:36 +01:00
Brian Gough
18f06a3daf
increase lock timeouts for archiving
2015-09-16 16:09:38 +01:00
Brian Gough
b4ffa7d57e
share the document lock between archiving and packing
2015-09-16 16:03:55 +01:00
Brian Gough
9d39012b49
add error handler to each stage of download pipeline
2015-09-16 16:00:37 +01:00
Brian Gough
d9085a5e5e
add error handler to each stage of upload pipeline
2015-09-16 16:00:25 +01:00
Brian Gough
1c1b1d9595
log the case where there are no entries in the document history
2015-09-16 15:34:30 +01:00
Brian Gough
82d0f4fce8
make unarchive more responsive by downloading documents in parallel
...
unarchive is triggered interactively so we should try to make it
reasonably fast
2015-09-16 15:33:59 +01:00
Brian Gough
dfa0036507
pause stream while writing to mongo
2015-09-16 15:32:36 +01:00
Brian Gough
70200a9cf1
only log document ids, not document content
...
avoid filling the log with large documents
2015-09-16 15:31:43 +01:00
Brian Gough
d3dff28bea
Merge remote-tracking branch 'origin/master' into heukirne-s3-archive
2015-09-15 15:19:43 +01:00
Brian Gough
092f98d3ad
suppress error in normal shutdown case
2015-09-12 11:07:54 +01:00
Shane Kilkelly
eab8b4b6c8
Null safe access of id
property, needed as user can be null.
2015-09-11 14:07:06 +01:00
Shane Kilkelly
0ad374556d
Add a comment for clarity.
2015-09-10 16:43:40 +01:00
Shane Kilkelly
8387383cb4
In _summarizeUpdates, allow null users through.
...
A null value represents a deleted or otherwise missing user record.
2015-09-10 14:32:47 +01:00
Shane Kilkelly
810bddb2cb
Log a message when the web api produces a 404 response.
2015-09-10 14:32:35 +01:00
Shane Kilkelly
522786d45e
Produce a null value, rather than crashing when the user info service returns 404.
2015-09-09 15:48:22 +01:00
Henry Oswald
18d817ee0a
added some missing error handling
2015-09-08 16:33:45 +01:00
Henry Oswald
17b0d99a65
rework the archiveDocChangesWithLock function
...
make it a bit more readable for me, struggle to trust indentation
based calls in coffeescript
2015-09-08 16:26:01 +01:00
Henry Oswald
0b3ebcff06
remove if statments checking if s3 is a backend
...
if its not enable then it can crash. In prod it should always be there
or not used at all
2015-09-08 16:23:15 +01:00
Henrique Dias
c5a8a249c6
add unarchive acceptance tests
2015-09-03 08:36:32 -03:00
Henrique Dias
da9e7dc7e1
init archive acceptance tests
2015-09-02 18:47:34 -03:00
Henrique Dias
d2b1243701
split MongoAWS files
2015-09-02 15:45:29 -03:00
Henrique Dias
1abcea1a66
add some unit test
2015-08-31 18:13:18 -03:00
Henrique Dias
efff026a79
handle easier propagation
2015-08-25 16:52:28 -03:00
Henrique Dias
f910e63e90
fix null case
2015-08-24 12:22:17 -03:00
Henrique Dias
fcbe4aa925
fix inS3 propagation
2015-08-24 12:19:19 -03:00
Henrique Dias
1ccba422c8
remove unused function
2015-08-24 10:55:27 -03:00
Henrique Dias
98ce03b2f2
replace docs collection to DocstoreHandler
2015-08-24 10:38:31 -03:00
Henrique Dias
04ec45529f
restore updates from S3 when exists
...
fix: avoid rearchiving
2015-08-18 17:11:19 -03:00
Henrique Dias
20c3e15f93
fix bulk insert limit
2015-08-14 19:58:38 -03:00
Henrique Dias
26c8048729
change mongo stream method (still have a bug in bulk insert limit)
2015-08-14 19:19:54 -03:00
Henrique Dias
fd4afb3574
Archive changes, care about: versioin, expiresAt and Lock
2015-08-14 15:07:16 -03:00
Henrique Dias
6bc9c9010a
handle auto unarchive track changes
2015-08-09 19:52:32 -03:00
Henrique Dias
3bc5380468
handle inS3 flag
2015-08-09 17:50:15 -03:00
Henrique Dias
daa42bcea0
change s3Stream lib
2015-08-09 15:47:47 -03:00
Henrique Dias
bca48ac117
add unarchive doc track from s3
2015-08-06 17:09:36 -03:00
Henrique Dias
438c4f4d0c
using mongoexport for s3 archive
2015-08-06 15:46:44 -03:00
Henrique Dias
028fe2fa03
archive docChanges list to s3
2015-08-06 11:11:43 -03:00
Henrique Dias
ae047ecf76
init s3 feature
2015-08-06 10:00:09 -03:00
Brian Gough
775f5ebbe1
add configurable limit, delay and timeout to /pack via query string
2015-06-05 13:38:47 +01:00
Brian Gough
23d2518ebb
added a /pack endpoint to fork a pack worker
2015-06-04 16:36:56 +01:00
Brian Gough
289616d3db
added a /doc/:doc_id/pack endpoint
2015-06-04 16:23:49 +01:00
Brian Gough
3f2e4b0c11
move pack script functionality into PackManager
2015-06-04 16:21:40 +01:00
Brian Gough
27a3511b37
update docHistoryStats after packing ops
2015-06-03 10:36:07 +01:00
Brian Gough
19d812734e
make PackManager parameters configurable
2015-06-03 10:35:36 +01:00
Brian Gough
66bca8d05c
include the current date in the updates to docHistoryStats
2015-05-26 11:00:55 +01:00
Brian Gough
1811ac2145
added support for cleaning old expired ops in packs
2015-05-22 15:40:02 +01:00
Brian Gough
5c4afd5303
add docHistoryStats collection to keep track of updates to docs
2015-05-22 15:40:01 +01:00
Brian Gough
78f0bdbae3
fix name of temporary parameter to match other methods
2015-05-22 15:40:01 +01:00
Brian Gough
adc2866a7d
add check to exclude temporary ops from packs
2015-05-22 15:40:01 +01:00
Brian Gough
ffeb1cccb6
move pack migration code into PackManager
2015-05-22 15:40:01 +01:00
Brian Gough
4c0eea9916
return an error if trying to pop the last update from a pack
2015-02-17 14:53:50 +00:00
Brian Gough
24701219ff
clean up comments and remove console.logs
2015-02-17 13:47:55 +00:00
Brian Gough
76cbf32e6a
cleanup
...
remove unnecessary comments, whitespace and unused modules
2015-02-17 13:41:31 +00:00
Brian Gough
448a7b7f36
rename MongoPackManager to PackManager
2015-02-17 11:14:13 +00:00
Brian Gough
92e67511d5
sort by doc_id to ensure consistent order for all changes with the same timestamp
2015-02-16 16:44:06 +00:00
Brian Gough
fa1e890282
move update deduplication to filterAndLimit method
2015-02-16 15:37:47 +00:00
Brian Gough
0516e9429d
ensure document results are in version order
2015-02-16 13:43:27 +00:00
Brian Gough
8615b7a086
fix project results order, sort by time
2015-02-16 13:43:11 +00:00
Brian Gough
4f36ccd519
moved pack code into MongoPackManager
2015-02-13 16:18:15 +00:00
Brian Gough
682e8e8ab6
add a sparse mongo index for finding packs
2015-02-10 16:54:58 +00:00
Brian Gough
1d7f0919a4
fix mongo logic for project search with packs
2015-02-10 16:54:45 +00:00
Brian Gough
7fdce8fc48
split pack handling code into separate parts for docs and projects
...
need to debug projects
2015-02-09 16:53:44 +00:00
Brian Gough
d8dc56b031
don't try to getLastCompressedUpdates from packs
2015-02-09 16:53:05 +00:00
Brian Gough
f50f091fc6
added comments
2015-02-06 16:59:09 +00:00
Brian Gough
e47476369a
clean up docHistory pack decoding
2015-02-06 15:04:46 +00:00
Brian Gough
8e810bab14
report error for inconsistent history results
2015-02-05 16:37:06 +00:00
Brian Gough
715b0df4a1
prototype for expanding packs
2015-02-05 16:36:41 +00:00
Henry Oswald
a0bb289fd3
works with sentinal v1
2014-09-26 17:21:33 +01:00
James Allen
eb7bcc6922
Remove old history deletion method
2014-05-29 15:37:16 +01:00
James Allen
cc962e1c44
Add indexes in background
2014-05-16 17:00:30 +01:00
James Allen
6a371c267f
Fix indexes
2014-05-16 16:41:40 +01:00
James Allen
29ad81c134
Use expiresAt rather than tempCreatedAt
2014-05-16 16:41:14 +01:00
James Allen
8b0b79bc32
Use TTL index to auto-delete updates after a week when versioning is not enabled
2014-05-16 15:59:12 +01:00
James Allen
24343a38cd
Add error handling of mismatched versions back in
2014-05-01 10:26:01 +01:00
James Allen
eebfa6499e
Be more forgiving with version mismatches
2014-04-15 08:02:49 +01:00
James Allen
953081c39d
Delete updates that are older than a week, unless versioning or preserveHistory is enabled
2014-03-28 16:01:34 +00:00
James Allen
9f1efe6dca
Add in getProjectDetails to WebApiManager
2014-03-28 13:05:16 +00:00
James Allen
22e3aac5b0
Don't trip up over empty ops
2014-03-25 11:40:48 +00:00
James Allen
bd4bb3d3cf
Add in project flushing end point
2014-03-21 15:57:17 +00:00
James Allen
8cae726838
Backport project_id onto doc updates that don't have one yet
2014-03-21 14:40:51 +00:00
James Allen
2cd85fefec
Flush all docs in a project before getting updates
2014-03-21 13:48:14 +00:00
James Allen
cd9cb51027
Delete doc_id from DocsWithHistoryOps set when processing
2014-03-21 13:17:58 +00:00
James Allen
6aaa7ba8d5
Group toV and fromV by the doc they correspond to
2014-03-20 13:37:23 +00:00
James Allen
5595910122
Return a summary of the entire project changes
2014-03-20 12:10:04 +00:00
James Allen
b7de4c9c6c
Implement project update getter methods
2014-03-19 17:44:16 +00:00
James Allen
962fc18329
Store project_id alongside each change
2014-03-19 16:40:55 +00:00
James Allen
d32b0ee12f
Fetch updates up to the limit, even after summarising
2014-03-18 18:09:25 +00:00
James Allen
81811d7cc5
Move update summarizing logic into UpdatesManager
2014-03-18 11:41:48 +00:00
James Allen
22a806a200
Fix issues with consuming delete updates beyond the end of the diff
2014-03-17 14:54:02 +00:00
James Allen
540aaf0672
Add in some null checks
2014-03-13 14:30:01 +00:00
James Allen
9977a418c3
Compare diffs with user.id, not user_id
2014-03-12 14:04:33 +00:00
James Allen
575afae048
Compress adjacent inserts and deletes into one diff item
2014-03-12 13:27:18 +00:00
James Allen
7fa88f6b41
Summarise updates when returning them
2014-03-11 17:40:07 +00:00
James Allen
4ecb17b16e
Put multiple ops in one update
2014-03-11 15:24:38 +00:00
James Allen
064bdc3eea
Take user id in from request header and pass to doc updater
2014-03-11 13:01:07 +00:00
James Allen
3d1d962501
Don't try to fetch user when id doesn't exist
2014-03-11 11:45:25 +00:00
James Allen
83b2aa3082
add in restore end point
2014-03-10 16:58:39 +00:00
James Allen
7eb8699b93
Refactor diff manager
2014-03-10 16:03:03 +00:00
James Allen
866084ee6b
Pop last version, not last timestamp
2014-03-07 14:02:16 +00:00
James Allen
b45db3aa2b
Return user details like email and name in updates
2014-03-06 18:04:00 +00:00
James Allen
5fc139db00
Return version numbers of updates
2014-03-06 11:00:49 +00:00
James Allen
aadce232a1
Use version numbers for sorting and querying, not dates
2014-03-06 10:45:51 +00:00
James Allen
2e1307bd82
Doc is always one version ahead of latest op version
2014-03-05 16:31:38 +00:00
James Allen
3660253fd4
Add in /updates end point to get updates
2014-03-05 15:59:40 +00:00
James Allen
a46963a349
Refactor arguments to MongoManager.getUpdatesBetweenDates
2014-03-05 15:06:46 +00:00
James Allen
188d620ce1
Log out and put the last update back when an error occurs
2014-03-05 13:22:38 +00:00
James Allen
e4d8cc7a11
Add in acceptance tests for getting a diff
2014-03-04 15:27:03 +00:00
James Allen
8b71d222d4
Create DiffManager.getDiff
2014-03-04 14:05:17 +00:00
James Allen
1d1dcdfa2f
Add in methods for retrieving updates and doc version
2014-03-04 13:02:48 +00:00
James Allen
20d70859aa
Create buildDiff function
2014-03-03 17:39:59 +00:00
James Allen
e0402692cf
Add in diff generating functions
2014-03-01 13:31:34 +00:00
James Allen
76cdd5cf98
Delete only the updates that were processed, not a full batch
2014-03-01 11:42:31 +00:00
James Allen
d27872c9bd
Get acceptance tests running
2014-02-26 12:11:45 +00:00
James Allen
45fe6978af
Add in /doc/<doc-id>/flush endpoint
2014-02-26 11:34:56 +00:00
James Allen
65360a9a2b
Continue to process raw updates until redis is empty
2014-02-26 11:18:26 +00:00
James Allen
8405a37c2c
Add a lock around processing updates
2014-02-26 10:55:20 +00:00
James Allen
f33a3bde3e
Create processUncompressedUpdates method
2014-02-25 16:48:42 +00:00
James Allen
34d3847fe4
Add Redis Manager for fetching and deleting uncompressed ops
2014-02-25 16:27:14 +00:00
James Allen
8ae9bcd60f
Ensure that updates are compressed in continuous incrementing order
2014-02-25 12:28:15 +00:00
James Allen
de783bf5b0
Pull out MongoManager module and add tests
2014-02-24 17:43:27 +00:00
James Allen
d040e7c410
Add acceptance tests for multiple appends
2014-01-27 18:20:38 +00:00
James Allen
d4295c2023
Create HttpController for HTTP requestS
2014-01-27 18:09:37 +00:00
James Allen
5dea123b13
Get append end point working with a simple acceptance test
2014-01-27 17:51:09 +00:00
James Allen
533b8e59a3
Big refactor to use better names and separation of concerns
2014-01-27 16:26:58 +00:00
James Allen
8a0aa55c91
Refactor to take doc ops from an HTTP request from doc updater
2013-09-03 17:17:27 +01:00
James Allen
dab2781e87
Do two passes better
2013-08-23 14:58:00 +01:00
James Allen
dbee4a57fb
Compress doc ops in two passes
2013-08-23 14:35:13 +01:00
James Allen
7a3b78da43
Return when calling callback
2013-08-09 13:51:02 +01:00
James Allen
07f078b527
Don't insert a null update if there are no updates or raw updates
2013-08-09 13:46:03 +01:00
James Allen
fe503010aa
Pop and set history in batches for speed
2013-08-09 13:27:35 +01:00
James Allen
3ab45c2a35
Handle if two updates cancel each other
2013-08-09 12:36:17 +01:00
James Allen
54c7959647
Match if both user ids are null/undefined
2013-08-09 12:26:23 +01:00
James Allen
64eb794c02
Compress all docs using compressHistory script
2013-08-09 11:36:19 +01:00
James Allen
c5e7f14ba1
Factor methods out into ConversionManager
2013-08-09 11:10:19 +01:00