github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-12-28 13:42:41 +00:00
Code Issues Projects Releases Packages Wiki Activity
555 commits 3 branches 7 tags 206 MiB
Commit graph

503 commits

Author SHA1 Message Date
Brian Gough
92e0b0f04c add logging to each stage of archiving 2015-09-24 09:10:06 +01:00
Brian Gough
e683b0275a bug fix for clear archive in progress flag 2015-09-24 09:09:49 +01:00
Brian Gough
692e8c657c Revert to the default lock timeout now we have write barriers 
Revert "increase lock timeouts for archiving"

This reverts commit 9eee1b383772adf058130d6e5eab409f57ce03cd.
 2015-09-24 08:53:09 +01:00
Brian Gough
2ab1778dd9 move default value of lastVersion into function body 2015-09-23 16:31:33 +01:00
Brian Gough
bdf1d267f0 fix tests 2015-09-23 15:34:36 +01:00
Brian Gough
dc0044020f only archive entries older than the current update 
to avoid a stale version of the current update ever being pulled back
from S3
 2015-09-23 14:33:40 +01:00
Brian Gough
696a866b67 pause the stream of ops, not the download 
the download is buffered in the lineStream so a lot comes out even
after pausing the S3 download.
 2015-09-23 13:38:57 +01:00
Brian Gough
847a553344 prevent double archiving by checking if any inS3 field is already present 2015-09-23 13:29:32 +01:00
Brian Gough
e49f260507 allow rollback/locking by setting inS3:false when starting the archive process 2015-09-23 13:28:07 +01:00
Brian Gough
551e8334cf compressedUpdates are now never inserted with inS3 
it is now always added later, and a new update is forced for any
addition to an archived update
 2015-09-23 13:25:10 +01:00
Brian Gough
d6b827426c support forcing new compressed update in popLastCompressedUpdate 
callback with a null update, passing the version as an additional
argument
 2015-09-23 13:22:38 +01:00
Brian Gough
dc35ef5cda fix tests for archiving 2015-09-21 13:24:06 +01:00
Brian Gough
a10dc4f898 Merge pull request #6 from heukirne/s3-archive 
Add S3 archive track changes feature
 2015-09-21 11:25:06 +01:00
Brian Gough
0e627c92d8 avoid clobbering global _ in loop 2015-09-18 16:26:05 +01:00
Henrique Dias
aa66c5ee8c improve size function 2015-09-17 10:41:53 -03:00
Henrique Dias
3f712c452a add size bulk limit 2015-09-17 09:23:13 -03:00
Henrique Dias
01cbfd5338 fix minor issues in acceptance test 2015-09-16 19:33:23 -03:00
Henrique Dias
a4575f178d fix unit test 2015-09-16 18:39:07 -03:00
Brian Gough
12c5098b48 fix lock in test 2015-09-16 16:50:36 +01:00
Brian Gough
7af5050370 add lock to unarchive doc 2015-09-16 16:18:36 +01:00
Brian Gough
18f06a3daf increase lock timeouts for archiving 2015-09-16 16:09:38 +01:00
Brian Gough
b4ffa7d57e share the document lock between archiving and packing 2015-09-16 16:03:55 +01:00
Brian Gough
9d39012b49 add error handler to each stage of download pipeline 2015-09-16 16:00:37 +01:00
Brian Gough
d9085a5e5e add error handler to each stage of upload pipeline 2015-09-16 16:00:25 +01:00
Brian Gough
1c1b1d9595 log the case where there are no entries in the document history 2015-09-16 15:34:30 +01:00
Brian Gough
82d0f4fce8 make unarchive more responsive by downloading documents in parallel 
unarchive is triggered interactively so we should try to make it
reasonably fast
 2015-09-16 15:33:59 +01:00
Brian Gough
dfa0036507 pause stream while writing to mongo 2015-09-16 15:32:36 +01:00
Brian Gough
70200a9cf1 only log document ids, not document content 
avoid filling the log with large documents
 2015-09-16 15:31:43 +01:00
Brian Gough
d3dff28bea Merge remote-tracking branch 'origin/master' into heukirne-s3-archive 2015-09-15 15:19:43 +01:00
Brian Gough
092f98d3ad suppress error in normal shutdown case 2015-09-12 11:07:54 +01:00
Shane Kilkelly
39f528bcbc Add a test to check that users are summarised properly even when a null user occurs earlier in the update list. 2015-09-11 14:12:01 +01:00
Shane Kilkelly
eab8b4b6c8 Null safe access of id property, needed as user can be null. 2015-09-11 14:07:06 +01:00
Shane Kilkelly
0ad374556d Add a comment for clarity. 2015-09-10 16:43:40 +01:00
Shane Kilkelly
97326308fa Update the Acceptance tests to include a case where a user doesn't exist. 2015-09-10 15:40:43 +01:00
Shane Kilkelly
8387383cb4 In _summarizeUpdates, allow null users through. 
A null value represents a deleted or otherwise missing user record.
 2015-09-10 14:32:47 +01:00
Shane Kilkelly
810bddb2cb Log a message when the web api produces a 404 response. 2015-09-10 14:32:35 +01:00
Shane Kilkelly
575bdc62ec Add a test for when the user can't be found. 2015-09-10 14:32:18 +01:00
Shane Kilkelly
522786d45e Produce a null value, rather than crashing when the user info service returns 404. 2015-09-09 15:48:22 +01:00
Henry Oswald
18d817ee0a added some missing error handling 2015-09-08 16:33:45 +01:00
Henry Oswald
17b0d99a65 rework the archiveDocChangesWithLock function 
make it a bit more readable for me, struggle to trust indentation
based calls in coffeescript
 2015-09-08 16:26:01 +01:00
Henry Oswald
0b3ebcff06 remove if statments checking if s3 is a backend 
if its not enable then it can crash. In prod it should always be there
or not used at all
 2015-09-08 16:23:15 +01:00
Henrique Dias
c5a8a249c6 add unarchive acceptance tests 2015-09-03 08:36:32 -03:00
Henrique Dias
da9e7dc7e1 init archive acceptance tests 2015-09-02 18:47:34 -03:00
Henrique Dias
7de103af68 fix unit scope error 2015-09-02 17:00:32 -03:00
Henrique Dias
d2b1243701 split MongoAWS files 2015-09-02 15:45:29 -03:00
Henrique Dias
0c16fbed88 add more unit tests 2015-09-02 15:39:19 -03:00
Henrique Dias
1abcea1a66 add some unit test 2015-08-31 18:13:18 -03:00
Henrique Dias
efff026a79 handle easier propagation 2015-08-25 16:52:28 -03:00
Henrique Dias
f910e63e90 fix null case 2015-08-24 12:22:17 -03:00
Henrique Dias
fcbe4aa925 fix inS3 propagation 2015-08-24 12:19:19 -03:00
Henrique Dias
1ccba422c8 remove unused function 2015-08-24 10:55:27 -03:00
Henrique Dias
98ce03b2f2 replace docs collection to DocstoreHandler 2015-08-24 10:38:31 -03:00
Henrique Dias
04ec45529f restore updates from S3 when exists 
fix: avoid rearchiving
 2015-08-18 17:11:19 -03:00
Henrique Dias
20c3e15f93 fix bulk insert limit 2015-08-14 19:58:38 -03:00
Henrique Dias
26c8048729 change mongo stream method (still have a bug in bulk insert limit) 2015-08-14 19:19:54 -03:00
Henrique Dias
fd4afb3574 Archive changes, care about: versioin, expiresAt and Lock 2015-08-14 15:07:16 -03:00
Henrique Dias
6bc9c9010a handle auto unarchive track changes 2015-08-09 19:52:32 -03:00
Henrique Dias
3bc5380468 handle inS3 flag 2015-08-09 17:50:15 -03:00
Henrique Dias
daa42bcea0 change s3Stream lib 2015-08-09 15:47:47 -03:00
Henrique Dias
bca48ac117 add unarchive doc track from s3 2015-08-06 17:09:36 -03:00
Henrique Dias
438c4f4d0c using mongoexport for s3 archive 2015-08-06 15:46:44 -03:00
Henrique Dias
028fe2fa03 archive docChanges list to s3 2015-08-06 11:11:43 -03:00
Henrique Dias
ae047ecf76 init s3 feature 2015-08-06 10:00:09 -03:00
Brian Gough
775f5ebbe1 add configurable limit, delay and timeout to /pack via query string 2015-06-05 13:38:47 +01:00
Brian Gough
76d1350593 fix unit tests 
packManager now uses LockManager, which tries to talk to redis and
needs to be replaced by a stub in the unit tests.
 2015-06-05 10:59:46 +01:00
Brian Gough
23d2518ebb added a /pack endpoint to fork a pack worker 2015-06-04 16:36:56 +01:00
Brian Gough
289616d3db added a /doc/:doc_id/pack endpoint 2015-06-04 16:23:49 +01:00
Brian Gough
3f2e4b0c11 move pack script functionality into PackManager 2015-06-04 16:21:40 +01:00
Brian Gough
8668bb9be1 use docHistoryStats to drive the packing script 2015-06-03 10:36:31 +01:00
Brian Gough
27a3511b37 update docHistoryStats after packing ops 2015-06-03 10:36:07 +01:00
Brian Gough
19d812734e make PackManager parameters configurable 2015-06-03 10:35:36 +01:00
Brian Gough
2521c8f751 added npm packages for pack.coffee script 2015-05-26 14:14:51 +01:00
Brian Gough
66bca8d05c include the current date in the updates to docHistoryStats 2015-05-26 11:00:55 +01:00
Brian Gough
1811ac2145 added support for cleaning old expired ops in packs 2015-05-22 15:40:02 +01:00
Brian Gough
5c4afd5303 add docHistoryStats collection to keep track of updates to docs 2015-05-22 15:40:01 +01:00
Brian Gough
78f0bdbae3 fix name of temporary parameter to match other methods 2015-05-22 15:40:01 +01:00
Brian Gough
adc2866a7d add check to exclude temporary ops from packs 2015-05-22 15:40:01 +01:00
Brian Gough
ffeb1cccb6 move pack migration code into PackManager 2015-05-22 15:40:01 +01:00
Brian Gough
8d3e0b2353 added consistency check to pack migration script 2015-05-21 15:46:29 +01:00
Henry Oswald
b37d0dd08a make startup message consistent 2015-04-30 15:07:11 +01:00
James Allen
24319f6d50 Release version 0.1.4 2015-03-20 15:27:34 +00:00
James Allen
79b9c48efd Release version 0.1.3 2015-02-26 11:35:14 +00:00
Brian Gough
34fa398cf6 increase pack migrate delay to 2 seconds 2015-02-23 17:00:54 +00:00
Brian Gough
09e24413e6 fix pack migration to run sequentially not in parallel 2015-02-23 17:00:43 +00:00
Brian Gough
ff7860b6dc added a time delay to the pack migration script 2015-02-23 14:10:09 +00:00
Brian Gough
b19f3835d5 added checkpointing and logging to pack script 2015-02-23 12:13:38 +00:00
Brian Gough
4c0eea9916 return an error if trying to pop the last update from a pack 2015-02-17 14:53:50 +00:00
Brian Gough
24701219ff clean up comments and remove console.logs 2015-02-17 13:47:55 +00:00
Brian Gough
76cbf32e6a cleanup 
remove unnecessary comments, whitespace and unused modules
 2015-02-17 13:41:31 +00:00
Brian Gough
f69f6452e9 Merge remote-tracking branch 'origin/master' into support-packs 2015-02-17 13:38:30 +00:00
Brian Gough
448a7b7f36 rename MongoPackManager to PackManager 2015-02-17 11:14:13 +00:00
Brian Gough
92e67511d5 sort by doc_id to ensure consistent order for all changes with the same timestamp 2015-02-16 16:44:06 +00:00
Brian Gough
7d6811559c test pack for getDocUpdates with time 2015-02-16 16:43:27 +00:00
Brian Gough
e55f2a0ff7 test pack for getDocUpdates with version 2015-02-16 15:59:58 +00:00
Brian Gough
f5a9184b57 update the tests to work with packs 2015-02-16 15:38:40 +00:00
Brian Gough
fa1e890282 move update deduplication to filterAndLimit method 2015-02-16 15:37:47 +00:00
Brian Gough
0516e9429d ensure document results are in version order 2015-02-16 13:43:27 +00:00
Brian Gough
8615b7a086 fix project results order, sort by time 2015-02-16 13:43:11 +00:00
Brian Gough
3ab8238013 update mongojs version to allow use of bulk updates 2015-02-13 17:01:55 +00:00
Brian Gough
18b34d1bcb clean up packing script 2015-02-13 17:01:37 +00:00
Brian Gough
cafda57710 command-line packing script 2015-02-13 16:23:36 +00:00
Brian Gough
4f36ccd519 moved pack code into MongoPackManager 2015-02-13 16:18:15 +00:00
Brian Gough
682e8e8ab6 add a sparse mongo index for finding packs 2015-02-10 16:54:58 +00:00
Brian Gough
1d7f0919a4 fix mongo logic for project search with packs 2015-02-10 16:54:45 +00:00
James Allen
2d25d54572 Release version 0.1.2 2015-02-10 13:18:12 +00:00
Brian Gough
7fdce8fc48 split pack handling code into separate parts for docs and projects 
need to debug projects
 2015-02-09 16:53:44 +00:00
Brian Gough
d8dc56b031 don't try to getLastCompressedUpdates from packs 2015-02-09 16:53:05 +00:00
Brian Gough
f50f091fc6 added comments 2015-02-06 16:59:09 +00:00
Brian Gough
e47476369a clean up docHistory pack decoding 2015-02-06 15:04:46 +00:00
Brian Gough
8e810bab14 report error for inconsistent history results 2015-02-05 16:37:06 +00:00
Brian Gough
715b0df4a1 prototype for expanding packs 2015-02-05 16:36:41 +00:00
Henry Oswald
3328acc82b bump sharelatex-redis 2014-10-07 08:33:00 +01:00
Henry Oswald
66bba0dec9 bump redis-sharelatex 2014-09-29 11:41:09 +01:00
Henry Oswald
3f4452e6af don't run acceptence tests on travis 2014-09-29 11:33:40 +01:00
Henry Oswald
1452eb4572 fixed tests 2014-09-26 17:51:41 +01:00
Henry Oswald
9891bc37d8 Merge branch 'master' of https://github.com/sharelatex/track-changes-sharelatex 2014-09-26 17:41:27 +01:00
Henry Oswald
a0bb289fd3 works with sentinal v1 2014-09-26 17:21:33 +01:00
James Allen
f0e5b914a6 Release version 0.1.0 2014-08-19 14:17:11 +01:00
James Allen
e7b4359c62 Lock down module versions 2014-08-19 14:15:44 +01:00
James Allen
556afc8776 Rename settings to defaults 2014-06-06 13:00:28 +01:00
James Allen
eb7bcc6922 Remove old history deletion method 2014-05-29 15:37:16 +01:00
James Allen
cc962e1c44 Add indexes in background 2014-05-16 17:00:30 +01:00
James Allen
6a371c267f Fix indexes 2014-05-16 16:41:40 +01:00
James Allen
29ad81c134 Use expiresAt rather than tempCreatedAt 2014-05-16 16:41:14 +01:00
James Allen
8b0b79bc32 Use TTL index to auto-delete updates after a week when versioning is not enabled 2014-05-16 15:59:12 +01:00
James Allen
c835528dea Add in metrics 2014-05-09 12:44:13 +01:00
James Allen
24343a38cd Add error handling of mismatched versions back in 2014-05-01 10:26:01 +01:00
James Allen
eebfa6499e Be more forgiving with version mismatches 2014-04-15 08:02:49 +01:00
James Allen
953081c39d Delete updates that are older than a week, unless versioning or preserveHistory is enabled 2014-03-28 16:01:34 +00:00
James Allen
9f1efe6dca Add in getProjectDetails to WebApiManager 2014-03-28 13:05:16 +00:00
James Allen
22e3aac5b0 Don't trip up over empty ops 2014-03-25 11:40:48 +00:00
James Allen
bd4bb3d3cf Add in project flushing end point 2014-03-21 15:57:17 +00:00
James Allen
8cae726838 Backport project_id onto doc updates that don't have one yet 2014-03-21 14:40:51 +00:00
James Allen
2cd85fefec Flush all docs in a project before getting updates 2014-03-21 13:48:14 +00:00
James Allen
cd9cb51027 Delete doc_id from DocsWithHistoryOps set when processing 2014-03-21 13:17:58 +00:00
James Allen
6aaa7ba8d5 Group toV and fromV by the doc they correspond to 2014-03-20 13:37:23 +00:00
James Allen
5595910122 Return a summary of the entire project changes 2014-03-20 12:10:04 +00:00
James Allen
b7de4c9c6c Implement project update getter methods 2014-03-19 17:44:16 +00:00
James Allen
962fc18329 Store project_id alongside each change 2014-03-19 16:40:55 +00:00
James Allen
fec648666c Add in project_id to flushing HTTP endpoint 2014-03-19 16:14:17 +00:00
James Allen
d32b0ee12f Fetch updates up to the limit, even after summarising 2014-03-18 18:09:25 +00:00
James Allen
81811d7cc5 Move update summarizing logic into UpdatesManager 2014-03-18 11:41:48 +00:00
James Allen
22a806a200 Fix issues with consuming delete updates beyond the end of the diff 2014-03-17 14:54:02 +00:00
James Allen
540aaf0672 Add in some null checks 2014-03-13 14:30:01 +00:00
James Allen
9977a418c3 Compare diffs with user.id, not user_id 2014-03-12 14:04:33 +00:00
James Allen
575afae048 Compress adjacent inserts and deletes into one diff item 2014-03-12 13:27:18 +00:00
James Allen
7fa88f6b41 Summarise updates when returning them 2014-03-11 17:40:07 +00:00
James Allen
4ecb17b16e Put multiple ops in one update 2014-03-11 15:24:38 +00:00
James Allen
064bdc3eea Take user id in from request header and pass to doc updater 2014-03-11 13:01:07 +00:00
James Allen
3d1d962501 Don't try to fetch user when id doesn't exist 2014-03-11 11:45:25 +00:00
James Allen
83b2aa3082 add in restore end point 2014-03-10 16:58:39 +00:00
James Allen
7eb8699b93 Refactor diff manager 2014-03-10 16:03:03 +00:00
James Allen
866084ee6b Pop last version, not last timestamp 2014-03-07 14:02:16 +00:00
James Allen
b45db3aa2b Return user details like email and name in updates 2014-03-06 18:04:00 +00:00
James Allen
5fc139db00 Return version numbers of updates 2014-03-06 11:00:49 +00:00
James Allen
aadce232a1 Use version numbers for sorting and querying, not dates 2014-03-06 10:45:51 +00:00
James Allen
2e1307bd82 Doc is always one version ahead of latest op version 2014-03-05 16:31:38 +00:00
James Allen
3660253fd4 Add in /updates end point to get updates 2014-03-05 15:59:40 +00:00
James Allen
a46963a349 Refactor arguments to MongoManager.getUpdatesBetweenDates 2014-03-05 15:06:46 +00:00
James Allen
188d620ce1 Log out and put the last update back when an error occurs 2014-03-05 13:22:38 +00:00
James Allen
e4d8cc7a11 Add in acceptance tests for getting a diff 2014-03-04 15:27:03 +00:00
James Allen
8b71d222d4 Create DiffManager.getDiff 2014-03-04 14:05:17 +00:00
James Allen
1d1dcdfa2f Add in methods for retrieving updates and doc version 2014-03-04 13:02:48 +00:00
James Allen
20d70859aa Create buildDiff function 2014-03-03 17:39:59 +00:00
James Allen
8d044b7f13 Fix acceptance tests 2014-03-03 13:36:05 +00:00
James Allen
ba2c4768d0 Use port 3015 by default 2014-03-03 13:31:10 +00:00
James Allen
c15df4c2aa Add in /status end point 2014-03-03 13:14:01 +00:00
James Allen
e0402692cf Add in diff generating functions 2014-03-01 13:31:34 +00:00
James Allen
76cdd5cf98 Delete only the updates that were processed, not a full batch 2014-03-01 11:42:31 +00:00
James Allen
d1e18d82aa Add simple benchmark script 2014-02-26 17:35:10 +00:00
James Allen
3570e8f408 Use grunt forever to run service daemonized for acceptance testing on travis 2014-02-26 16:44:13 +00:00
James Allen
53db11466b Fix YAML 2014-02-26 16:39:58 +00:00
James Allen
cef17a7fbc Experiment with acceptance tests in Travis 2014-02-26 16:36:53 +00:00
James Allen
6e9485c0c4 Add README and LICENSE 2014-02-26 16:29:14 +00:00
James Allen
1e7fc3dc29 Add Travis CI 2014-02-26 16:27:09 +00:00
James Allen
834f41eea1 Remove Rakefile 2014-02-26 16:17:26 +00:00
James Allen
ad18c1d5a5 Add Gruntfile 2014-02-26 16:17:15 +00:00
James Allen
a21f8a8004 Add acceptance test for batch processing 2014-02-26 12:44:13 +00:00
James Allen
80af34895b Refactor and improve acceptance tests 2014-02-26 12:38:47 +00:00
James Allen
d27872c9bd Get acceptance tests running 2014-02-26 12:11:45 +00:00
James Allen
45fe6978af Add in /doc/<doc-id>/flush endpoint 2014-02-26 11:34:56 +00:00
James Allen
65360a9a2b Continue to process raw updates until redis is empty 2014-02-26 11:18:26 +00:00
James Allen
8405a37c2c Add a lock around processing updates 2014-02-26 10:55:20 +00:00
James Allen
f33a3bde3e Create processUncompressedUpdates method 2014-02-25 16:48:42 +00:00
James Allen
34d3847fe4 Add Redis Manager for fetching and deleting uncompressed ops 2014-02-25 16:27:14 +00:00
James Allen
8ae9bcd60f Ensure that updates are compressed in continuous incrementing order 2014-02-25 12:28:15 +00:00
James Allen
de783bf5b0 Pull out MongoManager module and add tests 2014-02-24 17:43:27 +00:00
James Allen
d040e7c410 Add acceptance tests for multiple appends 2014-01-27 18:20:38 +00:00
James Allen
d4295c2023 Create HttpController for HTTP requestS 2014-01-27 18:09:37 +00:00
James Allen
5dea123b13 Get append end point working with a simple acceptance test 2014-01-27 17:51:09 +00:00
James Allen
533b8e59a3 Big refactor to use better names and separation of concerns 2014-01-27 16:26:58 +00:00
James Allen
8a0aa55c91 Refactor to take doc ops from an HTTP request from doc updater 2013-09-03 17:17:27 +01:00
James Allen
dab2781e87 Do two passes better 2013-08-23 14:58:00 +01:00
James Allen
dbee4a57fb Compress doc ops in two passes 2013-08-23 14:35:13 +01:00
James Allen
bef05c31f4 No need to print the entire history each time 2013-08-09 13:54:30 +01:00
James Allen
7a3b78da43 Return when calling callback 2013-08-09 13:51:02 +01:00
James Allen
07f078b527 Don't insert a null update if there are no updates or raw updates 2013-08-09 13:46:03 +01:00
James Allen
fe503010aa Pop and set history in batches for speed 2013-08-09 13:27:35 +01:00
James Allen
3ab45c2a35 Handle if two updates cancel each other 2013-08-09 12:36:17 +01:00
James Allen
54c7959647 Match if both user ids are null/undefined 2013-08-09 12:26:23 +01:00
James Allen
64eb794c02 Compress all docs using compressHistory script 2013-08-09 11:36:19 +01:00
James Allen
c5e7f14ba1 Factor methods out into ConversionManager 2013-08-09 11:10:19 +01:00
James Allen
7e96933cf2 Initial commit of script for compressing one docs history 2013-08-08 16:42:14 +01:00