github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-07 20:31:06 -05:00
Code Issues Projects Releases Packages Wiki Activity
18068 commits 3 branches 7 tags 201 MiB
Commit graph

601 commits

Author SHA1 Message Date
Brian Gough
692e8c657c Revert to the default lock timeout now we have write barriers 
Revert "increase lock timeouts for archiving"

This reverts commit 9eee1b383772adf058130d6e5eab409f57ce03cd.
 2015-09-24 08:53:09 +01:00
Brian Gough
2ab1778dd9 move default value of lastVersion into function body 2015-09-23 16:31:33 +01:00
Brian Gough
bdf1d267f0 fix tests 2015-09-23 15:34:36 +01:00
Brian Gough
dc0044020f only archive entries older than the current update 
to avoid a stale version of the current update ever being pulled back
from S3
 2015-09-23 14:33:40 +01:00
Brian Gough
696a866b67 pause the stream of ops, not the download 
the download is buffered in the lineStream so a lot comes out even
after pausing the S3 download.
 2015-09-23 13:38:57 +01:00
Brian Gough
847a553344 prevent double archiving by checking if any inS3 field is already present 2015-09-23 13:29:32 +01:00
Brian Gough
e49f260507 allow rollback/locking by setting inS3:false when starting the archive process 2015-09-23 13:28:07 +01:00
Brian Gough
551e8334cf compressedUpdates are now never inserted with inS3 
it is now always added later, and a new update is forced for any
addition to an archived update
 2015-09-23 13:25:10 +01:00
Brian Gough
d6b827426c support forcing new compressed update in popLastCompressedUpdate 
callback with a null update, passing the version as an additional
argument
 2015-09-23 13:22:38 +01:00
Brian Gough
dc35ef5cda fix tests for archiving 2015-09-21 13:24:06 +01:00
Brian Gough
a10dc4f898 Merge pull request #6 from heukirne/s3-archive 
Add S3 archive track changes feature
 2015-09-21 11:25:06 +01:00
Brian Gough
0e627c92d8 avoid clobbering global _ in loop 2015-09-18 16:26:05 +01:00
Henrique Dias
aa66c5ee8c improve size function 2015-09-17 10:41:53 -03:00
Henrique Dias
3f712c452a add size bulk limit 2015-09-17 09:23:13 -03:00
Henrique Dias
01cbfd5338 fix minor issues in acceptance test 2015-09-16 19:33:23 -03:00
Henrique Dias
a4575f178d fix unit test 2015-09-16 18:39:07 -03:00
Brian Gough
12c5098b48 fix lock in test 2015-09-16 16:50:36 +01:00
Brian Gough
7af5050370 add lock to unarchive doc 2015-09-16 16:18:36 +01:00
Brian Gough
18f06a3daf increase lock timeouts for archiving 2015-09-16 16:09:38 +01:00
Brian Gough
b4ffa7d57e share the document lock between archiving and packing 2015-09-16 16:03:55 +01:00
Brian Gough
9d39012b49 add error handler to each stage of download pipeline 2015-09-16 16:00:37 +01:00
Brian Gough
d9085a5e5e add error handler to each stage of upload pipeline 2015-09-16 16:00:25 +01:00
Brian Gough
1c1b1d9595 log the case where there are no entries in the document history 2015-09-16 15:34:30 +01:00
Brian Gough
82d0f4fce8 make unarchive more responsive by downloading documents in parallel 
unarchive is triggered interactively so we should try to make it
reasonably fast
 2015-09-16 15:33:59 +01:00
Brian Gough
dfa0036507 pause stream while writing to mongo 2015-09-16 15:32:36 +01:00
Brian Gough
70200a9cf1 only log document ids, not document content 
avoid filling the log with large documents
 2015-09-16 15:31:43 +01:00
Brian Gough
d3dff28bea Merge remote-tracking branch 'origin/master' into heukirne-s3-archive 2015-09-15 15:19:43 +01:00
Brian Gough
092f98d3ad suppress error in normal shutdown case 2015-09-12 11:07:54 +01:00
Shane Kilkelly
39f528bcbc Add a test to check that users are summarised properly even when a null user occurs earlier in the update list. 2015-09-11 14:12:01 +01:00
Shane Kilkelly
eab8b4b6c8 Null safe access of id property, needed as user can be null. 2015-09-11 14:07:06 +01:00
Shane Kilkelly
0ad374556d Add a comment for clarity. 2015-09-10 16:43:40 +01:00
Shane Kilkelly
97326308fa Update the Acceptance tests to include a case where a user doesn't exist. 2015-09-10 15:40:43 +01:00
Shane Kilkelly
8387383cb4 In _summarizeUpdates, allow null users through. 
A null value represents a deleted or otherwise missing user record.
 2015-09-10 14:32:47 +01:00
Shane Kilkelly
810bddb2cb Log a message when the web api produces a 404 response. 2015-09-10 14:32:35 +01:00
Shane Kilkelly
575bdc62ec Add a test for when the user can't be found. 2015-09-10 14:32:18 +01:00
Shane Kilkelly
522786d45e Produce a null value, rather than crashing when the user info service returns 404. 2015-09-09 15:48:22 +01:00
Henry Oswald
18d817ee0a added some missing error handling 2015-09-08 16:33:45 +01:00
Henry Oswald
17b0d99a65 rework the archiveDocChangesWithLock function 
make it a bit more readable for me, struggle to trust indentation
based calls in coffeescript
 2015-09-08 16:26:01 +01:00
Henry Oswald
0b3ebcff06 remove if statments checking if s3 is a backend 
if its not enable then it can crash. In prod it should always be there
or not used at all
 2015-09-08 16:23:15 +01:00
Henrique Dias
c5a8a249c6 add unarchive acceptance tests 2015-09-03 08:36:32 -03:00
Henrique Dias
da9e7dc7e1 init archive acceptance tests 2015-09-02 18:47:34 -03:00
Henrique Dias
7de103af68 fix unit scope error 2015-09-02 17:00:32 -03:00
Henrique Dias
d2b1243701 split MongoAWS files 2015-09-02 15:45:29 -03:00
Henrique Dias
0c16fbed88 add more unit tests 2015-09-02 15:39:19 -03:00
Henrique Dias
1abcea1a66 add some unit test 2015-08-31 18:13:18 -03:00
Henrique Dias
efff026a79 handle easier propagation 2015-08-25 16:52:28 -03:00
Henrique Dias
f910e63e90 fix null case 2015-08-24 12:22:17 -03:00
Henrique Dias
fcbe4aa925 fix inS3 propagation 2015-08-24 12:19:19 -03:00
Henrique Dias
1ccba422c8 remove unused function 2015-08-24 10:55:27 -03:00
Henrique Dias
98ce03b2f2 replace docs collection to DocstoreHandler 2015-08-24 10:38:31 -03:00
Henrique Dias
04ec45529f restore updates from S3 when exists 
fix: avoid rearchiving
 2015-08-18 17:11:19 -03:00
Henrique Dias
20c3e15f93 fix bulk insert limit 2015-08-14 19:58:38 -03:00
Henrique Dias
26c8048729 change mongo stream method (still have a bug in bulk insert limit) 2015-08-14 19:19:54 -03:00
Henrique Dias
fd4afb3574 Archive changes, care about: versioin, expiresAt and Lock 2015-08-14 15:07:16 -03:00
Henrique Dias
6bc9c9010a handle auto unarchive track changes 2015-08-09 19:52:32 -03:00
Henrique Dias
3bc5380468 handle inS3 flag 2015-08-09 17:50:15 -03:00
Henrique Dias
daa42bcea0 change s3Stream lib 2015-08-09 15:47:47 -03:00
Henrique Dias
bca48ac117 add unarchive doc track from s3 2015-08-06 17:09:36 -03:00
Henrique Dias
438c4f4d0c using mongoexport for s3 archive 2015-08-06 15:46:44 -03:00
Henrique Dias
028fe2fa03 archive docChanges list to s3 2015-08-06 11:11:43 -03:00
Henrique Dias
ae047ecf76 init s3 feature 2015-08-06 10:00:09 -03:00
Brian Gough
775f5ebbe1 add configurable limit, delay and timeout to /pack via query string 2015-06-05 13:38:47 +01:00
Brian Gough
76d1350593 fix unit tests 
packManager now uses LockManager, which tries to talk to redis and
needs to be replaced by a stub in the unit tests.
 2015-06-05 10:59:46 +01:00
Brian Gough
23d2518ebb added a /pack endpoint to fork a pack worker 2015-06-04 16:36:56 +01:00
Brian Gough
289616d3db added a /doc/:doc_id/pack endpoint 2015-06-04 16:23:49 +01:00
Brian Gough
3f2e4b0c11 move pack script functionality into PackManager 2015-06-04 16:21:40 +01:00
Brian Gough
8668bb9be1 use docHistoryStats to drive the packing script 2015-06-03 10:36:31 +01:00
Brian Gough
27a3511b37 update docHistoryStats after packing ops 2015-06-03 10:36:07 +01:00
Brian Gough
19d812734e make PackManager parameters configurable 2015-06-03 10:35:36 +01:00
Brian Gough
2521c8f751 added npm packages for pack.coffee script 2015-05-26 14:14:51 +01:00
Brian Gough
66bca8d05c include the current date in the updates to docHistoryStats 2015-05-26 11:00:55 +01:00
Brian Gough
1811ac2145 added support for cleaning old expired ops in packs 2015-05-22 15:40:02 +01:00
Brian Gough
5c4afd5303 add docHistoryStats collection to keep track of updates to docs 2015-05-22 15:40:01 +01:00
Brian Gough
78f0bdbae3 fix name of temporary parameter to match other methods 2015-05-22 15:40:01 +01:00
Brian Gough
adc2866a7d add check to exclude temporary ops from packs 2015-05-22 15:40:01 +01:00
Brian Gough
ffeb1cccb6 move pack migration code into PackManager 2015-05-22 15:40:01 +01:00
Brian Gough
8d3e0b2353 added consistency check to pack migration script 2015-05-21 15:46:29 +01:00
Henry Oswald
b37d0dd08a make startup message consistent 2015-04-30 15:07:11 +01:00
James Allen
24319f6d50 Release version 0.1.4 2015-03-20 15:27:34 +00:00
James Allen
79b9c48efd Release version 0.1.3 2015-02-26 11:35:14 +00:00
Brian Gough
34fa398cf6 increase pack migrate delay to 2 seconds 2015-02-23 17:00:54 +00:00
Brian Gough
09e24413e6 fix pack migration to run sequentially not in parallel 2015-02-23 17:00:43 +00:00
Brian Gough
ff7860b6dc added a time delay to the pack migration script 2015-02-23 14:10:09 +00:00
Brian Gough
b19f3835d5 added checkpointing and logging to pack script 2015-02-23 12:13:38 +00:00
Brian Gough
4c0eea9916 return an error if trying to pop the last update from a pack 2015-02-17 14:53:50 +00:00
Brian Gough
24701219ff clean up comments and remove console.logs 2015-02-17 13:47:55 +00:00
Brian Gough
76cbf32e6a cleanup 
remove unnecessary comments, whitespace and unused modules
 2015-02-17 13:41:31 +00:00
Brian Gough
f69f6452e9 Merge remote-tracking branch 'origin/master' into support-packs 2015-02-17 13:38:30 +00:00
Brian Gough
448a7b7f36 rename MongoPackManager to PackManager 2015-02-17 11:14:13 +00:00
Brian Gough
92e67511d5 sort by doc_id to ensure consistent order for all changes with the same timestamp 2015-02-16 16:44:06 +00:00
Brian Gough
7d6811559c test pack for getDocUpdates with time 2015-02-16 16:43:27 +00:00
Brian Gough
e55f2a0ff7 test pack for getDocUpdates with version 2015-02-16 15:59:58 +00:00
Brian Gough
f5a9184b57 update the tests to work with packs 2015-02-16 15:38:40 +00:00
Brian Gough
fa1e890282 move update deduplication to filterAndLimit method 2015-02-16 15:37:47 +00:00
Brian Gough
0516e9429d ensure document results are in version order 2015-02-16 13:43:27 +00:00
Brian Gough
8615b7a086 fix project results order, sort by time 2015-02-16 13:43:11 +00:00
Brian Gough
3ab8238013 update mongojs version to allow use of bulk updates 2015-02-13 17:01:55 +00:00
Brian Gough
18b34d1bcb clean up packing script 2015-02-13 17:01:37 +00:00
Brian Gough
cafda57710 command-line packing script 2015-02-13 16:23:36 +00:00
Brian Gough
4f36ccd519 moved pack code into MongoPackManager 2015-02-13 16:18:15 +00:00
Brian Gough
682e8e8ab6 add a sparse mongo index for finding packs 2015-02-10 16:54:58 +00:00
Brian Gough
1d7f0919a4 fix mongo logic for project search with packs 2015-02-10 16:54:45 +00:00
James Allen
2d25d54572 Release version 0.1.2 2015-02-10 13:18:12 +00:00
Brian Gough
7fdce8fc48 split pack handling code into separate parts for docs and projects 
need to debug projects
 2015-02-09 16:53:44 +00:00
Brian Gough
d8dc56b031 don't try to getLastCompressedUpdates from packs 2015-02-09 16:53:05 +00:00
Brian Gough
f50f091fc6 added comments 2015-02-06 16:59:09 +00:00
Brian Gough
e47476369a clean up docHistory pack decoding 2015-02-06 15:04:46 +00:00
Brian Gough
8e810bab14 report error for inconsistent history results 2015-02-05 16:37:06 +00:00
Brian Gough
715b0df4a1 prototype for expanding packs 2015-02-05 16:36:41 +00:00
Henry Oswald
3328acc82b bump sharelatex-redis 2014-10-07 08:33:00 +01:00
Henry Oswald
66bba0dec9 bump redis-sharelatex 2014-09-29 11:41:09 +01:00
Henry Oswald
3f4452e6af don't run acceptence tests on travis 2014-09-29 11:33:40 +01:00
Henry Oswald
1452eb4572 fixed tests 2014-09-26 17:51:41 +01:00
Henry Oswald
9891bc37d8 Merge branch 'master' of https://github.com/sharelatex/track-changes-sharelatex 2014-09-26 17:41:27 +01:00
Henry Oswald
a0bb289fd3 works with sentinal v1 2014-09-26 17:21:33 +01:00
James Allen
f0e5b914a6 Release version 0.1.0 2014-08-19 14:17:11 +01:00
James Allen
e7b4359c62 Lock down module versions 2014-08-19 14:15:44 +01:00
James Allen
556afc8776 Rename settings to defaults 2014-06-06 13:00:28 +01:00
James Allen
eb7bcc6922 Remove old history deletion method 2014-05-29 15:37:16 +01:00
James Allen
cc962e1c44 Add indexes in background 2014-05-16 17:00:30 +01:00
James Allen
6a371c267f Fix indexes 2014-05-16 16:41:40 +01:00
James Allen
29ad81c134 Use expiresAt rather than tempCreatedAt 2014-05-16 16:41:14 +01:00
James Allen
8b0b79bc32 Use TTL index to auto-delete updates after a week when versioning is not enabled 2014-05-16 15:59:12 +01:00
James Allen
c835528dea Add in metrics 2014-05-09 12:44:13 +01:00
James Allen
24343a38cd Add error handling of mismatched versions back in 2014-05-01 10:26:01 +01:00
James Allen
eebfa6499e Be more forgiving with version mismatches 2014-04-15 08:02:49 +01:00
James Allen
953081c39d Delete updates that are older than a week, unless versioning or preserveHistory is enabled 2014-03-28 16:01:34 +00:00
James Allen
9f1efe6dca Add in getProjectDetails to WebApiManager 2014-03-28 13:05:16 +00:00
James Allen
22e3aac5b0 Don't trip up over empty ops 2014-03-25 11:40:48 +00:00
James Allen
bd4bb3d3cf Add in project flushing end point 2014-03-21 15:57:17 +00:00
James Allen
8cae726838 Backport project_id onto doc updates that don't have one yet 2014-03-21 14:40:51 +00:00
James Allen
2cd85fefec Flush all docs in a project before getting updates 2014-03-21 13:48:14 +00:00
James Allen
cd9cb51027 Delete doc_id from DocsWithHistoryOps set when processing 2014-03-21 13:17:58 +00:00
James Allen
6aaa7ba8d5 Group toV and fromV by the doc they correspond to 2014-03-20 13:37:23 +00:00
James Allen
5595910122 Return a summary of the entire project changes 2014-03-20 12:10:04 +00:00
James Allen
b7de4c9c6c Implement project update getter methods 2014-03-19 17:44:16 +00:00
James Allen
962fc18329 Store project_id alongside each change 2014-03-19 16:40:55 +00:00
James Allen
fec648666c Add in project_id to flushing HTTP endpoint 2014-03-19 16:14:17 +00:00
James Allen
d32b0ee12f Fetch updates up to the limit, even after summarising 2014-03-18 18:09:25 +00:00
James Allen
81811d7cc5 Move update summarizing logic into UpdatesManager 2014-03-18 11:41:48 +00:00
James Allen
22a806a200 Fix issues with consuming delete updates beyond the end of the diff 2014-03-17 14:54:02 +00:00
James Allen
540aaf0672 Add in some null checks 2014-03-13 14:30:01 +00:00
James Allen
9977a418c3 Compare diffs with user.id, not user_id 2014-03-12 14:04:33 +00:00
James Allen
575afae048 Compress adjacent inserts and deletes into one diff item 2014-03-12 13:27:18 +00:00
James Allen
7fa88f6b41 Summarise updates when returning them 2014-03-11 17:40:07 +00:00
James Allen
4ecb17b16e Put multiple ops in one update 2014-03-11 15:24:38 +00:00
James Allen
064bdc3eea Take user id in from request header and pass to doc updater 2014-03-11 13:01:07 +00:00
James Allen
3d1d962501 Don't try to fetch user when id doesn't exist 2014-03-11 11:45:25 +00:00
James Allen
83b2aa3082 add in restore end point 2014-03-10 16:58:39 +00:00
James Allen
7eb8699b93 Refactor diff manager 2014-03-10 16:03:03 +00:00
James Allen
866084ee6b Pop last version, not last timestamp 2014-03-07 14:02:16 +00:00
James Allen
b45db3aa2b Return user details like email and name in updates 2014-03-06 18:04:00 +00:00
James Allen
5fc139db00 Return version numbers of updates 2014-03-06 11:00:49 +00:00
James Allen
aadce232a1 Use version numbers for sorting and querying, not dates 2014-03-06 10:45:51 +00:00
James Allen
2e1307bd82 Doc is always one version ahead of latest op version 2014-03-05 16:31:38 +00:00
James Allen
3660253fd4 Add in /updates end point to get updates 2014-03-05 15:59:40 +00:00
James Allen
a46963a349 Refactor arguments to MongoManager.getUpdatesBetweenDates 2014-03-05 15:06:46 +00:00
James Allen
188d620ce1 Log out and put the last update back when an error occurs 2014-03-05 13:22:38 +00:00
James Allen
e4d8cc7a11 Add in acceptance tests for getting a diff 2014-03-04 15:27:03 +00:00
James Allen
8b71d222d4 Create DiffManager.getDiff 2014-03-04 14:05:17 +00:00
James Allen
1d1dcdfa2f Add in methods for retrieving updates and doc version 2014-03-04 13:02:48 +00:00
James Allen
20d70859aa Create buildDiff function 2014-03-03 17:39:59 +00:00
James Allen
8d044b7f13 Fix acceptance tests 2014-03-03 13:36:05 +00:00
James Allen
ba2c4768d0 Use port 3015 by default 2014-03-03 13:31:10 +00:00
James Allen
c15df4c2aa Add in /status end point 2014-03-03 13:14:01 +00:00
James Allen
e0402692cf Add in diff generating functions 2014-03-01 13:31:34 +00:00
James Allen
76cdd5cf98 Delete only the updates that were processed, not a full batch 2014-03-01 11:42:31 +00:00
James Allen
d1e18d82aa Add simple benchmark script 2014-02-26 17:35:10 +00:00
James Allen
3570e8f408 Use grunt forever to run service daemonized for acceptance testing on travis 2014-02-26 16:44:13 +00:00
James Allen
53db11466b Fix YAML 2014-02-26 16:39:58 +00:00
James Allen
cef17a7fbc Experiment with acceptance tests in Travis 2014-02-26 16:36:53 +00:00
James Allen
6e9485c0c4 Add README and LICENSE 2014-02-26 16:29:14 +00:00
James Allen
1e7fc3dc29 Add Travis CI 2014-02-26 16:27:09 +00:00
James Allen
834f41eea1 Remove Rakefile 2014-02-26 16:17:26 +00:00
James Allen
ad18c1d5a5 Add Gruntfile 2014-02-26 16:17:15 +00:00
James Allen
a21f8a8004 Add acceptance test for batch processing 2014-02-26 12:44:13 +00:00
James Allen
80af34895b Refactor and improve acceptance tests 2014-02-26 12:38:47 +00:00
James Allen
d27872c9bd Get acceptance tests running 2014-02-26 12:11:45 +00:00
James Allen
45fe6978af Add in /doc/<doc-id>/flush endpoint 2014-02-26 11:34:56 +00:00
James Allen
65360a9a2b Continue to process raw updates until redis is empty 2014-02-26 11:18:26 +00:00
James Allen
8405a37c2c Add a lock around processing updates 2014-02-26 10:55:20 +00:00
James Allen
f33a3bde3e Create processUncompressedUpdates method 2014-02-25 16:48:42 +00:00
James Allen
34d3847fe4 Add Redis Manager for fetching and deleting uncompressed ops 2014-02-25 16:27:14 +00:00
James Allen
8ae9bcd60f Ensure that updates are compressed in continuous incrementing order 2014-02-25 12:28:15 +00:00
James Allen
de783bf5b0 Pull out MongoManager module and add tests 2014-02-24 17:43:27 +00:00
James Allen
d040e7c410 Add acceptance tests for multiple appends 2014-01-27 18:20:38 +00:00
James Allen
d4295c2023 Create HttpController for HTTP requestS 2014-01-27 18:09:37 +00:00
James Allen
5dea123b13 Get append end point working with a simple acceptance test 2014-01-27 17:51:09 +00:00
James Allen
533b8e59a3 Big refactor to use better names and separation of concerns 2014-01-27 16:26:58 +00:00
James Allen
8a0aa55c91 Refactor to take doc ops from an HTTP request from doc updater 2013-09-03 17:17:27 +01:00
James Allen
dab2781e87 Do two passes better 2013-08-23 14:58:00 +01:00
James Allen
dbee4a57fb Compress doc ops in two passes 2013-08-23 14:35:13 +01:00
James Allen
bef05c31f4 No need to print the entire history each time 2013-08-09 13:54:30 +01:00
James Allen
7a3b78da43 Return when calling callback 2013-08-09 13:51:02 +01:00
James Allen
07f078b527 Don't insert a null update if there are no updates or raw updates 2013-08-09 13:46:03 +01:00
James Allen
fe503010aa Pop and set history in batches for speed 2013-08-09 13:27:35 +01:00
James Allen
3ab45c2a35 Handle if two updates cancel each other 2013-08-09 12:36:17 +01:00
James Allen
54c7959647 Match if both user ids are null/undefined 2013-08-09 12:26:23 +01:00
James Allen
64eb794c02 Compress all docs using compressHistory script 2013-08-09 11:36:19 +01:00
James Allen
c5e7f14ba1 Factor methods out into ConversionManager 2013-08-09 11:10:19 +01:00
James Allen
7e96933cf2 Initial commit of script for compressing one docs history 2013-08-08 16:42:14 +01:00