github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-07 20:31:06 -05:00
Code Issues Projects Releases Packages Wiki Activity
579 commits 3 branches 7 tags 201 MiB
Commit graph

82 commits

Author SHA1 Message Date
Simon Detheridge
d073fe75ca Use single pipeline when calculating md5 2020-03-25 16:59:51 +00:00
Simon Detheridge
65f19346f0 Fix fallback to custom s3 buckets 2020-03-19 11:10:31 +00:00
Simon Detheridge
cb4bdd99f4 Use an ISODate for deleted file names 2020-03-18 20:19:12 +00:00
Simon Detheridge
9d32d4ec16 Don't modify 'opts' parameter 2020-03-16 15:57:37 +00:00
Simon Detheridge
b37c52fc3a Make GCS delete concurrency configurable 2020-03-16 15:54:05 +00:00
Simon Detheridge
9b658dda18 Copy-on-delete before unlocking 2020-03-16 15:53:45 +00:00
Simon Detheridge
06c4c0f74f Fix incorrect key when deleting projects 2020-03-16 11:35:01 +00:00
Simon Detheridge
ce52f8aa60 Update FSPersistor deleteDirectory behaviour to match S3 and GCS 2020-03-16 11:33:51 +00:00
Simon Detheridge
47e96a4d94 Add endpoint to delete a project 
Needs acceptance tests
 2020-03-14 14:56:29 +00:00
Simon Detheridge
fc80aa3954 Move directory key validation into FileHandler 2020-03-14 14:31:30 +00:00
Simon Detheridge
58db14456a Add timestamp to files in deleted bucket 2020-03-14 14:11:17 +00:00
Simon Detheridge
edf1ce1f7e Delete files from a directory in parallel 2020-03-14 14:02:58 +00:00
Simon Detheridge
183cb0179a Add support to GCS persistor for unlocking files and copying on delete 2020-03-13 16:18:45 +00:00
Simon Detheridge
28c3fe4a56 Validate key names when deleting directory with a configurable regex 2020-03-10 17:54:09 +00:00
Simon Detheridge
eb93ae4b10 Use Bucket.deleteFiles to delete directory contents, instead of iterating 2020-03-05 14:12:15 +00:00
Simon Detheridge
460dd96b17 Cosmetic clean-up of GCS Persistor & tests 2020-03-05 13:45:46 +00:00
Simon Detheridge
3bb956b38e Use http for the fake GCS server 2020-03-04 17:01:20 +00:00
Simon Detheridge
30114cd79b Remove unnecessary try/catch around 'createReadStream' 2020-03-04 16:38:05 +00:00
Simon Detheridge
a7198764cb Improve/add some comments for clarity 2020-03-04 16:25:11 +00:00
Simon Detheridge
def383574e Simplify getMeteredStream to record metric directly 2020-03-04 16:17:36 +00:00
Simon Detheridge
76243fd75a 'bucket' -> 'bucketName' in GCS Persistor 2020-03-04 16:04:26 +00:00
Simon Detheridge
e58284aefe Move base64/hex methods to PersistorHelper 
Also add some null-safety checks
 2020-03-04 15:38:18 +00:00
Simon Detheridge
366ce97169 Add GCS Persistor 2020-03-04 15:09:14 +00:00
Simon Detheridge
b5e4345855 Merge pull request #96 from overleaf/jpa-dependencies-cleanup 
[misc] dependencies cleanup
 2020-03-04 10:00:56 +00:00
Jakob Ackermann
2b15729658 [misc] promisify FileHandler and remove dependency on async 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-03-02 18:23:17 +01:00
Jakob Ackermann
32557ab1d7 [SafeExec] replace _.once with lodash.once 2020-03-02 17:08:42 +01:00
Jakob Ackermann
0db4a17a14 [HealthCheckController] use fs.copyFile instead of fs-extra.copy 
the fs-extra method has a HUGE overhead of JS code and also syscalls for
 no particular benefit in this case: just copy the tiny.pdf file.

Here is an overview of the major operations: paths are relative to
  https://github.com/jprichardson/node-fs-extra/blob/1.0.0

We start in /lib/copy/copy.js
- sys: check that the source file exists
- sys: check that the source file has an existing parent directory?!
Continue in /lib/copy/ncp.js
- sys: more stat calls on both source and dest to determine permissions
- read/write streams to pipe the file content through the process
- sys: chmod on the destination to match the source permissions

What we actually need is a call to the binding and let node/the os
 figure out the best way to copy the contents.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-03-02 17:02:35 +01:00
Simon Detheridge
8acc9ef0ae Make S3 partSize a setting 2020-02-17 14:06:15 +00:00
Simon Detheridge
7c4bf97a7f Avoid fetching file a second time when source md5 can't be fetched 2020-02-17 14:06:05 +00:00
Simon Detheridge
2f2a819b74 Re-fetch file to calculate md5 if etag is not in correct format 2020-02-17 14:05:43 +00:00
Simon Detheridge
5adfb3e2c0 Use large upload chunks 2020-02-17 14:05:31 +00:00
Simon Detheridge
4e507197fc Revert "Merge pull request #91 from overleaf/revert-78-spd-migration-persistor" 
This reverts commit 0cce792604c258538cf012486e7329fa1be7da0f, reversing
changes made to 08b606e22f855121b2d8a2832a5d7e2f45836c3f.
 2020-02-17 14:04:42 +00:00
Simon Detheridge
df2ddbe0e1 Revert "Add Migration Persistor, to send missing file requests to a fallback persistor" 2020-02-14 14:26:33 +00:00
Simon Detheridge
e4926e09a7 Move error handler to be the last middleware 2020-02-14 10:58:46 +00:00
Simon Detheridge
44fbc6a733 Merge pull request #82 from overleaf/spd-error-handler-error 
Fix log info assignment in error handler
 2020-02-14 09:18:16 +00:00
Simon Detheridge
3b011258d2 Tidy up md5 hash generation 2020-02-12 13:27:12 +00:00
Simon Detheridge
49ad408b30 Remove unused imports and format correctly 2020-02-12 13:27:12 +00:00
Simon Detheridge
93cd55fb79 Refactor persistors to use a helper for common things 2020-02-12 13:27:12 +00:00
Simon Detheridge
9e0b378948 Remove minipass as dependency and refactor to make things clearer 2020-02-12 13:21:52 +00:00
Simon Detheridge
5d5d325691 Preserve all error information when cleanup of copied file fails 2020-02-12 13:21:52 +00:00
Simon Detheridge
b4b7fd226e Add mechanisms to transfer files with md5-based integrity checks 
Fix error in settings and tidy up tests

Remove unused variable declaration

Remove .only from tests and update eslint rules to catch it in future

Use  to catch errors more safely getting md5 hash

Avoid unnecessary call to S3 to get md5 response
 2020-02-12 13:21:52 +00:00
Simon Detheridge
2625e03a31 Add MigrationPersistor for sending 404 requests to a fallback persistor 2020-02-12 13:19:48 +00:00
Simon Detheridge
f877f51775 Rename *PersistorManager to *Persistor 2020-02-12 13:19:48 +00:00
Simon Detheridge
e3613e52db Fix log info assignment in error handler 2020-02-12 10:34:56 +00:00
Simon Detheridge
035364b62e Fix RequestLogger object lifecycle 2020-01-14 17:15:01 +00:00
Simon Detheridge
28fb998719 Refactor request logger into a class 2020-01-14 12:02:39 +00:00
Simon Detheridge
9615a06e0f Improve logging middleware to add info methods to request object 2020-01-10 17:22:06 +00:00
Simon Detheridge
f40fbe77aa Simplify logging to log only once per http request 2020-01-10 15:41:36 +00:00
Simon Detheridge
e331a3130c Handle AccessDenied and stream-premature-close errors 
These errors tend to occur as part of normal operation and should not generate `logger.err` messages
 2020-01-09 14:15:01 +00:00
Simon Detheridge
d8470d4a59 Use correct object path for s3BucketCreds 2020-01-08 11:32:05 +00:00