Simon Detheridge
3637cd70ae
Add support for redirecting to signed URLs
2020-04-15 12:22:54 +01:00
Eric Mc Sween
003d54e8f0
Fix error handler returning two responses
...
The error handler mistakenly sent two responses on error, the first
being a 200.
2020-04-03 11:57:33 -04:00
Simon Detheridge
114883a9e9
Ensure streams are always drained on close or error
2020-04-02 15:56:23 +01:00
Simon Detheridge
cce4049153
Merge pull request #112 from overleaf/spd-more-stream-destruction
...
Destroy download streams on close
2020-03-31 20:29:20 +01:00
Simon Detheridge
60ede0bf46
Find more ways to destroy streams
2020-03-31 17:00:39 +01:00
Simon Detheridge
dceef85ccb
Update app/js/FileHandler.js
...
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
2020-03-31 16:03:38 +01:00
Simon Detheridge
4489670497
Only delete the converted-cache folder if conversions are enabled
2020-03-31 15:38:42 +01:00
Simon Detheridge
f440cec8ec
Check if streams are already destroyed before destroying
2020-03-31 11:07:31 +01:00
Simon Detheridge
5fadafaf96
Destroy streams handled by getReadyPipeline on error
2020-03-30 17:14:46 +01:00
Simon Detheridge
87b8f8e194
Destroy file-download stream on error
2020-03-30 17:13:40 +01:00
Simon Detheridge
4cb4d450be
Call end() when handing ERR_STREAM_PREMATURE_CLOSE
2020-03-30 15:03:09 +01:00
Simon Detheridge
a1ae68f6b5
Tidy up 'getReadyPipeline'
2020-03-26 16:57:55 +00:00
Simon Detheridge
1f037ef653
Catch errors from pipeline and stream when waiting for streams to be readable
2020-03-26 16:57:55 +00:00
Simon Detheridge
7e45a82c35
Use autodestroy on Transform stream
2020-03-26 15:09:56 +00:00
Simon Detheridge
d073fe75ca
Use single pipeline when calculating md5
2020-03-25 16:59:51 +00:00
Simon Detheridge
65f19346f0
Fix fallback to custom s3 buckets
2020-03-19 11:10:31 +00:00
Simon Detheridge
cb4bdd99f4
Use an ISODate for deleted file names
2020-03-18 20:19:12 +00:00
Simon Detheridge
9d32d4ec16
Don't modify 'opts' parameter
2020-03-16 15:57:37 +00:00
Simon Detheridge
b37c52fc3a
Make GCS delete concurrency configurable
2020-03-16 15:54:05 +00:00
Simon Detheridge
9b658dda18
Copy-on-delete before unlocking
2020-03-16 15:53:45 +00:00
Simon Detheridge
06c4c0f74f
Fix incorrect key when deleting projects
2020-03-16 11:35:01 +00:00
Simon Detheridge
ce52f8aa60
Update FSPersistor deleteDirectory behaviour to match S3 and GCS
2020-03-16 11:33:51 +00:00
Simon Detheridge
47e96a4d94
Add endpoint to delete a project
...
Needs acceptance tests
2020-03-14 14:56:29 +00:00
Simon Detheridge
fc80aa3954
Move directory key validation into FileHandler
2020-03-14 14:31:30 +00:00
Simon Detheridge
58db14456a
Add timestamp to files in deleted bucket
2020-03-14 14:11:17 +00:00
Simon Detheridge
edf1ce1f7e
Delete files from a directory in parallel
2020-03-14 14:02:58 +00:00
Simon Detheridge
183cb0179a
Add support to GCS persistor for unlocking files and copying on delete
2020-03-13 16:18:45 +00:00
Simon Detheridge
28c3fe4a56
Validate key names when deleting directory with a configurable regex
2020-03-10 17:54:09 +00:00
Simon Detheridge
eb93ae4b10
Use Bucket.deleteFiles to delete directory contents, instead of iterating
2020-03-05 14:12:15 +00:00
Simon Detheridge
460dd96b17
Cosmetic clean-up of GCS Persistor & tests
2020-03-05 13:45:46 +00:00
Simon Detheridge
3bb956b38e
Use http for the fake GCS server
2020-03-04 17:01:20 +00:00
Simon Detheridge
30114cd79b
Remove unnecessary try/catch around 'createReadStream'
2020-03-04 16:38:05 +00:00
Simon Detheridge
a7198764cb
Improve/add some comments for clarity
2020-03-04 16:25:11 +00:00
Simon Detheridge
def383574e
Simplify getMeteredStream to record metric directly
2020-03-04 16:17:36 +00:00
Simon Detheridge
76243fd75a
'bucket' -> 'bucketName' in GCS Persistor
2020-03-04 16:04:26 +00:00
Simon Detheridge
e58284aefe
Move base64/hex methods to PersistorHelper
...
Also add some null-safety checks
2020-03-04 15:38:18 +00:00
Simon Detheridge
366ce97169
Add GCS Persistor
2020-03-04 15:09:14 +00:00
Simon Detheridge
b5e4345855
Merge pull request #96 from overleaf/jpa-dependencies-cleanup
...
[misc] dependencies cleanup
2020-03-04 10:00:56 +00:00
Jakob Ackermann
2b15729658
[misc] promisify FileHandler and remove dependency on async
...
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-03-02 18:23:17 +01:00
Jakob Ackermann
32557ab1d7
[SafeExec] replace _.once with lodash.once
2020-03-02 17:08:42 +01:00
Jakob Ackermann
0db4a17a14
[HealthCheckController] use fs.copyFile instead of fs-extra.copy
...
the fs-extra method has a HUGE overhead of JS code and also syscalls for
no particular benefit in this case: just copy the tiny.pdf file.
Here is an overview of the major operations: paths are relative to
https://github.com/jprichardson/node-fs-extra/blob/1.0.0
We start in /lib/copy/copy.js
- sys: check that the source file exists
- sys: check that the source file has an existing parent directory?!
Continue in /lib/copy/ncp.js
- sys: more stat calls on both source and dest to determine permissions
- read/write streams to pipe the file content through the process
- sys: chmod on the destination to match the source permissions
What we actually need is a call to the binding and let node/the os
figure out the best way to copy the contents.
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-03-02 17:02:35 +01:00
Simon Detheridge
8acc9ef0ae
Make S3 partSize a setting
2020-02-17 14:06:15 +00:00
Simon Detheridge
7c4bf97a7f
Avoid fetching file a second time when source md5 can't be fetched
2020-02-17 14:06:05 +00:00
Simon Detheridge
2f2a819b74
Re-fetch file to calculate md5 if etag is not in correct format
2020-02-17 14:05:43 +00:00
Simon Detheridge
5adfb3e2c0
Use large upload chunks
2020-02-17 14:05:31 +00:00
Simon Detheridge
4e507197fc
Revert "Merge pull request #91 from overleaf/revert-78-spd-migration-persistor"
...
This reverts commit 0cce792604c258538cf012486e7329fa1be7da0f, reversing
changes made to 08b606e22f855121b2d8a2832a5d7e2f45836c3f.
2020-02-17 14:04:42 +00:00
Simon Detheridge
df2ddbe0e1
Revert "Add Migration Persistor, to send missing file requests to a fallback persistor"
2020-02-14 14:26:33 +00:00
Simon Detheridge
e4926e09a7
Move error handler to be the last middleware
2020-02-14 10:58:46 +00:00
Simon Detheridge
44fbc6a733
Merge pull request #82 from overleaf/spd-error-handler-error
...
Fix log info assignment in error handler
2020-02-14 09:18:16 +00:00
Simon Detheridge
3b011258d2
Tidy up md5 hash generation
2020-02-12 13:27:12 +00:00